Ticket #51 (new defect)

Opened 2 years ago

Last modified 2 years ago

Advice on handling deadlock is incorrect

Reported by: gvwilson Assigned to: gvwilson
Priority: major Milestone: Someday
Component: lectures Version:
Keywords: Cc:

Description

You advise to use Popen.communicate in order to pass lots of data to the process. Claiming it would solve the problem of buffers filling up. This sounded like good news to me, as AFAIK the only solution so far was to select() on the input and output, if you disliek threads that is.

The documentation however says """Note: The data read is buffered in memory, so do not use this method if the data size is large or unlimited."""

see: http://docs.python.org/lib/node239.html

am I missing something?

Change History

11/20/06 11:36:35 changed by gvwilson

I have attached a (very) crude example of how to solve the problem of deadlock. People might be concerned with the inefficiency of writing to a file and then reading it again, but for small(few tens of megabytes) this should not really be a problem, certainly not on linux as one can expect the file data to sit in cache, not quite sure about windows. I believe if you are advanced enough to worry about performance you will be able to read and understand select().

Please let me know if I should change something or you need anything else. Reasons to procrastinate are welcome...;]]

tim

Proposed change to line 128 in trunk/lec/integrate.swc : <b1>

<t>Solution is to use a temporary file.</t>

<b2><t>Write your data to a temporary file and use that as input

to the program</t></b2>

<b2><t>This will work fine for "small" data sets, if you want to

use sets of the order of gigabytes please read about <c>select()</c></t></b2> </b1>

An example(adapted freely from a post by Guido some years back) goes like this:

import tempfile tf = tempfile.TemporaryFile?()

try:

tf.write(text) tf.flush() #XXX don't know if this is really needed child = subprocess.Popen(['gzip', '-c'],

stdin=tf, stdout=subprocess.PIPE)

result = child.stdout.read()

finally:

tf.close()