473,320 Members | 2,020 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Question about StringIO

Hi all

I understand that StringIO creates a file-like object in memory.

Is it possible to invoke another program, using os.system() or
os.popen(), and use the < redirect operator, so that the other program
reads my StringIO object as its input?

I will provide more details if required, but hopefully this is enough
for a simple yes or no answer, and if so, how.

BTW, I have tried using popen2() and passing my data via stdin, but the
other program (psql) does not react well to this - again, I will give
more info if necessary.

Thanks

Frank Millman

Oct 9 '05 #1
11 1944
Frank Millman wrote:
Hi all

I understand that StringIO creates a file-like object in memory.

Is it possible to invoke another program, using os.system() or
os.popen(), and use the < redirect operator, so that the other program
reads my StringIO object as its input?

I will provide more details if required, but hopefully this is enough
for a simple yes or no answer, and if so, how.

BTW, I have tried using popen2() and passing my data via stdin, but the
other program (psql) does not react well to this - again, I will give
more info if necessary.

Unfortunately the StringIO module only creates instances inside the
process they are called: these objects have no existence to the
operating system or to other processes, and so can't be used for
inter-process communication.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

Oct 9 '05 #2
Frank Millman wrote:
Hi all

I understand that StringIO creates a file-like object in memory.

Is it possible to invoke another program, using os.system() or
os.popen(), and use the < redirect operator, so that the other program
reads my StringIO object as its input?
No. Processes don't share memory - thus you have to either use a temp
file, or pipes.
BTW, I have tried using popen2() and passing my data via stdin, but the
other program (psql) does not react well to this - again, I will give
more info if necessary.


Better do so :)

Diez
Oct 9 '05 #3

Diez B. Roggisch wrote:
Frank Millman wrote:
Hi all

I understand that StringIO creates a file-like object in memory.

Is it possible to invoke another program, using os.system() or
os.popen(), and use the < redirect operator, so that the other program
reads my StringIO object as its input?


No. Processes don't share memory - thus you have to either use a temp
file, or pipes.
BTW, I have tried using popen2() and passing my data via stdin, but the
other program (psql) does not react well to this - again, I will give
more info if necessary.


Better do so :)

Diez


Thanks, Steve and Diez, for the replies. I didn't think it was
possible, but it was worth asking :-)

I will try to explain my experience with popen() briefly.

I have some sql scripts to create tables, indexes, procedures, etc. At
present there are about 50 scripts, but this number will grow. I have
been running them manually so far. Now I want to automate the process.

I am supporting PostgreSQL and MS SQL Server, and the syntax is
slightly different in some cases. Rather than maintain two sets of
scripts, I prefix some lines with -pg- or -ms- to indicate the
platform, and then use Python to parse the scripts and generate a
correct output for each platform, passing it to 'psql' and 'osql'
respectively, using popen().

I have had a few problems, but it would take too long to describe them
all, and I just want a working solution, so I will focus on my latest
attempt.

I run through all the scripts and create a StringIO object with the
string I want to pass. It is about 250 000 bytes long. If I run psql
using popen(), and pass it the string via stdin, it works fine, but I
get all the messages on the screen. If I do the same, but end the
command with ' > fjm 2>&1' it works correctly and the messages end up
in the file fjm, which is about 40 000 bytes long. If I run it with
popen4(), it starts ok, but then hangs about 1/4 of the way through.
Exactly the same happens on MSW. It seems to be hitting a limit on the
size of the stdout file - is that possible?

For my purposes, I will be happy to use popen() and a choice of no
redirection, redirect to a file, or redirect to /dev/null. The question
about popen4() is therefore academic, though I would be interested to
know the answer.

BTW, is there an equivalent of /dev/null on MSW?

Thanks in advance for any suggestions.

Frank

Oct 10 '05 #4
Frank Millman wrote:
I will try to explain my experience with popen() briefly.

I have some sql scripts to create tables, indexes, procedures, etc. At
present there are about 50 scripts, but this number will grow. I have
been running them manually so far. Now I want to automate the process.

I am supporting PostgreSQL and MS SQL Server, and the syntax is
slightly different in some cases. Rather than maintain two sets of
scripts, I prefix some lines with -pg- or -ms- to indicate the
platform, and then use Python to parse the scripts and generate a
correct output for each platform, passing it to 'psql' and 'osql'
respectively, using popen().

I have had a few problems, but it would take too long to describe them
all, and I just want a working solution, so I will focus on my latest
attempt.

I run through all the scripts and create a StringIO object with the
string I want to pass. It is about 250 000 bytes long. If I run psql
using popen(), and pass it the string via stdin, it works fine, but I
get all the messages on the screen. If I do the same, but end the
command with ' > fjm 2>&1' it works correctly and the messages end up
in the file fjm, which is about 40 000 bytes long. If I run it with
popen4(), it starts ok, but then hangs about 1/4 of the way through.
Exactly the same happens on MSW. It seems to be hitting a limit on the
size of the stdout file - is that possible?

For my purposes, I will be happy to use popen() and a choice of no
redirection, redirect to a file, or redirect to /dev/null. The question
about popen4() is therefore academic, though I would be interested to
know the answer.
That's probably a deadlock as described in
<http://docs.python.org/lib/popen2-flow-control.html>
BTW, is there an equivalent of /dev/null on MSW?


Dunno - but as a last resort, you could create a tempfile with a unique name
(to be sure, not to override any existing data), dump your output there and
later os.unlink() it...

--
Benjamin Niemann
Email: pink at odahoda dot de
WWW: http://www.odahoda.de/
Oct 10 '05 #5
> Thanks, Steve and Diez, for the replies. I didn't think it was
possible, but it was worth asking :-)

I will try to explain my experience with popen() briefly.

I have some sql scripts to create tables, indexes, procedures, etc. At
present there are about 50 scripts, but this number will grow. I have
been running them manually so far. Now I want to automate the process.

I am supporting PostgreSQL and MS SQL Server, and the syntax is
slightly different in some cases. Rather than maintain two sets of
scripts, I prefix some lines with -pg- or -ms- to indicate the
platform, and then use Python to parse the scripts and generate a
correct output for each platform, passing it to 'psql' and 'osql'
respectively, using popen().


Why don't youn use te python DB-Api instead?
Regards,

Diez
Oct 10 '05 #6
Benjamin Niemann wrote:
Frank Millman wrote:
I will try to explain my experience with popen() briefly.

I run through all the scripts and create a StringIO object with the
string I want to pass. It is about 250 000 bytes long. If I run psql
using popen(), and pass it the string via stdin, it works fine, but I
get all the messages on the screen. If I do the same, but end the
command with ' > fjm 2>&1' it works correctly and the messages end up
in the file fjm, which is about 40 000 bytes long. If I run it with
popen4(), it starts ok, but then hangs about 1/4 of the way through.
Exactly the same happens on MSW. It seems to be hitting a limit on the
size of the stdout file - is that possible?


That's probably a deadlock as described in
<http://docs.python.org/lib/popen2-flow-control.html>


Thanks for this pointer. I have read it, but I don't think it applies
to my situation, as it talks about 'reading' from the child's stdout
while the child is 'writing' to stderr. I am not doing that, or at
least not consciously. Here is a code snippet. 's' is a StringIO object
that contains my input. It is about 6000 lines/250000 bytes long.

-------------

sql_stdin,sql_stdout = os.popen4('psql -U %s -d %s' % (user,database))

sql_stdin.writelines(s.readlines())
s.close()
sql_stdin.close()

-------------

It starts, and then hangs, after processing about 6% of my input.

If I add ' > fjm 2>&1' to the command, it works, so it is definitely
connected with the child writing to stdout/stderr.

I tried storing my input in a list, and passing ''.join(s), but it had
the same result. I also looped over the list and wrote one line at a
time to sql_stdin - same result.

I can work around this, so a solution is not critical. However, it
would be nice to know if this is a limitation of popen4(), or if I am
doing something wrong.
BTW, is there an equivalent of /dev/null on MSW?


Dunno - but as a last resort, you could create a tempfile with a unique name
(to be sure, not to override any existing data), dump your output there and
later os.unlink() it...


A quick google revealed the answer - there is a device called NUL which
achieves the same purpose.

Thanks

Frank

Oct 11 '05 #7
Diez B. Roggisch wrote:
Thanks, Steve and Diez, for the replies. I didn't think it was
possible, but it was worth asking :-)

I will try to explain my experience with popen() briefly.

I have some sql scripts to create tables, indexes, procedures, etc. At
present there are about 50 scripts, but this number will grow. I have
been running them manually so far. Now I want to automate the process.

I am supporting PostgreSQL and MS SQL Server, and the syntax is
slightly different in some cases. Rather than maintain two sets of
scripts, I prefix some lines with -pg- or -ms- to indicate the
platform, and then use Python to parse the scripts and generate a
correct output for each platform, passing it to 'psql' and 'osql'
respectively, using popen().


Why don't youn use te python DB-Api instead?
Regards,

Diez


My scripts are used to create the tables in the database. I didn't
think that DB-API covered that. However, even if it did, I don't think
it would handle differences such as the following.

For Unicode support, PostgreSQL uses the character-set specified when
the database is created. SQL Server allows you to specify it for each
column, using the datatype NCHAR and NVARCHAR instead of CHAR and
VARCHAR.

PostgreSQL uses data types called DATE and TIMESTAMP. SQL Server uses
DATETIME (it also uses TIMESTAMP, but that is used for something else).

Both DBMS's have the concept of a column which is automatically
assigned a 'next number' each time a row is created, but the syntax for
defining the column is completely different.

PostgreSQL allows the use of a WHERE clause when creating an INDEX,
which is useful if you only want to index a subset of a table.

SQL Server has the concept of a CLUSTERED INDEX, whereby it stores the
rows physically in index sequence. It defaults to using a clustered
index for the primary key. Often this is not what you want, so it is
desirable to specify the primary key as NONCLUSTERED, and then specify
a CLUSTERED index for a more frequently used column.

These are just a few of the differences, but you get the idea. If there
is a better way to do this in a cross-platform manner, I would love to
know how.

Thanks

Frank

Oct 11 '05 #8
> Thanks for this pointer. I have read it, but I don't think it applies
to my situation, as it talks about 'reading' from the child's stdout
while the child is 'writing' to stderr.


But that is exactly the point: the psql blocks because you don't read
away the buffered data. Start a thread, read that stdout/stderr and see
if things go smoothly.

Diez
Oct 11 '05 #9
> My scripts are used to create the tables in the database. I didn't
think that DB-API covered that.
The DB-Api covers executin arbirary SQL - either DDL or DML. It is
surely centered around DML, but that doesn't mean that its not usabel to
issue "create ..." statements.
However, even if it did, I don't think
it would handle differences such as the following.


<snip>

All that has nocthing to do with teh API - you'd still need your
differentiated DDL - but the communication with the programs would go away.

Diez
Oct 11 '05 #10

Diez B. Roggisch wrote:
Thanks for this pointer. I have read it, but I don't think it applies
to my situation, as it talks about 'reading' from the child's stdout
while the child is 'writing' to stderr.


But that is exactly the point: the psql blocks because you don't read
away the buffered data. Start a thread, read that stdout/stderr and see
if things go smoothly.

Diez


Of course (kicks himself), it is obvious now that you have explained
it. I tried your suggestion and it works perfectly.

Many thanks

Frank

Oct 11 '05 #11

Diez B. Roggisch wrote:
My scripts are used to create the tables in the database. I didn't
think that DB-API covered that.


The DB-Api covers executin arbirary SQL - either DDL or DML. It is
surely centered around DML, but that doesn't mean that its not usabel to
issue "create ..." statements.
>However, even if it did, I don't think
it would handle differences such as the following.


<snip>

All that has nocthing to do with teh API - you'd still need your
differentiated DDL - but the communication with the programs would go away.

Diez


I understand. It certainly gives me an alternative approach - I will
experiment to see which suits my purpose best.

Many thanks for your assistance.

Frank

Oct 11 '05 #12

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: michael | last post by:
Hello, Apologies if this seems like a trivial question, but it would help me with my python, coming from a Java background. What is the preferred naming convention for a .py file, that...
21
by: Paul Rubin | last post by:
I've always found the string-building idiom temp_list = for x in various_pieces_of_output(): v = go_figure_out_some_string() temp_list.append(v) final_string = ''.join(temp_list) ...
5
by: f pemberton | last post by:
I have kind of an interesting string, it looks like a couple hundred letters bunched together with no spaces. Anyway, i'm trying to put a "?" and a (\n) newline after every 100th character of the...
6
by: sebastian.noack | last post by:
Hi, is there a way to or at least a reason why I can not use tarfile to create a gzip or bunzip2 compressed archive in the memory? You might might wanna answer "use StringIO" but this isn't...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.