xmlrpc with Python and large datases

[writeson]

Hi all,

I helped one of my co-workers put together an XMLRPC Python script that
allowed him to get database data from remote machines. This was done
because the source of the data could be Oracle on a Sun/Solaris
machine, and MySQL on our linux machines. Doing the script in Python
allowed him to gather the date with a general purpose API and just send
over some SQL queries.

Since then he's run into a problem. If he sends a query that gets a
very large recordset from the database the script fails and it vaguely
reports about a "broken pipe", which I'm guessing is a problem with the
network connection. Has anyone else seen this use XMLRPC and Python and
large amounts of data being returned via XMLRPC? And if so, is there a
solution?

Thanks,
Doug

[Duncan Booth]

writeson wrote:

Since then he's run into a problem. If he sends a query that gets a
very large recordset from the database the script fails and it vaguely
reports about a "broken pipe", which I'm guessing is a problem with the
network connection. Has anyone else seen this use XMLRPC and Python and
large amounts of data being returned via XMLRPC? And if so, is there a
solution?

Most web servers have a configurable limit on the size of data sent to the
server for a single request. It sounds as though you may be exceeding this.
Check what setting has been used on your web server.

If it is Apache then look at the limitrequestbody parameter, it is
unlimited by default, but advisable to set a limit to deter certain types
of DoS attack.

[writeson]

Duncan,

Thanks for the reply. We are running this on an Apache server on the
linux box, and an iPlanet4.1 server on the solaris machines. However,
both these servers are strictly 'inside' the firewall. I checked the
apache configuration and there is no limitrequestbody parameter in the
file at all. So I'm assuming from what you've said that our
configuration would be unlimited. However, I will test that by
inserting limitrequestbody and setting it to 0 for unlimited to see if
that changes things.

Thanks,
Doug

[Steve Holden]

writeson wrote:

Duncan,

Thanks for the reply. We are running this on an Apache server on the
linux box, and an iPlanet4.1 server on the solaris machines. However,
both these servers are strictly 'inside' the firewall. I checked the
apache configuration and there is no limitrequestbody parameter in the
file at all. So I'm assuming from what you've said that our
configuration would be unlimited. However, I will test that by
inserting limitrequestbody and setting it to 0 for unlimited to see if
that changes things.

Thanks,
Doug

Usually a "broken pipe" message is to do with disruption of a Unix
pipeline rather than a network connection. I've seen the message mostly,
IIRC, in BSD-based environments.

A pipeline connects the standard output of one process to the standard
input of another. If this second process terminated while the first is
still writing its output the "broken pipe" message will be raised. It's
therefore probable that some consumer process is terminating before its
producer.

Whether this is due to timeouts in the server environment is open to
question, but at least you'll be looking for the right problem :-)

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.pycon.org/
Steve Holden http://www.holdenweb.com/

[Fredrik Lundh]

Steve Holden wrote:

Usually a "broken pipe" message is to do with disruption of a Unix pipeline rather than a network
connection. I've seen the message mostly, IIRC, in BSD-based environments.

the socket layer uses EPIPE to indicate that a socket has been shut down
by the remote end.

</F>

[writeson]

Steve,

Thanks for your reply, I'll look into things based on your comments.
Also, I've read your book "Python Web Programming" and wanted you to
know it has helped me a lot with various python projects, thanks!

Doug