By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,923 Members | 1,443 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,923 IT Pros & Developers. It's quick & easy.

xmlrpc with Python and large datases

P: n/a
Hi all,

I helped one of my co-workers put together an XMLRPC Python script that
allowed him to get database data from remote machines. This was done
because the source of the data could be Oracle on a Sun/Solaris
machine, and MySQL on our linux machines. Doing the script in Python
allowed him to gather the date with a general purpose API and just send
over some SQL queries.

Since then he's run into a problem. If he sends a query that gets a
very large recordset from the database the script fails and it vaguely
reports about a "broken pipe", which I'm guessing is a problem with the
network connection. Has anyone else seen this use XMLRPC and Python and
large amounts of data being returned via XMLRPC? And if so, is there a
solution?

Thanks,
Doug

Jul 18 '05 #1
Share this Question
Share on Google+
5 Replies


P: n/a
writeson wrote:
Since then he's run into a problem. If he sends a query that gets a
very large recordset from the database the script fails and it vaguely
reports about a "broken pipe", which I'm guessing is a problem with the
network connection. Has anyone else seen this use XMLRPC and Python and
large amounts of data being returned via XMLRPC? And if so, is there a
solution?

Most web servers have a configurable limit on the size of data sent to the
server for a single request. It sounds as though you may be exceeding this.
Check what setting has been used on your web server.

If it is Apache then look at the limitrequestbody parameter, it is
unlimited by default, but advisable to set a limit to deter certain types
of DoS attack.
Jul 18 '05 #2

P: n/a
Duncan,

Thanks for the reply. We are running this on an Apache server on the
linux box, and an iPlanet4.1 server on the solaris machines. However,
both these servers are strictly 'inside' the firewall. I checked the
apache configuration and there is no limitrequestbody parameter in the
file at all. So I'm assuming from what you've said that our
configuration would be unlimited. However, I will test that by
inserting limitrequestbody and setting it to 0 for unlimited to see if
that changes things.

Thanks,
Doug

Jul 18 '05 #3

P: n/a
writeson wrote:
Duncan,

Thanks for the reply. We are running this on an Apache server on the
linux box, and an iPlanet4.1 server on the solaris machines. However,
both these servers are strictly 'inside' the firewall. I checked the
apache configuration and there is no limitrequestbody parameter in the
file at all. So I'm assuming from what you've said that our
configuration would be unlimited. However, I will test that by
inserting limitrequestbody and setting it to 0 for unlimited to see if
that changes things.

Thanks,
Doug

Usually a "broken pipe" message is to do with disruption of a Unix
pipeline rather than a network connection. I've seen the message mostly,
IIRC, in BSD-based environments.

A pipeline connects the standard output of one process to the standard
input of another. If this second process terminated while the first is
still writing its output the "broken pipe" message will be raised. It's
therefore probable that some consumer process is terminating before its
producer.

Whether this is due to timeouts in the server environment is open to
question, but at least you'll be looking for the right problem :-)

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.pycon.org/
Steve Holden http://www.holdenweb.com/
Jul 18 '05 #4

P: n/a
Steve Holden wrote:
Usually a "broken pipe" message is to do with disruption of a Unix pipeline rather than a network
connection. I've seen the message mostly, IIRC, in BSD-based environments.


the socket layer uses EPIPE to indicate that a socket has been shut down
by the remote end.

</F>

Jul 18 '05 #5

P: n/a
Steve,

Thanks for your reply, I'll look into things based on your comments.
Also, I've read your book "Python Web Programming" and wanted you to
know it has helped me a lot with various python projects, thanks!

Doug

Jul 18 '05 #6

This discussion thread is closed

Replies have been disabled for this discussion.