473,888 Members | 1,369 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

httplib continuation packets

After a long debugging session while scripting my webmail,
I believe I have traced the problem to the way httplib sends
POST requests.

I have compared tcpdump listings from Python 2.4.3 and 2.5.0's
httplib (via urllib/urllib2), Perl's LWP::UserAgent 2.033 and
Firefox 2.0. Only Python sends the request in such a way that
the mailserver closes the connection before I get any data from
the POST request (immediate FIN packet after the POST request).

httplib always sends the urlencoded POST data in a separate packet
from the HTTP headers, and this seems to cause problems with
the web interface in Ipswitch-IMail/8.05 (the software running on
Doteasy's webmail).

Firefox 2.0 has most of the headers in a single packet, but unlike
httplib, it always places a couple of headers in the continuation
packet as well (usually the content-length and content-type
headers).

LWP::UserAgent 2.033 doesn't use continuation at all, and sends
everything in a single packet.

Is this a bug in httplib or the web server? Is there a
workaround, or should I use Perl for this?

--
Haakon
Nov 11 '06 #1
6 2417
Haakon Riiser wrote:
Is this a bug in httplib or the web server?
it could be that they're blocking requests from Python's urllib, of
course. have you tried overriding the user-agent string ?

</F>

Nov 11 '06 #2
[Fredrik Lundh]
Haakon Riiser wrote:
>Is this a bug in httplib or the web server?

it could be that they're blocking requests from Python's urllib, of
course. have you tried overriding the user-agent string ?
Yes, and it doesn't help.

By the way, this is the closest thing I've found in the bug tracker:
https://sourceforge.net/tracker/?fun...&group_id=5470
The bug was closed in 2002 with this comment:

"I changed httplib to send requests as a single packet in rev
1.60. The change was made to address a performance problem,
but happens to fix the problem you had with the bogus
server, too."

Has someone changed it back since then?

--
Haakon
Nov 11 '06 #3
Haakon Riiser wrote:
Yes, and it doesn't help.
then the server is mostly likely broken beyond repair.
By the way, this is the closest thing I've found in the bug tracker:
https://sourceforge.net/tracker/?fun...&group_id=5470
The bug was closed in 2002 with this comment:

"I changed httplib to send requests as a single packet in rev
1.60. The change was made to address a performance problem,
but happens to fix the problem you had with the bogus
server, too."

Has someone changed it back since then?
nope; that change buffers the *header* part of the request to avoid
problems with certain TCP/IP mechanisms; see

http://svn.python.org/view?rev=27644&view=rev

for a discussion. note that there's still no guarantee that the entire
header is sent in a single TCP packet.

to see if this really is the problem, you could try moving the call to
self._send_outp ut() from the end of the endheaders() method to the end
of the _send_request() method (around line 870 in httplib.py, at least
in 2.5).

</F>

Nov 11 '06 #4
[Fredrik Lundh]
Haakon Riiser wrote:
>Yes, and it doesn't help.

then the server is mostly likely broken beyond repair.
It's not in my power to upgrade the server, unfortunately.
Guess I'll have to use Perl.
to see if this really is the problem, you could try moving the call to
self._send_outp ut() from the end of the endheaders() method to the end
of the _send_request() method (around line 870 in httplib.py, at least
in 2.5).
Tried this, but the tcpdump still looks the same (two packets: one
with the headers, one with the body), and now it fails with

urllib2.HTTPErr or: HTTP Error 501: Not Implemented

Nevertheless, I'm fairly sure that the packet fragmentation is
the culprit. It works perfectly with Perl, even when I make
no effort at all to spoof the browser (no user-agent, referer,
cookies, etc.).

--
Haakon
Nov 11 '06 #5
Haakon Riiser wrote:
[Fredrik Lundh]
>Haakon Riiser wrote:
>>Yes, and it doesn't help.
then the server is mostly likely broken beyond repair.

It's not in my power to upgrade the server, unfortunately.
Guess I'll have to use Perl.
>to see if this really is the problem, you could try moving the call to
self._send_out put() from the end of the endheaders() method to the end
of the _send_request() method (around line 870 in httplib.py, at least
in 2.5).

Tried this, but the tcpdump still looks the same (two packets: one
with the headers, one with the body), and now it fails with

urllib2.HTTPErr or: HTTP Error 501: Not Implemented

Nevertheless, I'm fairly sure that the packet fragmentation is
the culprit. It works perfectly with Perl, even when I make
no effort at all to spoof the browser (no user-agent, referer,
cookies, etc.).
It really does seem quite bizarre that a server should respond
differently to the same TCP request when it is split differently into IP
datagrams.

There really is nothing wrong (from a standards point of view) with
sending FIN with your last data segment. FIN means "I guarantee to send
no more data, and will continue to acknowledge your segments until I see
your FIN".

Are you planning to report this bug to Ipswitch? It certainly sounds
like someone should.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

Nov 13 '06 #6
[Steve Holden]
It really does seem quite bizarre that a server should respond
differently to the same TCP request when it is split differently into IP
datagrams.

There really is nothing wrong (from a standards point of view) with
sending FIN with your last data segment. FIN means "I guarantee to send
no more data, and will continue to acknowledge your segments until I see
your FIN".
It is the server that sends the FIN. What happens is this (each
line corresponds to one packet):

client: POST request headers
client: POST request body
server: FIN + ACK

On receiving the FIN + ACK packet, Python gets immediate
end-of-file on the POST request. Unless the order of the
parameters in the POST request matters (I haven't yet tested this),
I have no other explanation than the fragmentation. If Ipswitch
bothers to reply to my bug report, I'll look into it. Otherwise,
I'm not wasting any more time on this -- it's not that big a deal
for me personally, since I have already scripted the stuff I needed
with Perl.
Are you planning to report this bug to Ipswitch? It certainly sounds
like someone should.
I quickly browsed through ipswitch.com, but couldn't find any good
place to submit bugs. I ended up using the product feedback web
form. Wrote a one-line summary, and referred to this thread on
Google Groups.

--
Haakon
Nov 13 '06 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
15942
by: scummer | last post by:
Hi, I am having a problem with the httplib HTTPConnection object. While I can easily send requests that don't have any payload (ie. "get"), I encounter issues if I want to post xml data. If you look at the class below, when req == 'getFreeBusyInfo' all functions perfectly, but when req == 'conflictRequest' I run into problems of the nature "400 Bad Request". As you will notice, I also tried the shorter HTTPConnection instance method...
0
1910
by: Milos Prudek | last post by:
How can I set httplib timeout for httplib.request() ? httplib.request timeouts after 3:10 with "socket" timeout error message. I found socket.settimeout() and I believe there is a way to add ".settimeout()" function to httplib without hacking httplib source. Is it possible? --
1
2885
by: Brian Beck | last post by:
Hi. I'm having some problems with code based directly on the following httplib documentation code: http://www.zvon.org/other/python/doc21/lib/httplib-examples.html I've included the code and traceback at the end of this post. The odd thing is, using DEPRECATED FUNCTIONS to perform the same function works fine!
0
1481
by: Laszlo Zsolt Nagy | last post by:
Hello, This is from the docs, from section 11.6.1 (HTTPConnection Objects) HTTPConnection instances have the following methods: request( method, url]) The headers argument should be a mapping of extra HTTP headers to send with the request.
0
1296
by: knguyen | last post by:
Hi, For some reason, httplib request() method splits the request packet into two packets, the first packet contains only HTTP headers, the body in the second packet. The first packet size is way below the MTU size. Is there a way I can send everything in one packet? Below is a piece of my code: self.conn.request(method, url, msg, headers) response = self.conn.getresponse()
0
2943
by: Robert | last post by:
did you solve this problem? It seems to be still present here with py2.3.5. Robert -- From: Manish Jethani <manish.j@gmx.net> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6b) Gecko/20031205 Thunderbird/0.4 X-Accept-Language: en-us, en
0
1445
by: philip20060308 | last post by:
Hi all, Has anyone ever seen Python 2.4.1's httplib choke when reading chunked content? I'm using it via urrlib2, and I ran into a particular server that returns something that httplib doesn't expect. Specifically, in the code below where the error occurs, line == ''. Python 2.4.1 (#2, Oct 12 2005, 01:36:32) 20050518] on freebsd6 Type "help", "copyright", "credits" or "license" for more information. >>> import urllib2
4
2328
by: Patrick Altman | last post by:
I am attempting to use a HEAD request against Amazon S3 to check whether a file exists or not and if it does parse the md5 hash from the ETag in the response to verify the contents of the file so as to save on bandwidth of uploading files when it is not necessary. If the file exist, the HEAD works as expected and I get valid headers back that I can parse and pull the ETag out of the dictionary using getheader('ETag') (using the slice to...
3
4080
by: rhXX | last post by:
hi all, i'm using this tutorial example import httplib h = httplib.HTTP("www.python.org") h.putrequest('GET','/index.html') h.putheader('User-Agent','Lame Tutorial Code') h.putheader('Accept','text/html')
0
11185
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10778
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10887
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10439
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7148
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5824
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4642
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4247
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
3252
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.