473,387 Members | 1,517 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,387 software developers and data experts.

urllib2 and transfer-encoding = chunked

I am having errors which appear to be linked to a previous bug in
urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
Has anyone established a standard workaround? I keep finding old
posts about it, that basically give up and say "well it's a known bug."
Any help would be greatly appreciated.

Jan 18 '07 #1
3 5960
<jd****@gmail.comescribió en el mensaje
news:11**********************@l53g2000cwa.googlegr oups.com...
>I am having errors which appear to be linked to a previous bug in
urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
Has anyone established a standard workaround? I keep finding old
posts about it, that basically give up and say "well it's a known bug."
Any help would be greatly appreciated.
Perhaps if you said what the supposed error is...

--
Gabriel Genellina
Jan 18 '07 #2
Haha! My mistake.

The error is that when a web server is chunking a web page only the
first chunk appears to be acquired by the urllib2.urlopen call. If you
check the headers, there is no 'Content-length' (as expected) and
instead there is 'transfer-encoding' = 'chunked'. I am getting about
the first 30Kb, and then nothing else.

I don't get a ValueError like described at the following post:

http://groups.google.com/group/comp....3ea9ca84b28669

Here's the code that I think is failing, it's basically textbook Python
for accessing a url:

file = urllib2.urlopen(url)
contenttype = file.info().type
if contenttype and contenttype.find('text') -1:
return file.read()
#
# I am checking the content type because I don't want to download
..jpegs and the like
#

I have typed similar commands into the interpreter, which also produces
only about the first 30KB of the url.

Sorry for the confusion.

Gabriel Genellina wrote:
<jd****@gmail.comescribió en el mensaje
news:11**********************@l53g2000cwa.googlegr oups.com...
I am having errors which appear to be linked to a previous bug in
urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
Has anyone established a standard workaround? I keep finding old
posts about it, that basically give up and say "well it's a known bug."
Any help would be greatly appreciated.

Perhaps if you said what the supposed error is...

--
Gabriel Genellina
Jan 18 '07 #3
In article <11**********************@51g2000cwl.googlegroups. com>,
jd****@gmail.com wrote:
Haha! My mistake.

The error is that when a web server is chunking a web page only the
first chunk appears to be acquired by the urllib2.urlopen call. If you
check the headers, there is no 'Content-length' (as expected) and
instead there is 'transfer-encoding' = 'chunked'. I am getting about
the first 30Kb, and then nothing else.

I don't get a ValueError like described at the following post:
Hi jdvolz,
What error *do* you get? Or is it that no error is raised; you're just
not getting all of the data? If it is the latter, then the sending
server might be at fault for not properly following the chunked transfer
protocol. One way to find out would be to fire up Ethereal and see
what's coming down the wire.
I am having errors which appear to be linked to a previous bug in
urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
Has anyone established a standard workaround? I keep finding old
posts about it, that basically give up and say "well it's a known bug."
Can you give us some pointers to some of these old posts? And tell us
what version of Python you're using.

--
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more
Jan 20 '07 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Matthew Wilson | last post by:
I am writing a script to check on my router's external IP address. My ISP refreshes my IP very often and I use dyndns for the hostname for my computer. My Netgear mr814 router has a webserver that...
0
by: mw | last post by:
Hi, I get an error when trying to use urllib2 with the ftp daemon of publicfile: $ python Python 2.2.3 (#1, Oct 15 2003, 23:33:35) on linux2 Type "help", "copyright", "credits" or...
2
by: John F Dutcher | last post by:
Can anyone comment on why the code shown in the Python error is in some way incorrect...or is there a problem with Python on my hoster's site ?? The highlites don't seem to show here...but line...
0
by: jacob c. | last post by:
When I request a URL using urllib2, it appears that urllib2 always makes the request using HTTP 1.0, and not HTTP 1.1. I'm trying to use the "If-None-Match"/"ETag" HTTP headers to conserve...
3
by: Artificial Life | last post by:
urllib2 does not seem to be able to handle META-REFRESH in an html document. I just get back the html to the page that is supposed to forward me to the intended page. Any way around this?
0
by: Gil Tal | last post by:
Hi, I use urllib2 to download a redirected url and I get an exception from the bowels of urllib2. It seems that urllib2 implements some super sophisticated self check and tries to control the...
1
by: Ray Slakinski | last post by:
Hello, I have defined a function to set an opener for urllib2, this opener defines any proxy and http authentication that is required. If the proxy has authencation itself and requests an...
0
by: Ali.Sabil | last post by:
hello all, I just maybe hit a bug in both urllib and urllib2, actually urllib doesn't support proxy authentication, and if you setup the http_proxy env var to...
3
by: m.banaouas | last post by:
Hi all, I started to use urllib2 library and HTTPBasicAuthHandler class in order to authenticate with a http server (Zope in this case). I don't know why but it doesn't work, while authenticating...
1
by: Alessandro Fachin | last post by:
I write this simply code that should give me the access to private page with htaccess using a proxy, i don't known because it's wrong... import urllib,urllib2 #input url...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.