473,325 Members | 2,480 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,325 software developers and data experts.

urllib2 question


Hi All,

I would like to save a URL into a file. I need to manage cookies and use
my own HTTP headers so I'm using urllib2 and a custom openerdirector.
Here is a code fragment:

while True:
data = openerdirector.read(1024)
fd.write(data)
if not data:
break

The main reason for doing this is that the URL can contain big amounts
of data, and I do not want to store it in memory. The other way would be:

fd.write(openerdirector.read())

My question is that am I doing this the right way? I used the
openerdirector as a file here, but I'm not sure if it works like a file.
A file object should block when read() is called, and then return at
most 1024 bytes of data when available. It will return with an empty
string only after EOF reached. But is the same true for the
openerdirector instance? I did not find useful documentation in the
python docs about this. The documentation says:

*class OpenerDirector*( )

The OpenerDirector class opens URLs via BaseHandlers chained
together. It manages the chaining of handlers, and recovery from
errors.

But I'm not sure if it can be used as a file or not.

Thanks,

Laszlo
Jan 23 '07 #1
0 868

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Matthew Wilson | last post by:
I am writing a script to check on my router's external IP address. My ISP refreshes my IP very often and I use dyndns for the hostname for my computer. My Netgear mr814 router has a webserver that...
4
by: Fuzzyman | last post by:
urllib2 (under windows) will auto-detect your proxy settings and use those. Normally that's a good thing (I guess), except when it's not ! How do I switch off this behaviour ? I'm behind a...
2
by: ken | last post by:
Hi, i have the following code to load a url. My question is what if I try to load an invalide url ("http:// www.heise.de/"), will I get an IOException? or it will wait forever? Thanks for any...
6
by: Jack | last post by:
I'm trying to use a proxy server with urllib2. So I have managed to get it to work by setting the environment variable: export HTTP_PROXY=127.0.0.1:8081 But I wanted to set it from the code....
6
by: robean | last post by:
Hi everyone, I have a question about using urllib2. I like urllib2 better than urllib at least in part because it has more elaborate support for handling errors: there is built in support for...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.