By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
457,903 Members | 1,677 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 457,903 IT Pros & Developers. It's quick & easy.

Urllib(1/2) how to open multiple client sockets?

P: n/a
Hey there i made a script to download all images from a web site but
it runs damn slow though I have a lot of bandwidth waiting to be used
please tell me a way to use urllib to open many connections to the
server to download many pics simultaneously.... Any off question
suggestions are also ok...
Jun 27 '08 #1
Share this Question
Share on Google+
2 Replies


P: n/a
ShashiGowda wrote:
Hey there i made a script to download all images from a web site but
it runs damn slow though I have a lot of bandwidth waiting to be used
please tell me a way to use urllib to open many connections to the
server to download many pics simultaneously.... Any off question
suggestions are also ok...
"Simultaneously" means using multiple threads (or processes).

You could create worker threads that do the downloading (use the
threading module for this) and communicate with them via a queue (use
the Queue module for this).

For example the main thread would then do the HTML parsing and push the
image URLs to the worker threads via the queue.

Each worker thread then polls the queue and downloads the images it gets.

One issue is how to terminate the multithreaded application. A simple
approach is to use a sentinel value to push to the queue to signal the
worker threads to quit. Something like "QUIT" will do. Be sure to push
at least as many sentinel values as you have worker threads, then.

-- Gerhard

Jun 27 '08 #2

P: n/a
On Jun 26, 11:48*am, ShashiGowda <connect2sha...@gmail.comwrote:
Hey there i made a script to download all images from a web site but
it runs damn slow though I have a lot of bandwidth waiting to be used
please tell me a way to use urllib to open many connections to the
server to download many pics simultaneously.... Any off question
suggestions are also ok...
You also need to remember the bandwidth of the website itself, which
is being shared with other users.
Jun 27 '08 #3

This discussion thread is closed

Replies have been disabled for this discussion.