By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,649 Members | 2,142 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,649 IT Pros & Developers. It's quick & easy.

Multiple threaded download streams?

P: n/a
Hello.
Though Python supports threading, I think it is limited to python code
- as soon as you issue a command that uses an external (C?) module, all
of your python threads hang until this command returns.
Is that true?
I'm using urllib2 to download many files, and I have a double problem:
1. downloading all of them is painfully slow since it's serial - one
has to finish before the next request gets sent.
2. my GUI becomes non responsive during the downloads - major problem!

Is there any way to work around that?
I want to run multiple download streams, in parallel, and while keeping
my program responsive.
Are there alternative modules that I can use for that?

Any ideas?
Thanks a lot!

Feb 25 '06 #1
Share this Question
Share on Google+
5 Replies


P: n/a
On 2006-02-25, gj*******@gmail.com <gj*******@gmail.com> wrote:
Though Python supports threading, I think it is limited to
python code - as soon as you issue a command that uses an
external (C?) module, all of your python threads hang until
this command returns. Is that true?
No. Not unless the C modules is broken.
I'm using urllib2 to download many files, and I have a double
problem:

1. downloading all of them is painfully slow since it's serial
- one has to finish before the next request gets sent.
Then don't do it that way.
2. my GUI becomes non responsive during the downloads - major
problem!
Use a separate thread for downloading.
Is there any way to work around that?
Yes. Use threads.
I want to run multiple download streams, in parallel, and while keeping
my program responsive.
Then do it that way.
Are there alternative modules that I can use for that?


And what, exactly, didn't work right when you tried the
threading module?

--
Grant Edwards grante Yow! I demand IMPUNITY!
at
visi.com
Feb 25 '06 #2

P: n/a
> Use a separate thread for downloading.

Or the twisted select-reactor. No threads needed.

Diez
Feb 25 '06 #3

P: n/a
Diez B. Roggisch wrote:
Use a separate thread for downloading.


Or the twisted select-reactor. No threads needed.


Although depending on what the GUI is like, and what platform is
involved, one might still want at least a second thread for the reactor
itself.

-Peter

Feb 25 '06 #4

P: n/a
Diez B. Roggisch wrote:
Use a separate thread for downloading.


Or the twisted select-reactor. No threads needed.


He's using urllib2, which does not use Twisted's select-reactor.

Fortunately urllib2 downloads run fine in their own threads; no
rewrite-all-the-code-as-Twisted-state-machines needed.
--
--Bryan
Feb 25 '06 #5

P: n/a
gj*******@gmail.com wrote:
Hello.
Though Python supports threading, I think it is limited to python code
- as soon as you issue a command that uses an external (C?) module, all
of your python threads hang until this command returns.
Is that true?
I'm using urllib2 to download many files, and I have a double problem:
1. downloading all of them is painfully slow since it's serial - one
has to finish before the next request gets sent.
2. my GUI becomes non responsive during the downloads - major problem!

Is there any way to work around that?
I want to run multiple download streams, in parallel, and while keeping
my program responsive.
Are there alternative modules that I can use for that?

Any ideas?
Thanks a lot!

Others have spoken to the specifics of threads, etc. What I wanted
to ask was why you think that parallel downloads will be faster?
Unless you are downloading from multiple and different sites and
your Internet download bandwidth is much greater than any individual
site you wish to download from, you won't speed things up. If you
wish to download several files from the SAME site, I doubt that
threading is going to help. Most likely you are limited by the
upload bandwidth of that site. Now if the site you are downloading
from has some throttling turned on to limit the speed of an
individual download, you could gain some benefit. You can't push
bits through the pipes faster than their upper bandwidth.

Just some thoughts to consider.

-Larry Bates
Feb 25 '06 #6

This discussion thread is closed

Replies have been disabled for this discussion.