473,545 Members | 1,947 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Urllib(1/2) how to open multiple client sockets?

Hey there i made a script to download all images from a web site but
it runs damn slow though I have a lot of bandwidth waiting to be used
please tell me a way to use urllib to open many connections to the
server to download many pics simultaneously. ... Any off question
suggestions are also ok...
Jun 27 '08 #1
2 1988
ShashiGowda wrote:
Hey there i made a script to download all images from a web site but
it runs damn slow though I have a lot of bandwidth waiting to be used
please tell me a way to use urllib to open many connections to the
server to download many pics simultaneously. ... Any off question
suggestions are also ok...
"Simultaneously " means using multiple threads (or processes).

You could create worker threads that do the downloading (use the
threading module for this) and communicate with them via a queue (use
the Queue module for this).

For example the main thread would then do the HTML parsing and push the
image URLs to the worker threads via the queue.

Each worker thread then polls the queue and downloads the images it gets.

One issue is how to terminate the multithreaded application. A simple
approach is to use a sentinel value to push to the queue to signal the
worker threads to quit. Something like "QUIT" will do. Be sure to push
at least as many sentinel values as you have worker threads, then.

-- Gerhard

Jun 27 '08 #2
On Jun 26, 11:48*am, ShashiGowda <connect2sha... @gmail.comwrote :
Hey there i made a script to download all images from a web site but
it runs damn slow though I have a lot of bandwidth waiting to be used
please tell me a way to use urllib to open many connections to the
server to download many pics simultaneously. ... Any off question
suggestions are also ok...
You also need to remember the bandwidth of the website itself, which
is being shared with other users.
Jun 27 '08 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
2394
by: Andreas Dahl | last post by:
Hi, I use urllib to retrieve data via HTTP. Unfortunately my program crashes after a while (after some loops) because the connection timed out. raise socket.error, msg IOError: (60, 'Connection timed out') I am not so familiar with python, but is there a possibility to configure the 'waiting time'? Or how can I handle such an event?...
1
2485
by: John Hunter | last post by:
I have a test script below which I use to fetch urls into strings, either over https or http. When over https, I use m2crypto.urllib and when over http I use the standard urllib. Whenever, I import sockets and setdefaulttimeout, however, using m2crypto.urllib tends to cause a http.BadStatusLine to be raised, even if the timeout is set to be...
0
1285
by: Shane Hathaway | last post by:
I started experimenting with SOAPpy yesterday and immediately hit a snag. Both web services I tried simply hung and never replied. After a lot of digging, I found out what was going wrong: urllib.urlopen() is issuing an HTTP/1.0 request, but Microsoft IIS 5 ignores the client HTTP version and replies with an HTTP/1.1 response. This is a...
3
2218
by: Chris Tavares | last post by:
Hi all. I'm currently tracking down a problem in a little script I have, and I was hoping that those more experienced than myself could weigh in. The script's job is to grab the status page off a DLink home router. This is a really simple job: I just use urllib.urlopen() to grab the status page. The router uses HTTP Basic authentication, so...
1
1492
by: Astan Chee | last post by:
Hi Guys, I have a python script which runs perfectly on my machine. However a machine that I tested it on gives the following error message: Traceback (most recent call last): File "whip.py", line 616, in OnRebootRunning File "whip.py", line 626, in RebootCustom File "urllib.pyc", line 77, in urlopen File "urllib.pyc", line 170, in open
11
3551
by: Johnny Lee | last post by:
Hi, I was using urllib to grab urls from web. here is the work flow of my program: 1. Get base url and max number of urls from user 2. Call filter to validate the base url 3. Read the source of the base url and grab all the urls from "href" property of "a" tag 4. Call filter to validate every url grabbed 5. Continue 3-4 until the number...
11
2549
by: Stefan Palme | last post by:
Hi all, is there a way to modify the time a call of urllib.open(...) waits for an answer from the other side? Have a tool which automatically checks a list of websites for certain content. The tool "hangs" when one of the contacted websites behaves badly and "never" answers...
5
7676
by: John Nagle | last post by:
I thought I had all the timeout problems with urllib worked around, but no. socket.setdefaulttimeout is useful, but not always effective. I'm setting that to 15 seconds. If the host end won't open the connection within 15 seconds, urllib times out. But if the host end opens the connection, then never sends anything, urllib waits for many...
5
13028
by: chrispoliquin | last post by:
Hi, I have a small Python script to fetch some pages from the internet. There are a lot of pages and I am looping through them and then downloading the page using urlretrieve() in the urllib module. The problem is that after 110 pages or so the script sort of hangs and then I get the following traceback: Traceback (most recent call...
0
7484
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7928
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
7775
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
4963
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3470
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3451
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1902
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1030
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
726
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.