473,903 Members | 5,218 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Multiple FTP download using Muliti thread

I have taken a look at the code that dose one download at time, in
multi threaded manner:
http://aspn.activestate.com/ASPN/Coo.../Recipe/465531

What I wanted to do is, make it download multiple files at the same
time. I am new to python and have gone over "Dive In To Python"
yesterday. Can some give a idea what I need to do. Once I understand
the big picture, then I can piece together.

Dec 1 '06
12 6205
johnny wrote:
It places the ftp downloaded contents on the same folder as the this
ftp python script. How do I set a diffrent download folder location?
by prepending a directory name to the filename in the open(p, 'wb') call.

</F>

Dec 5 '06 #11
I am getting the following error:

raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Here is the code:

import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host )
c.connect()
try:
c.login()
p = posixpath.basen ame(e)
fp = open('H:/eclipse/workspace/src/ftp_download/' + p,
'wb')
try: c.retrbinary('R ETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()

tq.task_done()

if __name__ == '__main__':
q = TaskQueue()
#host = 'ftp.microsoft. com'
host = 'mysite.com'
c = ftplib.FTP(host )
c.connect()
try:
#c.login()
c.login("te**@m ysite.com","tem ppass" )

#folder = '/deskapps/kids/'
folder = ''
for n in c.nlst(folder):
#if n.lower().endsw ith('.exe'):
# q.put((host, n))
if n.lower().endsw ith('.jpg'):
q.put((host, n))
elif n.lower().endsw ith('.jpeg'):
q.put((host, n))

finally: c.close()

numworkers = 4
for i in range(numworker s):
t = threading.Threa d(target=worker , args=(q,))
t.setDaemon(Tru e)
t.start()

q.join()
print 'Done.'
Justin Ezequiel wrote:
johnny wrote:
When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Dose the host should allow 4 simultaneous login at a time?

does it work using ftp.microsoft.c om?

post your code
Dec 5 '06 #12
Ok I fixed it. Needed to put in username, and password in the c.login
inside while True loop.

while True:
host, e = tq.get()

c = ftplib.FTP(host )
c.connect()
try:
c.login()
p = posixpath.basen ame(e)
fp = open('H:/eclipse/workspace/src/ftp_download/' + p,
'wb')

johnny wrote:
I am getting the following error:

raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Here is the code:

import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host )
c.connect()
try:
c.login()
p = posixpath.basen ame(e)
fp = open('H:/eclipse/workspace/src/ftp_download/' + p,
'wb')
try: c.retrbinary('R ETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()

tq.task_done()

if __name__ == '__main__':
q = TaskQueue()
#host = 'ftp.microsoft. com'
host = 'mysite.com'
c = ftplib.FTP(host )
c.connect()
try:
#c.login()
c.login("te**@m ysite.com","tem ppass" )

#folder = '/deskapps/kids/'
folder = ''
for n in c.nlst(folder):
#if n.lower().endsw ith('.exe'):
# q.put((host, n))
if n.lower().endsw ith('.jpg'):
q.put((host, n))
elif n.lower().endsw ith('.jpeg'):
q.put((host, n))

finally: c.close()

numworkers = 4
for i in range(numworker s):
t = threading.Threa d(target=worker , args=(q,))
t.setDaemon(Tru e)
t.start()

q.join()
print 'Done.'
Justin Ezequiel wrote:
johnny wrote:
When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP
>
Dose the host should allow 4 simultaneous login at a time?
>
does it work using ftp.microsoft.c om?

post your code
Dec 5 '06 #13

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
2691
by: Dominik | last post by:
I am writing application which must download multiple files simultaneous. Initially I wanted to use multiple threads to achieve this task but later I heard that Net is "using" separate thread from ThradPool for each connection. What is the best (fastest) method to download multiple files from one server in .NET ???? (sorry for my english)
6
5012
by: James Radke | last post by:
Hello, I have a multithreaded windows NT service application (vb.net 2003) that I am working on (my first one), which reads a message queue and creates multiple threads to perform the processing for long running reports. When the processing is complete it uses crystal reports to load a template file, populate it, and then export it to a PDF. It works fine so far....
1
3449
by: Ryan Malone | last post by:
I have a situation where I need to download multiple files in a vb.net application. To speed up the process, I am trying to download multiple files at one time looping through each of the files and launching them in their own thread (code below). The problem is that when there are hundreds of files to download, it launches them all at the same time and half don't get downloaded. Any idea how I could control it to launch say 4 at a time,...
4
1747
by: VBTricks.de.vu Webmaster | last post by:
Hello, at the moment I'm not able to finish my program. That means, I have a problem I can't solve myself. Therefore I'm asking those who already worked with HttpWebRequest... and threads. I'm writing some kind of download-manager, which is able to download several files parallely. Therefore I've created a class implementing the download function and use four threads which use four instances of the class. This works fine, except the...
12
17682
by: Scott | last post by:
Front-end Access 2000 I have a stored procedure that has 2 parameters BusinessUnitID and Year. It returns multiple record sets (5 to be exact). I thought I could use a Pass through query but that only returns the first record set. I thought I could use ADO but that does not seem to work. I get an Error
19
3072
by: santosh | last post by:
Hi all, In the following program I allocate a block of pointers to type char, initialised to zero. I then point each of those pointers to a block of allocated memory of fixed size (33 bytes). A unique 'string' is then generated using itoa() and rand() for each block of memory. Finally using pointer-to-pointer of type char, I print the contents of the blocks of memory, i.e. the strings, using both printf() and manually, character by...
5
2405
by: gjzusenet | last post by:
Hello. Though Python supports threading, I think it is limited to python code - as soon as you issue a command that uses an external (C?) module, all of your python threads hang until this command returns. Is that true? I'm using urllib2 to download many files, and I have a double problem: 1. downloading all of them is painfully slow since it's serial - one has to finish before the next request gets sent. 2. my GUI becomes non responsive...
35
9382
by: keerthyragavendran | last post by:
hi i'm downloading a single file using multiple threads... how can i specify a particular range of bytes alone from a single large file... for example say if i need only bytes ranging from 500000 to 3200000 of a file whose size is say 20MB... how do i request a download which starts directly at 500000th byte... thank u cheers
2
2005
by: ShashiGowda | last post by:
Hey there i made a script to download all images from a web site but it runs damn slow though I have a lot of bandwidth waiting to be used please tell me a way to use urllib to open many connections to the server to download many pics simultaneously.... Any off question suggestions are also ok...
0
9847
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
11283
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10875
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10986
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10501
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9685
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
7206
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
6093
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
4308
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.