By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,400 Members | 903 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,400 IT Pros & Developers. It's quick & easy.

Multiple FTP download using Muliti thread

P: n/a
I have taken a look at the code that dose one download at time, in
multi threaded manner:
http://aspn.activestate.com/ASPN/Coo.../Recipe/465531

What I wanted to do is, make it download multiple files at the same
time. I am new to python and have gone over "Dive In To Python"
yesterday. Can some give a idea what I need to do. Once I understand
the big picture, then I can piece together.

Dec 1 '06 #1
Share this Question
Share on Google+
12 Replies


P: n/a
import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
fp = open(p, 'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()

tq.task_done()

if __name__ == '__main__':
q = TaskQueue()
host = 'ftp.microsoft.com'

c = ftplib.FTP(host)
c.connect()
try:
c.login()
folder = '/deskapps/kids/'
for n in c.nlst(folder):
if n.lower().endswith('.exe'):
q.put((host, n))
finally: c.close()

numworkers = 4
for i in range(numworkers):
t = threading.Thread(target=worker, args=(q,))
t.setDaemon(True)
t.start()

q.join()
print 'Done.'

Dec 4 '06 #2

P: n/a
Justin Ezequiel wrote:
from TaskQueue import TaskQueue
what Python version is this ?

</F>

Dec 4 '06 #3

P: n/a

Fredrik Lundh wrote:
Justin Ezequiel wrote:
from TaskQueue import TaskQueue

what Python version is this ?

</F>
oops. forgot to note...

http://aspn.activestate.com/ASPN/Coo.../Recipe/475160

Dec 4 '06 #4

P: n/a
Justin Ezequiel wrote:
Fredrik Lundh wrote:
>Justin Ezequiel wrote:
from TaskQueue import TaskQueue

what Python version is this ?

</F>

oops. forgot to note...

http://aspn.activestate.com/ASPN/Coo.../Recipe/475160
Also noteworthy: "This recipe was accepted for inclusion in Py2.5.", i. e.
in Python 2.5 you can use Queue instead of TaskQueue.

Peter
Dec 4 '06 #5

P: n/a
Where or What folder does the ftp files get downloaded to?

Justin Ezequiel wrote:
import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
fp = open(p, 'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()

tq.task_done()

if __name__ == '__main__':
q = TaskQueue()
host = 'ftp.microsoft.com'

c = ftplib.FTP(host)
c.connect()
try:
c.login()
folder = '/deskapps/kids/'
for n in c.nlst(folder):
if n.lower().endswith('.exe'):
q.put((host, n))
finally: c.close()

numworkers = 4
for i in range(numworkers):
t = threading.Thread(target=worker, args=(q,))
t.setDaemon(True)
t.start()

q.join()
print 'Done.'
Dec 4 '06 #6

P: n/a
When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Dose the host should allow 4 simultaneous login at a time?

Justin Ezequiel wrote:
import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
fp = open(p, 'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()

tq.task_done()

if __name__ == '__main__':
q = TaskQueue()
host = 'ftp.microsoft.com'

c = ftplib.FTP(host)
c.connect()
try:
c.login()
folder = '/deskapps/kids/'
for n in c.nlst(folder):
if n.lower().endswith('.exe'):
q.put((host, n))
finally: c.close()

numworkers = 4
for i in range(numworkers):
t = threading.Thread(target=worker, args=(q,))
t.setDaemon(True)
t.start()

q.join()
print 'Done.'
Dec 4 '06 #7

P: n/a

johnny wrote:
When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Dose the host should allow 4 simultaneous login at a time?
does it work using ftp.microsoft.com?

post your code

Dec 5 '06 #8

P: n/a
It works using ftp.microsoft.com. But where does it put the downloaded
files? can I specify a download folder location?

Justin Ezequiel wrote:
johnny wrote:
When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Dose the host should allow 4 simultaneous login at a time?

does it work using ftp.microsoft.com?

post your code
Dec 5 '06 #9

P: n/a
It places the ftp downloaded contents on the same folder as the this
ftp python script. How do I set a diffrent download folder location?

johnny wrote:
It works using ftp.microsoft.com. But where does it put the downloaded
files? can I specify a download folder location?

Justin Ezequiel wrote:
johnny wrote:
When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP
>
Dose the host should allow 4 simultaneous login at a time?
>
does it work using ftp.microsoft.com?

post your code
Dec 5 '06 #10

P: n/a
johnny wrote:
It places the ftp downloaded contents on the same folder as the this
ftp python script. How do I set a diffrent download folder location?
by prepending a directory name to the filename in the open(p, 'wb') call.

</F>

Dec 5 '06 #11

P: n/a
I am getting the following error:

raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Here is the code:

import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
fp = open('H:/eclipse/workspace/src/ftp_download/' + p,
'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()

tq.task_done()

if __name__ == '__main__':
q = TaskQueue()
#host = 'ftp.microsoft.com'
host = 'mysite.com'
c = ftplib.FTP(host)
c.connect()
try:
#c.login()
c.login("te**@mysite.com","temppass" )

#folder = '/deskapps/kids/'
folder = ''
for n in c.nlst(folder):
#if n.lower().endswith('.exe'):
# q.put((host, n))
if n.lower().endswith('.jpg'):
q.put((host, n))
elif n.lower().endswith('.jpeg'):
q.put((host, n))

finally: c.close()

numworkers = 4
for i in range(numworkers):
t = threading.Thread(target=worker, args=(q,))
t.setDaemon(True)
t.start()

q.join()
print 'Done.'
Justin Ezequiel wrote:
johnny wrote:
When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Dose the host should allow 4 simultaneous login at a time?

does it work using ftp.microsoft.com?

post your code
Dec 5 '06 #12

P: n/a
Ok I fixed it. Needed to put in username, and password in the c.login
inside while True loop.

while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
fp = open('H:/eclipse/workspace/src/ftp_download/' + p,
'wb')

johnny wrote:
I am getting the following error:

raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP

Here is the code:

import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
fp = open('H:/eclipse/workspace/src/ftp_download/' + p,
'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()

tq.task_done()

if __name__ == '__main__':
q = TaskQueue()
#host = 'ftp.microsoft.com'
host = 'mysite.com'
c = ftplib.FTP(host)
c.connect()
try:
#c.login()
c.login("te**@mysite.com","temppass" )

#folder = '/deskapps/kids/'
folder = ''
for n in c.nlst(folder):
#if n.lower().endswith('.exe'):
# q.put((host, n))
if n.lower().endswith('.jpg'):
q.put((host, n))
elif n.lower().endswith('.jpeg'):
q.put((host, n))

finally: c.close()

numworkers = 4
for i in range(numworkers):
t = threading.Thread(target=worker, args=(q,))
t.setDaemon(True)
t.start()

q.join()
print 'Done.'
Justin Ezequiel wrote:
johnny wrote:
When I run the following script, with host and password and username
changed, I get the following errors:
raise error_temp, resp
error_temp: 421 Unable to set up secure anonymous FTP
>
Dose the host should allow 4 simultaneous login at a time?
>
does it work using ftp.microsoft.com?

post your code
Dec 5 '06 #13

This discussion thread is closed

Replies have been disabled for this discussion.