473,226 Members | 1,258 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,226 software developers and data experts.

ftplib error- Large file

Hi all,

I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?

Traceback (most recent call last):

File "/usr/local/sw/bin/FTPput.py", line 142, in execute
self.ftp.put(file.path)

File "/usr/local/sw/lib/FTPconn.py", line 32, in put
return self.storbinary("STOR %s" % destfile, open(f.path, 'rb'))

File "/usr/lib/python2.2/ftplib.py", line 422, in storbinary
return self.voidresp()

File "/usr/lib/python2.2/ftplib.py", line 222, in voidresp
resp = self.getresp()

File "/usr/lib/python2.2/ftplib.py", line 208, in getresp
resp = self.getmultiline()

File "/usr/lib/python2.2/ftplib.py", line 194, in getmultiline
line = self.getline()

File "/usr/lib/python2.2/ftplib.py", line 181, in getline
line = self.file.readline()

IOError: [Errno 104] Connection reset by peer

I'd appreciate any input or ideas.

~Sean

Jun 6 '07 #1
6 5714
En Wed, 06 Jun 2007 13:39:57 -0300, <ha**********@gmail.comescribió:
I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?
IOError: [Errno 104] Connection reset by peer
You always have to consider a connection failure - it's just more likely
to happen on a big file.
The REST ftp command is used to restart the transfer from a certain offset.

--
Gabriel Genellina

Jun 6 '07 #2
<hal...an@gmail.comwrote:
Hi all,

I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?
20 gig is a big thing - what is your link speed, and how long does the thing
run before falling over - Is your ISP maybe giving you or the remote site a
new IP addy periodically?

Are there any "anti jabber" precautions taken anywhere?

you may have to break it up into smaller chunks and glue them together
remotely...

hth - Hendrik
Jun 7 '07 #3
On Jun 6, 11:21 pm, "Hendrik van Rooyen" <m...@microcorp.co.zawrote:
<hal.....@gmail.comwrote:
Hi all,
I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?

20 gig is a big thing - what is your link speed, and how long does the thing
run before falling over - Is your ISP maybe giving you or the remote site a
new IP addy periodically?

Are there any "anti jabber" precautions taken anywhere?

you may have to break it up into smaller chunks and glue them together
remotely...

hth - Hendrik
It's different sites (state to state) within the same company, so I'm
sure of the IP addresses. Whats odd is that the transfers don't fail
most of the time. The transfer usually completes properly but
ftplib.FTP.storbinary() throws an IOError at the end of the file
instead of returning the status of the transfer. I've resorted to
just catching and logging the exception and then comparing remote vs.
local md5's to check for integrity, but it seems like if a connection
error occurs, storbinary should handle the error and return an error
code rather than throwing an exception. The 20gb files take about an
hour to transfer, and I've had success with files up to 60gb so far...
once I figured out what ftplib was doing and starting cathing the
IOError.

Why would storbinary throw an exception even when the file transfer
was successful? Why would the exception not be thrown until after the
file was sent? Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error? Is my
reasoning off somewhere?

~Sean
Jun 7 '07 #4
ha**********@gmail.com wrote:
Why would storbinary throw an exception even when the file transfer
was successful? Why would the exception not be thrown until after the
file was sent? Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error? Is my
reasoning off somewhere?
There was an error, the connection was reset by peer. The error code is
104. As it *was* an error, an exception is correct.

You always can hide it yourself, with try/except...

Regards,

--
.. Facundo
..
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
Jun 7 '07 #5
On Jun 7, 8:54 am, Facundo Batista <facu...@taniquetil.com.arwrote:
half.ital...@gmail.com wrote:
Why would storbinary throw an exception even when the file transfer
was successful? Why would the exception not be thrown until after the
file was sent? Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error? Is my
reasoning off somewhere?

There was an error, the connection was reset by peer. The error code is
104. As it *was* an error, an exception is correct.

You always can hide it yourself, with try/except...

Regards,

--
. Facundo
.
Blog:http://www.taniquetil.com.ar/plog/
PyAr:http://www.python.org/ar/
Ok. I guess that makes sense. But what about the other
questions...mainly: Why would it throw an exception even though the
file was properly transferred?

I guess this could be blamed on the FTP server.

~Sean

Jun 7 '07 #6
ha**********@gmail.com wrote:
Ok. I guess that makes sense. But what about the other
questions...mainly: Why would it throw an exception even though the
file was properly transferred?
Je, well, I answered the one I knew about, :)

Regarding the error... es hard to say.

What happens if you transmit the file using a standard ftp program?

Or better, could you sniff your network to see what is the TCP dialog?

I guess this could be blamed on the FTP server.
Maybe. Maybe not...

--
.. Facundo
..
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
Jun 8 '07 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: python | last post by:
Hi: I want to write a procedure to automatically upload some files for me, but I'm getting stuck. Before I write my own gruesome put() function, I wanted to check if there is an easier way. ...
1
by: John fabiani | last post by:
Hi, I have a script that is not running on a freeBSD 4.4 with Python 2.1 that uses ftplib. I'm wondering if it's possible to update only the ftplib module from 2.1 to 2.3. I only ask because I...
0
by: Mikael | last post by:
Hello, I am using the ftplib.FTP.nlst() method to list the files in a directory on a FTP server. But when the directoty is empty, python raises an error: ftplib.error_perm: 550 No files...
0
by: Beeyah | last post by:
in my inability to find a working ftp program for my mac, i decided to write one in python. i wrote a class (link at end of post) that inherited ftplib.FTP and made the process slightly less...
2
by: Matija Papec | last post by:
I would like to reimplement ftplib "nlst" using ftplib.dir (ftp server doesn't support nlst) so I'm trying to guess how to use ftp callbacks. Any help is appreciated. tia! ============...
1
by: Peter A. Schott | last post by:
Using ftplib.FTP object for a project we have here to upload/download files. I know that I can wrap everything in try/except blocks, but I'm having trouble getting the exact error messages out of...
3
by: Peter A. Schott | last post by:
Does the ftplib support SFTP or FTPS? Is that part of a different module? We have a handful of partners who use FTPS or SFTP and I need to pull/push files to/from them. Thank you for all of...
0
by: vedran | last post by:
Hello, One simple question.Which security ftplib support,and how to I 'load' these security to ftputil. (Please don't give me security modules) e.g. ...
1
by: loial | last post by:
Trying to use ftplib.FTP.nlst() method to list the files in a directory on a FTP server. It works fine except when there are no files in the directory. Then it gives the error ...
1
by: Laszlo Nagy | last post by:
Hi All, I'm using a simple program that uploads a file on a remote ftp server. This is an example (not the whole program): def store(self,hostname,username,password,destdir,srcpath):...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.