Hi all,
I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?
Traceback (most recent call last):
File "/usr/local/sw/bin/FTPput.py", line 142, in execute
self.ftp.put(fi le.path)
File "/usr/local/sw/lib/FTPconn.py", line 32, in put
return self.storbinary ("STOR %s" % destfile, open(f.path, 'rb'))
File "/usr/lib/python2.2/ftplib.py", line 422, in storbinary
return self.voidresp()
File "/usr/lib/python2.2/ftplib.py", line 222, in voidresp
resp = self.getresp()
File "/usr/lib/python2.2/ftplib.py", line 208, in getresp
resp = self.getmultili ne()
File "/usr/lib/python2.2/ftplib.py", line 194, in getmultiline
line = self.getline()
File "/usr/lib/python2.2/ftplib.py", line 181, in getline
line = self.file.readl ine()
IOError: [Errno 104] Connection reset by peer
I'd appreciate any input or ideas.
~Sean 6 5772
En Wed, 06 Jun 2007 13:39:57 -0300, <ha**********@g mail.comescribi ó:
I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?
IOError: [Errno 104] Connection reset by peer
You always have to consider a connection failure - it's just more likely
to happen on a big file.
The REST ftp command is used to restart the transfer from a certain offset.
--
Gabriel Genellina
<hal...an@gmail .comwrote:
Hi all,
I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?
20 gig is a big thing - what is your link speed, and how long does the thing
run before falling over - Is your ISP maybe giving you or the remote site a
new IP addy periodically?
Are there any "anti jabber" precautions taken anywhere?
you may have to break it up into smaller chunks and glue them together
remotely...
hth - Hendrik
On Jun 6, 11:21 pm, "Hendrik van Rooyen" <m...@microcorp .co.zawrote:
<hal.....@gmail .comwrote:
Hi all,
I'm using ftplib to transfer large files to remote sites. The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete. I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site? Packet loss?
20 gig is a big thing - what is your link speed, and how long does the thing
run before falling over - Is your ISP maybe giving you or the remote site a
new IP addy periodically?
Are there any "anti jabber" precautions taken anywhere?
you may have to break it up into smaller chunks and glue them together
remotely...
hth - Hendrik
It's different sites (state to state) within the same company, so I'm
sure of the IP addresses. Whats odd is that the transfers don't fail
most of the time. The transfer usually completes properly but
ftplib.FTP.stor binary() throws an IOError at the end of the file
instead of returning the status of the transfer. I've resorted to
just catching and logging the exception and then comparing remote vs.
local md5's to check for integrity, but it seems like if a connection
error occurs, storbinary should handle the error and return an error
code rather than throwing an exception. The 20gb files take about an
hour to transfer, and I've had success with files up to 60gb so far...
once I figured out what ftplib was doing and starting cathing the
IOError.
Why would storbinary throw an exception even when the file transfer
was successful? Why would the exception not be thrown until after the
file was sent? Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error? Is my
reasoning off somewhere?
~Sean ha**********@gm ail.com wrote:
Why would storbinary throw an exception even when the file transfer
was successful? Why would the exception not be thrown until after the
file was sent? Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error? Is my
reasoning off somewhere?
There was an error, the connection was reset by peer. The error code is
104. As it *was* an error, an exception is correct.
You always can hide it yourself, with try/except...
Regards,
--
.. Facundo
..
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
On Jun 7, 8:54 am, Facundo Batista <facu...@taniqu etil.com.arwrot e:
half.ital...@gm ail.com wrote:
Why would storbinary throw an exception even when the file transfer
was successful? Why would the exception not be thrown until after the
file was sent? Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error? Is my
reasoning off somewhere?
There was an error, the connection was reset by peer. The error code is
104. As it *was* an error, an exception is correct.
You always can hide it yourself, with try/except...
Regards,
--
. Facundo
.
Blog:http://www.taniquetil.com.ar/plog/
PyAr:http://www.python.org/ar/
Ok. I guess that makes sense. But what about the other
questions...mai nly: Why would it throw an exception even though the
file was properly transferred?
I guess this could be blamed on the FTP server.
~Sean ha**********@gm ail.com wrote:
Ok. I guess that makes sense. But what about the other
questions...mai nly: Why would it throw an exception even though the
file was properly transferred?
Je, well, I answered the one I knew about, :)
Regarding the error... es hard to say.
What happens if you transmit the file using a standard ftp program?
Or better, could you sniff your network to see what is the TCP dialog?
I guess this could be blamed on the FTP server.
Maybe. Maybe not...
--
.. Facundo
..
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/ This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: python |
last post by:
Hi:
I want to write a procedure to automatically upload some files for me, but
I'm getting stuck. Before I write my own gruesome put() function, I
wanted to check if there is an easier way.
Here's what I can do so far:
>>>import ftplib
>>>conn = ftplib.FTP('ftp.example.com')
>>>conn.login(user='userid', passwd='passwd')
|
by: John fabiani |
last post by:
Hi,
I have a script that is not running on a freeBSD 4.4 with Python 2.1
that uses ftplib. I'm wondering if it's possible to update only the
ftplib module from 2.1 to 2.3. I only ask because I have no idea how to
upgrade anything for freeBSD 4.4. I know there is are doc's available
but I sure don't want to screw anything up on the freeBSD 4.4 server
because I didn't understand what I was doing.
Thanks
John
|
by: Mikael |
last post by:
Hello,
I am using the ftplib.FTP.nlst() method to list the files in
a directory on a FTP server. But when the directoty is empty,
python raises an error:
ftplib.error_perm: 550 No files found.
I find this behaviour very strange. Why not simply return an empty
list?
|
by: Beeyah |
last post by:
in my inability to find a working ftp program for my mac, i decided to
write one in python. i wrote a class (link at end of post) that
inherited ftplib.FTP and made the process slightly less involved, but
it brought up a few questions:
using the sendcmd() method to do a simple "LIST" command returns an
error explaining that a data connection could not be opened, just as
if you connected to a ftp server using telnet. so, you use the...
|
by: Matija Papec |
last post by:
I would like to reimplement ftplib "nlst" using ftplib.dir (ftp server
doesn't support nlst) so I'm trying to guess how to use ftp callbacks.
Any help is appreciated.
tia!
============
#!/usr/bin/python
# nlst vs list
| |
by: Peter A. Schott |
last post by:
Using ftplib.FTP object for a project we have here to upload/download files. I
know that I can wrap everything in try/except blocks, but I'm having trouble
getting the exact error messages out of the Exceptions.
I'd like to either send an e-mail or log the failure to a database. It would
also be nice to see what the message is and perhaps add some additional logic
for delay/retry or something similar.
Has anyone done any of the...
|
by: Peter A. Schott |
last post by:
Does the ftplib support SFTP or FTPS? Is that part of a different module? We
have a handful of partners who use FTPS or SFTP and I need to pull/push files
to/from them.
Thank you for all of your help.
-Pete Schott
|
by: vedran |
last post by:
Hello,
One simple question.Which security ftplib support,and how to I 'load'
these security to ftputil.
(Please don't give me security modules)
e.g.
ftp=FTPHost('myhost','myusername','mypassword',session_factory=ftplib.FTP)
|
by: loial |
last post by:
Trying to use ftplib.FTP.nlst() method to list the files in
a directory on a FTP server.
It works fine except when there are no files in the directory. Then it
gives the error
ftplib.error_perm: 550 No files found.
How can I handle this cleanly?
|
by: Laszlo Nagy |
last post by:
Hi All,
I'm using a simple program that uploads a file on a remote ftp server.
This is an example (not the whole program):
def store(self,hostname,username,password,destdir,srcpath):
self.ftp = ftplib.FTP(hostname)
self.ftp.login(username,password)
self.ftp.set_pasv(False)
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
| |
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |