Re: Fastest way to write huge files

[Mohamed Yousef]

Thanks all ,
but there is still something i forget to state -sorry - all
communication will be via Http with a server
so data is received via Http
so local network solutions won't work
the problem really starts after receiving data in storing them without
much of a CPU/Memory usage and with a good speed

@James Mills : didn't understand fully what you mean and how it will
improve writting effciency

Thanks,

Regards,
Mohamed Yousef

2008/8/29 Tim Golden <ma**@timgolden.me.uk>:

Terry Reedy wrote:

>>

Mohamed Yousef wrote:

>>>
let's say , I'm moving large files through network between devices
what is the fastest way to do this ?
what i came up with :-

Use your OS's network copy command. On unix, that was once uucp. On
Windows, I drag-and-drop to/from a Network Neighborhood location, including
to a printer, so I don't know whether you can use copy and if so how.

For completeness' sake, on Windows you could use any of the following
techniques with a UNC as the destination (and/or source):

http://timgolden.me.uk/python/win32_...py-a-file.html

TJG
--
http://mail.python.org/mailman/listinfo/python-list

[Larry Bates]

Mohamed Yousef wrote:

Thanks all ,
but there is still something i forget to state -sorry - all
communication will be via Http with a server
so data is received via Http
so local network solutions won't work
the problem really starts after receiving data in storing them without
much of a CPU/Memory usage and with a good speed

@James Mills : didn't understand fully what you mean and how it will
improve writting effciency

Thanks,

Regards,
Mohamed Yousef

2008/8/29 Tim Golden <ma**@timgolden.me.uk>:

>Terry Reedy wrote:

>>>
Mohamed Yousef wrote:
let's say , I'm moving large files through network between devices
what is the fastest way to do this ?
what i came up with :-
Use your OS's network copy command. On unix, that was once uucp. On
Windows, I drag-and-drop to/from a Network Neighborhood location, including
to a printer, so I don't know whether you can use copy and if so how.

For completeness' sake, on Windows you could use any of the following
techniques with a UNC as the destination (and/or source):

http://timgolden.me.uk/python/win32_...py-a-file.html

TJG
--
http://mail.python.org/mailman/listinfo/python-list

If connection is over Internet via HTTP the connection speed is so slow in
relation to the speed of your CPU that it doesn't really matter. You are
prematurely optimizing your application. Get it working first and then see if
the file writing is the bottleneck (it probably won't be).

-Larry

[Mohamed Yousef]

>

If connection is over Internet via HTTP the connection speed is so slow in
relation to the speed of your CPU that it doesn't really matter.

this is not always true , espicially when using a localhost or a local
netwtork server
the problem is the increase in cpu and memory usage make it a painful bug
such that downloading a big file (say 1 GB ) would introduce a big cpu
usage (already tested)

Thanks ,

Regards,
Mohamed Yousef

[Larry Bates]

Mohamed Yousef wrote:

>If connection is over Internet via HTTP the connection speed is so slow in
relation to the speed of your CPU that it doesn't really matter.

this is not always true , espicially when using a localhost or a local
netwtork server
the problem is the increase in cpu and memory usage make it a painful bug
such that downloading a big file (say 1 GB ) would introduce a big cpu
usage (already tested)

Thanks ,

Regards,
Mohamed Yousef

I disagree. There is very little overhead in file writing if you stream your
writes to disk in small blocks as they arrive via HTTP. Don't wait for the
entire 1Gb to arrive and then write it. Python can write small to a file
blazingly fast with normal file writing I/O. Any extra CPU overhead you may see
is probably due to reading the entire 1Gb into memory and seeing swapping to
disk as you exhaust main memory. Interleaving your HTTP reading with file
writing should be very fast.

-Larry