470,619 Members | 1,817 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,619 developers. It's quick & easy.

Re: Fastest way to write huge files

Thanks all ,
but there is still something i forget to state -sorry - all
communication will be via Http with a server
so data is received via Http
so local network solutions won't work
the problem really starts after receiving data in storing them without
much of a CPU/Memory usage and with a good speed

@James Mills : didn't understand fully what you mean and how it will
improve writting effciency

Thanks,

Regards,
Mohamed Yousef

2008/8/29 Tim Golden <ma**@timgolden.me.uk>:
Terry Reedy wrote:
>>

Mohamed Yousef wrote:
>>>
let's say , I'm moving large files through network between devices
what is the fastest way to do this ?
what i came up with :-

Use your OS's network copy command. On unix, that was once uucp. On
Windows, I drag-and-drop to/from a Network Neighborhood location, including
to a printer, so I don't know whether you can use copy and if so how.

For completeness' sake, on Windows you could use any of the following
techniques with a UNC as the destination (and/or source):

http://timgolden.me.uk/python/win32_...py-a-file.html

TJG
--
http://mail.python.org/mailman/listinfo/python-list
Aug 29 '08 #1
3 2569
Mohamed Yousef wrote:
Thanks all ,
but there is still something i forget to state -sorry - all
communication will be via Http with a server
so data is received via Http
so local network solutions won't work
the problem really starts after receiving data in storing them without
much of a CPU/Memory usage and with a good speed

@James Mills : didn't understand fully what you mean and how it will
improve writting effciency

Thanks,

Regards,
Mohamed Yousef

2008/8/29 Tim Golden <ma**@timgolden.me.uk>:
>Terry Reedy wrote:
>>>
Mohamed Yousef wrote:
let's say , I'm moving large files through network between devices
what is the fastest way to do this ?
what i came up with :-
Use your OS's network copy command. On unix, that was once uucp. On
Windows, I drag-and-drop to/from a Network Neighborhood location, including
to a printer, so I don't know whether you can use copy and if so how.
For completeness' sake, on Windows you could use any of the following
techniques with a UNC as the destination (and/or source):

http://timgolden.me.uk/python/win32_...py-a-file.html

TJG
--
http://mail.python.org/mailman/listinfo/python-list
If connection is over Internet via HTTP the connection speed is so slow in
relation to the speed of your CPU that it doesn't really matter. You are
prematurely optimizing your application. Get it working first and then see if
the file writing is the bottleneck (it probably won't be).

-Larry
Aug 30 '08 #2
>
If connection is over Internet via HTTP the connection speed is so slow in
relation to the speed of your CPU that it doesn't really matter.
this is not always true , espicially when using a localhost or a local
netwtork server
the problem is the increase in cpu and memory usage make it a painful bug
such that downloading a big file (say 1 GB ) would introduce a big cpu
usage (already tested)

Thanks ,

Regards,
Mohamed Yousef
Aug 30 '08 #3
Mohamed Yousef wrote:
>If connection is over Internet via HTTP the connection speed is so slow in
relation to the speed of your CPU that it doesn't really matter.
this is not always true , espicially when using a localhost or a local
netwtork server
the problem is the increase in cpu and memory usage make it a painful bug
such that downloading a big file (say 1 GB ) would introduce a big cpu
usage (already tested)

Thanks ,

Regards,
Mohamed Yousef
I disagree. There is very little overhead in file writing if you stream your
writes to disk in small blocks as they arrive via HTTP. Don't wait for the
entire 1Gb to arrive and then write it. Python can write small to a file
blazingly fast with normal file writing I/O. Any extra CPU overhead you may see
is probably due to reading the entire 1Gb into memory and seeing swapping to
disk as you exhaust main memory. Interleaving your HTTP reading with file
writing should be very fast.

-Larry
Aug 30 '08 #4

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

8 posts views Thread by David P. Jessup | last post: by
1 post views Thread by Mohamed Yousef | last post: by
reply views Thread by James Mills | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.