471,873 Members | 1,841 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,873 software developers and data experts.

Fastest way to write huge files

Hello ,

let's say , I'm moving large files through network between devices
what is the fastest way to do this ?
what i came up with :-

1) using regular file operations with an in memory limit of 4MB which
when filled written to disk and re-filled again

2) using memory mapped files in the following way , map first 20MB of
file and after finishing them flush and map next 20MB
------
which of them should be faster in writting to disk ? and less memory consuming

or is there faster ways ?

Thanks,

Regards ,
Mohamed Yousef
Aug 29 '08 #1
1 1500
On Aug 28, 5:35*pm, "Mohamed Yousef" <harrr...@gmail.comwrote:
Hello ,

let's say , I'm moving large files through network between devices
what is the fastest way to do this ?
what i came up with :-

1) using regular file operations with an in memory limit of 4MB which
when filled written to disk and re-filled again

2) using memory mapped files in the following way , map first 20MB of
file and after finishing them flush and map next 20MB
------
which of them should be faster in writting to disk ? and less memory consuming

or is there faster ways ?

Thanks,

Regards ,
Mohamed Yousef
In the send side :

You may want to use sendfile and here is python package for same
http://pypi.python.org/pypi/py-sendfile/1.2.3

In the receive side :

You may want to use splice (Linux 2.6.17 onwards) - Just read from a
network fd and write to a file fd

I just assumed you are using Linux.:-)

Thanks
-Saswat
Aug 29 '08 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

8 posts views Thread by David P. Jessup | last post: by
reply views Thread by James Mills | last post: by
reply views Thread by zermasroor | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.