By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
435,148 Members | 739 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 435,148 IT Pros & Developers. It's quick & easy.

Huge data file

P: n/a
I have a data file which is 2.1GB (zipped), when the exe runs, it
copies the content of the zip file to a destination folder. What is
the best way to achive this? Current implementation is to zip the file
at the source and at the destination unzip the file and if the file
dates are different then copy the files over. Is there a better way to
do this other than zipping and unzipping?
Thanks.
Jul 23 '08 #1
Share this Question
Share on Google+
5 Replies


P: n/a
On Wed, 23 Jul 2008 13:32:16 -0700 (PDT), CSharper <cs******@gmx.com>
wrote:
>I have a data file which is 2.1GB (zipped), when the exe runs, it
copies the content of the zip file to a destination folder. What is
the best way to achive this? Current implementation is to zip the file
at the source and at the destination unzip the file and if the file
dates are different then copy the files over. Is there a better way to
do this other than zipping and unzipping?
Thanks.
When you create the big zipped file, also create a very small file at
the same time at the source. The small file contains a date/copy
number/serial number/whatever identifying the current iteration of the
large file.

Copy the small file to the destination first and check its date/etc.
against the date/etc. of the current version of the small file. If
they are different then update the small file and copy over the large
file. This avoids copying over the large file unless you have to -
the small file tells you enough to know whether there is an updated
version of the large file without having to copy the whole thing over
the network.

Zipping is a good way to reduce network load when you do have to get a
new copy of the large file.

A more radical solution would be for the update file to merely contain
the changes from the last time, though this might be too much effort
to set up from scratch. Probably easier to use a database at both
ends and to use the built-in synchronisation routines.

rossum

Jul 23 '08 #2

P: n/a
CSharper wrote:
I have a data file which is 2.1GB (zipped), when the exe runs, it
copies the content of the zip file to a destination folder. What is
the best way to achive this? Current implementation is to zip the file
at the source and at the destination unzip the file and if the file
dates are different then copy the files over. Is there a better way to
do this other than zipping and unzipping?
You can not avoid zipping and unzipping if you use a zip file.

You can avoid any intermediate files, if your app checks
dates and if necessary extract the file directly to final
destination.

Arne
Jul 24 '08 #3

P: n/a
Can I ask what is in the data file? Not specifics - but is it
* an archive of lots of other files
* raw binary data
* serialized .NET entities, such as xml

In the first bullet, you could perhaps read the archive with tools like
#ZipLib and only extract / copy the interesting files - but note that
you are on the limit of what this will work with. Alternatively, tools
like "robocopy" will work on the *extracted* files (not the single
archive) to efficiently copy on the changed data; you can use
OS/file-system compression rather a zip to allow this to work happily.

In the second bullet; well - that is tricky; the data is likely to be
fairly dense, so I'd be surprised if it compressed very well anyway...

In the third bullet - perhaps try an alternative data format to reduce
space? I have lots of current ideas in this area if you are interested...

Marc
Jul 24 '08 #4

P: n/a
On Wed, 23 Jul 2008 13:32:16 -0700, CSharper wrote:
I have a data file which is 2.1GB (zipped), when the exe runs, it copies
the content of the zip file to a destination folder. What is the best
way to achive this? Current implementation is to zip the file at the
source and at the destination unzip the file and if the file dates are
different then copy the files over. Is there a better way to do this
other than zipping and unzipping? Thanks.
You could look at the application rsync to allow synchronisation of files
that only change a little bit each time.

If you are talking a Database (eg Access) then look at replication.

Ta
Ken
Jul 24 '08 #5

P: n/a
On Jul 24, 6:14*am, Ken Foskey <rmove.fos...@optushome.com.auwrote:
On Wed, 23 Jul 2008 13:32:16 -0700, CSharper wrote:
I have a data file which is 2.1GB (zipped), when the exe runs, it copies
the content of the zip file to a destination folder. What is the best
way to achive this? Current implementation is to zip the file at the
source and at the destination unzip the file and if the file dates are
different then copy the files over. Is there a better way to do this
other than zipping and unzipping? Thanks.

You could look at the application rsync to allow synchronisation of files
that only change a little bit each time.

If you are talking a Database (eg Access) then look at replication.

Ta
Ken
Thanks to you all. I really appriciate your comments.
Jul 24 '08 #6

This discussion thread is closed

Replies have been disabled for this discussion.