473,811 Members | 3,135 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

File Synchro (Checksum, Fragments, ...)

Hello,

What would be the best way to synchronise 2 big files ?

The files have only small modifications (not more than 10Kb changed /
added insided).
It would a bad choice to transfer 5Mb regarding only 10Kb of changes.
If only a few bytes are added/modified it's does not worth it to
transfer the whole file.

The idea would be the following one :
Make a dichotomy to find out only modified fragments and only transfer
changes (extending the file length if necessary).
Client and Server would only exchange checksums and only transfer
fragments of the files that differ.

Would you have hints ?
- Checksum algorythms
- Transfer by fragment
- OpenSource Projects
- Samples
- Ideas
Thanks.

Cybertof.

Nov 16 '05 #1
1 1919
Cybertof <cy************ @ifrance.com> wrote:
What would be the best way to synchronise 2 big files ?

The files have only small modifications (not more than 10Kb changed /
added insided).
It would a bad choice to transfer 5Mb regarding only 10Kb of changes.
If only a few bytes are added/modified it's does not worth it to
transfer the whole file.

The idea would be the following one :
Make a dichotomy to find out only modified fragments and only transfer
changes (extending the file length if necessary).
Client and Server would only exchange checksums and only transfer
fragments of the files that differ.

Would you have hints ?
- Checksum algorythms
- Transfer by fragment
- OpenSource Projects
- Samples
- Ideas


There's a good binary diff file format called vcdiff. I've got a
decoder that you're welcome to use at
http://www.pobox.com/~skeet/csharp/miscutil

However, it doesn't have an encoder - you can use one called xdelta3
though, which is freely available (but not in .NET).

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 16 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
2721
by: Bart Nessux | last post by:
Python may not be suitable for this, but I thought I'd ask the experts: I'm doing a summer project that will attempt to measure exactly how file fragmentation affects disk drive and OS performance. I'd like to use Python for some of this. In particular, I'd like to write a file fragmentor in Python that will randomly fragment x% of files on a NTFS filesystem into y number of fragments. For example, assuming that we start with a 100%...
2
5274
by: pradeep | last post by:
I have 2 data files, DATA1 and DATA2 , both same. My task is to: Open DATA1, compute the checksum and put it in the end of the file(don't bother about boundary conditions).close DATA1 Open DATA2,compute the checksum and put it in the end(don't bother about boundary conditions).close DATA2. Now again open DATA1, compute the checksum of the file(leaving the
9
1528
by: Sharktyyfa | last post by:
Hi, hoping someone can help. Access 2003, WinXP. I have built a database that uses synchro to co-ordinate with the mothership. All is well. The person i built it for wants the satellites to be able to edit the
26
3009
by: Michel Rouzic | last post by:
I have a binary file used to store the values of variables in order to use them again. I easily know whether the file exists or not, but the problem is, in case the program has been earlier interupted before it could write the variables to the file, the file is gonna be empty, and then it's gonna load a load of crap into variables, which i want to avoid. That file is always 36 bytes big (it contains 4 double-precision floats and one...
24
3187
by: pristo | last post by:
hello All, how can i insert unique ID into binary file (that created by compiler)? (so after compiling i can to identify the src that i use) thx
6
4472
by: Chang | last post by:
How to get SHA1 or MD5 of a big file (+5MB - 20GB) as I can't read 20GB into memory. -- Chang.
1
5318
by: xoinki | last post by:
hi experts, I need a little help in debugging this code.. would u pleeze kindly help me? here this program sends a datagram every 10 seconds and on reception it cheks whether the source IP is already written in file or not.. if it is not already entered into the file then a file is opened and that IP is entered. the problem is with this part.. if the filewrite() function is tested independently it is working but in this program...
6
6523
by: Vicente García | last post by:
hi, I have a xml file as shown: <?xml version="1.0" encoding="utf-8"?> <PIM> <fragments> </fragments> </PIM> </xml>
0
968
by: gajjar123 | last post by:
hi , i m trying to generate MD5 checksum (32 char) for xml file or other file using string Result ; Result=System.Web.Security.FormsAuthentication.HashPasswordForStoringInConfigFile(str, "md5"); but it is not working for PDF file bcoz i dont have pdf reader class
0
9603
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10644
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10379
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10393
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10124
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7664
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6882
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5690
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
3863
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.