Data Transfer Dilemma

[goralondos]

Hopefully this is the right forum, it didn't seem to quite fit into any of the others.

My company has 20 remote sites that backup date to our main building.
Presently, a scheduled batch job is setup that zips (tar+gnu) all data in one large file, then FTP the zip file to the main location. The size of the data coming over ranges from 50MB to 1.5GB per location, 12 GB total.

The size of the data will continue to grow as time goes on.
On a day to day basis, only 1-5% of the files coming over will have actually changed, but those files can be 20-25% of the total size of the transfer.

For me, it seemed like a waste to bring over the rest of the files that haven’t changed.

Why I came here: I’m hoping to get some advice about sequence of events and/or programs to accomplish just bring over the files that have changed.

I hope to have a zipped copy of the files in a backup directory at the remote site and at my location, with as little extra data transferred, in the least amount of time, possible.

It should be pretty simple…
Data Directories – (sync) -> Backup Directory
Backup Directory – (sync) -> FTP Site
It would be nice to have the Backup Directory and FTP Site sync zipped files instead, so the files travel across the internet compressed.
Is it possible to compare zipped archives through an ftp(sftp, etc) type connect and only send the files out of one archive to the other, that have changed?

A secondary option could be instead of creating one large zip file, zip all the files as individual archives in the backup directory, then sync?

Any thoughts would be greatly appreciated.

Thanks!
Adam

[Killer42]

Don't know about through the FTP facility, but it should be possible to check the CRC of files within the archive, in order to get a comparison without having to transfer the entire archive.

Hm... how about this?

Along with the archive (zip) file, perhaps you could store a text file listing the contents of the archive. This could be FTP'd across, and examined to determine what else has to be done.

Ideally, of course, you want as much as possble to happen on the sending side (for example if anything needs to be extracted/re-zipped), to minimise traffic. You might be better off looking into real backup software rather than rolling your own.

Maybe you could just do a weekly full and daily incremental zip files? Zip utilities usually allow you to add only those files which have been touched.

[goralondos]

Thank You for the response!

Ideally, of course, you want as much as possible to happen on the sending side (for example if anything needs to be extracted/re-zipped), to minimize traffic. You might be better off looking into real backup software rather than rolling your own.

All the remote sites use one kind of backup software or another already. Coordinating all the transferred around the backups is another fun “side” project . This backup is meant for disaster recovery. Just to give a little more detail, all the remote sites do different accounting related work. Some remote sites print payroll checks, some accept payments, etc. If a disaster struck on the right day, it could affect the payroll checks getting out on time, this backup is meant to minimize that chance.

Maybe you could just do a weekly full and daily incremental zip files? Zip utilities usually allow you to add only those files which have been touched.

I’ve been playing around with a utility I found on the web called “SyncronEx”
It will do a one-way sync of a set of backup directories to another location. Even though it would end up being a full backup in both the backup directory at the remote site and on the ftp site, it is incremental in the sense that it’s only moves the files to 2 other locations that have changed. The main problem I have with the program, and in general with this is I would feel more comfortable creating a secondary directory and syncing off that because of the time factor that can be involved in the transfer.

[NeoPa]

Personally, I'd maintain a copy on each remote site of the current status of the files backed up to the central site (Keep separate files, the higher the granularity, the better the efficiency).
Every time a backup is scheduled, it will compare the current file with the file that is a duplicate of what is on the central site (No bandwidth used in this), only Zipping and copying those files which have been changed.
Alternatively, Zip them all before the comparison to waste less local space.

[Banfa]

I’ve been playing around with a utility I found on the web called “SyncronEx” It will do a one-way sync of a set of backup directories to another location. Even though it would end up being a full backup in both the backup directory at the remote site and on the ftp site, it is incremental in the sense that it’s only moves the files to 2 other locations that have changed. The main problem I have with the program, and in general with this is I would feel more comfortable creating a secondary directory and syncing off that because of the time factor that can be involved in the transfer.

Surely you have just answered your own question.

Create a batch/command file that copies the files at the remote site to an alternate directory and then uses SyncronEx on that directory to backup to the server.

[drhowarddrfine]

This sounds like a job for 'dump' in Unix/BSD.

[Killer42]

Personally, I'd maintain a copy on each remote site of the current status of the files backed up to the central site (Keep separate files, the higher the granularity, the better the efficiency).
Every time a backup is scheduled, it will compare the current file with the file that is a duplicate of what is on the central site (No bandwidth used in this), only Zipping and copying those files which have been changed.
Alternatively, Zip them all before the comparison to waste less local space.

One bonus of this sort of approach is that, if you keep a complete copy at each site rather than just the status, you automatically have a number of off-site backups in case of disaster.

Of course, I may be totally destroying the bandwidth-minimisation idea there. Maybe they just need a faster connection. :-)

[NeoPa]

The OP mentioned 12GB of data.
That's a delayed response even at LAN speeds. I think the most important issue here is to avoid uploading as much as possible. Uploading just the Deltas may be a step too far though ;)

[goralondos]

Thank you for all the replies!

I did some testing with SynchronEx utility over the weekend. It works very well for Syncing directories if anyone needs that function in a simple command line style utility that works on multiple platforms.

Let me know what you guys think about a configuration like this…

Live Data Directory is compressed to Backup directory at Remote Site 1.
Instead of compressing the data in one large archive, compress each file in an individual archive in the backup directory at Remote Site 1. Then run a utility like SynchronEx, syncing the backup directory at Remote Site 1 to backup directory at my location.

[NeoPa]

Thank you for all the replies!

I did some testing with SynchronEx utility over the weekend. It works very well for Syncing directories if anyone needs that function in a simple command line style utility that works on multiple platforms.

Let me know what you guys think about a configuration like this…

Live Data Directory is compressed to Backup directory at Remote Site 1.
Instead of compressing the data in one large archive, compress each file in an individual archive in the backup directory at Remote Site 1. Then run a utility like SynchronEx, syncing the backup directory at Remote Site 1 to backup directory at my location.

If "Live Data Directory is compressed to Backup directory at Remote Site 1." refers to a "Live Data Directory" at that same "Remote Site 1" then this sounds ideal. Obviously, this would not be limited to "Remote Site 1" but would have to be repeated at each of the remote sites.

[goralondos]

If "Live Data Directory is compressed to Backup directory at Remote Site 1." refers to a "Live Data Directory" at that same "Remote Site 1" then this sounds ideal. Obviously, this would not be limited to "Remote Site 1" but would have to be repeated at each of the remote sites.

Yes, my apologizes for not being clear about that, the Live Data Directory and Remote Site 1 are at the same location.
Once I have it working at one remote site, I will need to repeat the new setup at the other remote sites.

Now I just have to find a good command line program that can zip multiple directories to backup directory as individual archives.

I do have to say, as much as projects like this can be frustrating, they are quite fun and I do greatly appreciate the input provided here.

[NeoPa]

I have a routine which depends on having WinZip Command Line Interface (CLI) installed on the PC.

'Zip zips up the files in strFiles into strZip.  Returns success state.
Public Function Zip(strZip As String, strFiles As String) As Boolean
    Dim strCmd As String, strExe As String
 
    Zip = True
    On Error GoTo ErrorZ
    strExe = RegRead(conHKLM, conZipKey, "")
    strCmd = ParamReplace("""%E"" -a+ -ex -ybc ""%Z"" ""%F""", "%E", strExe, _
                                                               "%Z", strZip, _
                                                               "%F", strFiles)
    Call Shell(PathName:=strCmd, WindowStyle:=vbNormalFocus)
    Exit Function
 
ErrorZ:
    strCmd = ParamReplace("Unable to zip {%F} into '%Z'", "%F", strFiles, _
                                                          "%Z", strZip)
    Call ShowMsg(strMsg:=strCmd, strTitle:="Zip", intButtons:=vbInformation)
    Zip = False
End Function

[NeoPa]

There are two function calls in there I should really explain further :
RegRead()
This is provided as a separate module in Module to Read from the Windows Registry.
ParamReplace()
This is simply Replace() but for multiple parameter sets. I'll include it here for your reference (or usage) but you may prefer simply to change the code to issue multiple Replace() references.

'ParamReplace replaces all occurrences of varParam in strMain with varReplace.
'Using VbBinaryCompare means that case is not ignored.
Public Function ParamReplace(ByRef strMain As String, _
                             ByVal varParam As Variant, _
                             ByVal varReplace As Variant, _
                             ParamArray avarArgs())
    Dim intIdx As Integer
 
    If (UBound(avarArgs) - LBound(avarArgs)) Mod 2 = 0 Then Stop
    ParamReplace = Replace(Expression:=strMain, _
                           Find:=Nz(varParam, ""), _
                           Replace:=Nz(varReplace, ""), _
                           Compare:=vbBinaryCompare)
    For intIdx = LBound(avarArgs) To UBound(avarArgs) Step 2
        ParamReplace = Replace(Expression:=ParamReplace, _
                               Find:=Nz(avarArgs(intIdx), ""), _
                               Replace:=Nz(avarArgs(intIdx + 1), ""), _
                               Compare:=vbBinaryCompare)
    Next intIdx
End Function