Data transfer problem - ideas/solutions wanted (please)

[E.T. Grey]

Hi,

I have an interesting problem. I have a (LARGE) set of historical data
that I want to keep on a central server, as several separate files. I
want a client process to be able to request the data in a specific file
by specifying the file name, start date/time and end date/time.

The files are in binary format, to conserve space on the server (as well
as to increase processing time). The data in each file can be quite
large, covering several years of data. New data will be appended to
these files each day, by a (PHP) script. The server machine is likely to
be a Unix machine, whereas my clients will be running on windows
machines. My clients program is written in C++.

My two main problems/questions are as follows:

1). Transfer method issue:
What is the best (i.e. most efficient and fast way) to transfer data
from the server to clients ?. I think SOAP is likely to be too slow,
because of the sheer size of the data

2). Cross platform issue:
How can I insure that that the (binary?) data sent from the Unix server
can be correctly interpreted at the client side?

2). Security issue:
How can I prevent clients from directly accessing the files (to prevent
malicious or accidental corruption of the data files.?

[E.T. Grey]

noone wrote:

NC wrote:

E.T. Grey wrote:

> I have a (LARGE) set of historical data that I want to keep
> on a central server, as several separate files.

How large exactly?
At last count, there are about 65,000 distinct files (and increasing)

...

Each file has the equivalent of approx 1M rows (yes - thats 1 million)

...

If you multiply the number of rows (on avg) by the number of files -
you can quickly see why using a db as a repository would be a
poor design choice.

Sorry, I can't. 65 million records is a manageable database.

I agree... I have designed and deployed binary and ascii data loads in
excess of 250Million records/day. Searching the data was a piece of
cake - if you know how to actually designed the database correctly.

65M records is peanuts to a database - even MySql. With proper indexing
you can do a direct-row lookup in < 4-8 I/O's - not so with the path
you are currently trying to traverse... you are looking at up to 65M
reads - and reads are very expensive!!

Use the proper tools/mechanisms for the job at hand...

Michael Austin
DBA
(stuff snipped)

Please do not patronise me. Like NC, you completely overlooked the
obvious fact that the number of records we are talking about (if a
database design is used) runs into billions - not millions. Furthermore,
the datasets are time series data and therefore order is of paramount
importance. Instead of trying to impose a design on me (without fully
understanding the problem), it would have been infinetely preferable if
you had simply answered the question I had asked in the first place. But
judging by the way you have overlooked basic facts - whilst being hell
bent that a db solution is *definitely* the way forward - you have
instantly lost any credibility you may have had - and consequently, I
will ignore any "advice" you care to offer in the future.

[NC]

E.T. Grey wrote:

NC wrote:

E.T. Grey wrote:

At last count, there are about 65,000 distinct files (and increasing)

...

Each file has the equivalent of approx 1M rows (yes - thats 1 million)

...

If you multiply the number of rows (on avg) by the number of files -
you can quickly see why using a db as a repository would be a
poor design choice.

Sorry, I can't. 65 million records is a manageable database.

Its amazing how some people once having set their mind on one thing,
wont change it - even when presented with the facts. Last time I
checked, 65,000 x 1 million = 65 billion - not 65 million.

OK, I obviously made a stupid typo; I'll gladly correct it:

65 billion records is a manageable database

Even MySQL (which is often thought of as a departmental rather than
enterprise system, although with MySQL 5.0 available this may be
reconsidered) is capable of maintaining large databases. Since MySQL
3.23, you can store up to 65536 terabytes using the MyISAM storage
engine (which effectively means that the size of your table is limited
only by your operating system's file size limit) or mere 64 TB using
the InnoDB storage engine (but in this case, the file size limit does
not apply, because an InnoDB table can be spread over several files).
You stated earlier that a compressed set of one million records takes
10-15 kilobytes to store, so an uncompressed record would probably be
just a few bytes long. This is a load that a single server with a
properly configured RAID could handle...

Cheers,
NC