469,317 Members | 1,906 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,317 developers. It's quick & easy.

C database :: compared to a relational database engine?

Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

Thanks to any and all replies.

Aug 24 '07 #1
24 1809
sonos wrote:
Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
That's your call. You can either start out with a database, or start
out with your own implementation, depending on the complexity of the
data set, the target platform, performance requirements and a host of
other variables.

There isn't a simple answer.

--
Ian Collins.
Aug 24 '07 #2
On Aug 24, 3:42 pm, Ian Collins <ian-n...@hotmail.comwrote:
sonos wrote:
Hi,
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

That's your call. You can either start out with a database, or start
out with your own implementation, depending on the complexity of the
data set, the target platform, performance requirements and a host of
other variables.

There isn't a simple answer.

--
Ian Collins.
OK, thanks. Can you give an example of a simple C data set, a
moderately complex data set, and a data set clearly meant for a
relational database engine?

Aug 24 '07 #3
sonos wrote:
On Aug 24, 3:42 pm, Ian Collins <ian-n...@hotmail.comwrote:
>sonos wrote:
>>Hi,
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
That's your call. You can either start out with a database, or start
out with your own implementation, depending on the complexity of the
data set, the target platform, performance requirements and a host of
other variables.

There isn't a simple answer.
*Please don't quote signatures*
>
OK, thanks. Can you give an example of a simple C data set, a
moderately complex data set, and a data set clearly meant for a
relational database engine?
No, because of the other variables I mentioned. I have implemented very
complex database engines in C for embedded platforms where we couldn't
find a database engine that would work on that platform. I have also
used a database (because it was there) to perform simple archiving on
hosted platforms.

You have to weigh up the situation and go for what's best for you. On a
hosted platform, I'd use a database for all but the most basic
applications so long as performance wasn't a issue, just to save the
trouble of writing and testing my own.

--
Ian Collins.
Aug 24 '07 #4
On Aug 24, 1:17 pm, sonos <dspcyp...@gmail.comwrote:
Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
Only when you care about your data.
Thanks to any and all replies.

Aug 24 '07 #5
sonos skrev:
Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

Thanks to any and all replies.

I would sugest using PostgreSQL and it's c library...
http://www.postgresql.org/docs/8.1/static/libpq.html
Aug 26 '07 #6
Carramba said:
sonos skrev:
>>
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

I would sugest using PostgreSQL and it's c library...
It might be better to move comparative databasology discussions to a
database programming newsgroup. FTR, I don't see any reason for the OP
to avoid MySQL; a database programming newsgroup might, so why not ask
them?

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Aug 26 '07 #7
On 2007-08-26 18:00, Carramba <ca******@example.comwrote:
sonos skrev:
>I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
* If you need or want to manipulate your data with other tools than
your program.

* If your data fits well into the relational model.

* If you want to access your data from other hosts

* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.
>Thanks to any and all replies.

I would sugest using PostgreSQL and it's c library...
http://www.postgresql.org/docs/8.1/static/libpq.html
No. PostgreSQL is not a C library. It comes with a C library which
simplifies sending queries to the server. The database engine itself is
running as a separate process.

There are SQL engines implemented as libraries: SQLite, the MS Jet
engine (used by MS Access), embedded MySQL, etc.

hp
--
_ | Peter J. Holzer | I know I'd be respectful of a pirate
|_|_) | Sysadmin WSR | with an emu on his shoulder.
| | | hj*@hjp.at |
__/ | http://www.hjp.at/ | -- Sam in "Freefall"
Aug 26 '07 #8
On 2007-08-26 19:33, Peter J. Holzer <hj*********@hjp.atwrote:
On 2007-08-26 18:00, Carramba <ca******@example.comwrote:
>sonos skrev:
>>I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

* If you need or want to manipulate your data with other tools than
your program.

* If your data fits well into the relational model.

* If you want to access your data from other hosts

* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.
Forgot the most important reason (and the one least off-topic here):

* The queries you want to make are easier to express in SQL than in C.

hp
--
_ | Peter J. Holzer | I know I'd be respectful of a pirate
|_|_) | Sysadmin WSR | with an emu on his shoulder.
| | | hj*@hjp.at |
__/ | http://www.hjp.at/ | -- Sam in "Freefall"
Aug 26 '07 #9
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

* If you need or want to manipulate your data with other tools than
your program.

* If your data fits well into the relational model.

* If you want to access your data from other hosts

* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C. Or is it that a SQL query is faster to code than
to hard code the C program? As to remote access, that does not really
pose a problem w. POSIX. Or, were you thinking about concurrent use of
a file? i.e. is contention easier to manage w. a RDB engine compared
to root file systems? And I guess that poses another perspective; C
programs must have a root filesystem (and less portability) whereas
a .db file can be easily ported to another OS?

Aug 26 '07 #10
sonos wrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
>On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
>>sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
* If you need or want to manipulate your data with other tools than
your program.

* If your data fits well into the relational model.

* If you want to access your data from other hosts

* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.

It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C. Or is it that a SQL query is faster to code than
to hard code the C program?
It's not a matter of performance, it's more a case of not reinventing
the wheel. If you are targeting a platform or platforms where a
database engine is available, just use it for what it does and spend
your time on your application, not worrying about the consistency of you
data.
As to remote access, that does not really
pose a problem w. POSIX. Or, were you thinking about concurrent use of
a file? i.e. is contention easier to manage w. a RDB engine compared
to root file systems? And I guess that poses another perspective; C
programs must have a root filesystem (and less portability) whereas
a .db file can be easily ported to another OS?
Remote access may not be a problem to implement, but with a database the
implementation is provide for you.

One aspect of database storage I've come to like is the ability to
manipulate and view data from the command line or scripts.

--
Ian Collins.
Aug 26 '07 #11
On Aug 26, 5:32 pm, Ian Collins <ian-n...@hotmail.comwrote:
sonos wrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
>sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
* If you need or want to manipulate your data with other tools than
your program.
* If your data fits well into the relational model.
* If you want to access your data from other hosts
* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C. Or is it that a SQL query is faster to code than
to hard code the C program?

It's not a matter of performance, it's more a case of not reinventing
the wheel. If you are targeting a platform or platforms where a
database engine is available, just use it for what it does and spend
your time on your application, not worrying about the consistency of you
data.
As to remote access, that does not really
pose a problem w. POSIX. Or, were you thinking about concurrent use of
a file? i.e. is contention easier to manage w. a RDB engine compared
to root file systems? And I guess that poses another perspective; C
programs must have a root filesystem (and less portability) whereas
a .db file can be easily ported to another OS?

Remote access may not be a problem to implement, but with a database the
implementation is provide for you.

One aspect of database storage I've come to like is the ability to
manipulate and view data from the command line or scripts.

--
Ian Collins.
Ian, thanks for your comments. You commented earlier that your
embedded system was in C, and you also seem to know about the RDB too.

The current problem is that the system data is mainly housed on a
linux server, yet there are several embedded systems downloading just
a chunk of data, not the entire database. The embedded system then
updates a few variables before uploading information to the server. In
the meantime, another user may have added information too but on the
serever, so there is a contention problem. The data really isn't a
prime candidate for relational queries, and it seems that a RDB engine
is overkill in that respect. It's the interchange of data streams that
is the real problem. So maybe it is a matter of server data streams to
the headless system by way of scripting in C. I don't know how to
script for SQL engines, and didn't know if it was worth the effort to
learn for this system, or to recode the server in C.

Aug 27 '07 #12
sonos wrote:
>>
Please don't quote signatures.
>
Ian, thanks for your comments. You commented earlier that your
embedded system was in C, and you also seem to know about the RDB too.

The current problem is that the system data is mainly housed on a
linux server, yet there are several embedded systems downloading just
a chunk of data, not the entire database. The embedded system then
updates a few variables before uploading information to the server. In
the meantime, another user may have added information too but on the
serever, so there is a contention problem. The data really isn't a
prime candidate for relational queries, and it seems that a RDB engine
is overkill in that respect. It's the interchange of data streams that
is the real problem. So maybe it is a matter of server data streams to
the headless system by way of scripting in C. I don't know how to
script for SQL engines, and didn't know if it was worth the effort to
learn for this system, or to recode the server in C.
Wandering a bit OT here, but if you embedded system supports sockets
(which it sounds like it does to access a remote sever), you should be
able to compile the interface library for one of the popular opensource
databases and use database connections to access the data. The
interface is very simple to use, the only SQL you have to understand is
"select" and "insert".

Whether the data is a prime candidate for relational queries or not, it
sounds like you would benefit from the data integrity offered by
transactions.

A database engine is a good solutions to the requirement for multiple
readers and writers to a pool of data.

--
Ian Collins.
Aug 27 '07 #13
A database engine is a good solutions to the requirement for multiple
readers and writers to a pool of data.
So it's possible then to capture data input from more than one source
on the same data set, even when several are not 'online'? I understood
the RDB engines must lock a record and not allow edits until it is
released.
Aug 27 '07 #14
sonos wrote:
>A database engine is a good solutions to the requirement for multiple
readers and writers to a pool of data.
So it's possible then to capture data input from more than one source
on the same data set, even when several are not 'online'? I understood
the RDB engines must lock a record and not allow edits until it is
released.
That's what transactions are for.

Do some background research and then ask on a database group (no, I
don't know which), this has gone beyond the scope of this group.

--
Ian Collins.
Aug 27 '07 #15
Peter J. Holzer said:
On 2007-08-26 18:00, Carramba <ca******@example.comwrote:
<snip>
>I would sugest using PostgreSQL and it's c library...

No. PostgreSQL is not a C library. It comes with a C library
He meant "using PostgreSQL and its C library".

Careless apostrophes cost meaning.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Aug 27 '07 #16
Richard Heathfield skrev:
Peter J. Holzer said:
>On 2007-08-26 18:00, Carramba <ca******@example.comwrote:

<snip>
>>I would sugest using PostgreSQL and it's c library...
No. PostgreSQL is not a C library. It comes with a C library

He meant "using PostgreSQL and its C library".

Careless apostrophes cost meaning.
Thank you!
Aug 27 '07 #17
On 2007-08-27 03:54, Richard Heathfield <rj*@see.sig.invalidwrote:
Peter J. Holzer said:
>On 2007-08-26 18:00, Carramba <ca******@example.comwrote:

<snip>
>>I would sugest using PostgreSQL and it's c library...

No. PostgreSQL is not a C library. It comes with a C library

He meant "using PostgreSQL and its C library".
Makes sense. Sorry to Carramba.
Careless apostrophes cost meaning.
The interesting thing is that I mentally (mis-)corrected "and it's c library"
to "and it's a c library" without even noticing it, even though I used
to write "it's" instead of "its" a lot, too. One would assume that one
tends to reverse their own errors, but that doesn't seem to be the case.

hp

--
_ | Peter J. Holzer | I know I'd be respectful of a pirate
|_|_) | Sysadmin WSR | with an emu on his shoulder.
| | | hj*@hjp.at |
__/ | http://www.hjp.at/ | -- Sam in "Freefall"
Aug 27 '07 #18
On 2007-08-26 22:23, sonos <ds*******@gmail.comwrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
>On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
sonos skrev:
I am working on a program to archive data to disk.
>At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
[...]
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C.
I didn't realize from your initial posting that you were working on
embedded systems. That adds two additional criteria:

* Is there a client library available for your target system at all?
Even if your system is POSIX-compatible it may have restrictions which
the authors (which were developing for servers and workstations)
simply didn't think of.

* Does it fit on your system? The MySQL and PostgreSQL client libraries
seem to be rather small (a few 100 kB), but for some RDBMSs the client
libs (plus support files: charset conversion tables, error message
catalogs, etc.) are huge (tens of MB).

hp

--
_ | Peter J. Holzer | I know I'd be respectful of a pirate
|_|_) | Sysadmin WSR | with an emu on his shoulder.
| | | hj*@hjp.at |
__/ | http://www.hjp.at/ | -- Sam in "Freefall"
Aug 27 '07 #19
On Aug 27, 12:41 am, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 22:23, sonos <dspcyp...@gmail.comwrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
[...]
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C.

I didn't realize from your initial posting that you were working on
embedded systems. That adds two additional criteria:

* Is there a client library available for your target system at all?
Even if your system is POSIX-compatible it may have restrictions which
the authors (which were developing for servers and workstations)
simply didn't think of.

* Does it fit on your system? The MySQL and PostgreSQL client libraries
seem to be rather small (a few 100 kB), but for some RDBMSs the client
libs (plus support files: charset conversion tables, error message
catalogs, etc.) are huge (tens of MB).

hp
MySQL and PostgreSQL have huge footprints and require a server
installation.

Perhaps SQLite or FastDB would be more appropriate for embedded work.
http://www.sqlite.org/
http://www.fastdb.org/fastdb.html

Just a guess.

Aug 27 '07 #20
"sonos" <ds*******@gmail.comwrote in message
news:11**********************@x40g2000prg.googlegr oups.com...
Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
Just remember that you can code a relational database in C. However, that is
not very practical at all... So, if you don't feel comfortable wrt creating
your own solution, well, you can always license somebody else's hard work.

Aug 27 '07 #21
On Aug 27, 2:41 am, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 22:23, sonos <dspcyp...@gmail.comwrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
[...]
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C.

I didn't realize from your initial posting that you were working on
embedded systems. That adds two additional criteria:

* Is there a client library available for your target system at all?
Even if your system is POSIX-compatible it may have restrictions which
the authors (which were developing for servers and workstations)
simply didn't think of.

* Does it fit on your system? The MySQL and PostgreSQL client libraries
seem to be rather small (a few 100 kB), but for some RDBMSs the client
libs (plus support files: charset conversion tables, error message
catalogs, etc.) are huge (tens of MB).

hp
MySQL is available in embedded form (~1MB), but the customer is
wanting to avoid vendor lock in. MySQL AB is leaning in that direction
by locking src to paying customers only. Amazon's S3 service basically
puts the entire database in cache, so advancing hardware technology
appears to be making inroads on the RDB software performance. Clusters
are an option I suppose, and C would be the primary language anyway.

I suppose it's a matter of develpment budget and time to market. With
C, I am presuming a well coded embedded<>server data system will
continue to be a viable solution for the years ahead, and one I'll
place my dollar on. Might I be so bold as to say within 10 years RDB
will not always be the 'knee-jerk' solution to data management apps.
And, if quantum computing ever arrives, well, hardware solutions win
hands down.

Anyway, thanks for your insights on the matter. I know much has been
OT, but asking the RDB usenet groups wasn't the perspective I was
looking for either.

Aug 27 '07 #22
On Aug 27, 4:14 pm, user923005 <dcor...@connx.comwrote:
A database will give ACID properties to your transactions.
A database will give security to your transactions.
Do you value your data?
Will you elaborate?

Aug 27 '07 #23
On Aug 27, 3:01 pm, sonos <dspcyp...@gmail.comwrote:
On Aug 27, 4:14 pm, user923005 <dcor...@connx.comwrote:
A database will give ACID properties to your transactions.
A database will give security to your transactions.
Do you value your data?

Will you elaborate?
ACID:
http://en.wikipedia.org/wiki/ACID

What happens to your data if the power fails or someone kicks the plug
out of the wall in the middle of a transaction?
If you use a database, then nothing bad will happen. If you use a
disk file, then lots of bad things can happen.

Security:
http://ieeexplore.ieee.org/Xplore/lo...number=1416861

There are bad people who would like to peer into your data and get
unauthorized access to it or damage it.
Database systems provide access and audit mechanisms to prevent that.

If your data is disposable, then any sort of dictionary structure is
probably good enough. If your data needs to be durable, then a
database is one obvious solution that solves lots of problems.

Aug 27 '07 #24
user923005 wrote:

<snip>

MySQL and PostgreSQL have huge footprints and require a server
installation.

Perhaps SQLite or FastDB would be more appropriate for embedded work.
http://www.sqlite.org/
http://www.fastdb.org/fastdb.html
Berkeley DB is another option, and doesn't require a DB server.

--
Tor <torust [at] online [dot] no>
Aug 31 '07 #25

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

14 posts views Thread by Ruby Tuesdays | last post: by
1 post views Thread by Marek Kotowski | last post: by
3 posts views Thread by canigou9 (remove your socks to reply) | last post: by
6 posts views Thread by Jerry Spence1 | last post: by
4 posts views Thread by ImOk | last post: by
14 posts views Thread by Johnny Jörgensen | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by suresh191 | last post: by
reply views Thread by harlem98 | last post: by
1 post views Thread by Geralt96 | last post: by
reply views Thread by harlem98 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.