473,372 Members | 928 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,372 software developers and data experts.

C database :: compared to a relational database engine?

Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

Thanks to any and all replies.

Aug 24 '07 #1
24 2061
sonos wrote:
Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
That's your call. You can either start out with a database, or start
out with your own implementation, depending on the complexity of the
data set, the target platform, performance requirements and a host of
other variables.

There isn't a simple answer.

--
Ian Collins.
Aug 24 '07 #2
On Aug 24, 3:42 pm, Ian Collins <ian-n...@hotmail.comwrote:
sonos wrote:
Hi,
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

That's your call. You can either start out with a database, or start
out with your own implementation, depending on the complexity of the
data set, the target platform, performance requirements and a host of
other variables.

There isn't a simple answer.

--
Ian Collins.
OK, thanks. Can you give an example of a simple C data set, a
moderately complex data set, and a data set clearly meant for a
relational database engine?

Aug 24 '07 #3
sonos wrote:
On Aug 24, 3:42 pm, Ian Collins <ian-n...@hotmail.comwrote:
>sonos wrote:
>>Hi,
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
That's your call. You can either start out with a database, or start
out with your own implementation, depending on the complexity of the
data set, the target platform, performance requirements and a host of
other variables.

There isn't a simple answer.
*Please don't quote signatures*
>
OK, thanks. Can you give an example of a simple C data set, a
moderately complex data set, and a data set clearly meant for a
relational database engine?
No, because of the other variables I mentioned. I have implemented very
complex database engines in C for embedded platforms where we couldn't
find a database engine that would work on that platform. I have also
used a database (because it was there) to perform simple archiving on
hosted platforms.

You have to weigh up the situation and go for what's best for you. On a
hosted platform, I'd use a database for all but the most basic
applications so long as performance wasn't a issue, just to save the
trouble of writing and testing my own.

--
Ian Collins.
Aug 24 '07 #4
On Aug 24, 1:17 pm, sonos <dspcyp...@gmail.comwrote:
Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
Only when you care about your data.
Thanks to any and all replies.

Aug 24 '07 #5
sonos skrev:
Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

Thanks to any and all replies.

I would sugest using PostgreSQL and it's c library...
http://www.postgresql.org/docs/8.1/static/libpq.html
Aug 26 '07 #6
Carramba said:
sonos skrev:
>>
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

I would sugest using PostgreSQL and it's c library...
It might be better to move comparative databasology discussions to a
database programming newsgroup. FTR, I don't see any reason for the OP
to avoid MySQL; a database programming newsgroup might, so why not ask
them?

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Aug 26 '07 #7
On 2007-08-26 18:00, Carramba <ca******@example.comwrote:
sonos skrev:
>I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
* If you need or want to manipulate your data with other tools than
your program.

* If your data fits well into the relational model.

* If you want to access your data from other hosts

* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.
>Thanks to any and all replies.

I would sugest using PostgreSQL and it's c library...
http://www.postgresql.org/docs/8.1/static/libpq.html
No. PostgreSQL is not a C library. It comes with a C library which
simplifies sending queries to the server. The database engine itself is
running as a separate process.

There are SQL engines implemented as libraries: SQLite, the MS Jet
engine (used by MS Access), embedded MySQL, etc.

hp
--
_ | Peter J. Holzer | I know I'd be respectful of a pirate
|_|_) | Sysadmin WSR | with an emu on his shoulder.
| | | hj*@hjp.at |
__/ | http://www.hjp.at/ | -- Sam in "Freefall"
Aug 26 '07 #8
On 2007-08-26 19:33, Peter J. Holzer <hj*********@hjp.atwrote:
On 2007-08-26 18:00, Carramba <ca******@example.comwrote:
>sonos skrev:
>>I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

* If you need or want to manipulate your data with other tools than
your program.

* If your data fits well into the relational model.

* If you want to access your data from other hosts

* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.
Forgot the most important reason (and the one least off-topic here):

* The queries you want to make are easier to express in SQL than in C.

hp
--
_ | Peter J. Holzer | I know I'd be respectful of a pirate
|_|_) | Sysadmin WSR | with an emu on his shoulder.
| | | hj*@hjp.at |
__/ | http://www.hjp.at/ | -- Sam in "Freefall"
Aug 26 '07 #9
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?

* If you need or want to manipulate your data with other tools than
your program.

* If your data fits well into the relational model.

* If you want to access your data from other hosts

* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C. Or is it that a SQL query is faster to code than
to hard code the C program? As to remote access, that does not really
pose a problem w. POSIX. Or, were you thinking about concurrent use of
a file? i.e. is contention easier to manage w. a RDB engine compared
to root file systems? And I guess that poses another perspective; C
programs must have a root filesystem (and less portability) whereas
a .db file can be easily ported to another OS?

Aug 26 '07 #10
sonos wrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
>On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
>>sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
* If you need or want to manipulate your data with other tools than
your program.

* If your data fits well into the relational model.

* If you want to access your data from other hosts

* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.

It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C. Or is it that a SQL query is faster to code than
to hard code the C program?
It's not a matter of performance, it's more a case of not reinventing
the wheel. If you are targeting a platform or platforms where a
database engine is available, just use it for what it does and spend
your time on your application, not worrying about the consistency of you
data.
As to remote access, that does not really
pose a problem w. POSIX. Or, were you thinking about concurrent use of
a file? i.e. is contention easier to manage w. a RDB engine compared
to root file systems? And I guess that poses another perspective; C
programs must have a root filesystem (and less portability) whereas
a .db file can be easily ported to another OS?
Remote access may not be a problem to implement, but with a database the
implementation is provide for you.

One aspect of database storage I've come to like is the ability to
manipulate and view data from the command line or scripts.

--
Ian Collins.
Aug 26 '07 #11
On Aug 26, 5:32 pm, Ian Collins <ian-n...@hotmail.comwrote:
sonos wrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
>sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
* If you need or want to manipulate your data with other tools than
your program.
* If your data fits well into the relational model.
* If you want to access your data from other hosts
* If you need transactions, constraints, or other services
ordinarily provided by an RDBMS.
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C. Or is it that a SQL query is faster to code than
to hard code the C program?

It's not a matter of performance, it's more a case of not reinventing
the wheel. If you are targeting a platform or platforms where a
database engine is available, just use it for what it does and spend
your time on your application, not worrying about the consistency of you
data.
As to remote access, that does not really
pose a problem w. POSIX. Or, were you thinking about concurrent use of
a file? i.e. is contention easier to manage w. a RDB engine compared
to root file systems? And I guess that poses another perspective; C
programs must have a root filesystem (and less portability) whereas
a .db file can be easily ported to another OS?

Remote access may not be a problem to implement, but with a database the
implementation is provide for you.

One aspect of database storage I've come to like is the ability to
manipulate and view data from the command line or scripts.

--
Ian Collins.
Ian, thanks for your comments. You commented earlier that your
embedded system was in C, and you also seem to know about the RDB too.

The current problem is that the system data is mainly housed on a
linux server, yet there are several embedded systems downloading just
a chunk of data, not the entire database. The embedded system then
updates a few variables before uploading information to the server. In
the meantime, another user may have added information too but on the
serever, so there is a contention problem. The data really isn't a
prime candidate for relational queries, and it seems that a RDB engine
is overkill in that respect. It's the interchange of data streams that
is the real problem. So maybe it is a matter of server data streams to
the headless system by way of scripting in C. I don't know how to
script for SQL engines, and didn't know if it was worth the effort to
learn for this system, or to recode the server in C.

Aug 27 '07 #12
sonos wrote:
>>
Please don't quote signatures.
>
Ian, thanks for your comments. You commented earlier that your
embedded system was in C, and you also seem to know about the RDB too.

The current problem is that the system data is mainly housed on a
linux server, yet there are several embedded systems downloading just
a chunk of data, not the entire database. The embedded system then
updates a few variables before uploading information to the server. In
the meantime, another user may have added information too but on the
serever, so there is a contention problem. The data really isn't a
prime candidate for relational queries, and it seems that a RDB engine
is overkill in that respect. It's the interchange of data streams that
is the real problem. So maybe it is a matter of server data streams to
the headless system by way of scripting in C. I don't know how to
script for SQL engines, and didn't know if it was worth the effort to
learn for this system, or to recode the server in C.
Wandering a bit OT here, but if you embedded system supports sockets
(which it sounds like it does to access a remote sever), you should be
able to compile the interface library for one of the popular opensource
databases and use database connections to access the data. The
interface is very simple to use, the only SQL you have to understand is
"select" and "insert".

Whether the data is a prime candidate for relational queries or not, it
sounds like you would benefit from the data integrity offered by
transactions.

A database engine is a good solutions to the requirement for multiple
readers and writers to a pool of data.

--
Ian Collins.
Aug 27 '07 #13
A database engine is a good solutions to the requirement for multiple
readers and writers to a pool of data.
So it's possible then to capture data input from more than one source
on the same data set, even when several are not 'online'? I understood
the RDB engines must lock a record and not allow edits until it is
released.
Aug 27 '07 #14
sonos wrote:
>A database engine is a good solutions to the requirement for multiple
readers and writers to a pool of data.
So it's possible then to capture data input from more than one source
on the same data set, even when several are not 'online'? I understood
the RDB engines must lock a record and not allow edits until it is
released.
That's what transactions are for.

Do some background research and then ask on a database group (no, I
don't know which), this has gone beyond the scope of this group.

--
Ian Collins.
Aug 27 '07 #15
Peter J. Holzer said:
On 2007-08-26 18:00, Carramba <ca******@example.comwrote:
<snip>
>I would sugest using PostgreSQL and it's c library...

No. PostgreSQL is not a C library. It comes with a C library
He meant "using PostgreSQL and its C library".

Careless apostrophes cost meaning.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Aug 27 '07 #16
Richard Heathfield skrev:
Peter J. Holzer said:
>On 2007-08-26 18:00, Carramba <ca******@example.comwrote:

<snip>
>>I would sugest using PostgreSQL and it's c library...
No. PostgreSQL is not a C library. It comes with a C library

He meant "using PostgreSQL and its C library".

Careless apostrophes cost meaning.
Thank you!
Aug 27 '07 #17
On 2007-08-27 03:54, Richard Heathfield <rj*@see.sig.invalidwrote:
Peter J. Holzer said:
>On 2007-08-26 18:00, Carramba <ca******@example.comwrote:

<snip>
>>I would sugest using PostgreSQL and it's c library...

No. PostgreSQL is not a C library. It comes with a C library

He meant "using PostgreSQL and its C library".
Makes sense. Sorry to Carramba.
Careless apostrophes cost meaning.
The interesting thing is that I mentally (mis-)corrected "and it's c library"
to "and it's a c library" without even noticing it, even though I used
to write "it's" instead of "its" a lot, too. One would assume that one
tends to reverse their own errors, but that doesn't seem to be the case.

hp

--
_ | Peter J. Holzer | I know I'd be respectful of a pirate
|_|_) | Sysadmin WSR | with an emu on his shoulder.
| | | hj*@hjp.at |
__/ | http://www.hjp.at/ | -- Sam in "Freefall"
Aug 27 '07 #18
On 2007-08-26 22:23, sonos <ds*******@gmail.comwrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
>On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
sonos skrev:
I am working on a program to archive data to disk.
>At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
[...]
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C.
I didn't realize from your initial posting that you were working on
embedded systems. That adds two additional criteria:

* Is there a client library available for your target system at all?
Even if your system is POSIX-compatible it may have restrictions which
the authors (which were developing for servers and workstations)
simply didn't think of.

* Does it fit on your system? The MySQL and PostgreSQL client libraries
seem to be rather small (a few 100 kB), but for some RDBMSs the client
libs (plus support files: charset conversion tables, error message
catalogs, etc.) are huge (tens of MB).

hp

--
_ | Peter J. Holzer | I know I'd be respectful of a pirate
|_|_) | Sysadmin WSR | with an emu on his shoulder.
| | | hj*@hjp.at |
__/ | http://www.hjp.at/ | -- Sam in "Freefall"
Aug 27 '07 #19
On Aug 27, 12:41 am, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 22:23, sonos <dspcyp...@gmail.comwrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
[...]
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C.

I didn't realize from your initial posting that you were working on
embedded systems. That adds two additional criteria:

* Is there a client library available for your target system at all?
Even if your system is POSIX-compatible it may have restrictions which
the authors (which were developing for servers and workstations)
simply didn't think of.

* Does it fit on your system? The MySQL and PostgreSQL client libraries
seem to be rather small (a few 100 kB), but for some RDBMSs the client
libs (plus support files: charset conversion tables, error message
catalogs, etc.) are huge (tens of MB).

hp
MySQL and PostgreSQL have huge footprints and require a server
installation.

Perhaps SQLite or FastDB would be more appropriate for embedded work.
http://www.sqlite.org/
http://www.fastdb.org/fastdb.html

Just a guess.

Aug 27 '07 #20
"sonos" <ds*******@gmail.comwrote in message
news:11**********************@x40g2000prg.googlegr oups.com...
Hi,
I am working on a program to archive data to disk.

At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
Just remember that you can code a relational database in C. However, that is
not very practical at all... So, if you don't feel comfortable wrt creating
your own solution, well, you can always license somebody else's hard work.

Aug 27 '07 #21
On Aug 27, 2:41 am, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 22:23, sonos <dspcyp...@gmail.comwrote:
On Aug 26, 2:33 pm, "Peter J. Holzer" <hjp-usen...@hjp.atwrote:
On 2007-08-26 18:00, Carramba <carra...@example.comwrote:
sonos skrev:
I am working on a program to archive data to disk.
At what point is it best to use a relational database like MySQL as
the backend instead of a C program alone?
[...]
It just seems to me that w/ the current uC computational power,
performance return from a relational database engine is not really
much better than C.

I didn't realize from your initial posting that you were working on
embedded systems. That adds two additional criteria:

* Is there a client library available for your target system at all?
Even if your system is POSIX-compatible it may have restrictions which
the authors (which were developing for servers and workstations)
simply didn't think of.

* Does it fit on your system? The MySQL and PostgreSQL client libraries
seem to be rather small (a few 100 kB), but for some RDBMSs the client
libs (plus support files: charset conversion tables, error message
catalogs, etc.) are huge (tens of MB).

hp
MySQL is available in embedded form (~1MB), but the customer is
wanting to avoid vendor lock in. MySQL AB is leaning in that direction
by locking src to paying customers only. Amazon's S3 service basically
puts the entire database in cache, so advancing hardware technology
appears to be making inroads on the RDB software performance. Clusters
are an option I suppose, and C would be the primary language anyway.

I suppose it's a matter of develpment budget and time to market. With
C, I am presuming a well coded embedded<>server data system will
continue to be a viable solution for the years ahead, and one I'll
place my dollar on. Might I be so bold as to say within 10 years RDB
will not always be the 'knee-jerk' solution to data management apps.
And, if quantum computing ever arrives, well, hardware solutions win
hands down.

Anyway, thanks for your insights on the matter. I know much has been
OT, but asking the RDB usenet groups wasn't the perspective I was
looking for either.

Aug 27 '07 #22
On Aug 27, 4:14 pm, user923005 <dcor...@connx.comwrote:
A database will give ACID properties to your transactions.
A database will give security to your transactions.
Do you value your data?
Will you elaborate?

Aug 27 '07 #23
On Aug 27, 3:01 pm, sonos <dspcyp...@gmail.comwrote:
On Aug 27, 4:14 pm, user923005 <dcor...@connx.comwrote:
A database will give ACID properties to your transactions.
A database will give security to your transactions.
Do you value your data?

Will you elaborate?
ACID:
http://en.wikipedia.org/wiki/ACID

What happens to your data if the power fails or someone kicks the plug
out of the wall in the middle of a transaction?
If you use a database, then nothing bad will happen. If you use a
disk file, then lots of bad things can happen.

Security:
http://ieeexplore.ieee.org/Xplore/lo...number=1416861

There are bad people who would like to peer into your data and get
unauthorized access to it or damage it.
Database systems provide access and audit mechanisms to prevent that.

If your data is disposable, then any sort of dictionary structure is
probably good enough. If your data needs to be durable, then a
database is one obvious solution that solves lots of problems.

Aug 27 '07 #24
user923005 wrote:

<snip>

MySQL and PostgreSQL have huge footprints and require a server
installation.

Perhaps SQLite or FastDB would be more appropriate for embedded work.
http://www.sqlite.org/
http://www.fastdb.org/fastdb.html
Berkeley DB is another option, and doesn't require a DB server.

--
Tor <torust [at] online [dot] no>
Aug 31 '07 #25

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

14
by: Ruby Tuesdays | last post by:
Perhaps you database guru able to suggest what would be a good choice for opensource database platform to use to develop projects. At the moment the project is small/medium, but it will grow in...
5
by: Alex | last post by:
Hi all, We're looking at a vendor who uses the InterSystems Cache Database Platform, but our IT department has zero experience with this system. This software package will have a pivotal and...
1
by: Marek Kotowski | last post by:
I'm preparing short dictionary and this is the question: are 'RDBMS' and 'database server' synonyms? If not, what are the differences? Thanx in advance. Regards Marek Kotowski Warsaw
3
by: canigou9 (remove your socks to reply) | last post by:
(cross posted - comp.databases.ms-access, microsoft.public.access) Hello folks - this is my first post here, after a short lurk. I have written an application in Access2002 for a friend's...
6
by: Jerry Spence1 | last post by:
Why doesn't the following work in my ASP program? I have imported ADOX I am trying to create a temporary database on the user's PC. The example is taken from Microsoft. Dim cat As Catalog =...
4
by: ImOk | last post by:
I need to program generically. I am familiar with the ODBC layer but never used PDO or DB. Does anyone have any experience with these database layers? Thanks
9
by: TC | last post by:
Like a lot of database developers, I often choose Jet for the back- end. I'm starting to worry about what will happen when Jet is deprecated. Ostensibly, Jet users like me must switch to SQL Server...
5
by: Dave | last post by:
I need to filter an Access 2000 result set in ASP 30 using the ADO recordset.filter. I build the filter in pieces. The first clause of the filter is this... WHERE word LIKE 'S%' ... to...
14
by: Johnny Jörgensen | last post by:
Hi all I just saw an ad in the latest ComponentSource newsletter for a database engine called VistaDB 3.0. http://www.vistadb.net/default.asp It looks very interesting as a database engine...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.