473,320 Members | 1,848 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

speed: file vs. DB

Hi

I am the administrator of TSEP - The Search Engine Project (www.tsep.info).
We developer are unclear about the speed of two different methods:
Is it faster to
a) write to a file
or
b) update a record in a database

Olaf

--
http://www.team-noehring.de
Jul 17 '05 #1
7 2207
Writing is not the issue, reading is. If you have a text file containing
100,000 records how quickly can you read just one particular record?
Database tables have keys, so it is very easy to read a record with a
particular key value. Text files don't have keys, so you cannot go straight
to a particular record. You have to read serially through the whole file
until you happen to find the record you want.

If you know so little about databases then what makes you think your little
project will be of use to anyone?

--
Tony Marston

http://www.tonymarston.net

"Olaf Noehring" <ol******@team-noehring.de(0000=year)> wrote in message
news:Xn**************************@195.14.215.228.. .
Hi

I am the administrator of TSEP - The Search Engine Project
(www.tsep.info).
We developer are unclear about the speed of two different methods:
Is it faster to
a) write to a file
or
b) update a record in a database

Olaf

--
http://www.team-noehring.de

Jul 17 '05 #2
0000=year wrote:
Hi

I am the administrator of TSEP - The Search Engine Project
(www.tsep.info). We developer are unclear about the speed of two different
methods: Is it faster to
a) write to a file
or
b) update a record in a database

Olaf


Hi Olaf,

Do you only want to write some data away, or do you want to retrieve the
data later?
If you want to retrieve it later: use a database.
A database can use indexing, making retrieval of data much faster.

If you just want to write it away: I do not know. I guess filewriting speed
depends on filesystem you use, HD-speed, fragmentation, and the number of
files already on the filesystem.

My guess is your best bet is a database for most situations.

Regards,
Erwin Moller
Jul 17 '05 #3
Hi

to add some more information what we need this for. Sorry, that I forgot
to throw in this information right away:

We need to write only about 5 lines to the file - or 1 record to the DB.
It's only needed to tell the indexer (of TSEP) which reads the contents of
the files that another instance of the indexer is already running. So, the
first instance started will write to the file / db record when it has been
started and that it is running at time X (every 30 seconds)
The reading (from the file/record) is only done if a instance of the
indexer is started and there IS a record already.

Olaf
I am the administrator of TSEP - The Search Engine Project
(www.tsep.info). We developer are unclear about the speed of two
different methods: Is it faster to
a) write to a file
or
b) update a record in a database

Olaf


--
http://www.team-noehring.de
Jul 17 '05 #4
0000=year wrote:
Hi

to add some more information what we need this for. Sorry, that I forgot
to throw in this information right away:

We need to write only about 5 lines to the file - or 1 record to the DB.
It's only needed to tell the indexer (of TSEP) which reads the contents of
the files that another instance of the indexer is already running. So, the
first instance started will write to the file / db record when it has been
started and that it is running at time X (every 30 seconds)
The reading (from the file/record) is only done if a instance of the
indexer is started and there IS a record already.

Olaf


Use a database. :-)
You ARE accessing the data (files) again, so you better switch to a real
solution (database) right away.

Just think about it: You have 5.000.000 files in a directory on a filesystem
like NTFS. You need a file with the name xyzwtq.txt
How to find it?
The system has to scan through all filenames stored and compare it to your
filename.
If you have some smart indexing going on, this will seriously reduce the
seekspeed.

Maybe there are some fancy filesystems out there that can compete with a
database by using some smart indexing, I do not know to be honest.

Regards,
Erwin Moller
Jul 17 '05 #5
Olaf Noehring wrote:
Hi

I am the administrator of TSEP - The Search Engine Project (www.tsep.info).
We developer are unclear about the speed of two different methods:
Is it faster to
a) write to a file
or
b) update a record in a database

Olaf

Designers of databases have done all the very fast coding for you, why
shouldn't you use it?
Jul 17 '05 #6
In that case I would use a file. Using a database seems overkill.

Jul 17 '05 #7
SOR
<comp.lang.php , Tony Marston , to**@NOSPAM.demon.co.uk>
<da*******************@news.demon.co.uk>
<Tue, 5 Jul 2005 11:00:10 +0100>
Writing is not the issue, reading is. If you have a text file containing
100,000 records how quickly can you read just one particular record?
Database tables have keys, so it is very easy to read a record with a
particular key value. Text files don't have keys, so you cannot go straight
to a particular record. You have to read serially through the whole file
until you happen to find the record you want.


Not always , As a newish user I done www.sparesorrepair.co.uk using
flatfiles and each advert is saved on its own and called/displayed by
the advert number .

(obviously not saying its the best way to do it)
Jul 17 '05 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: J. Campbell | last post by:
Hi everyone. I'm sure that this is common knowledge for many/most here. However, it was rather illuminating for me (someone learning c++) and I thought it might be helpful for someone else. I'm...
60
by: Neil | last post by:
I have a situation with an ODBC linked view in an Access 2000 MDB with a SQL 7 back end. The view is scrolling very slowly. However, if I open the view in an ADP file, it scrolls quickly. I...
2
by: Abubakar | last post by:
Hi, I have a small sockets application and I'm adding a feature in it to transfer files. Since I also want to give it a transfer resume feature so this means I'll be transfering in specific bytes...
6
by: Ham | last post by:
Yeah, Gotto work with my VB.Net graphic application for days, do any possible type of code optimization, check for unhandled errors and finally come up with sth that can't process 2D graphics and...
3
by: Mike Kelly | last post by:
Hi. I've built a page using standard ASP.NET 2.0 features and when I upload a large file (>20MB) to our intranet server, I get a paltry 100KB/s on our 100Mb/s LAN. Simply copying the file, I get...
2
by: Mike Kober | last post by:
I am having issues with the File Upload control for sending files to the server via HTTP. The speed of the upload is often between 20kbs and 40kbs. If I use the LAN at work to the server, it...
9
by: sturnfie | last post by:
Hi all, I am attempting to write a tool that would get (via something of a stopwatch action) the amount of time it takes to copy a file from a local disk to a connected USB drive on a Windows XP...
19
by: llothar | last post by:
I must say i didn't expect this. I just did some measures on FreeBSD 6.2 with gcc 3.4.6 and there is absolutely no significant difference between 32 and 64 bit mode - neither in compilation speed,...
8
by: SaltyBoat | last post by:
Needing to import and parse data from a large PDF file into an Access 2002 table: I start by converted the PDF file to a html file. Then I read this html text file, line by line, into a table...
10
by: Devang | last post by:
Hello, I am using php script to upload file. some times if file size is too big(1GB) it takes too much time to upload. Can someone suggest me the way to increase upload speed. thanks
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.