473,320 Members | 2,122 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

How to read a text file really fast??

Friends,
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer
Nov 16 '05 #1
6 23735
G.Esmeijer <ge****@nomail.nl> wrote:
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers


It would help if you showed the code you're currently using. I suspect
it's not the reading which is taking the time, but the process you're
using for handling each line.

Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 16 '05 #2
G.Esmeijer wrote:
Friends,
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer


I've done it before, and StreamReader line-by-line was as fast as it gets in
C#. I was able to process 100 MB text files in under 10 seconds on my average
machine.

What does "takes to long" mean to you?

I'd suspect that most of your time is spent in processing the text to the
database, and not the action of reading the text. Have you profiled to
definitively determine that the StreamReader is the slow part?
Nov 16 '05 #3
> I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme


The fastest way I can think of is ugly, but perhaps worth investigating. If
your lines of text are all exactly the same length, and the fields within
the line are in exactly the same places, you could read the entire file into
an array of char, and use a struct with explicit layout to get the fields.
For instance, if your lines have fields A, B, C, and D starting at positions
0, 8, 12, and 27, with a total line length of 60 characters, then you could
do something like this:

[StructLayoutAttribute(LayoutKind.Explicit)]
struct RecordParser {
[FieldOffsetAttribute(0)] char[8] A;
[FieldOffsetAttribute(8)] char[4] B;
[FieldOffsetAttribute(12)] char[15] C;
[FieldOffsetAttribute(27)] char[33] D;
}

....

public unsafe void ParseFile() {
char[] theFile = ...; // do whatever you need to do to read the file
into a char[] buffer
int lineLength = 60;
RecordParser* record;
for(int RecordPosition=0; RecordPosition < theFile.Length;
RecordPosition+=lineLength) {
record = (RecordParser *) (&(theFile[RecordPosition]));
// do something useful with record->A, record->B, etc
}
}

Depending on the actual types you're trying to extract out of your lines of
text, you may have to do some additional work "do something useful" stage to
parse the individual fields into the proper types (e.g. converting "120.17"
to an actual float or double value, trim trailing whitespace, etc.)

Note that because we're using unmanaged pointers to coerce raw memory
(slices of the char array) into acting like structs, we're violating the
Common Language Runtime's type system and therefore we must mark the
ParseFile method as "unsafe" to satisfy the compiler that we know what we're
doing (for that matter, you may also need to mark the RecordParser struct as
"unsafe" as well; I'm not sure about that offhand). Also, you must add the
"/unsafe" command line switch to your compilation command.

For a much more thorough treatment of this type of memory access from C#,
see chapter 10 of Don Box's excellent book "Essential .NET Volume 1".
Nov 16 '05 #4
G.Esmeijer wrote:
Friends,
I would like to read a text file (fixed length formaated) really fast
and store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line
into string just takes to long.

When I Import the file with access manually goes fast. But how to use
this fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer


I had this issue as well, and not from .NET but a Win32 app, the issue
for me was the actual insertion into the database (it was access, but
using MSDE backend) the symptoms were it would start off fast enough,
but quickly slow down.

so no matter how fast I read the file for the bulk insert it didn't
help. In the end what I did was create a Data Transformation Service
(DTS) package and used that for the insert, it then inserted as fast as
an import directly from Access. Note you can create and run DTS
packages programatically. (link below)

http://tinyurl.com/6s3cx

Now this helped me when using Access with a MSDE backend (I note you
are using Access 2003 hence the assumption that you may possibly be
using MSDE) if it is a simple .mdb database then I don't think DTS
packages are suitable.

Regards Tim.
Nov 16 '05 #5
IIRC, there is a ReadEntireFile() or ReadAll() ... something
like that. I'm not sure where it is in the framework...

It reads the entire file into memory, then you use
the other methods to parse it.

As others have said, "iff the file is the problem". Usually,
bottlenecks are not where you "know" they are until
you measure.
--
Grace + Peace,
Peter N Roth
Engineering Objects International
http://engineeringobjects.com
Home of Matrix.NET
"G.Esmeijer" <ge****@nomail.nl> wrote in message
news:41**********************@dreader2.news.tiscal i.nl...
Friends,
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer

Nov 16 '05 #6
Thanks for all the responses. The general response pointed me into the right
direction. Not the reading but the processing of each line took so much
time.
Since I know the number of line to read I add that many row to the grid and
the put the results in there. It really is very fast.
Gerrit Esmeijer
"G.Esmeijer" <ge****@nomail.nl> schreef in bericht
news:41**********************@dreader2.news.tiscal i.nl...
Friends,
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer

Nov 16 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: rishka | last post by:
Rishka Mar 17, 5:40 am show options Newsgroups: comp.databases.oracle.tools From: "Rishka" <ris...@webmail.co.za> - Find messages by this author Date: 17 Mar 2005 05:40:45 -0800 Local:...
3
by: JenHu | last post by:
Hi, I want read line by line and characters. The characters are fix length text file, no specific delimited method between each fields. The first line is header line, the last line is footer. ...
5
by: Mika M | last post by:
Hi! I'm trying to read text file like... "Field1";"Field2";"Field3";"Field4" "ABCD";"EFGH";"1234";"20051020" "AABB";"CCDD";"2468";"20051021" "CCDD";"XXYY";"4321";"20051022" ....using OLE...
1
by: sallu2000 | last post by:
hi to all, I am new to php,can u give any idea to read text file from folder based on datewise .for eg. in that folder there are three text file. one.txt 10/3/2007 two.txt 13/3/2007...
1
by: abtet | last post by:
Hello people, I am new to php and have question on how to read text file and convert it to XML with php. . Can it be done if the text file contains text, image and also tables? please help...
1
by: neveen | last post by:
i want to open and read text file using j2me that can run on mobile 6630 then i want to make button called read that when pressed the data inside text display
6
by: portCo | last post by:
Hello there, I am creating a vb application which is some like like a questionare. Application read a text file which contains many questions and display one question and the input is needed...
28
by: tlpell | last post by:
Hey, read some tips/pointers on PHP.net but can't seem to solve this problem. I have a php page that reads the contents of a file and then displays the last XX lines of the file. Problem is...
8
by: karimkhan | last post by:
I am trying to read text file , here is the code , I cant use activex control so plz give me some way.... <html> <head> <script type="text/javascript"> function r() {
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.