473,847 Members | 1,472 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

How to read a text file really fast??

Friends,
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer
Nov 16 '05 #1
6 23782
G.Esmeijer <ge****@nomail. nl> wrote:
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers


It would help if you showed the code you're currently using. I suspect
it's not the reading which is taking the time, but the process you're
using for handling each line.

Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 16 '05 #2
G.Esmeijer wrote:
Friends,
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer


I've done it before, and StreamReader line-by-line was as fast as it gets in
C#. I was able to process 100 MB text files in under 10 seconds on my average
machine.

What does "takes to long" mean to you?

I'd suspect that most of your time is spent in processing the text to the
database, and not the action of reading the text. Have you profiled to
definitively determine that the StreamReader is the slow part?
Nov 16 '05 #3
> I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme


The fastest way I can think of is ugly, but perhaps worth investigating. If
your lines of text are all exactly the same length, and the fields within
the line are in exactly the same places, you could read the entire file into
an array of char, and use a struct with explicit layout to get the fields.
For instance, if your lines have fields A, B, C, and D starting at positions
0, 8, 12, and 27, with a total line length of 60 characters, then you could
do something like this:

[StructLayoutAtt ribute(LayoutKi nd.Explicit)]
struct RecordParser {
[FieldOffsetAttr ibute(0)] char[8] A;
[FieldOffsetAttr ibute(8)] char[4] B;
[FieldOffsetAttr ibute(12)] char[15] C;
[FieldOffsetAttr ibute(27)] char[33] D;
}

....

public unsafe void ParseFile() {
char[] theFile = ...; // do whatever you need to do to read the file
into a char[] buffer
int lineLength = 60;
RecordParser* record;
for(int RecordPosition= 0; RecordPosition < theFile.Length;
RecordPosition+ =lineLength) {
record = (RecordParser *) (&(theFile[RecordPosition]));
// do something useful with record->A, record->B, etc
}
}

Depending on the actual types you're trying to extract out of your lines of
text, you may have to do some additional work "do something useful" stage to
parse the individual fields into the proper types (e.g. converting "120.17"
to an actual float or double value, trim trailing whitespace, etc.)

Note that because we're using unmanaged pointers to coerce raw memory
(slices of the char array) into acting like structs, we're violating the
Common Language Runtime's type system and therefore we must mark the
ParseFile method as "unsafe" to satisfy the compiler that we know what we're
doing (for that matter, you may also need to mark the RecordParser struct as
"unsafe" as well; I'm not sure about that offhand). Also, you must add the
"/unsafe" command line switch to your compilation command.

For a much more thorough treatment of this type of memory access from C#,
see chapter 10 of Don Box's excellent book "Essential .NET Volume 1".
Nov 16 '05 #4
G.Esmeijer wrote:
Friends,
I would like to read a text file (fixed length formaated) really fast
and store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line
into string just takes to long.

When I Import the file with access manually goes fast. But how to use
this fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer


I had this issue as well, and not from .NET but a Win32 app, the issue
for me was the actual insertion into the database (it was access, but
using MSDE backend) the symptoms were it would start off fast enough,
but quickly slow down.

so no matter how fast I read the file for the bulk insert it didn't
help. In the end what I did was create a Data Transformation Service
(DTS) package and used that for the insert, it then inserted as fast as
an import directly from Access. Note you can create and run DTS
packages programatically . (link below)

http://tinyurl.com/6s3cx

Now this helped me when using Access with a MSDE backend (I note you
are using Access 2003 hence the assumption that you may possibly be
using MSDE) if it is a simple .mdb database then I don't think DTS
packages are suitable.

Regards Tim.
Nov 16 '05 #5
IIRC, there is a ReadEntireFile( ) or ReadAll() ... something
like that. I'm not sure where it is in the framework...

It reads the entire file into memory, then you use
the other methods to parse it.

As others have said, "iff the file is the problem". Usually,
bottlenecks are not where you "know" they are until
you measure.
--
Grace + Peace,
Peter N Roth
Engineering Objects International
http://engineeringobjects.com
Home of Matrix.NET
"G.Esmeijer " <ge****@nomail. nl> wrote in message
news:41******** **************@ dreader2.news.t iscali.nl...
Friends,
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer

Nov 16 '05 #6
Thanks for all the responses. The general response pointed me into the right
direction. Not the reading but the processing of each line took so much
time.
Since I know the number of line to read I add that many row to the grid and
the put the results in there. It really is very fast.
Gerrit Esmeijer
"G.Esmeijer " <ge****@nomail. nl> schreef in bericht
news:41******** **************@ dreader2.news.t iscali.nl...
Friends,
I would like to read a text file (fixed length formaated) really fast and
store the data into an Access database (2003).
Using the streamreader and reading line by line, separating the line into
string just takes to long.

When I Import the file with access manually goes fast. But how to use this
fromout a C# programme

who has done this before and who can give met some answers

Gerrit Esmeijer

Nov 16 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
5938
by: rishka | last post by:
Rishka Mar 17, 5:40 am show options Newsgroups: comp.databases.oracle.tools From: "Rishka" <ris...@webmail.co.za> - Find messages by this author Date: 17 Mar 2005 05:40:45 -0800 Local: Thurs, Mar 17 2005 5:40 am Subject: Read Text file into Oracle Reports Reply | Reply to Author | Forward | Print | Individual Message | Show original | Remove | Report Abuse
3
5227
by: JenHu | last post by:
Hi, I want read line by line and characters. The characters are fix length text file, no specific delimited method between each fields. The first line is header line, the last line is footer. Between the first line and last line is the content, and I will have to load the content to database. How to read line by line? I use For lineNum = LBound(lines) To (UBound(lines) - 1) which generate error because it does read the
5
7650
by: Mika M | last post by:
Hi! I'm trying to read text file like... "Field1";"Field2";"Field3";"Field4" "ABCD";"EFGH";"1234";"20051020" "AABB";"CCDD";"2468";"20051021" "CCDD";"XXYY";"4321";"20051022" ....using OLE DB-provider. Unfortunately I can't affect in what kind of
1
2360
by: sallu2000 | last post by:
hi to all, I am new to php,can u give any idea to read text file from folder based on datewise .for eg. in that folder there are three text file. one.txt 10/3/2007 two.txt 13/3/2007 three.txt 13/3/2007
1
2690
by: abtet | last post by:
Hello people, I am new to php and have question on how to read text file and convert it to XML with php. . Can it be done if the text file contains text, image and also tables? please help and if anyone has the sources that could help solve my problem, please direct me to the source. Thank you very much -abtet-
1
8801
by: neveen | last post by:
i want to open and read text file using j2me that can run on mobile 6630 then i want to make button called read that when pressed the data inside text display
6
2493
by: portCo | last post by:
Hello there, I am creating a vb application which is some like like a questionare. Application read a text file which contains many questions and display one question and the input is needed from user to calculate the score. Here is a problem. I can read a text file. However, it's read whole file at a time. So,
28
3289
by: tlpell | last post by:
Hey, read some tips/pointers on PHP.net but can't seem to solve this problem. I have a php page that reads the contents of a file and then displays the last XX lines of the file. Problem is this...whenever the file gets larger that ~5MB, the page just displays nothing, as though a timeout has occurred but I get no error. At 4.8MB (last confirmed size)...the function still works. Any ideas what code below is lacking?? <? $handle=...
8
12450
by: karimkhan | last post by:
I am trying to read text file , here is the code , I cant use activex control so plz give me some way.... <html> <head> <script type="text/javascript"> function r() {
0
9889
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9732
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10987
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10652
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
10343
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7885
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5914
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4535
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3166
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.