Reading text data to double crlf

I have an application that is reading data from text files to the first
occurrence of a double crlf. I have this working using streamreader
and using readline.

However, I am hitting 500+ files at a time and this is a bit slow.

I was thinking of testing this using FileStream or something similar,
but I am not sure how to detect the occurrence of crlf crlf?

Thoughts?
Amy

re: Reading text data to double crlf

amyl@paxemail.com wrote:
[color=blue]
> I have an application that is reading data from text files to the first
> occurrence of a double crlf. I have this working using streamreader
> and using readline.
>
> However, I am hitting 500+ files at a time and this is a bit slow.
>
> I was thinking of testing this using FileStream or something similar,
> but I am not sure how to detect the occurrence of crlf crlf?
>
> Thoughts?
> Amy[/color]
In the studio there is a configuration setting. Perhaps that is accessible
elsewhere?

re: Reading text data to double crlf

It's doubtful switch to FileStream will make it any faster.
StreamReader, when constructed using a file name, just creates a
FileStream internally. Your test would be just duplicating
StreamReader.

How are you reading it now (using streamreader & readline)?

re: Reading text data to double crlf

Hi Amy,

A couple of questions first, if I may:

1. Are these files always using the same encoding (Unicode, UTF8, ASCII,
etc)? It makes a difference as you can handle different encodings in
different ways. For example, ASCII is always 1 byte per character, whereas
Unicode is always 2 bytes per character, and needs a bit more effort to
read.

2. What is the expected (and projected) maximum size of any of these files?
You say that your app is processing 500+ at a time, but not what the maximum
size of any of these files is. If the files are small enough, you can read
each file all at once, rather than using ReadLine. Then you can simply
search the resulting string for "\r\n\r\n".

--
HTH,

Kevin Spencer
Microsoft MVP
Professional Numbskull

Hard work is a medication for which
there is no placebo.

<amyl@paxemail.com> wrote in message
news:1146151800.344480.232990@j33g2000cwa.googlegr oups.com...[color=blue]
>I have an application that is reading data from text files to the first
> occurrence of a double crlf. I have this working using streamreader
> and using readline.
>
> However, I am hitting 500+ files at a time and this is a bit slow.
>
> I was thinking of testing this using FileStream or something similar,
> but I am not sure how to detect the occurrence of crlf crlf?
>
> Thoughts?
> Amy
>[/color]

re: Reading text data to double crlf

james.curran@gmail.com wrote:[color=blue]
> It's doubtful switch to FileStream will make it any faster.
> StreamReader, when constructed using a file name, just creates a
> FileStream internally. Your test would be just duplicating
> StreamReader.
>
> How are you reading it now (using streamreader & readline)?[/color]

Yes, streamreader and readline.

It seems the file reading and parsing is fast. I beleive my major
delay is the listview being built.

Amy

re: Reading text data to double crlf

1.) No, the encoding could be different.
2.) File size is random. Most are less < 20k, some could be as large
as 10mb.

Currently I am going line by line and stopping at the second \r\n\.

It seems my issue is not the reading of the files, but the listview
being created. It takes several seconds (5-10) to add about 1000
entries into a 6 column listview. Unfortuantly it still happens even
when using BeginUpdate and EndUpdate. Are there any other listview
tricks?

Amy

re: Reading text data to double crlf

<amyl@paxemail.com> wrote:[color=blue]
> 1.) No, the encoding could be different.
> 2.) File size is random. Most are less < 20k, some could be as large
> as 10mb.
>
> Currently I am going line by line and stopping at the second \r\n\.
>
> It seems my issue is not the reading of the files, but the listview
> being created. It takes several seconds (5-10) to add about 1000
> entries into a 6 column listview. Unfortuantly it still happens even
> when using BeginUpdate and EndUpdate. Are there any other listview
> tricks?[/color]

You should read the file in another thread and update the UI using
BeginInvoke.

See http://www.pobox.com/~skeet/csharp/t...winforms.shtml

--
Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

re: Reading text data to double crlf

>> > How are you reading it now (using streamreader & readline)?[color=blue][color=green]
>> Yes, streamreader and readline.[/color][/color]

That wasn't intended as a yes/no question.

Rephrasing: "Your current method (the one that uses streamreader &
readline), what is it?"

However, your next statement would make that irrelevant.

re: Reading text data to double crlf

Thanks Jon,

I am currently going this avenue. I am using .Net 2.0 and seen they
now have a BackgroundWorker. Have you used this class yet?

Amy.

Jon wrote:[color=blue]
> <amyl@paxemail.com> wrote:[color=green]
> > 1.) No, the encoding could be different.
> > 2.) File size is random. Most are less < 20k, some could be as large
> > as 10mb.
> >
> > Currently I am going line by line and stopping at the second \r\n\.
> >
> > It seems my issue is not the reading of the files, but the listview
> > being created. It takes several seconds (5-10) to add about 1000
> > entries into a 6 column listview. Unfortuantly it still happens even
> > when using BeginUpdate and EndUpdate. Are there any other listview
> > tricks?[/color]
>
> You should read the file in another thread and update the UI using
> BeginInvoke.
>
> See http://www.pobox.com/~skeet/csharp/t...winforms.shtml
>
> --
> Jon Skeet - <skeet@pobox.com>
> http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
> If replying to the group, please do not mail me too[/color]

re: Reading text data to double crlf

<amyl@paxemail.com> wrote:[color=blue]
> I am currently going this avenue. I am using .Net 2.0 and seen they
> now have a BackgroundWorker. Have you used this class yet?[/color]

I haven't used it much, but that would do the trick, certainly. You'd
still need to be careful not to manipulate the UI directly from the
wrong thread. Read the docs carefully and you should be okay.

--
Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too