Best way to delete the first and the last records of a very big file

[booksnore]

I am reading some very large files greater than 10 GB. Some of the files
(not all) contain a header and footer record identified by "***" in the
first three characters of the record. I need to delete the header or
footer record before reading the file into a database. Whats the best
way to do this in C#?
Any help appreciated.

Joe

*** Sent via Developersdex http://www.developersdex.com ***

[jeremiah johnson]

booksnore wrote:

I am reading some very large files greater than 10 GB. Some of the files
(not all) contain a header and footer record identified by "***" in the
first three characters of the record. I need to delete the header or
footer record before reading the file into a database. Whats the best
way to do this in C#?
Any help appreciated.

Joe

*** Sent via Developersdex http://www.developersdex.com ***

if you're able to loop through the records in the very large files, skip
processing when you see that the record you're on starts with "***".

using System.Text.RegularExpressions;
....

foreach (string s in records) { // example code
Match m = Regex.Match(s, "^***");

if (m.Success) { // this is a header or footer
continue;
} else {
processRecord();
}

}

[Greg Bacon]

In article <#t**************@TK2MSFTNGP12.phx.gbl>,
jeremiah johnson <na*******@gmail.com> wrote:

: booksnore wrote:

: > I am reading some very large files greater than 10 GB. Some of the files
: > (not all) contain a header and footer record identified by "***" in the
: > first three characters of the record. I need to delete the header or
: > footer record before reading the file into a database. Whats the best
: > way to do this in C#?
: > Any help appreciated.
:
: if you're able to loop through the records in the very large
: files, skip processing when you see that the record you're on
: starts with "***".
:
: using System.Text.RegularExpressions;
: ...
:
: foreach (string s in records) { // example code
: Match m = Regex.Match(s, "^***");

Of course, that should be @"^\*\*\*"; otherwise, you'll get an exception
at runtime.

: if (m.Success) { // this is a header or footer
: continue;
: } else {
: processRecord();
: }

If you like guards instead, you could write

if (Regex.IsMatch(input, @"^\*\*\*"))
continue;

ObJonSkeet: You could also write the test as

if (input.IndexOf("***") == 0)

Greg
--
The idea that the vote of a people, no matter how nearly unanimous, makes or
creates or determines what is right or just becomes as absurd and
unacceptable as the idea that right and justice are simply whatever a king
says they are. -- Robert Welch

[Gordon Smith \(eMVP\)]

booksnore wrote:

I am reading some very large files greater than 10 GB. Some of the
files (not all) contain a header and footer record identified by
"***" in the first three characters of the record. I need to delete
the header or footer record before reading the file into a database.
Whats the best way to do this in C#?
Any help appreciated.

Joe

*** Sent via Developersdex http://www.developersdex.com ***

That depends on how you are "reading the file into a database". If you are
parsing the file yourself, simply skip the input records you want to (as
identified above as having "***" in the first 3 characters). In other
words, do you really need to DELETE the records or just ignore them?

--
Gordon Smith (eMVP)
-- Avnet Applied Computing Solutions

[Jon Skeet [C# MVP]]

Greg Bacon <gb****@hiwaay.net> wrote:

: if (m.Success) { // this is a header or footer
: continue;
: } else {
: processRecord();
: }

If you like guards instead, you could write

if (Regex.IsMatch(input, @"^\*\*\*"))
continue;

ObJonSkeet: You could also write the test as

if (input.IndexOf("***") == 0)

Or, even more readably:

if (input.StartsWith("***"))

I challenge anyone to claim with a straight face that the regex is a
more readable option than the above...

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[booksnore]

Thank you for the replies. Yes I will be ignoring the records rather
than deleting them. I'm testing your suggestions now to see which will
provide the best performance on very large input.

Thank you again

Joe

*** Sent via Developersdex http://www.developersdex.com ***