HELP!! FAST way to read Parts of Big Files

[DraguVaso]

Hi,

I have files I need to read, which contains records with a variable lenght.
What I need to do is Copy a Part of such a File to a new File, based on the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyF ile, MakePathFile(st rDirS, strFileS), OpenMode.Input,
OpenAccess.Read , OpenShare.Share d, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMy File)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(Make PathFile(strDir S, strFileS), FileMode.Open,
FileAccess.Read )
Dim brFile As New StreamReader(fs File,
System.Text.Enc oding.GetEncodi ng(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(Make PathFile(strDir S, strFileS),
FileMode.Open, FileAccess.Read )
Dim brFile As New StreamReader(fs File,
System.Text.Enc oding.GetEncodi ng(1252), False, fsFile.Length - 1)
Dim strChar((intEnd Rec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBloc k(strChar, intStartRec * 128, (intEndRec - intStartRec) *
128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what about
spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem,
and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

[Jon Skeet [C# MVP]]

DraguVaso <pi**********@h otmail.com> wrote:

I have files I need to read, which contains records with a variable lenght.
What I need to do is Copy a Part of such a File to a new File, based on the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyF ile, MakePathFile(st rDirS, strFileS), OpenMode.Input,
OpenAccess.Read , OpenShare.Share d, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMy File)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.

The problem is your string concatenation. Either create a StreamWriter
to start with, and just copy the relevant lines straight to disk, or
use a StringBuilder to build up the string in memory.

Just reading through the file should be very fast indeed.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

DraguVaso <pi**********@h otmail.com> wrote:

I have files I need to read, which contains records with a variable lenght.
What I need to do is Copy a Part of such a File to a new File, based on the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyF ile, MakePathFile(st rDirS, strFileS), OpenMode.Input,
OpenAccess.Read , OpenShare.Share d, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMy File)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.

The problem is your string concatenation. Either create a StreamWriter
to start with, and just copy the relevant lines straight to disk, or
use a StringBuilder to build up the string in memory.

Just reading through the file should be very fast indeed.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[DraguVaso]

Shit :-S
*enormous blush* :-(
I spend 2 hours trying to get it faster and getting frustrated, and I didn't
think about the string-concatenation :-(

Thanks a lot! You made my day! hehe :-)

Pieter

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...

DraguVaso <pi**********@h otmail.com> wrote:

I have files I need to read, which contains records with a variable lenght. What I need to do is Copy a Part of such a File to a new File, based on the a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyF ile, MakePathFile(st rDirS, strFileS), OpenMode.Input, OpenAccess.Read , OpenShare.Share d, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMy File)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.

The problem is your string concatenation. Either create a StreamWriter
to start with, and just copy the relevant lines straight to disk, or
use a StringBuilder to build up the string in memory.

Just reading through the file should be very fast indeed.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[DraguVaso]

Shit :-S
*enormous blush* :-(
I spend 2 hours trying to get it faster and getting frustrated, and I didn't
think about the string-concatenation :-(

Thanks a lot! You made my day! hehe :-)

Pieter

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...

DraguVaso <pi**********@h otmail.com> wrote:

I have files I need to read, which contains records with a variable lenght. What I need to do is Copy a Part of such a File to a new File, based on the a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyF ile, MakePathFile(st rDirS, strFileS), OpenMode.Input, OpenAccess.Read , OpenShare.Share d, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMy File)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.

The problem is your string concatenation. Either create a StreamWriter
to start with, and just copy the relevant lines straight to disk, or
use a StringBuilder to build up the string in memory.

Just reading through the file should be very fast indeed.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too