Urgent: Fast way to read Parts of Big Files

DraguVaso

Hi,

I have files I need to read, which contains records with a variable lenght.
What I need to do is Copy a Part of such a File to a new File, based on the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS), OpenMode.Input,
OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS), FileMode.Open,
FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) *
128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what about
spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem,
and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

Nov 20 '05 #1

CJ Taylor

you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...

Hi,

I have files I need to read, which contains records with a variable lenght. What I need to do is Copy a Part of such a File to a new File, based on the a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS), OpenMode.Input, OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS), FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) *
128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what about spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem,
and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

Nov 20 '05 #2

CJ Taylor

you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...

Hi,

I have files I need to read, which contains records with a variable lenght. What I need to do is Copy a Part of such a File to a new File, based on the a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS), OpenMode.Input, OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS), FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) *
128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what about spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem,
and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

Nov 20 '05 #3

DraguVaso

Hihihi damn I'm affraid I don't dare to come back to this newsgroups, hihi
:-) I really made some stupid mistake :-/

That's what happens when you make love to your girlfriend during the
lunch-break, come too late back at work and skip accidently a meeting... No
wonder I didn't think clear, hahahaha :-)

Thanks guys!

And my apologizes for the 2 postings, but my Outlook Express deleted the
first message, and 30 minutes later it suddenly came back :-/

Pieter

"CJ Taylor" <no****@blowgoats.com> wrote in message
news:10*************@corp.supernews.com...

you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...

Hi,

I have files I need to read, which contains records with a variable

lenght.

What I need to do is Copy a Part of such a File to a new File, based on

the

a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS),

OpenMode.Input,

OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),

FileMode.Open,

FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) * 128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what

about

spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem, and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

Nov 20 '05 #4

DraguVaso

Hihihi damn I'm affraid I don't dare to come back to this newsgroups, hihi
:-) I really made some stupid mistake :-/

That's what happens when you make love to your girlfriend during the
lunch-break, come too late back at work and skip accidently a meeting... No
wonder I didn't think clear, hahahaha :-)

Thanks guys!

And my apologizes for the 2 postings, but my Outlook Express deleted the
first message, and 30 minutes later it suddenly came back :-/

Pieter

"CJ Taylor" <no****@blowgoats.com> wrote in message
news:10*************@corp.supernews.com...

you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...

Hi,

I have files I need to read, which contains records with a variable

lenght.

What I need to do is Copy a Part of such a File to a new File, based on

the

a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS),

OpenMode.Input,

OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),

FileMode.Open,

FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) * 128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what

about

spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem, and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

Nov 20 '05 #5

CJ Taylor

Wow.

"DraguVaso" <pi**********@hotmail.com> wrote in message
news:Ow***************@tk2msftngp13.phx.gbl...

Hihihi damn I'm affraid I don't dare to come back to this newsgroups, hihi
:-) I really made some stupid mistake :-/

That's what happens when you make love to your girlfriend during the
lunch-break, come too late back at work and skip accidently a meeting... No wonder I didn't think clear, hahahaha :-)

Thanks guys!

And my apologizes for the 2 postings, but my Outlook Express deleted the
first message, and 30 minutes later it suddenly came back :-/

Pieter

"CJ Taylor" <no****@blowgoats.com> wrote in message
news:10*************@corp.supernews.com...

you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...

Hi,

I have files I need to read, which contains records with a variable lenght.

What I need to do is Copy a Part of such a File to a new File, based on

the

a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS),

OpenMode.Input,

OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),

FileMode.Open,

FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec -

intStartRec) * 128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to

a String (I tryed filing the Char with spaces and than Trim it but what

about

spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this

problem, and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

Nov 20 '05 #6

CJ Taylor

Wow.

"DraguVaso" <pi**********@hotmail.com> wrote in message
news:Ow***************@tk2msftngp13.phx.gbl...

Hihihi damn I'm affraid I don't dare to come back to this newsgroups, hihi
:-) I really made some stupid mistake :-/

That's what happens when you make love to your girlfriend during the
lunch-break, come too late back at work and skip accidently a meeting... No wonder I didn't think clear, hahahaha :-)

Thanks guys!

And my apologizes for the 2 postings, but my Outlook Express deleted the
first message, and 30 minutes later it suddenly came back :-/

Pieter

"CJ Taylor" <no****@blowgoats.com> wrote in message
news:10*************@corp.supernews.com...

you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...

Hi,

I have files I need to read, which contains records with a variable lenght.

What I need to do is Copy a Part of such a File to a new File, based on

the

a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS),

OpenMode.Input,

OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),

FileMode.Open,

FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec -

intStartRec) * 128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to

a String (I tryed filing the Char with spaces and than Trim it but what

about

spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this

problem, and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

Nov 20 '05 #7