472,353 Members | 2,059 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,353 software developers and data experts.

Urgent: Fast way to read Parts of Big Files

Hi,

I have files I need to read, which contains records with a variable lenght.
What I need to do is Copy a Part of such a File to a new File, based on the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS), OpenMode.Input,
OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS), FileMode.Open,
FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) *
128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what about
spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem,
and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter
Nov 20 '05 #1
6 1640
you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...
Hi,

I have files I need to read, which contains records with a variable lenght. What I need to do is Copy a Part of such a File to a new File, based on the a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS), OpenMode.Input, OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS), FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) *
128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what about spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem,
and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

Nov 20 '05 #2
you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...
Hi,

I have files I need to read, which contains records with a variable lenght. What I need to do is Copy a Part of such a File to a new File, based on the a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS), OpenMode.Input, OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10
Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS), FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) *
128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what about spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem,
and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter

Nov 20 '05 #3
Hihihi damn I'm affraid I don't dare to come back to this newsgroups, hihi
:-) I really made some stupid mistake :-/

That's what happens when you make love to your girlfriend during the
lunch-break, come too late back at work and skip accidently a meeting... No
wonder I didn't think clear, hahahaha :-)

Thanks guys!

And my apologizes for the 2 postings, but my Outlook Express deleted the
first message, and 30 minutes later it suddenly came back :-/

Pieter

"CJ Taylor" <no****@blowgoats.com> wrote in message
news:10*************@corp.supernews.com...
you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...
Hi,

I have files I need to read, which contains records with a variable

lenght.
What I need to do is Copy a Part of such a File to a new File, based on

the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS),

OpenMode.Input,
OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),

FileMode.Open,
FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) * 128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what

about
spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem, and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter


Nov 20 '05 #4
Hihihi damn I'm affraid I don't dare to come back to this newsgroups, hihi
:-) I really made some stupid mistake :-/

That's what happens when you make love to your girlfriend during the
lunch-break, come too late back at work and skip accidently a meeting... No
wonder I didn't think clear, hahahaha :-)

Thanks guys!

And my apologizes for the 2 postings, but my Outlook Express deleted the
first message, and 30 minutes later it suddenly came back :-/

Pieter

"CJ Taylor" <no****@blowgoats.com> wrote in message
news:10*************@corp.supernews.com...
you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...
Hi,

I have files I need to read, which contains records with a variable

lenght.
What I need to do is Copy a Part of such a File to a new File, based on

the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS),

OpenMode.Input,
OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),

FileMode.Open,
FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec - intStartRec) * 128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to a
String (I tryed filing the Char with spaces and than Trim it but what

about
spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is
there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this problem, and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter


Nov 20 '05 #5
Wow.

"DraguVaso" <pi**********@hotmail.com> wrote in message
news:Ow***************@tk2msftngp13.phx.gbl...
Hihihi damn I'm affraid I don't dare to come back to this newsgroups, hihi
:-) I really made some stupid mistake :-/

That's what happens when you make love to your girlfriend during the
lunch-break, come too late back at work and skip accidently a meeting... No wonder I didn't think clear, hahahaha :-)

Thanks guys!

And my apologizes for the 2 postings, but my Outlook Express deleted the
first message, and 30 minutes later it suddenly came back :-/

Pieter

"CJ Taylor" <no****@blowgoats.com> wrote in message
news:10*************@corp.supernews.com...
you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...
Hi,

I have files I need to read, which contains records with a variable lenght.
What I need to do is Copy a Part of such a File to a new File, based on
the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS),

OpenMode.Input,
OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),

FileMode.Open,
FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec -
intStartRec) * 128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to
a String (I tryed filing the Char with spaces and than Trim it but what

about
spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this

problem, and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter



Nov 20 '05 #6
Wow.

"DraguVaso" <pi**********@hotmail.com> wrote in message
news:Ow***************@tk2msftngp13.phx.gbl...
Hihihi damn I'm affraid I don't dare to come back to this newsgroups, hihi
:-) I really made some stupid mistake :-/

That's what happens when you make love to your girlfriend during the
lunch-break, come too late back at work and skip accidently a meeting... No wonder I didn't think clear, hahahaha :-)

Thanks guys!

And my apologizes for the 2 postings, but my Outlook Express deleted the
first message, and 30 minutes later it suddenly came back :-/

Pieter

"CJ Taylor" <no****@blowgoats.com> wrote in message
news:10*************@corp.supernews.com...
you know better than this pieter.
"DraguVaso" <pi**********@hotmail.com> wrote in message
news:%2******************@TK2MSFTNGP10.phx.gbl...
Hi,

I have files I need to read, which contains records with a variable lenght.
What I need to do is Copy a Part of such a File to a new File, based on
the
a Begin- and End-record.

I used this functions:
Dim intMyFile As Integer = FreeFile()
FileOpen(intMyFile, MakePathFile(strDirS, strFileS),

OpenMode.Input,
OpenAccess.Read, OpenShare.Shared, -1)
Do While Not EOF(intMyFile)
strLine = LineInput(intMyFile)
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

It worked fine until I met some really big files. I have some files of 10 Mb, containing 75000 records... After 20 minutes my application still
doesn't have read the exact part.
I tryed this:
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),

FileMode.Open,
FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
intX = 0
Do While intX <= intEndRec
strLine = brFile.ReadLine
If (intX >= intStartRec) And (intX <= intEndRec) Then
strNew = strNew & strLine & vbCrLf
End If
intX = intX + 1
Loop

But it's as slow as the other one.

Only one thing was really quick (only 10 seconds):
Dim fsFile As New FileStream(MakePathFile(strDirS, strFileS),
FileMode.Open, FileAccess.Read)
Dim brFile As New StreamReader(fsFile,
System.Text.Encoding.GetEncoding(1252), False, fsFile.Length - 1)
Dim strChar((intEndRec * 128) - 1) As Char
For intX = 0 To strChar.Length - 1
strChar(intX) = " "
Next
brFile.ReadBlock(strChar, intStartRec * 128, (intEndRec -
intStartRec) * 128)

But here I had several stupid problems for which I din't really find a
solution:
- first of all: I'm having really big problems converting the Char to
a String (I tryed filing the Char with spaces and than Trim it but what

about
spaces in the end of my File?)
- the ReadBlock works with character-positioning, and not with lines. Is there a way to convert a line-position to a character-position or do a
ReadBlock with lines or something like that?
Anyhelp would really be appreciated!! I'm really stuck with this

problem, and it's kidn of urgent!! Any help regarding the ReadBlock or
StremReader-stuff, or on other (FAST!) way to do this would really be
appreciated!!

Thanks a lot in advance!!

Pieter



Nov 20 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

20
by: GS | last post by:
The stdint.h header definition mentions five integer categories, 1) exact width, eg., int32_t 2) at least as wide as, eg., int_least32_t 3) as...
6
by: Cable | last post by:
Hello, I am hoping that someone can answer a question or two regarding file access. I have created an app that reads an image from a file then...
4
by: DraguVaso | last post by:
Hi, I have files I need to read, which contains records with a variable lenght. What I need to do is Copy a Part of such a File to a new File,...
0
by: DraguVaso | last post by:
Hi, I have files I need to read, which contains records with a variable lenght. What I need to do is Copy a Part of such a File to a new File,...
0
by: Együd Csaba | last post by:
> -----Original Message----- > From: pgsql-admin-owner@postgresql.org > On Behalf Of Együd Csaba > Sent: 2004. július 16. 12:23 > To: pgsql-admin...
8
by: ginnisharma1 | last post by:
Hi All, I am very new to C language and I got really big assignment in my work.I am wondering if anyone can help me.........I need to port...
1
by: devil123 | last post by:
hey .. i am a beginner in visual basic ... i want a urgent help i have created a application which is having 2 text files as input ..now i have...
14
by: Umesh | last post by:
CoffeeVendor is a program, which simulate a coffee machine. The coffee machine serves coffee with sugar and creamer for Rs.10/- and Rs.6/- without...
9
by: Salad | last post by:
I have access, for testing at my client's site, a Win2000 computer running A2003 retail. He recently upgraded all of his other machines to...
1
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and...
0
jalbright99669
by: jalbright99669 | last post by:
Am having a bit of a time with URL Rewrite. I need to incorporate http to https redirect with a reverse proxy. I have the URL Rewrite rules made...
0
by: antdb | last post by:
Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was...
0
by: Matthew3360 | last post by:
Hi there. I have been struggling to find out how to use a variable as my location in my header redirect function. Here is my code. ...
2
by: Matthew3360 | last post by:
Hi, I have a python app that i want to be able to get variables from a php page on my webserver. My python app is on my computer. How would I make it...
0
hi
by: WisdomUfot | last post by:
It's an interesting question you've got about how Gmail hides the HTTP referrer when a link in an email is clicked. While I don't have the specific...
0
by: Matthew3360 | last post by:
Hi, I have been trying to connect to a local host using php curl. But I am finding it hard to do this. I am doing the curl get request from my web...
0
Oralloy
by: Oralloy | last post by:
Hello Folks, I am trying to hook up a CPU which I designed using SystemC to I/O pins on an FPGA. My problem (spelled failure) is with the...
0
by: Carina712 | last post by:
Setting background colors for Excel documents can help to improve the visual appeal of the document and make it easier to read and understand....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.