Out of Memory Error during reDim Statment

[Trevor Fairchild]

I've got a program that parses text files. The text files come to me
in Unicode and they contain goofy characters that VB chokes on -
treats them as eof markers. I have already been through this issue in
another thread and I have received an answer that works (code is
below)

However, my test data thus far has been in the 10-50MB size range. I
now have a case where the text file is 650MB in size!

I need to convert it to ANSI for my program to parse it correctly
using the line input method. There is over 1 million lines of text in
this file, and each line contains about 12 fields' worth of
information.

Here is the code:

Dim bHTML() as Byte
Dim myLong as Long
myLong = FreeFile
Open myPath & myName For Binary as #myLong
ReDim bHTML(1 to LOF(myLong))
Get #myLong, , bHTML
Close #myLong

Open myPath & "temp_" & myName for Output as #myLong
Print #myLong, bHTML
Close #myLong

When running this code against the 650MB file - I get an Out of Memory
error instantaneously. I am not very familiar with arrays, so do not
know what to do to work-around this type of error.

Does anyone have any suggestions?

[Mike and Jo]

"Trevor Fairchild" <MR*******@e-crime.on.ca> wrote in message
news:f5**************************@posting.google.c om...

I've got a program that parses text files. The text files come to me
in Unicode and they contain goofy characters that VB chokes on -
treats them as eof markers. I have already been through this issue in
another thread and I have received an answer that works (code is
below)

However, my test data thus far has been in the 10-50MB size range. I
now have a case where the text file is 650MB in size!

I need to convert it to ANSI for my program to parse it correctly
using the line input method. There is over 1 million lines of text in
this file, and each line contains about 12 fields' worth of
information.

Here is the code:

Dim bHTML() as Byte
Dim myLong as Long
myLong = FreeFile
Open myPath & myName For Binary as #myLong
ReDim bHTML(1 to LOF(myLong))
If the file is 650 MB you are trying to allocate 650 MB of memory. You get
the error because you don't have that much memory availiable.

Try creating an array that your system can handle and step through the file
with the same array lengths. Try something like 1 MB. Access and transfer
the file in 1 MB portions until you have made it through the file. You can
do this using the Get #myLong, [File Byte Position], bHTML.

Just increment the [File Byte Position] in steps of the length of the array
you declare. Make sure when you get to the last segment you ReDim the bHTML
array to the remaining byte size (your file may not be exactly 650 MB)

Regards
Mike

Get #myLong, , bHTML
Close #myLong

Open myPath & "temp_" & myName for Output as #myLong
Print #myLong, bHTML
Close #myLong

When running this code against the 650MB file - I get an Out of Memory
error instantaneously. I am not very familiar with arrays, so do not
know what to do to work-around this type of error.

Does anyone have any suggestions?

[Trevor Fairchild]

This was what I was trying to do. However, I cannot get the ReDim
Statement to only take out 1,000,000 characters at a time.

I put in a loop to cycle through, using

ReDim bHTML(c, c + 1000000)

but it does not seem to work right - I keep on getting the first
1,000,000 characters, regardless of what c is equal to. It doesn't
make much sense to me, but I get the impression this isn't just
spaces, but array dimensions, and I'm out of my league in that.

The "Get #myLong, STARTING, bHTML" simply tells it where to start
getting the data, not where to end - so I just get the 1,000,000 - to
EOF, instead of all of it. Furthermore, the system crashes at the
ReDim statement - I have to redim in smaller chunks - that's my
problem, I guess.

I've tested a fair amount on the redim and cannot seem to figure it
out. The file is in Unicode, so each character is actually 2 bytes in
size, not 1 - I tried ReDim bHTML(2 to 4) and that gets what I would
expect - the first 2 characters in the file. When I ReDim bHTML(4 to
8) I still only get the first 2 characters in the file - I would
expect to get the 3rd & 4th characters.

I don't get it.

[the Wiz]

MR*******@e-crime.on.ca (Trevor Fairchild) wrote:

I've got a program that parses text files. The text files come to me
in Unicode and they contain goofy characters that VB chokes on -
treats them as eof markers. I have already been through this issue in
another thread and I have received an answer that works (code is
below)

However, my test data thus far has been in the 10-50MB size range. I
now have a case where the text file is 650MB in size!

I need to convert it to ANSI for my program to parse it correctly
using the line input method. There is over 1 million lines of text in
this file, and each line contains about 12 fields' worth of
information.

Air code - you'll need to make adjustments ;-)
It's based on a compiled DOS Basic program from 1991 that read 300MB of trashed
UNIX file system in 32K chunks to recover email.

Dim FileSize as long
Dim CurrentPosition as long
Dim MyArray(1000000) as Byte
Dim LastBytes as Long

Open the file and use LOF to get the FileSize.

LastBytes = FileSize mod 1,000,000

CurrentPosition = 0

Do While CurrentPosition < FileSize

Get MyArray from files, starting at CurrentPosition. 'read in 1,000,000 bytes

Increment CurrentPosition by 1,000,000 'move pointer to next block of bytes

Process 1,000,000 bytes in MyArray.
End Do

Get MyArray from files, starting at CurrentPosition.

Process LastBytes of valid data in MyArray

No ReDim, no running out of memory.

If you have enough available memory, MyArray could be 5 or 10 million.
More about me: http://www.jecarter.com/
VB3/VB6/C/PowerBasic source code: http://www.jecarter.com/programs.html
Freeware for the Palm with NS Basic source code: http://nsb.jecarter.com
Drivers for Pablo graphics tablet and JamCam cameras: http://home.earthlink.net/~mwbt/
johnecarter at@at mindspring dot.dot com. Fix the obvious to reply by email.

[Steve Gerrard]

"Trevor Fairchild" <MR*******@e-crime.on.ca> wrote in message
news:f5**************************@posting.google.c om...

You are a bit confused about what redimming an array does. It has no
affect on which characters are read from the input file. Whether you
number four elements as (0 to 3) or (1 to 4) or even (123 to 126) makes
no difference - they are still just four elements, and the first time
you read, you will get the first four items.

Your original program is basically reading two byte unicode characters
as individual bytes, then implicitly converting the byte array to an
ANSI string by using the Print statement to output them. As long as you
read an even number of bytes each time, it works just as well in blocks.

Here is an example of how to go about reading and writing a file of any
size in blocks of 16384, a reasonable power of 2 size for doing file IO
stuff:

Private Sub Convert(Path As String, FileName As String)
Dim bHTML() As Byte
Dim nFileIn As Long
Dim nFileOut As Long
Dim nBytes As Long
Const BlockSize As Long = 16384
Dim nBlocks As Long
Dim nExtra As Long
Dim nCnt As Long

nFileIn = FreeFile
Open Path & FileName For Binary As #nFileIn

nBytes = LOF(nFileIn)
nBlocks = nBytes \ BlockSize
nExtra = nBytes Mod BlockSize

nFileOut = FreeFile
Open Path & "temp_" & FileName For Output As #nFileOut

ReDim bHTML(1 To BlockSize)

For nCnt = 1 To nBlocks
Get #nFileIn, , bHTML
Print #nFileOut, bHTML
Next n

If nExtra > 0 Then
ReDim bHTML(1 To nExtra)
Get #nFileIn, , bHTML
Print #nFileOut, bHTML
End If

Close #nFileOut
Close #nFileIn

End Sub