467,113 Members | 1,318 Online
Bytes | Developer Community
Ask Question

Home New Posts Topics Members FAQ

Post your question to a community of 467,113 developers. It's quick & easy.

Using .Read(buffer, 0, totalSize) with low RAM

I have only 1GB of RAM so I cannot work with files too big with the ReadtoEnd method.
Here is the code Im trying so that the file would split in pieces of 256MB:

Expand|Select|Wrap|Line Numbers
  1.  Try
  2.                Dim BInput As New FileStream(Filename, FileMode.Open, FileAccess.Read)
  3.                Dim Reader As New BinaryReader(BInput)
  5.                Dim info As New FileInfo(Filename)
  6.                totalSize = info.Length
  7.                Dim i As Integer = totalSize / Dtblock 'Dtblock is a const 256mb
  8.                Dim completed As Integer = 0
  10.                Dim firstround As Boolean = True
  12.                If (totalSize > Dtblock) Then
  13.                   'The file is bigger then 256MB
  14.                   'Prepare to split
  15.                   Dim EngineObj As New Engine()
  16.                   ReDim buffer(Dtblock)
  18.                   While completed < totalSize
  20.                      Reader.Read(buffer, completed, completed + Dtblock)
  21.                      '..
  22. 'here I work with the buffer, then I have to save it
  23.   '..
  25.                      completed += Dtblock + 1
  27.                      If firstround Then 'here I check if I should create or append to output
  28.                         'Write
  29.                         firstround = False
  30.                         Dim BOutput As New FileStream(TxtBoxOutput.Text, FileMode.Create, FileAccess.Write)
  31.                         Dim writer As New BinaryWriter(BOutput)
  32.                         writer.Write(buffer)
  33.                      Else
  34.                         'Append
  35.                         Dim BOutput As New FileStream(TxtBoxOutput.Text, FileMode.Append, FileAccess.Write)
  36.                         Dim writer As New BinaryWriter(BOutput)
  37.                         writer.Write(buffer)
  38.                      End If
  40.                   End While
  42.                Else
  43.                   'The file is smaller then 256MB
  45.                   ReDim buffer(totalSize)
  46.                   Reader.Read(buffer, 0, totalSize)
  47. ..
  48.                  'Here I work with the buffer..
  49. ..
  50.                   'Write buffer to new file
  51.                   Dim BOutput As New FileStream(TxtBoxOutput.Text, FileMode.Create, FileAccess.Write)
  52.                   Dim writer As New BinaryWriter(BOutput)
  53.                   writer.Write(buffer)
  54.                   'Done, close instances of file streams
  55.                   Reader.Close()
  56.                   writer.Close()
  58.                End If
  60.                BInput.Dispose()
  61.                MessageBox.Show("Operation finished!")
  62.             Catch ex As Exception
  63.                MessageBox.Show(ex.ToString)
  64.             End Try
I know it might be horrible, but its the only idea I came up with.
anyone knows a better way to accomplish this?

Or if somebody find knows why I always get an exception on the "Reader.Read(buffer, completed, completed + Dtblock)" saying :

"Offset and length for the array lie outside of the valid field, or the number is greater than the number of the elements of the index to at the end of the source listing."

would be even better..

but till now I dont know why that happens because I supposed that the parameters for Read was like this:

Read(<WheretoStore>,<starting point>, <how many to read from starting point>)

Aug 3 '07 #1
  • viewed: 1767
4 Replies
Expert 512MB
I am not sure, but I think the error is because when you read, you move the start location of the read. For example:

Expand|Select|Wrap|Line Numbers
  1. The dog jumped over the moon
If you wrote:

Expand|Select|Wrap|Line Numbers
  1. Read(myBuffer, 0, 3)
You would get 'The'

If you did it again, you would get ' do'. But if instead you did:

Expand|Select|Wrap|Line Numbers
  1. Read(myBuffer, 2, 3)
you would get 'og '

This is all purely speculation, a short test should prove this wrong/right.
Aug 3 '07 #2
thank you very much for your reply.

you were right about the Read's parameter.

I just didnt know that it had an internal pointer that keeps track of the last byte processed ;)

thanks again!

Aug 12 '07 #3
Expert 512MB
No problem, glad to have helped.
Aug 13 '07 #4
Expert 4TB
For Read the paremters are:
[number of bytes actually read] Read( [name of byte[] you want the data to go into], [starting index in the GIVEN byte[] (the previous parameter)], [total number of bytes to attempt to read] );

so if I wanted to try to read 1024 bytes into a buffer starting at the first index in my array I woul do this:
Expand|Select|Wrap|Line Numbers
  1. Byte[] buff=new Byte[1024];
  2. int actualread=-1;
  3. actaulread=mystream.Read(buff,0,buff.Length);
  4. //actualread will be set to how many bytes I actually was able to read, or 0 if end of stream
Aug 13 '07 #5

Post your reply

Sign in to post your reply or Sign up for a free account.

Similar topics

reply views Thread by Craig Buchanan | last post: by
4 posts views Thread by chenatcr@yahoo.com | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.