By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,364 Members | 1,272 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,364 IT Pros & Developers. It's quick & easy.

Using .Read(buffer, 0, totalSize) with low RAM

P: 89
I have only 1GB of RAM so I cannot work with files too big with the ReadtoEnd method.
Here is the code Im trying so that the file would split in pieces of 256MB:

Expand|Select|Wrap|Line Numbers
  1.  Try
  2.                Dim BInput As New FileStream(Filename, FileMode.Open, FileAccess.Read)
  3.                Dim Reader As New BinaryReader(BInput)
  4.  
  5.                Dim info As New FileInfo(Filename)
  6.                totalSize = info.Length
  7.                Dim i As Integer = totalSize / Dtblock 'Dtblock is a const 256mb
  8.                Dim completed As Integer = 0
  9.  
  10.                Dim firstround As Boolean = True
  11.  
  12.                If (totalSize > Dtblock) Then
  13.                   'The file is bigger then 256MB
  14.                   'Prepare to split
  15.                   Dim EngineObj As New Engine()
  16.                   ReDim buffer(Dtblock)
  17.  
  18.                   While completed < totalSize
  19.  
  20.                      Reader.Read(buffer, completed, completed + Dtblock)
  21.                      '..
  22. 'here I work with the buffer, then I have to save it
  23.   '..
  24.  
  25.                      completed += Dtblock + 1
  26.  
  27.                      If firstround Then 'here I check if I should create or append to output
  28.                         'Write
  29.                         firstround = False
  30.                         Dim BOutput As New FileStream(TxtBoxOutput.Text, FileMode.Create, FileAccess.Write)
  31.                         Dim writer As New BinaryWriter(BOutput)
  32.                         writer.Write(buffer)
  33.                      Else
  34.                         'Append
  35.                         Dim BOutput As New FileStream(TxtBoxOutput.Text, FileMode.Append, FileAccess.Write)
  36.                         Dim writer As New BinaryWriter(BOutput)
  37.                         writer.Write(buffer)
  38.                      End If
  39.  
  40.                   End While
  41.  
  42.                Else
  43.                   'The file is smaller then 256MB
  44.  
  45.                   ReDim buffer(totalSize)
  46.                   Reader.Read(buffer, 0, totalSize)
  47. ..
  48.                  'Here I work with the buffer..
  49. ..
  50.                   'Write buffer to new file
  51.                   Dim BOutput As New FileStream(TxtBoxOutput.Text, FileMode.Create, FileAccess.Write)
  52.                   Dim writer As New BinaryWriter(BOutput)
  53.                   writer.Write(buffer)
  54.                   'Done, close instances of file streams
  55.                   Reader.Close()
  56.                   writer.Close()
  57.  
  58.                End If
  59.  
  60.                BInput.Dispose()
  61.                MessageBox.Show("Operation finished!")
  62.             Catch ex As Exception
  63.                MessageBox.Show(ex.ToString)
  64.             End Try
I know it might be horrible, but its the only idea I came up with.
anyone knows a better way to accomplish this?

Or if somebody find knows why I always get an exception on the "Reader.Read(buffer, completed, completed + Dtblock)" saying :

"Offset and length for the array lie outside of the valid field, or the number is greater than the number of the elements of the index to at the end of the source listing."

would be even better..

but till now I dont know why that happens because I supposed that the parameters for Read was like this:

Read(<WheretoStore>,<starting point>, <how many to read from starting point>)

correct?
Aug 3 '07 #1
Share this Question
Share on Google+
4 Replies


TRScheel
Expert 100+
P: 638
I am not sure, but I think the error is because when you read, you move the start location of the read. For example:

Expand|Select|Wrap|Line Numbers
  1. The dog jumped over the moon
If you wrote:

Expand|Select|Wrap|Line Numbers
  1. Read(myBuffer, 0, 3)
You would get 'The'

If you did it again, you would get ' do'. But if instead you did:

Expand|Select|Wrap|Line Numbers
  1. Read(myBuffer, 2, 3)
you would get 'og '

This is all purely speculation, a short test should prove this wrong/right.
Aug 3 '07 #2

P: 89
Hey!
thank you very much for your reply.

you were right about the Read's parameter.

I just didnt know that it had an internal pointer that keeps track of the last byte processed ;)

thanks again!

later,
Thiago.
Aug 12 '07 #3

TRScheel
Expert 100+
P: 638
No problem, glad to have helped.
Aug 13 '07 #4

Plater
Expert 5K+
P: 7,872
For Read the paremters are:
[number of bytes actually read] Read( [name of byte[] you want the data to go into], [starting index in the GIVEN byte[] (the previous parameter)], [total number of bytes to attempt to read] );

so if I wanted to try to read 1024 bytes into a buffer starting at the first index in my array I woul do this:
Expand|Select|Wrap|Line Numbers
  1. Byte[] buff=new Byte[1024];
  2. int actualread=-1;
  3. actaulread=mystream.Read(buff,0,buff.Length);
  4. //actualread will be set to how many bytes I actually was able to read, or 0 if end of stream
  5.  
Aug 13 '07 #5

Post your reply

Sign in to post your reply or Sign up for a free account.