file I/O

[shiva]

Inorder to handle large files over 100 MB, is there a way in dotnet to open
only certain bytes of the file by buffer ?

It is shaky if the entire file is opened and splitted. Instead is there a
way or any alternatives on opening small chunk of the file.

[Peter Rilling]

I don't think there is any different between opening a 1M file or 100M file
if you use a stream. Streams are designed with buffers and will only read
in as much information as you want. So, you could open your file using
FileStream and seek to a specified location, then read as many bytes as you
want.

Now, if you need more complex processing like getting the 200th row of some
file that are delimited by newlines, then that is something that you might
have to process yourself or rely on something like a Stream reader.
"shiva" <sh***@discussions.microsoft.com> wrote in message
news:C7**********************************@microsof t.com...

Inorder to handle large files over 100 MB, is there a way in dotnet to
open
only certain bytes of the file by buffer ?

It is shaky if the entire file is opened and splitted. Instead is there a
way or any alternatives on opening small chunk of the file.

[shiva]

Thank you for the feedback Sir. Well, i am just trying to print a 30MB file
and its shaky. It gets hogged sometimes amd prints sometimes as if it works
when we have luck. Unless or otherwise something is wrong with the following
code. There is lot more transactions to do than just printing, but it is
skaky right here when printing.

Private Sub Button1_Click(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Button1.Click
'Open a file for reading
Dim FILENAME As String
Dim FileContents As String
Dim iCount As Integer
iCount = 1

FILENAME = "c:\\Filereader\\File1.txt"

Try

'Get a StreamReader class that can be used to read the file
Dim objStreamReader As StreamReader
objStreamReader = File.OpenText(FILENAME)

'Read Line By Line
While True
FileContents = CType(objStreamReader.ReadLine, String)
If Not FileContents = Nothing Then
Response.Write(iCount & "Begin Line1" & "<br>")
Response.Write(FileContents)
Response.Write("<br>" & iCount & "End Line1" & "<br>")
iCount = iCount + 1
'lblRawOutput.Text &= FileContents & vbCrLf
Else : Exit While
End If
End While

'Set the text of the file to a Web control
'lblRawOutput.Text = contents

objStreamReader.Close()

Catch ex As Exception
Response.Write(ex.Message)
End Try

End Sub

"Peter Rilling" wrote:

I don't think there is any different between opening a 1M file or 100M file
if you use a stream. Streams are designed with buffers and will only read
in as much information as you want. So, you could open your file using
FileStream and seek to a specified location, then read as many bytes as you
want.

Now, if you need more complex processing like getting the 200th row of some
file that are delimited by newlines, then that is something that you might
have to process yourself or rely on something like a Stream reader.
"shiva" <sh***@discussions.microsoft.com> wrote in message
news:C7**********************************@microsof t.com...

Inorder to handle large files over 100 MB, is there a way in dotnet to
open
only certain bytes of the file by buffer ?

It is shaky if the entire file is opened and splitted. Instead is there a
way or any alternatives on opening small chunk of the file.

[shiva]

Thank you for the feedback Sir. Well, i am just trying to print a 30MB file
and its shaky. It gets hogged sometimes amd prints sometimes as if it works
when we have luck. Unless or otherwise something is wrong with the following
code. There is lot more transactions to do than just printing, but it is
skaky right here when printing.

Private Sub Button1_Click(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Button1.Click
'Open a file for reading
Dim FILENAME As String
Dim FileContents As String
Dim iCount As Integer
iCount = 1
 
FILENAME = "c:\\Filereader\\File1.txt"
 
Try
 
'Get a StreamReader class that can be used to read the file
Dim objStreamReader As StreamReader
objStreamReader = File.OpenText(FILENAME)
 
'Read Line By Line
While True
FileContents = CType(objStreamReader.ReadLine, String)
If Not FileContents = Nothing Then
Response.Write(iCount & "Begin Line1" & "<br>")
Response.Write(FileContents)
Response.Write("<br>" & iCount & "End Line1" & "<br>")
iCount = iCount + 1
'lblRawOutput.Text &= FileContents & vbCrLf
Else : Exit While
End If
End While
 
'Set the text of the file to a Web control
'lblRawOutput.Text = contents
 
objStreamReader.Close()
 
Catch ex As Exception
Response.Write(ex.Message)
End Try
 
End Sub
 
 

"Peter Rilling" wrote:

I don't think there is any different between opening a 1M file or 100M file
if you use a stream. Streams are designed with buffers and will only read
in as much information as you want. So, you could open your file using
FileStream and seek to a specified location, then read as many bytes as you
want.

Now, if you need more complex processing like getting the 200th row of some
file that are delimited by newlines, then that is something that you might
have to process yourself or rely on something like a Stream reader.
"shiva" <sh***@discussions.microsoft.com> wrote in message
news:C7**********************************@microsof t.com...

Inorder to handle large files over 100 MB, is there a way in dotnet to
open
only certain bytes of the file by buffer ?

It is shaky if the entire file is opened and splitted. Instead is there a
way or any alternatives on opening small chunk of the file.

[shiva]

Thank you for the feedback Sir. Well, i am just trying to print a 30MB file
and its shaky. It gets hogged sometimes amd prints sometimes as if it works
when we have luck. Unless or otherwise something is wrong with the following
code. There is lot more transactions to do than just printing, but it is
skaky right here when printing.

Private Sub Button1_Click(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Button1.Click
'Open a file for reading
Dim FILENAME As String
Dim FileContents As String
Dim iCount As Integer
iCount = 1
 
FILENAME = "c:\\Filereader\\File1.txt"
 
Try
 
'Get a StreamReader class that can be used to read the file
Dim objStreamReader As StreamReader
objStreamReader = File.OpenText(FILENAME)
 
'Read Line By Line
While True
FileContents = CType(objStreamReader.ReadLine, String)
If Not FileContents = Nothing Then
Response.Write(iCount & "Begin Line1" & "<br>")
Response.Write(FileContents)
Response.Write("<br>" & iCount & "End Line1" & "<br>")
iCount = iCount + 1
'lblRawOutput.Text &= FileContents & vbCrLf
Else : Exit While
End If
End While
 
'Set the text of the file to a Web control
'lblRawOutput.Text = contents
 
objStreamReader.Close()
 
Catch ex As Exception
Response.Write(ex.Message)
End Try
 
End Sub
 
 

"Peter Rilling" wrote:

I don't think there is any different between opening a 1M file or 100M file
if you use a stream. Streams are designed with buffers and will only read
in as much information as you want. So, you could open your file using
FileStream and seek to a specified location, then read as many bytes as you
want.

Now, if you need more complex processing like getting the 200th row of some
file that are delimited by newlines, then that is something that you might
have to process yourself or rely on something like a Stream reader.
"shiva" <sh***@discussions.microsoft.com> wrote in message
news:C7**********************************@microsof t.com...

Inorder to handle large files over 100 MB, is there a way in dotnet to
open
only certain bytes of the file by buffer ?

It is shaky if the entire file is opened and splitted. Instead is there a
way or any alternatives on opening small chunk of the file.

[Cor Ligthert]

Shiva,

Are you sure you want it byte by byte, used are often Unicodecharters which
used two bytes.

Than the encoding class will probably one of the first things too look at.

http://msdn.microsoft.com/library/de...classtopic.asp

I hope this helps?

Copr
"shiva" <sh***@discussions.microsoft.com>

Inorder to handle large files over 100 MB, is there a way in dotnet to open only certain bytes of the file by buffer ?

It is shaky if the entire file is opened and splitted. Instead is there a
way or any alternatives on opening small chunk of the file.

[Cor Ligthert]

Shiva,

Tell us first what type of file it is, text, a kind of own database, are
there char in it etc. or maybe it are pictures. Now it is just shooting in
the dark.
Cor

[Andrew Faust]

"shiva" <sh***@discussions.microsoft.com> wrote in message
news:B9**********************************@microsof t.com...

Thank you for the feedback Sir. Well, i am just trying to print
a 30MB file
and its shaky. It gets hogged sometimes amd prints sometimes as
if it works
when we have luck. Unless or otherwise something is wrong with
the following
code. There is lot more transactions to do than just printing,
but it is
skaky right here when printing.

Private Sub Button1_Click(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Button1.Click
'Open a file for reading
Dim FILENAME As String
Dim FileContents As String
Dim iCount As Integer
iCount = 1

FILENAME = "c:\\Filereader\\File1.txt"

Try

'Get a StreamReader class that can be used to read
the file
Dim objStreamReader As StreamReader
objStreamReader = File.OpenText(FILENAME)

'Read Line By Line
While True
FileContents = CType(objStreamReader.ReadLine,
String)
ReadLine returns a string already, so there is no reason to
re-caste it.
If Not FileContents = Nothing Then
Response.Write(iCount & "Begin Line1" &
"<br>")
Response.Write(FileContents)
Response.Write("<br>" & iCount & "End Line1"
& "<br>")
iCount = iCount + 1
'lblRawOutput.Text &= FileContents & vbCrLf
Else : Exit While
End If
End While

It doesn't look like your speed issues are due to the way you are
working with the files, but rather with what you are doing with
all the data.

What is this Response object you are working with? I'm assuming
it's something like a web response. If this is the case then it
can take quite a while to send the 30 MB of data via the network.
Even on a local network it can take a while depending on network
configuration and current traffic loads.

Also you have
'lblRawOutput.Text &= FileContents & vbCrLf
commented out. However, if your tests were done with this
uncommented it could have a drastic performance hit for large
files, as the larger a string gets, the longer it takes to
concatenate more data on to the end of it.

Andrew Faust

[Andrew Faust]

"shiva" <sh***@discussions.microsoft.com> wrote in message
news:53**********************************@microsof t.com...

Dear Cor,

For example if the file is a 30MB file, i am looking for
opening for 1MB
at a time.

You can't open only 1 MB at a time. You either have a file open
or you don't. Size is irrelevant. However, you can read in to
memory only a small portion at a time.

Andrew Faust

[Cor Ligthert]

Shiva,

What is than the reason you do than not proces it line by line?

http://msdn.microsoft.com/library/de...xtFromFile.asp

Cor

"shiva" <sh***@discussions.microsoft.com> schreef in bericht
news:9D**********************************@microsof t.com...

Hi Cor,

Its a regular text file, but the formats are pretty complex like a EDI file. There are no pictures or any kind of picture data in the file. Its a
regular file

Header1<Information>endofline
Detail1<Information>endofline
Detail2<Information>endofline
Header2<Information>endofline
Detail1<Information>endofline
Detail2<Information>endofline
Detail3<Information>endofline
Header3<Information>endofline
Detail1<Information>endofline
Header4<Information>endofline
Detail1<Information>endofline
Detail2<Information>endofline
Detail3<Information>endofline

I understand if i read only a 1MB out of the 30MB i may not get an entire
record, but i can come up with some other algorithm for that.

But the main question is, is there a way i can open 1MB out of the 30MB.

Thank you
Shiva
"Cor Ligthert" wrote:

Shiva,

Tell us first what type of file it is, text, a kind of own database, are
there char in it etc. or maybe it are pictures. Now it is just shooting in the dark.
Cor

[Cor Ligthert]

Shiva,

When you process it line by line why it can than not be in a kind of pseudo
Private rowcounter = 0

Process 10000rows(not endoffile)

Procedure Process10000rows
do while not endoffile
For rowcounter to 10000
read line into whatever
next

When this can fit and you cannot make this in real code tell than?

Cor

"shiva" <sh***@discussions.microsoft.com>

Dear Cor,

The reason i don't want to do line by line was it runs out of memory.

Dear Andrew,

I removed the Ctype and the Response.write but the difference in
performance is very less.

Thank you
Siva
"Cor Ligthert" wrote:

Shiva,

What is than the reason you do than not proces it line by line?

http://msdn.microsoft.com/library/de...xtFromFile.asp
Cor

"shiva" <sh***@discussions.microsoft.com> schreef in bericht
news:9D**********************************@microsof t.com...

Hi Cor,

Its a regular text file, but the formats are pretty complex like a EDI

file.

There are no pictures or any kind of picture data in the file. Its a
regular file

Header1<Information>endofline
Detail1<Information>endofline
Detail2<Information>endofline
Header2<Information>endofline
Detail1<Information>endofline
Detail2<Information>endofline
Detail3<Information>endofline
Header3<Information>endofline
Detail1<Information>endofline
Header4<Information>endofline
Detail1<Information>endofline
Detail2<Information>endofline
Detail3<Information>endofline

I understand if i read only a 1MB out of the 30MB i may not get an

entire record, but i can come up with some other algorithm for that.

But the main question is, is there a way i can open 1MB out of the 30MB.
Thank you
Shiva
"Cor Ligthert" wrote:

> Shiva,
>
> Tell us first what type of file it is, text, a kind of own database, are > there char in it etc. or maybe it are pictures. Now it is just

shooting in

> the dark.
>
>
> Cor
>
>
>