Hi,
Is it possible to read a file in reverse and only get the last 100 bytes in
the file without reading the whole file from the begining? I have to get info
from files that are in the last 100 bytes of the file and some of these files
are 600Mb -1 GB in size. I am getting "outofMemory.." exceptions on the
largest files and the other files take "forever" to get the last 100 bytes.
This is the code I have currently that works with smaller files:
Dim fileData() As Byte
FilePath = "C:\Temp\1DD04336CA1A45CB81A89167048D2594.abk"
Try
Dim oFile As System.IO.FileInfo
oFile = New System.IO.FileInfo(FilePath)
Dim oFileStream As System.IO.FileStream = oFile.Open
_(IO.FileMode.Open, IO.FileAccess.Read)
Dim lBytes As Long = oFileStream.Length
ReDim fileData(lBytes)
Dim StartReadAt As Long = lBytes - 100
Dim StopReadAt As Long = 100
oFileStream.Read(fileData, StartReadAt, StopReadAt)
oFileStream.Close()
Catch ex As Exception
MsgBox(ex.Message)
End Try
The problem with this code is an Array is created in the size of the file
resulting in upper bounds of maybe 200 000 000, so this is time consuming to
get the last 100 bytes.
Any idéas?
Thanks /Thomas
6  5764 
Try seeking to oFileStream.Length - 100. Then read from there to the end of
the file. The array size only needs to be 100 bytes. It may take the OS a
while to do the seek, but you won't get the out of memory errors.
Mike Ober.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:4C**********************************@microsof t.com...
Hi,
Is it possible to read a file in reverse and only get the last 100 bytes
in
the file without reading the whole file from the begining? I have to get
info
from files that are in the last 100 bytes of the file and some of these
files
are 600Mb -1 GB in size. I am getting "outofMemory.." exceptions on the
largest files and the other files take "forever" to get the last 100
bytes.
This is the code I have currently that works with smaller files:
Dim fileData() As Byte
FilePath = "C:\Temp\1DD04336CA1A45CB81A89167048D2594.abk"
Try
Dim oFile As System.IO.FileInfo
oFile = New System.IO.FileInfo(FilePath)
Dim oFileStream As System.IO.FileStream = oFile.Open
_(IO.FileMode.Open, IO.FileAccess.Read)
Dim lBytes As Long = oFileStream.Length
ReDim fileData(lBytes)
Dim StartReadAt As Long = lBytes - 100
Dim StopReadAt As Long = 100
oFileStream.Read(fileData, StartReadAt, StopReadAt)
oFileStream.Close()
Catch ex As Exception
MsgBox(ex.Message)
End Try
The problem with this code is an Array is created in the size of the file
resulting in upper bounds of maybe 200 000 000, so this is time consuming
to
get the last 100 bytes.
Any idéas?
Thanks /Thomas
Ahh, thanks for the "Eye opener"
I used the binaryreader instead and set the "cursor" to the End -100
position and step to the end.
This code worked for me:
Dim fileData(100) As Byte
Dim oFileStream As New FileStream(FilePath, FileMode.Open,
FileAccess.Read)
Dim oBinaryReader As New BinaryReader(oFileStream)
Dim lBytes As Long = oFileStream.Length
oBinaryReader.BaseStream.Position = lBytes - 100
Dim fdi As Integer = 0
For i As Long = 0 To 99
fileData(i) = oBinaryReader.ReadByte()
Next i
oBinaryReader.Close()
oFileStream.Close()
Thanx
/Thomas
"Michael D. Ober" wrote:
Try seeking to oFileStream.Length - 100. Then read from there to the end of
the file. The array size only needs to be 100 bytes. It may take the OS a
while to do the seek, but you won't get the out of memory errors.
Mike Ober.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:4C**********************************@microsof t.com...
Hi,
Is it possible to read a file in reverse and only get the last 100 bytes
in
the file without reading the whole file from the begining? I have to get
info
from files that are in the last 100 bytes of the file and some of these
files
are 600Mb -1 GB in size. I am getting "outofMemory.." exceptions on the
largest files and the other files take "forever" to get the last 100
bytes.
This is the code I have currently that works with smaller files:
Dim fileData() As Byte
FilePath = "C:\Temp\1DD04336CA1A45CB81A89167048D2594.abk"
Try
Dim oFile As System.IO.FileInfo
oFile = New System.IO.FileInfo(FilePath)
Dim oFileStream As System.IO.FileStream = oFile.Open
_(IO.FileMode.Open, IO.FileAccess.Read)
Dim lBytes As Long = oFileStream.Length
ReDim fileData(lBytes)
Dim StartReadAt As Long = lBytes - 100
Dim StopReadAt As Long = 100
oFileStream.Read(fileData, StartReadAt, StopReadAt)
oFileStream.Close()
Catch ex As Exception
MsgBox(ex.Message)
End Try
The problem with this code is an Array is created in the size of the file
resulting in upper bounds of maybe 200 000 000, so this is time consuming
to
get the last 100 bytes.
Any idéas?
Thanks /Thomas
Out of curiosity, is there a performance difference when seeking to end -100
on a 1K file vs a 10Gb file. I know the old FAT file system would take a
lot longer for the seek on the larger file. Also, why not use ReadBytes and
avoid your read loop?
Dim oBinaryReader As New BinaryReader(New FileStream(FilePath,
FileMode.Open, FileAccess.Read))
oBinaryReader.BaseStream.Position = oBinaryReader.BaseStream.Length - 100
Dim fileData(100) As Byte = oBinaryReader.ReadBytes(100)
oBinaryReader.Close() ' Close both the binary reader and the base
stream
Mike.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:34**********************************@microsof t.com...
Ahh, thanks for the "Eye opener"
I used the binaryreader instead and set the "cursor" to the End -100
position and step to the end.
This code worked for me:
Dim fileData(100) As Byte
Dim oFileStream As New FileStream(FilePath, FileMode.Open,
FileAccess.Read)
Dim oBinaryReader As New BinaryReader(oFileStream)
Dim lBytes As Long = oFileStream.Length
oBinaryReader.BaseStream.Position = lBytes - 100
Dim fdi As Integer = 0
For i As Long = 0 To 99
fileData(i) = oBinaryReader.ReadByte()
Next i
oBinaryReader.Close()
oFileStream.Close()
Thanx
/Thomas
"Michael D. Ober" wrote:
>Try seeking to oFileStream.Length - 100. Then read from there to the end
of
the file. The array size only needs to be 100 bytes. It may take the OS
a
while to do the seek, but you won't get the out of memory errors.
Mike Ober.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:4C**********************************@microso ft.com...
Hi,
Is it possible to read a file in reverse and only get the last 100
bytes
in
the file without reading the whole file from the begining? I have to
get
info
from files that are in the last 100 bytes of the file and some of these
files
are 600Mb -1 GB in size. I am getting "outofMemory.." exceptions on the
largest files and the other files take "forever" to get the last 100
bytes.
This is the code I have currently that works with smaller files:
Dim fileData() As Byte
FilePath = "C:\Temp\1DD04336CA1A45CB81A89167048D2594.abk"
Try
Dim oFile As System.IO.FileInfo
oFile = New System.IO.FileInfo(FilePath)
Dim oFileStream As System.IO.FileStream = oFile.Open
_(IO.FileMode.Open, IO.FileAccess.Read)
Dim lBytes As Long = oFileStream.Length
ReDim fileData(lBytes)
Dim StartReadAt As Long = lBytes - 100
Dim StopReadAt As Long = 100
oFileStream.Read(fileData, StartReadAt, StopReadAt)
oFileStream.Close()
Catch ex As Exception
MsgBox(ex.Message)
End Try
The problem with this code is an Array is created in the size of the
file
resulting in upper bounds of maybe 200 000 000, so this is time
consuming
to
get the last 100 bytes.
Any idéas?
Thanks /Thomas
Regardless of the file size it takes about 156 milliseconds to get the last
100 bytes.
Measured with a 100kb file and a 656MB file over the network. This is very
nice since I have to read hundreds of files on a regular basis.
I get this error when trying youre example with ReadBytes(100) "Explicit
initialization is not permitted for arrays declared with explicit bounds."
But the idéa is good and I'll look further into it.
Thanks again
/Thomas
"Michael D. Ober" wrote:
Out of curiosity, is there a performance difference when seeking to end -100
on a 1K file vs a 10Gb file. I know the old FAT file system would take a
lot longer for the seek on the larger file. Also, why not use ReadBytes and
avoid your read loop?
Dim oBinaryReader As New BinaryReader(New FileStream(FilePath,
FileMode.Open, FileAccess.Read))
oBinaryReader.BaseStream.Position = oBinaryReader.BaseStream.Length - 100
Dim fileData(100) As Byte = oBinaryReader.ReadBytes(100)
oBinaryReader.Close() ' Close both the binary reader and the base
stream
Mike.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:34**********************************@microsof t.com...
Ahh, thanks for the "Eye opener"
I used the binaryreader instead and set the "cursor" to the End -100
position and step to the end.
This code worked for me:
Dim fileData(100) As Byte
Dim oFileStream As New FileStream(FilePath, FileMode.Open,
FileAccess.Read)
Dim oBinaryReader As New BinaryReader(oFileStream)
Dim lBytes As Long = oFileStream.Length
oBinaryReader.BaseStream.Position = lBytes - 100
Dim fdi As Integer = 0
For i As Long = 0 To 99
fileData(i) = oBinaryReader.ReadByte()
Next i
oBinaryReader.Close()
oFileStream.Close()
Thanx
/Thomas
"Michael D. Ober" wrote:
Try seeking to oFileStream.Length - 100. Then read from there to the end
of
the file. The array size only needs to be 100 bytes. It may take the OS
a
while to do the seek, but you won't get the out of memory errors.
Mike Ober.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:4C**********************************@microsof t.com...
Hi,
Is it possible to read a file in reverse and only get the last 100
bytes
in
the file without reading the whole file from the begining? I have to
get
info
from files that are in the last 100 bytes of the file and some of these
files
are 600Mb -1 GB in size. I am getting "outofMemory.." exceptions on the
largest files and the other files take "forever" to get the last 100
bytes.
This is the code I have currently that works with smaller files:
Dim fileData() As Byte
FilePath = "C:\Temp\1DD04336CA1A45CB81A89167048D2594.abk"
Try
Dim oFile As System.IO.FileInfo
oFile = New System.IO.FileInfo(FilePath)
Dim oFileStream As System.IO.FileStream = oFile.Open
_(IO.FileMode.Open, IO.FileAccess.Read)
Dim lBytes As Long = oFileStream.Length
ReDim fileData(lBytes)
Dim StartReadAt As Long = lBytes - 100
Dim StopReadAt As Long = 100
oFileStream.Read(fileData, StartReadAt, StopReadAt)
oFileStream.Close()
Catch ex As Exception
MsgBox(ex.Message)
End Try
The problem with this code is an Array is created in the size of the
file
resulting in upper bounds of maybe 200 000 000, so this is time
consuming
to
get the last 100 bytes.
Any idéas?
Thanks /Thomas
Got it, Dim as Byte array:
Dim oFileStream As New FileStream(FilePath, FileMode.Open, FileAccess.Read)
Dim oBinaryReader As New BinaryReader(oFileStream)
Dim lBytes As Long = oFileStream.Length
oBinaryReader.BaseStream.Position = lBytes - 100
Dim fileData As Byte() = oBinaryReader.ReadBytes(100)
oBinaryReader.Close()
oFileStream.Close()
Did you mean that oBinaryReader.Close automatically closes oFileStream.Close?
"Michael D. Ober" wrote:
Out of curiosity, is there a performance difference when seeking to end -100
on a 1K file vs a 10Gb file. I know the old FAT file system would take a
lot longer for the seek on the larger file. Also, why not use ReadBytes and
avoid your read loop?
Dim oBinaryReader As New BinaryReader(New FileStream(FilePath,
FileMode.Open, FileAccess.Read))
oBinaryReader.BaseStream.Position = oBinaryReader.BaseStream.Length - 100
Dim fileData(100) As Byte = oBinaryReader.ReadBytes(100)
oBinaryReader.Close() ' Close both the binary reader and the base
stream
Mike.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:34**********************************@microsof t.com...
Ahh, thanks for the "Eye opener"
I used the binaryreader instead and set the "cursor" to the End -100
position and step to the end.
This code worked for me:
Dim fileData(100) As Byte
Dim oFileStream As New FileStream(FilePath, FileMode.Open,
FileAccess.Read)
Dim oBinaryReader As New BinaryReader(oFileStream)
Dim lBytes As Long = oFileStream.Length
oBinaryReader.BaseStream.Position = lBytes - 100
Dim fdi As Integer = 0
For i As Long = 0 To 99
fileData(i) = oBinaryReader.ReadByte()
Next i
oBinaryReader.Close()
oFileStream.Close()
Thanx
/Thomas
"Michael D. Ober" wrote:
Try seeking to oFileStream.Length - 100. Then read from there to the end
of
the file. The array size only needs to be 100 bytes. It may take the OS
a
while to do the seek, but you won't get the out of memory errors.
Mike Ober.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:4C**********************************@microsof t.com...
Hi,
Is it possible to read a file in reverse and only get the last 100
bytes
in
the file without reading the whole file from the begining? I have to
get
info
from files that are in the last 100 bytes of the file and some of these
files
are 600Mb -1 GB in size. I am getting "outofMemory.." exceptions on the
largest files and the other files take "forever" to get the last 100
bytes.
This is the code I have currently that works with smaller files:
Dim fileData() As Byte
FilePath = "C:\Temp\1DD04336CA1A45CB81A89167048D2594.abk"
Try
Dim oFile As System.IO.FileInfo
oFile = New System.IO.FileInfo(FilePath)
Dim oFileStream As System.IO.FileStream = oFile.Open
_(IO.FileMode.Open, IO.FileAccess.Read)
Dim lBytes As Long = oFileStream.Length
ReDim fileData(lBytes)
Dim StartReadAt As Long = lBytes - 100
Dim StopReadAt As Long = 100
oFileStream.Read(fileData, StartReadAt, StopReadAt)
oFileStream.Close()
Catch ex As Exception
MsgBox(ex.Message)
End Try
The problem with this code is an Array is created in the size of the
file
resulting in upper bounds of maybe 200 000 000, so this is time
consuming
to
get the last 100 bytes.
Any idéas?
Thanks /Thomas
According to the documentation, BinaryReader.Close() closes both the binary
reader and the underlying stream. That's why you can use a single Binary
Reader and access it's BaseStream property to get the file length and set
your desired position.
Mike.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:1A**********************************@microsof t.com...
Got it, Dim as Byte array:
Dim oFileStream As New FileStream(FilePath, FileMode.Open,
FileAccess.Read)
Dim oBinaryReader As New BinaryReader(oFileStream)
Dim lBytes As Long = oFileStream.Length
oBinaryReader.BaseStream.Position = lBytes - 100
Dim fileData As Byte() = oBinaryReader.ReadBytes(100)
oBinaryReader.Close()
oFileStream.Close()
Did you mean that oBinaryReader.Close automatically closes
oFileStream.Close?
"Michael D. Ober" wrote:
>Out of curiosity, is there a performance difference when seeking to
end -100
on a 1K file vs a 10Gb file. I know the old FAT file system would take a
lot longer for the seek on the larger file. Also, why not use ReadBytes
and
avoid your read loop?
Dim oBinaryReader As New BinaryReader(New FileStream(FilePath,
FileMode.Open, FileAccess.Read))
oBinaryReader.BaseStream.Position = oBinaryReader.BaseStream.Length - 100
Dim fileData(100) As Byte = oBinaryReader.ReadBytes(100)
oBinaryReader.Close() ' Close both the binary reader and the
base
stream
Mike.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:34**********************************@microso ft.com...
Ahh, thanks for the "Eye opener"
I used the binaryreader instead and set the "cursor" to the End -100
position and step to the end.
This code worked for me:
Dim fileData(100) As Byte
Dim oFileStream As New FileStream(FilePath, FileMode.Open,
FileAccess.Read)
Dim oBinaryReader As New BinaryReader(oFileStream)
Dim lBytes As Long = oFileStream.Length
oBinaryReader.BaseStream.Position = lBytes - 100
Dim fdi As Integer = 0
For i As Long = 0 To 99
fileData(i) = oBinaryReader.ReadByte()
Next i
oBinaryReader.Close()
oFileStream.Close()
Thanx
/Thomas
"Michael D. Ober" wrote:
Try seeking to oFileStream.Length - 100. Then read from there to the
end
of
the file. The array size only needs to be 100 bytes. It may take the
OS
a
while to do the seek, but you won't get the out of memory errors.
Mike Ober.
"ThomasZ" <Th*****@discussions.microsoft.comwrote in message
news:4C**********************************@microso ft.com...
Hi,
Is it possible to read a file in reverse and only get the last 100
bytes
in
the file without reading the whole file from the begining? I have to
get
info
from files that are in the last 100 bytes of the file and some of
these
files
are 600Mb -1 GB in size. I am getting "outofMemory.." exceptions on
the
largest files and the other files take "forever" to get the last 100
bytes.
This is the code I have currently that works with smaller files:
Dim fileData() As Byte
FilePath = "C:\Temp\1DD04336CA1A45CB81A89167048D2594.abk"
Try
Dim oFile As System.IO.FileInfo
oFile = New System.IO.FileInfo(FilePath)
Dim oFileStream As System.IO.FileStream = oFile.Open
_(IO.FileMode.Open, IO.FileAccess.Read)
Dim lBytes As Long = oFileStream.Length
ReDim fileData(lBytes)
Dim StartReadAt As Long = lBytes - 100
Dim StopReadAt As Long = 100
oFileStream.Read(fileData, StartReadAt, StopReadAt)
oFileStream.Close()
Catch ex As Exception
MsgBox(ex.Message)
End Try
The problem with this code is an Array is created in the size of the
file
resulting in upper bounds of maybe 200 000 000, so this is time
consuming
to
get the last 100 bytes.
Any idéas?
Thanks /Thomas
This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Axel |
last post by:
Hiho,
I want to read the content of a binary file into a string. It works, but
only the first 4 chars are left after the reading and the stringsize is,
of course, 4 bytes.
I got this method...
|
by: Vladimir |
last post by:
Can anyone halp me with some questions?
Is there multiplatform standart in binary representation of primitive
datatypes when you writing it through BinaryWriter?
In other words if I write by...
|
by: baustin75 |
last post by:
Posted: Mon Oct 03, 2005 1:41 pm Post subject: cannot mail() in ie
only when debugging in php designer 2005
--------------------------------------------------------------------------------
...
|
by: RyanS09 |
last post by:
Hello-
I am trying to write a snippet which will open a text file with an
integer on each line. I would like to read the last integer in the
file. I am currently using:
file = fopen("f.txt",...
|
by: Vijay |
last post by:
Hi ,
I am doing a small project in c.
I have a Hexadecimal file and want to convert into ascii value.
(i.e., Hexadecimal to Ascii conversion from a file).
Could anyone help me?
Thanks in...
|
by: ShaneO |
last post by:
Hello,
I wish to extract embedded string data from a file using a Binary Read
method.
The following code sample is used in VB.NET and similar code is used in
VB6 -
(Assume variable...
|
by: Gary Wessle |
last post by:
Hi there
I have a method which returns time_t and another two methods return
double data types and I cann't change that since the library is
provided by Big Bucks Inc. I think time_t is long but...
|
by: Mastastealth |
last post by:
I'm trying to create a program to read a certain binary format. I have
the format's spec which goes something like:
First 6 bytes: String
Next 4 bytes: 3 digit number and a blank byte
---
Next...
|
by: tom thomas |
last post by:
Dear sir,
i am very new to c# and i request you to... please look into the problem
The following is the code(please see the code below:) which i uses to open the binary file,currently it works...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
| |
