473,287 Members | 1,492 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

reading strings from binary files - performance issue

I am working on a code library which needs to read in the data from
large binary files. The files hold int, double and string data. This
is the code for reading in the strings:

protected internal override string ReadString()
{
stringLength = fileStream.ReadByte();
moInput.Read(byteArrayBuffer, 0, stringLength);
return asciiEncoding.GetString(byteArrayBuffer, 0, stringLength );
}

At the moment the code that reads in the binary file data is
unacceptably slow (it takes several minutes), and most of the time
(according to my profiler) is taken up by the thousands of calls to
this ReadString function. Within this function, it is the third line -
the call to asciiEncoding.GetString() - that is taking up all the time.
Is there a way I can rewrite this line, or the whole function, to
speed the process up?

I have total control over the format of the binary files, so I can
change the way the string length and/or the string data itself is
encoded in the binary file. Since this is for a code library, I would
like the data to be saved and read out in a way that will work
correctly on machines with different language and format settings, etc.
- it would also be an advantage to support unicode, though I found the
unicode Encoding object's GetString method to be even slower.
Thanks for any help in advance,

Richard

Nov 16 '05 #1
5 6412
Sorry, the code should be:
protected internal override string ReadString()
{
stringLength = fileStream.ReadByte();
fileStream.Read(byteArrayBuffer, 0, stringLength);
return asciiEncoding.GetString(byteArrayBuffer, 0, stringLength );
}


That'll teach me to rename my variables to try and explain their
purpose better!

int stringLength
System.IO.Stream fileStream
System.Text.Encoding asciiEncoding
byte[] byteArrayBuffer
are all class level private variables.

Cheers,

Richard

Nov 16 '05 #2
<rn********@hotmail.com> wrote:
I am working on a code library which needs to read in the data from
large binary files. The files hold int, double and string data. This
is the code for reading in the strings:

protected internal override string ReadString()
{
stringLength = fileStream.ReadByte();
moInput.Read(byteArrayBuffer, 0, stringLength);
return asciiEncoding.GetString(byteArrayBuffer, 0, stringLength );
}

At the moment the code that reads in the binary file data is
unacceptably slow (it takes several minutes), and most of the time
(according to my profiler) is taken up by the thousands of calls to
this ReadString function. Within this function, it is the third line -
the call to asciiEncoding.GetString() - that is taking up all the time.
Is there a way I can rewrite this line, or the whole function, to
speed the process up?

I have total control over the format of the binary files, so I can
change the way the string length and/or the string data itself is
encoded in the binary file. Since this is for a code library, I would
like the data to be saved and read out in a way that will work
correctly on machines with different language and format settings, etc.
- it would also be an advantage to support unicode, though I found the
unicode Encoding object's GetString method to be even slower.
Thanks for any help in advance,


Just how large are these files? ASCIIEncoding.GetString should be very
fast indeed...

Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.

Note that you should really use the return result of Stream.Read to
check how much data has *actually* been read.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 16 '05 #3
Thanks for your reply.
Just how large are these files? ASCIIEncoding.GetString should be very fast indeed...
The largest of the files I am working with is around 31Mb - the average
is more like 8Mb.
Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.
I read your page and wrote a short program. I'll post again in a while
- I am still investigating. Next time, I will do what you suggest and
write the short test program *before* posting.
Note that you should really use the return result of Stream.Read to
check how much data has *actually* been read.


Yes, I know. I took out all the error handling to try and speed the
thing up, based on the fact that since I am writing the files and know
whether the write succeeded, the read should work correctly as it is
the mirror of the write.

Thanks,

Richard

Nov 16 '05 #4
<rn********@hotmail.com> wrote:
Just how large are these files? ASCIIEncoding.GetString should be
very fast indeed...


The largest of the files I am working with is around 31Mb - the average
is more like 8Mb.


That shouldn't take long, certainly. I quick test converting 31 bytes a
million times takes a fraction of a second on my laptop.
Could you post a short but complete program which demonstrates the
problem?

See http://www.pobox.com/~skeet/csharp/complete.html for details of
what I mean by that.


I read your page and wrote a short program. I'll post again in a while
- I am still investigating. Next time, I will do what you suggest and
write the short test program *before* posting.


Goodo :)
Note that you should really use the return result of Stream.Read to
check how much data has *actually* been read.


Yes, I know. I took out all the error handling to try and speed the
thing up, based on the fact that since I am writing the files and know
whether the write succeeded, the read should work correctly as it is
the mirror of the write.


Not necessarily - there's nothing which guarantees that a Stream.Read
will return all the requested bytes, even if those bytes are actually
present in the stream. While FileStream.Read *probably* always does
this, it might not if it's (say) reading over a network.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 16 '05 #5
> > I read your page and wrote a short program. I'll post again in a
while
- I am still investigating. Next time, I will do what you suggest and write the short test program *before* posting.
Goodo :)


My current best guess is that when I read in the data from the binary
file, the working variables that the data is loaded into is taking so
much memory that some behind-the-scenes memory management is going on
that isn't related to any particular line of code, and the profiler is
adding that time to the execution time of whatever line the application
is on at the time. So now I am on the trail of the memory-gobbling
objects...
Not necessarily - there's nothing which guarantees that a Stream.Read will return all the requested bytes, even if those bytes are actually present in the stream. While FileStream.Read *probably* always does
this, it might not if it's (say) reading over a network.

Thanks, I will bear that in mind.

Richard

Nov 16 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Fredrik Normann | last post by:
Hello, I'm trying to read the binary files under /var/spool/rwho/ so I'm wondering if anyone has done that before or could give me some clues on how to read those files. I've tried to use the...
4
by: nightflyer | last post by:
Hi all, [code snippet appended at the end.) my question: A class has a few string variables with not know length at design time. Now I declare lets say a 1000 of those classes and put them...
24
by: Hendrik Schober | last post by:
Hi, I have a 'std::istream' and need to read its whole contents into a string. How can I do this? TIA; Schobi
8
by: Yeow | last post by:
hello, i was trying to use the fread function on SunOS and ran into some trouble. i made a simple test as follows: i'm trying to read in a binary file (generated from a fortran code) that...
5
by: Robert Manea | last post by:
Hello everyone, I wrote, simply as an exercise, a small piece of code to find 'strings' (defined as an amount of at least 3 ASCII characters followed by a non ASCII character) in binary files. ...
7
by: John Dann | last post by:
I'm trying to read some binary data from a file created by another program. I know the binary file format but can't change or control the format. The binary data is organised such that it should...
6
by: arne.muller | last post by:
Hello, I've come across some problems reading strucutres from binary files. Basically I've some strutures typedef struct { int i; double x; int n; double *mz;
6
by: efrenba | last post by:
Hi, I came from delphi world and now I'm doing my first steps in C++. I'm using C++builder because its ide is like delphi although I'm trying to avoid the vcl. I need to insert new features...
6
by: jcasique.torres | last post by:
Hi everyboy. I trying to create a C promang in an AIX System to read JPG files but when it read just the first 4 bytes when it found a DLE character (^P) doesn't read anymore. I using fread...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.