parsing a file..

[broli]

I need to parse a file which has about 2000 lines and I'm getting
told that reading the file in ascii would be a slower way to do it and
so i need to resort to binary by reading it in large chunks. Can any
one please explain what is all
this about ?

[Richard Bos]

ri*****@cogsci. ed.ac.uk (Richard Tobin) wrote:

In article <fr**********@r egistered.motza rella.org>,
Richard <de***@gmail.co mwrote:

Reading in large chunks is unrelated to whether it's binary or
ascii.

I would question that statement. Reading in binary will be a LOT faster
,if its the same platform. for reading in the same NUMBER of
readings.

I didn't say whether it's in binary is unrelated to *speed*.

I meant: there are two separate issues; whether you read it in large
chunks, and whether it's binary. You can read each of text or binary
in small or large chunks. Each of these choices will separately affect
the speed.

Besides, he _has_ a text file. Yes, it's a lot larger than a binary file
would be, and therefore slower to read. But the fact that the _file_ is
text is not the OP's doing. Reading this file as text or as binary won't
make a large difference. _Writing_ it as a binary file would have; but
that's not something the OP can do.

Richard

[Bartc]

"Chris Dollin" <ch**********@h p.comwrote in message
news:fr******** **@news-pa1.hpl.hp.com. ..

Richard wrote:

>ri*****@cogsci. ed.ac.uk (Richard Tobin) writes:

>>In article
<4e********** *************** *********@s19g2 000prg.googlegr oups.com>,
broli <Br*****@gmail. comwrote:

I need to parse a file which has about 2000 lines and I'm getting
told that reading the file in ascii would be a slower way to do it and
so i need to resort to binary by reading it in large chunks. Can any
one please explain what is all this about ?

Reading in large chunks is unrelated to whether it's binary or
ascii.

I would question that statement. Reading in binary will be a LOT faster
,if its the same platform. for reading in the same NUMBER of
readings.

Quick test, one file, 2000 lines, each line with two floats (1.12345
and 7.890), about 28Kb total.

One single big-enough fread:

real 0m0.002s
user 0m0.000s
sys 0m0.001s

Repeat fscanf( ... "%lf %lf" ... ) until EOF:

real 0m0.004s
user 0m0.002s
sys 0m0.002s

Yes, in this test it's twice as slow. The data file is probably
cached (it's been read several other times already as I /cough/

My own tests:

(A) 100,000 lines of text, each with 3 doubles (2900000 bytes):

2.1 seconds to read a number at a time, using sscanf() (but I use a wrapper
or two with some extra overhead)

(B) The same data as 300,000 doubles written as binary (2400000 bytes):

0.8 seconds to read a number at a time, using fread() 8 bytes at a time

(C) Same binary data as (B)

0.004 seconds to read as a single block into memory (possibly straight into
the array or whatever datastructure is used). Using fread() on 2400000
bytes.

So about 200-500 times faster in binary mode, when done properly.

--
Bart

[Richard]

ri*****@cogsci. ed.ac.uk (Richard Tobin) writes:

In article <fr**********@r egistered.motza rella.org>,
Richard <de***@gmail.co mwrote:

>>Reading in large chunks is unrelated to whether it's binary or
ascii.

>>I would question that statement. Reading in binary will be a LOT faster
,if its the same platform. for reading in the same NUMBER of
readings.

I didn't say whether it's in binary is unrelated to *speed*.

I'm not sure that parses :-;

>
I meant: there are two separate issues; whether you read it in large
chunks, and whether it's binary. You can read each of text or binary
in small or large chunks. Each of these choices will separately affect
the speed.

Yes, I agree.

>
-- Richard

[Willem]

Bartc wrote:
) My own tests:
)
) (A) 100,000 lines of text, each with 3 doubles (2900000 bytes):
)
) 2.1 seconds to read a number at a time, using sscanf() (but I use a wrapper
) or two with some extra overhead)
)
) (B) The same data as 300,000 doubles written as binary (2400000 bytes):
)
) 0.8 seconds to read a number at a time, using fread() 8 bytes at a time
)
) (C) Same binary data as (B)
)
) 0.004 seconds to read as a single block into memory (possibly straight into
) the array or whatever datastructure is used). Using fread() on 2400000
) bytes.
)
) So about 200-500 times faster in binary mode, when done properly.

Have you tried reading the text file into memory as a single block
and then using sscanf() to parse it ?
SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

[Richard]

Chris Dollin <ch**********@h p.comwrites:

Richard wrote:

>ri*****@cogsci. ed.ac.uk (Richard Tobin) writes:

>>In article <4e************ *************** *******@s19g200 0prg.googlegrou ps.com>,
broli <Br*****@gmail. comwrote:

I need to parse a file which has about 2000 lines and I'm getting
told that reading the file in ascii would be a slower way to do it and
so i need to resort to binary by reading it in large chunks. Can any
one please explain what is all this about ?

Reading in large chunks is unrelated to whether it's binary or
ascii.

I would question that statement. Reading in binary will be a LOT faster
,if its the same platform. for reading in the same NUMBER of
readings.

>> Perhaps they meant that character-at-a-time reading with
getchar() is slow, which it is on some systems. You can perfectly
well use fread() on text files.

The text file will be larger. There is a need to parse the ascii text
into the destination formats.

It will be slower in the great majority of cases.

Quick test, one file, 2000 lines, each line with two floats (1.12345
and 7.890), about 28Kb total.

One single big-enough fread:

real 0m0.002s
user 0m0.000s
sys 0m0.001s

Repeat fscanf( ... "%lf %lf" ... ) until EOF:

real 0m0.004s
user 0m0.002s
sys 0m0.002s

Yes, in this test it's twice as slow. The data file is probably
cached (it's been read several other times already as I /cough/
debugged my code). It includes program start-up time (I just did
`time ./a.out` to get the numbers) so the actual reading time will
be less.

Myself I wouldn't count that as "LOTS faster" for binary data,
but doubtless there are applications where it is so counted;
I don't think the OPs case is one of them, and it does look as
though he's reading a text file anyway.

Then why not take the static noise out? Make the file a lot bigger and
report back.

But even these results do indicate quite a large % difference .....

And we do not know how often this data sample is written or read. I
could be thousands of times an hour leading to considerable unnecessary
overhead if using ascii over binary.

[Richard]

"Bartc" <[e-mail] rites:

"Chris Dollin" <ch**********@h p.comwrote in message
news:fr******** **@news-pa1.hpl.hp.com. ..

>Richard wrote:

>>ri*****@cogsci. ed.ac.uk (Richard Tobin) writes:

In article
<4e********* *************** **********@s19g 2000prg.googleg roups.com>,
broli <Br*****@gmail. comwrote:

>I need to parse a file which has about 2000 lines and I'm getting
>told that reading the file in ascii would be a slower way to do it and
>so i need to resort to binary by reading it in large chunks. Can any
>one please explain what is all this about ?

Reading in large chunks is unrelated to whether it's binary or
ascii.

I would question that statement. Reading in binary will be a LOT faster
,if its the same platform. for reading in the same NUMBER of
readings.

>Quick test, one file, 2000 lines, each line with two floats (1.12345
and 7.890), about 28Kb total.

One single big-enough fread:

real 0m0.002s
user 0m0.000s
sys 0m0.001s

Repeat fscanf( ... "%lf %lf" ... ) until EOF:

real 0m0.004s
user 0m0.002s
sys 0m0.002s

Yes, in this test it's twice as slow. The data file is probably
cached (it's been read several other times already as I /cough/

My own tests:

(A) 100,000 lines of text, each with 3 doubles (2900000 bytes):

2.1 seconds to read a number at a time, using sscanf() (but I use a wrapper
or two with some extra overhead)

(B) The same data as 300,000 doubles written as binary (2400000 bytes):

0.8 seconds to read a number at a time, using fread() 8 bytes at a time

(C) Same binary data as (B)

0.004 seconds to read as a single block into memory (possibly straight into
the array or whatever datastructure is used). Using fread() on 2400000
bytes.

So about 200-500 times faster in binary mode, when done properly.

I'm surprised this is even being contested.

[Bartc]

"Willem" <wi****@stack.n lwrote in message
news:sl******** ***********@sna il.stack.nl...

Bartc wrote:
) My own tests:
)
) (A) 100,000 lines of text, each with 3 doubles (2900000 bytes):
)
) 2.1 seconds to read a number at a time, using sscanf() (but I use a
wrapper
) or two with some extra overhead)
)
) (B) The same data as 300,000 doubles written as binary (2400000 bytes):
)
) 0.8 seconds to read a number at a time, using fread() 8 bytes at a time
)
) (C) Same binary data as (B)
)
) 0.004 seconds to read as a single block into memory (possibly straight
into
) the array or whatever datastructure is used). Using fread() on 2400000
) bytes.
)
) So about 200-500 times faster in binary mode, when done properly.

Have you tried reading the text file into memory as a single block
and then using sscanf() to parse it ?

No. I would imagine it would add a second or so to the time.

However, I left out the word 'apparently' when quoting the 200+ speed-up for
the binary block. I'm sure the disk cache has a big effect here, unless my
harddrive has a 600MB/sec transfer rate.

--
Bart

[Richard Tobin]

In article <fr**********@r egistered.motza rella.org>,
Richard <de***@gmail.co mwrote:

>I didn't say whether it's in binary is unrelated to *speed*.

>I'm not sure that parses :-;

I didn't say { { whether it's in binary } is unrelated to { *speed* } }.

-- Richard
--
:wq

[Chris Dollin]

Richard wrote:

Chris Dollin <ch**********@h p.comwrites:

>Myself I wouldn't count that as "LOTS faster" for binary data,
but doubtless there are applications where it is so counted;
I don't think the OPs case is one of them, and it does look as
though he's reading a text file anyway.

Then why not take the static noise out? Make the file a lot bigger and
report back.

Because 2000 lines was the OPs file size, and for that file size
and context, the difference in timing is unimportant, and because
life being finite, I'd already spent what time I had available.

But even these results do indicate quite a large % difference .....

And we do not know how often this data sample is written or read. I
could be thousands of times an hour leading to considerable unnecessary
overhead if using ascii over binary.

Yes, and it could be once a day. Or a week. And for all we know -- hey,
if you can invent facts, so can I -- his code will be run on machines
with different floating-point formats, making binary transfer a clear
road to the Pit and text transfer more of a Dragons of Bel'kwinith thing.

--
"Creation began." - James Blish, /A Clash of Cymbals/

Hewlett-Packard Limited registered office: Cain Road, Bracknell,
registered no: 690597 England Berks RG12 1HN

[Chris Dollin]

Richard wrote:

"Bartc" <[e-mail] rites:

>So about 200-500 times faster in binary mode, when done properly.

I'm surprised this is even being contested.

It's not being contested; it's being /quantified/, which is part of
deciding whether whatever is the right thing to do.

[You can drive along the M4 at 70mph or at 120mph [1]; the latter is
certainly faster.]

[1] And, Just In Case Someone Suspects A Weasel, at a whole bunch of
other speeds as well, including at times 0; I don't /think/ I've
ever had to go negative, though.

--
"It was the dawn of the third age of mankind." /Babylon 5/

Hewlett-Packard Limited registered office: Cain Road, Bracknell,
registered no: 690597 England Berks RG12 1HN