472,353 Members | 1,856 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,353 software developers and data experts.

Help with "read" issue please

Thanks in advance for any tips that might get me going in the right
direction.

I am working on a simple exercise for school that is supposed to use read to
read a file (about 10 MB). I am supposed to change the buffer size and see
how this affects the read time. In other words, the buffer is supposed to
limit how much of the file gets read per call, and cause some change in
speed. I am supposed to do the same with fread as well, but I'm just asking
for help on read for now.

Everything works like a charm if I have the buffer set to the same size as
the file, but if I make the buffer size smaller than the file, I get a core
dump - segmentation fault.

Here's all the code. Do I need to put the data somewhere or clear the
buffer after a pass or something? I'm guessing I'm stepping out of my
buffer when it gets full.

#include <fstream>
#include <time.h>
#include <iostream>
using namespace std;
void main(){

cout << "start \n" ;

char buff[100];
int length;
time_t seconds;

ifstream inputFile ("TEST", ios::binary);

inputFile.seekg (0, ios::end);
length = inputFile.tellg();
inputFile.seekg (0, ios::beg);

seconds = time (NULL);

cout << seconds << " Seconds at start \n" << length << " bytes\n" ;

// If buffer is same size as the file, I have no problem, but if I
drop the buff size - CRASH!
while(inputFile.read(buff,10000000)){
}

seconds = time(NULL);

cout << seconds << " Seconds at finish \n" ;

}
Any help is appreciated!
Shane

Jul 22 '05 #1
10 2162
ZafT wrote:
char buff[100]; while(inputFile.read(buff,10000000)){


The only thing read() knows about your buf is where it begins. read() relies
on you to tell it where the buf array ends. That's why lying in the second
argument is not a good idea. Pass the actual length of the buf array.

You are learning that the C languages are very mechanical, and will compile
any well-formed source code you give it, regardless of whether it makes
sense.

--
Phlip
http://industrialxp.org/community/bi...UserInterfaces
Jul 22 '05 #2

"Phlip" <ph*******@yahoo.com> wrote in message
news:XH*****************@newssvr15.news.prodigy.co m...
ZafT wrote:
char buff[100];
while(inputFile.read(buff,10000000)){


The only thing read() knows about your buf is where it begins. read()

relies on you to tell it where the buf array ends. That's why lying in the second
argument is not a good idea. Pass the actual length of the buf array.
--

Phlip
http://industrialxp.org/community/bi...UserInterfaces


Hmmm...

I agree that normally things would work better if I passed a buffer the size
of the file, but I have been asked to do the opposite for the sake of
reading the file in chunks to test performance. Is there no way to do this?

Thanks for letting me know I'm not crazy at least :-) The help is very
appreciated.

Shane

Jul 22 '05 #3

"ZafT" <bu**@inhell.com> wrote in message
news:g8********************@comcast.com...

"Phlip" <ph*******@yahoo.com> wrote in message
news:XH*****************@newssvr15.news.prodigy.co m...
ZafT wrote:
char buff[100];
while(inputFile.read(buff,10000000)){


The only thing read() knows about your buf is where it begins. read()

relies
on you to tell it where the buf array ends. That's why lying in the second argument is not a good idea. Pass the actual length of the buf array.
--

Phlip
http://industrialxp.org/community/bi...UserInterfaces


Hmmm...

I agree that normally things would work better if I passed a buffer the

size of the file, but I have been asked to do the opposite for the sake of
reading the file in chunks to test performance. Is there no way to do this?
Thanks for letting me know I'm not crazy at least :-) The help is very
appreciated.

Shane


Here's the description of my exercise -

Your program will read the input file several times: for each of the two
cases above, you will use a read size of 1, 2, 5, 10, 20, 50, 100, 200, 500,
1000, 2000, 5000, 10000, 20000, and 50000. In other words, your program will
first open the input file, read its entire contents 1-byte at a time using
the read system call, then close the file. Your program will then open the
file again and read though the file, reading 2-bytes at a time, and so on,
up to 50,000 bytes at a time.

I'm not sure how to accomplish this without changing the buffer.


Jul 22 '05 #4
ZafT wrote:
char buff[100];
while(inputFile.read(buff,10000000)){


The only thing read() knows about your buf is where it begins. read()

relies
on you to tell it where the buf array ends. That's why lying in the second argument is not a good idea. Pass the actual length of the buf array.

Hmmm...

I agree that normally things would work better if I passed a buffer the size of the file, but I have been asked to do the opposite for the sake of
reading the file in chunks to test performance. Is there no way to do

this?

You didn't read my reply, or the help files for read, or something.

read() doesn't give a rat's ass how big your buffer is. It will read the
amount specified in its second argument. If that number is too big. read()
will write characters outside your buf array. Your buf is 100 and your read
says 10000000. You are writing all over your stack.

Look these topics up in your tutorial before proceding.

--
Phlip
http://industrialxp.org/community/bi...UserInterfaces
Jul 22 '05 #5
On Thu, 22 Jul 2004 22:08:00 -0700, ZafT <bu**@inhell.com> wrote:

"ZafT" <bu**@inhell.com> wrote in message
news:g8********************@comcast.com...

"Phlip" <ph*******@yahoo.com> wrote in message
news:XH*****************@newssvr15.news.prodigy.co m...
> ZafT wrote:
>
> > char buff[100];
>
> > while(inputFile.read(buff,10000000)){
>
> The only thing read() knows about your buf is where it begins. read()

relies
> on you to tell it where the buf array ends. That's why lying in the second > argument is not a good idea. Pass the actual length of the buf array.
>> --
> Phlip
> http://industrialxp.org/community/bi...UserInterfaces >
>


Hmmm...

I agree that normally things would work better if I passed a buffer the

size
of the file, but I have been asked to do the opposite for the sake of
reading the file in chunks to test performance. Is there no way to do

this?

Thanks for letting me know I'm not crazy at least :-) The help is very
appreciated.

Shane


Here's the description of my exercise -

Your program will read the input file several times: for each of the two
cases above, you will use a read size of 1, 2, 5, 10, 20, 50, 100, 200,
500,
1000, 2000, 5000, 10000, 20000, and 50000. In other words, your program
will
first open the input file, read its entire contents 1-byte at a time
using
the read system call, then close the file. Your program will then open
the
file again and read though the file, reading 2-bytes at a time, and so
on,
up to 50,000 bytes at a time.

I'm not sure how to accomplish this without changing the buffer.


You obviously have some sort of blind spot here, here's two cases

// read the entire file 100 bytes at a time
char buf[100];
while (file.read(buf, 100))
{
}

// read the entire file 50000 bytes at a time
char buf[50000];
while (file.read(buf, 50000))
{
}

That's all there is to it.

john
Jul 22 '05 #6
read() doesn't give a rat's ass how big your buffer is. It will read the
amount specified in its second argument. If that number is too big. read()
will write characters outside your buf array. Your buf is 100 and your read says 10000000. You are writing all over your stack.

Look these topics up in your tutorial before proceding.

--
Phlip
http://industrialxp.org/community/bi...UserInterfaces


Thanks for the help. I guessed that I was going a bit out of bounds, but
since the teacher referred to the buffer as the point of manipulation in
class, I thought read() would go until the buffer was full, then make
another pass at it or something. At least now I know I'm wrong.
Unfortionately, I'm not sure how to proceed now.

Can I read 100 byte chunks, ie..

char buff[100];

while(inputFile.read(buff,100)){
// fill buffer
// empty buffer
// go back and read the next 100 bytes
}

??

btw, I did not intentionally ignore your advice - I just misinterpreted it.

Shane

Jul 22 '05 #7
You obviously have some sort of blind spot here, here's two cases

// read the entire file 100 bytes at a time
char buf[100];
while (file.read(buf, 100))
{
}

// read the entire file 50000 bytes at a time
char buf[50000];
while (file.read(buf, 50000))
{
}

That's all there is to it.

john


It's sad that I actually tried that, but both read and fread read in the
file so fast that I assumed that I was doing something wrong. I thought
that I was reading only the first ___ bytes and ending without continuing on
through the rest of the file. At least I can call the program done now and
report my results!

Thanks for all of the help!
Jul 22 '05 #8
On Thu, 22 Jul 2004 22:41:34 -0700, ZafT <bu**@inhell.com> wrote:
You obviously have some sort of blind spot here, here's two cases

// read the entire file 100 bytes at a time
char buf[100];
while (file.read(buf, 100))
{
}

// read the entire file 50000 bytes at a time
char buf[50000];
while (file.read(buf, 50000))
{
}

That's all there is to it.

john


It's sad that I actually tried that, but both read and fread read in the
file so fast that I assumed that I was doing something wrong. I thought
that I was reading only the first ___ bytes and ending without
continuing on
through the rest of the file. At least I can call the program done now
and
report my results!

Thanks for all of the help!


There are several things you should be aware of in interpreting your
results. First there are several other buffers between the buffer in your
code and the file. ifstream will maintain its own buffer, and it very
likely that the operating system will have its own buffers as well, and
the hardware could also have its own buffers.

So don't assume that

char buf[100];
while (file.read(buf, 100))
{
}

results in bytes being phyically read 100 at a time from the file.

Another thing to be aware of it that on some operating systems you could
get different results on different occasions. For instance it could be
that if you run your program twice it will run faster the second time.
This is because the operating system is 'remembering' that you read the
file before and so storing part or all of the file in memory assuming that
you will want to read it again. Another case of buffering.

john
Jul 22 '05 #9
..............

So don't assume that

char buf[100];
while (file.read(buf, 100))
{
}

results in bytes being phyically read 100 at a time from the file.

Another thing to be aware of it that on some operating systems you could
get different results on different occasions. For instance it could be
that if you run your program twice it will run faster the second time.
This is because the operating system is 'remembering' that you read the
file before and so storing part or all of the file in memory assuming that
you will want to read it again. Another case of buffering.

john


It's like you were reading my mind. I was just wondering why it ran so much
faster the second time. heh. Well it seems as though this experiment for
comparing read and fread is not going to give any kind of raw result.
That's okay though. I've learned a few things tonight. I appreciate your
time very much. This group has been good to me as long as I try everything
myself first and post enough code so people don't have to read my mind. :-)


Jul 22 '05 #10
ZafT wrote:
.............
So don't assume that

char buf[100];
while (file.read(buf, 100))
{
}

results in bytes being phyically read 100 at a time from the file.

Another thing to be aware of it that on some operating systems you could
get different results on different occasions. For instance it could be
that if you run your program twice it will run faster the second time.
This is because the operating system is 'remembering' that you read the
file before and so storing part or all of the file in memory assuming that
you will want to read it again. Another case of buffering.

john

It's like you were reading my mind. I was just wondering why it ran so much
faster the second time. heh. Well it seems as though this experiment for
comparing read and fread is not going to give any kind of raw result.
That's okay though. I've learned a few things tonight. I appreciate your
time very much. This group has been good to me as long as I try everything
myself first and post enough code so people don't have to read my mind. :-)


#include <iostream>
#include <fstream>
#include <cstdlib>
using namespace std; // for toy programs.

const unsigned int MAX_BUFFER_SIZE = 50000;
const unsigned int chunk_sizes[] =
{
1, 2, 5, 10, 20, 50, 100, 200, 500,
1000, 2000, 5000, 10000, 20000, 50000
}
const unsigned int NUM_CHUNK_SIZES =
sizeof(chunk_sizes) / sizeof(chunk_sizes[0]);

unsigned char buffer[MAX_BUFFER_SIZE];

int main(void)
{
ifstream data_file;

for (unsigned int i = 0; i < NUM_CHUNK_SIZES; ++i)
{
unsigned int chunk_size = chunk_sizes[i];
data_file.open(/*...*/); // Per requirements.
while (data_file.read(buffer, chunk_size))
{
// Do stuff here.
}
data_file.close(); // Per requirements.
}
return EXIT_SUCCESS;
}
If your compiler cannot allocate an automatic array
of 50,000 then you will have to use the new operator
and allocate memory before you start reading. I
highly recommend allocating the largest size and only
do this once.

--
Thomas Matthews

C++ newsgroup welcome message:
http://www.slack.net/~shiva/welcome.txt
C++ Faq: http://www.parashift.com/c++-faq-lite
C Faq: http://www.eskimo.com/~scs/c-faq/top.html
alt.comp.lang.learn.c-c++ faq:
http://www.raos.demon.uk/acllc-c++/faq.html
Other sites:
http://www.josuttis.com -- C++ STL Library book

Jul 22 '05 #11

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Askari | last post by:
Yesterday, ALL code in python work and nothing when I close(finish) a code. Today, when I close, some raise this windows error : Instruction at...
1
by: Poseidon | last post by:
I use "read" method to read some blocks into a buffer from a filestream, and send data in the buffer to remote server with "send" method of cocket...
1
by: Venkat | last post by:
Hi, I am buried with the following issue on Linux platform, please let me know if someone can help me out. I am using ifstream to open and...
3
by: Vic | last post by:
Dear All, I have a database of laboratory records in Access 2000. There is one form which acts as an interface to input experimetal data. This...
2
by: Ryan Linneman via .NET 247 | last post by:
Any help appreciated... I am using a third party DLL file to access an HMI using ethernet. The function is declared in a VB module as: 'Read...
6
by: Lighter | last post by:
How to read "The lvalue-to-rvalue, array-to-pointer, and function-to- pointer standard conversionsare not applied to the left expressions"? In...
3
kickingthehabbit
by: kickingthehabbit | last post by:
Hi All new to all of this I just downloaded reactor server. everything works fine except Mysql - SQL import files. I am trying to import...
3
by: mohammadazim | last post by:
Hi guys I wrote a socket program in c++ and used blocking system call read () for receiving data on socket. But it gives following error while...
1
by: kamcap | last post by:
I have referenced a COM dll (unmanagged code) in a C# console application and this DLL is referenced in C# program using Interop facility in .NET....
0
by: Naresh1 | last post by:
What is WebLogic Admin Training? WebLogic Admin Training is a specialized program designed to equip individuals with the skills and knowledge...
0
jalbright99669
by: jalbright99669 | last post by:
Am having a bit of a time with URL Rewrite. I need to incorporate http to https redirect with a reverse proxy. I have the URL Rewrite rules made...
2
by: Matthew3360 | last post by:
Hi, I have a python app that i want to be able to get variables from a php page on my webserver. My python app is on my computer. How would I make it...
0
by: Arjunsri | last post by:
I have a Redshift database that I need to use as an import data source. I have configured the DSN connection using the server, port, database, and...
0
by: Matthew3360 | last post by:
Hi, I have been trying to connect to a local host using php curl. But I am finding it hard to do this. I am doing the curl get request from my web...
0
Oralloy
by: Oralloy | last post by:
Hello Folks, I am trying to hook up a CPU which I designed using SystemC to I/O pins on an FPGA. My problem (spelled failure) is with the...
0
by: Carina712 | last post by:
Setting background colors for Excel documents can help to improve the visual appeal of the document and make it easier to read and understand....
0
BLUEPANDA
by: BLUEPANDA | last post by:
At BluePanda Dev, we're passionate about building high-quality software and sharing our knowledge with the community. That's why we've created a SaaS...
0
by: Rahul1995seven | last post by:
Introduction: In the realm of programming languages, Python has emerged as a powerhouse. With its simplicity, versatility, and robustness, Python...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.