473,396 Members | 1,772 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,396 software developers and data experts.

filecopy with std::copy()

I was wondering if anyone knew of a way to use std::copy() and
istream_iterator<>/ostream_iterator<> write a file copy function that is
quick and efficient.

Doing this messes up the file because it seems to ignore '\n'

ifstream in("somefile");
ofstream out("someOtherFile");

std::copy(std::istream_iterator<unsigned char>(in),
std::istream_iterator<unsigned char>(),
std::ostream_iterator<unsigned char>(out));

Now, I figured out how to do it correctly but it is dog slow. I was
wondering if anyone knew how to do this in an ellegant manner?

thomas j. clancy
Jul 19 '05 #1
9 42739
Hi Thomas,
"Thomas J. Clancy" <tj******@comcast.net> wrote in message
news:Rq********************@comcast.com...
I was wondering if anyone knew of a way to use std::copy() and
istream_iterator<>/ostream_iterator<> write a file copy function that is
quick and efficient. .... Now, I figured out how to do it correctly but it is dog slow. I was
wondering if anyone knew how to do this in an ellegant manner?


Unless you insist on using std::copy, the elegant and efficient
manner to copy an entire file (or stream):
dstStream << srcStream.rdbuf();
A C++ implementation should be able to ultimately optimize this
operation (but performance may vary...).

hth,
Ivan
--
http://www.post1.com/~ivec <> Ivan Vecerina
Jul 19 '05 #2
"Ivan Vecerina" <ivecATmyrealboxDOTcom> wrote in message
news:3f********@news.swissonline.ch...
Hi Thomas,
"Thomas J. Clancy" <tj******@comcast.net> wrote in message
news:Rq********************@comcast.com...
I was wondering if anyone knew of a way to use std::copy() and
istream_iterator<>/ostream_iterator<> write a file copy function that is
quick and efficient. ...
Now, I figured out how to do it correctly but it is dog slow. I was
wondering if anyone knew how to do this in an ellegant manner?


Unless you insist on using std::copy, the elegant and efficient
manner to copy an entire file (or stream):
dstStream << srcStream.rdbuf();
A C++ implementation should be able to ultimately optimize this
operation (but performance may vary...).

Elegant, yes... this I already knew about, but boy is it
sloooooooooooooooowwwwww.... I came up with a different solution using the
std::copy and a type (class) that contains a buffer of chars and uses
stream::read() and stream::write() within the input stream operator (>>) and
the output stream operator (<<), respectively. And man does it scream.

Anyway, I was just wondering if there were alternatives to creating this
sort of thing using or extending the stream stuff.

hth,
Ivan
--
http://www.post1.com/~ivec <> Ivan Vecerina

Jul 19 '05 #3
On Sun, 07 Sep 2003 09:44:26 -0400, Thomas J. Clancy wrote:
"Ivan Vecerina" <ivecATmyrealboxDOTcom> wrote in message
news:3f********@news.swissonline.ch...
Unless you insist on using std::copy, the elegant and efficient
manner to copy an entire file (or stream):
dstStream << srcStream.rdbuf();
A C++ implementation should be able to ultimately optimize this
operation (but performance may vary...).

Elegant, yes... this I already knew about, but boy is it
sloooooooooooooooowwwwww....


Nothing using IOStreams is going to be faster. File copies are best left
to OS routines.

Josh
Jul 19 '05 #4
On Sun, 07 Sep 2003 10:11:13 -0400, Thomas J. Clancy wrote:
Ummm... the rest of my previous reply talks about what I did to make it
much, much faster than the solution you mentioned, so what do you mean by
your statement above?


It was actually faster using a copy than rdbuf? That's a messed-up
IOStreams implementation. :-}
Jul 19 '05 #5
On Sun, 07 Sep 2003 11:05:28 -0400, Thomas J. Clancy wrote:

Not at all... when you use the output iterator of rdbuf(), I believe that it
is doing it byte by byte and not in chunks. At least this is the behaviour
I am seeing with VC7.1's implementation, which they get from Dinkumware, I
believe. Now I could try this using STLPort.


It shouldn't be, there should be buffering done both by your OS and by
IOStreams. For example

curien@balar:~/prog$ uname -a
Linux balar 2.4.18 #1 Sun Aug 10 12:24:29 EDT 2003 i686 GNU/Linux
curien@balar:~/prog$ cat blah.cpp
#include <fstream>
#include <ios>

int main() {
std::ifstream infile("test.dat", std::ios_base::binary);
std::ofstream outfile("test~.dat", std::ios_base::binary);

outfile << infile.rdbuf();
}
curien@balar:~/prog$ dd if=/dev/zero of=test.dat bs=1024 count=50K
51200+0 records in
51200+0 records out
52428800 bytes transferred in 0.700340 seconds (74861922 bytes/sec)
curien@balar:~/prog$ g++ -v
Reading specs from /usr/lib/gcc-lib/i486-linux/3.3.2/specs
Configured with: ../src/configure -v --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared --with-system-zlib --enable-nls --without-included-gettext --enable-__cxa_atexit --enable-clocale=gnu --enable-debug --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc i486-linux
Thread model: posix
gcc version 3.3.2 20030812 (Debian prerelease)
curien@balar:~/prog$ g++ -ansi -pedantic -W -Wall -O2 blah.cpp
curien@balar:~/prog$ time ./a.out

real 0m0.619s
user 0m0.030s
sys 0m0.540s

Josh
Jul 19 '05 #6

"Josh Sebastian" <cu****@cox.net> wrote in message
news:pa****************************@cox.net...
On Sun, 07 Sep 2003 11:05:28 -0400, Thomas J. Clancy wrote:

Not at all... when you use the output iterator of rdbuf(), I believe that it is doing it byte by byte and not in chunks. At least this is the behaviour I am seeing with VC7.1's implementation, which they get from Dinkumware, I believe. Now I could try this using STLPort.
It shouldn't be, there should be buffering done both by your OS and by
IOStreams. For example

curien@balar:~/prog$ uname -a
Linux balar 2.4.18 #1 Sun Aug 10 12:24:29 EDT 2003 i686 GNU/Linux
curien@balar:~/prog$ cat blah.cpp
#include <fstream>
#include <ios>

int main() {
std::ifstream infile("test.dat", std::ios_base::binary);
std::ofstream outfile("test~.dat", std::ios_base::binary);

outfile << infile.rdbuf();
}
curien@balar:~/prog$ dd if=/dev/zero of=test.dat bs=1024 count=50K
51200+0 records in
51200+0 records out
52428800 bytes transferred in 0.700340 seconds (74861922 bytes/sec)


Hey man, I have the numbers, too, and believe me, they suck. I wonder if
Microsoft is pulling a fast one? :-)
curien@balar:~/prog$ g++ -v
Reading specs from /usr/lib/gcc-lib/i486-linux/3.3.2/specs
Configured with: .../src/configure -v --enable-languages=c,c++,java,f77,pascal,objc,ada,treela
ng --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-gx
x-include-dir=/usr/include/c++/3.3 --enable-shared --with-system-zlib --enab
le-nls --without-included-gettext --enable-__cxa_atexit --enable-clocale=gnu
--enable-debug --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-
gc i486-linux Thread model: posix
gcc version 3.3.2 20030812 (Debian prerelease)
curien@balar:~/prog$ g++ -ansi -pedantic -W -Wall -O2 blah.cpp
curien@balar:~/prog$ time ./a.out

real 0m0.619s
user 0m0.030s
sys 0m0.540s

Josh

Jul 19 '05 #7
> It shouldn't be, there should be buffering done both by your OS and by
IOStreams. For example


My bad, you're right. Under microsoft, if you build this little application
in debug, it is dog slow. I thought I had been building in release mode.
Once I set it to release mode and rebuilt the thing flew! Thanks for the
information on this.

Tom
Jul 19 '05 #8
Thomas J. Clancy wrote:
"Ivan Vecerina" <ivecATmyrealboxDOTcom> wrote in message
news:3f********@news.swissonline.ch...

Unless you insist on using std::copy, the elegant and efficient
manner to copy an entire file (or stream):
dstStream << srcStream.rdbuf();
A C++ implementation should be able to ultimately optimize this
operation (but performance may vary...).
Elegant, yes... this I already knew about, but boy is it
sloooooooooooooooowwwwww....


Are you using Visual C++ 5 or 6 by chance? There's a known bug in the
iostream library that causes buffering to be wrongly disabled in file
streams that are opened by name. That could account for this being slow,
I think. Check here:

http://www.dinkumware.com/vc_fixes.html
I came up with a different solution using the
std::copy and a type (class) that contains a buffer of chars and uses
stream::read() and stream::write() within the input stream operator (>>) and
the output stream operator (<<), respectively. And man does it scream.


Buffering should be automatic, making this unnecessary. I guess that it
should be possible to make the solution using standard stream classes
perform just as well or better than this solution, but I wouldn't know
exactly how to do it.

-Kevin
--
My email address is valid, but changes periodically.
To contact me please use the address from a recent posting.

Jul 19 '05 #9

"Kevin Goodsell" <us*********************@neverbox.com> wrote in message
news:B0*****************@newsread3.news.pas.earthl ink.net...
Thomas J. Clancy wrote:
"Ivan Vecerina" <ivecATmyrealboxDOTcom> wrote in message
news:3f********@news.swissonline.ch...

Buffering should be automatic, making this unnecessary. I guess that it
should be possible to make the solution using standard stream classes
perform just as well or better than this solution, but I wouldn't know
exactly how to do it.


Here was my solution before I realized that using stream::rdbuf() worked
well while NOT in debug mode using VC++7.1 (.NET 2003).

/**
* A block buffer type that can be used with std::copy() and
istream_iterators without
* having to write a special form of copy or an istream_iterator.
*/
class ByteBlock
{
public:
ByteBlock()
: m_bytesRead(0),
m_fileSize(-1),
m_totalRead(0)
{
}

private:
unsigned char m_block[10240];
int m_bytesRead;
long m_fileSize;
long m_totalRead;
friend std::istream& operator >> (std::istream& stream, ByteBlock& byte);
friend std::ostream& operator << (std::ostream& stream, const ByteBlock&
byte);
};

std::istream& operator >> (std::istream& stream, ByteBlock& block)
{
if (block.m_fileSize == -1)
{
stream.seekg(0, std::ios::end);
block.m_fileSize = stream.tellg();
stream.seekg(0, std::ios::beg);
}
std::size_t leftToRead = block.m_fileSize - block.m_totalRead;
if (leftToRead)
{
stream.read((char*)block.m_block, std::min(sizeof(block.m_block),
leftToRead));
block.m_bytesRead = stream.gcount();
block.m_totalRead += block.m_bytesRead;
}
else
{
stream.setstate(std::ios_base::eofbit | std::ios_base::badbit);
}
return stream;
}

std::ostream& operator << (std::ostream& stream, const ByteBlock& block)
{
stream.write((char*)block.m_block, block.m_bytesRead);
return stream;
}

void blockCopyFile(const char* source, const char* dest)
{
ifstream in(source, ios::in | ios::binary);
ofstream out(dest, ios::out | ios::binary);
copy(istream_iterator<tjc_std::ByteBlock>(in),
istream_iterator<tjc_std::ByteBlock>(),
ostream_iterator<tjc_std::ByteBlock>(out));
}

Yes, this was a naive approach, but it worked quickly and in fact for some
reason this approach still seemed to work slightly faster than:

out << in.rdbuf();

I don't know why that would be, especially since what I've recently read and
what I've been told by others here in this newsgroup. But hey, I just need
a way to copy files without relying on the OS, so both of these ideas seems
to work just fine.

Tom
Jul 19 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

30
by: franky.backeljauw | last post by:
Hello, I am wondering which of these two methods is the fastest: std::copy, which is included in the standard library, or a manually written pointer copy? Do any of you have any experience with...
4
by: Simon Elliott | last post by:
Is there an equivalent of std::copy which works on STL containers for overlapping ranges? -- Simon Elliott http://www.ctsn.co.uk
3
by: kathy | last post by:
I have array: double a; double b; std::vector <double> vDouble; when I use: std::copy(a,a+1024,vDouble.begin());
1
by: Siegfried Heintze | last post by:
What is the minimum I must type to create a custom iterator that will allow me display my iterator on std::cout using std::copy? Thanks, Siegfried
1
by: Siegfried Heintze | last post by:
Should the following work? It does on some compilers. How can I get it to work on g++ 3.2? On g++ 3.2 it keeps telling me that there is no such function operator << for ostream and...
6
by: Chris Johnson | last post by:
Greetings all. I am really stuck on this one as I can't seem to grok if I am abusing the C++ language or if I am simply using components of the C++ Standard Library incorrectly. Here is the code:...
5
by: Pradeep | last post by:
Hi All, I am facing some problem using istream_iterator for reading the contents of a file and copying it in a vector of strings.However the same thing works for a vector of integers. The...
3
by: none | last post by:
Hi, Consider the following piece of code: int t={1,2,3,4,5,6}; vector<intv; std::copy (t, t+sizeof(t)/sizeof(t), std::back_inserter (v)); Could someone explain me why we can pass "t" as an...
9
by: arnuld | last post by:
Earlier, I have posted a program like this, a month ago IIRC. I have created it again, without looking at the old program. Can I have your opinions on this: 1) I wanted my program to be...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.