473,854 Members | 1,491 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

making an istream from a char array


I'm working with two libraries, one written
in old school C, that returns a very large
chunk of data in the form of a C-style,
NUL-terminated string.

The other written in a more modern C++
is a parser for the chunk of bytes returned by
the first. It expects a reference to a
std::istream as its argument.

The chunk of data is very large.
I'd like to feed the output of the first to
the second WITHOUT MAKING AN EXTRA IN-MEMORY COPY.

My attempts to create an istringstream from the
chunk of data all seem to at least double the
amount of VM used. Here's a short program demonstrating
what I've tried. Is there any way to get "inside"
the istringstream and tell it to use the 'chunk'
directly, rather than insisting on making a copy?

Thanks,
John Salmon

[jsalmon@river c++]$ cat chararraytostre am.cpp
#include <string>
#include <sstream>
#include <cstdlib>
#include <cstring>
#include <cstdio>
using namespace std;

char *getLotsOfBytes ();
istream& streamParser(is tream &s);
void linuxChkMem(con st char *msg);

void withImplicitStr ing(){
linuxChkMem("Be fore getLotsOfBytes: ");
char *chunk = getLotsOfBytes( );
linuxChkMem("Af ter getLotsOfBytes( ):");
{
istringstream iss(chunk);
linuxChkMem("Af ter iss(p): ");
streamParser(is s);
linuxChkMem("Af ter streamParser(is s): ");
}
linuxChkMem("Af ter iss goes out of scope: ");
free(chunk);
linuxChkMem("Af ter free(p): ");
}

void withExplicitStr ing(){
linuxChkMem("Be fore getLotsOfBytes: ");
char *chunk = getLotsOfBytes( );
linuxChkMem("Af ter getLotsOfBytes( ):");
{
string s(chunk);
linuxChkMem("Af ter s(chunk): ");
free(chunk);
linuxChkMem("Af ter free(p): ");
istringstream iss(s);
linuxChkMem("Af ter iss(s): ");
streamParser(is s);
linuxChkMem("Af ter streamParser(is s): ");
}
linuxChkMem("Af ter iss goes out of scope: ");
}

int main(int argc, char **argv){
printf("with an implicit string constructor\n") ;
withImplicitStr ing();
printf("\nwith an explicit string constructor\n") ;
withExplicitStr ing();
return 0;
}

// On linux, tell us how much data space we're using
// in the VM.
void linuxChkMem(con st char *msg){
printf("%s", msg);
fflush(stdout);
char cmd[50];
sprintf(cmd, "grep VmData /proc/%d/status", getpid());
system(cmd);
}

static const int SZ = 100*1024*1024;
// A rough approximation to getLotsOfBytes. In the
// real application, getLotsOfBytes has these characteristics :
// - it returns a malloced pointer to a NUL-terminated array of chars.
// - it is out of my control. E.g., I can't rewrite it in a way
// that might be more friendly to C++ streams.
char *getLotsOfBytes (){
char *p = (char *)malloc(SZ);
memset(p, ' ', SZ);
strcpy(p+SZ-50, "3.1415 2.718 1.414");
return p;
}

// A rough approximation to streamParser. In the real
// application, streamParser takes a ref to an istream
// and does what it does. Again, I can't easily redefine
// the interface.
istream& streamParser(is tream& s){
double x, y, z;
s > x >y >z;
printf("x: %f y: %f z: %f\n", x, y, z);
return s;
}

[jsalmon@river c++]$ g++ -O3 chararraytostre am.cpp
[jsalmon@river c++]$ a.out
with an implicit string constructor
Before getLotsOfBytes: VmData: 40 kB
After getLotsOfBytes( ):VmData: 102444 kB
After iss(p): VmData: 204848 kB
x: 3.141500 y: 2.718000 z: 1.414000
After streamParser(is s): VmData: 204980 kB
After iss goes out of scope: VmData: 102576 kB
After free(p): VmData: 172 kB

with an explicit string constructor
Before getLotsOfBytes: VmData: 172 kB
After getLotsOfBytes( ):VmData: 102576 kB
After s(chunk): VmData: 204980 kB
After free(p): VmData: 102576 kB
After iss(s): VmData: 204980 kB
x: 3.141500 y: 2.718000 z: 1.414000
After streamParser(is s): VmData: 204980 kB
After iss goes out of scope: VmData: 172 kB
[jsalmon@river c++]$

Dec 30 '06 #1
7 20645
Hello John!
John Salmon wrote:
My attempts to create an istringstream from the
chunk of data all seem to at least double the
amount of VM used.
std::istringstr eam takes a std::string. For creating this
std::string from a char array, a copy is created. This copy
is then copied into the std::istringstr eam. For this purpose,
you probably don't want to use an std::istringstr eam. Instead,
you could use a simple homegrown stream buffer (code see
below).

Good luck, Denise!
--- CUT HERE ---
#include <istream>
#include <iostream>
#include <streambuf>
#include <string>
#include <string.h>

struct membuf:
std::streambuf
{
membuf(char* b, char* e) { this->setg(b, b, e); }
};

int main()
{
char* buffer = get_huge_buffer _with_data();
membuf sbuf(buffer, std::find(buffe r, buffer + strlen(buffer), 0));
std::istream in(&sbuf);
for (std::string line; std::getline(in , line); )
std::cout << "line: " << line << "\n";
}

Dec 30 '06 #2
John Salmon wrote:
I'm working with two libraries, one written
in old school C, that returns a very large
chunk of data in the form of a C-style,
NUL-terminated string.

The other written in a more modern C++
is a parser for the chunk of bytes returned by
the first. It expects a reference to a
std::istream as its argument.

The chunk of data is very large.
I'd like to feed the output of the first to
the second WITHOUT MAKING AN EXTRA IN-MEMORY COPY.
The "without making a copy" might be a little tricky with istringstream.

I'm no expert on c++ streams but something like this might work.

#include <istream>

class Xistream
: public std::istream,
public std::streambuf
{
public:
Xistream( const char * begin, const char * end )
: std::istream( this )
{
setg( const_cast<char *>(begin), const_cast<char *>(begin),
const_cast<char *>(end) );
}
};

#include <iostream>

int main()
{
const char xx[] = "1 22 33";

Xistream xi( xx, xx + sizeof(xx) -1);

int i;
xi >i;

std::cout << i << "\n";

xi >i;

std::cout << i << "\n";

}
Dec 30 '06 #3
>>>>"Denise" == Denise Kleingeist <de************ ***@googlemail. comwrites:

DeniseHello John!
DeniseJohn Salmon wrote:
>My attempts to create an istringstream from the
chunk of data all seem to at least double the
amount of VM used.
Denisestd::istr ingstream takes a std::string. For creating this
Denisestd::stri ng from a char array, a copy is created. This copy
Deniseis then copied into the std::istringstr eam. For this purpose,
Deniseyou probably don't want to use an std::istringstr eam. Instead,
Deniseyou could use a simple homegrown stream buffer (code see
Denisebelow).

DeniseGood luck, Denise!
Denise--- CUT HERE ---
Denise #include <istream>
Denise #include <iostream>
Denise #include <streambuf>
Denise #include <string>
Denise #include <string.h>

Denise struct membuf:
Denise std::streambuf
Denise {
Denise membuf(char* b, char* e) { this->setg(b, b, e); }
Denise };

Denise int main()
Denise {
Denise char* buffer = get_huge_buffer _with_data();
Denise membuf sbuf(buffer, std::find(buffe r, buffer + strlen(buffer), 0));
Denise std::istream in(&sbuf);
Denise for (std::string line; std::getline(in , line); )
Denise std::cout << "line: " << line << "\n";
Denise }

Thanks! This is exactly what I needed.

One question - what's the point of the std::find()?

I don't see how std::find(buffe r, buffer+strlen(b uffer), 0);
could ever be different from buffer+strlen(b uffer)??

Cheers,
John Salmon
Dec 30 '06 #4
Hello John!
John Salmon wrote:
>>>"Denise" == Denise Kleingeist <de************ ***@googlemail. comwrites:
Denise membuf sbuf(buffer, std::find(buffe r, buffer + strlen(buffer), 0));
One question - what's the point of the std::find()?

I don't see how std::find(buffe r, buffer+strlen(b uffer), 0);
could ever be different from buffer+strlen(b uffer)??
You are right: it is a left over from a discarded attempt to use
std::find() instead of strlen()! Just use buffer + strlen(buffer)
instead.

Sorry for any confusion caused, Denise!

Dec 30 '06 #5
"John Salmon" <js*****@thesal mons.orgwrote in message
news:m3******** ****@river.fish net...
I'm working with two libraries, one written
in old school C, that returns a very large
chunk of data in the form of a C-style,
NUL-terminated string.

The other written in a more modern C++
is a parser for the chunk of bytes returned by
the first. It expects a reference to a
std::istream as its argument.

The chunk of data is very large.
I'd like to feed the output of the first to
the second WITHOUT MAKING AN EXTRA IN-MEMORY COPY.

My attempts to create an istringstream from the
chunk of data all seem to at least double the
amount of VM used. Here's a short program demonstrating
what I've tried. Is there any way to get "inside"
the istringstream and tell it to use the 'chunk'
directly, rather than insisting on making a copy?
See the header <strstream>. It does exactly what you want,
and it's part of the C++ Standard (albeit a bit old
fashioned).

P.J. Plauger
Dinkumware, Ltd.
http://www.dinkumware.com
Dec 30 '06 #6
>>>>"PJ" == P J Plauger <pj*@dinkumware .comwrites:

PJ"John Salmon" <js*****@thesal mons.orgwrote in message
PJnews:m3****** ******@river.fi shnet...
>I'm working with two libraries, one written
in old school C, that returns a very large
chunk of data in the form of a C-style,
NUL-terminated string.

The other written in a more modern C++
is a parser for the chunk of bytes returned by
the first. It expects a reference to a
std::istream as its argument.

The chunk of data is very large.
I'd like to feed the output of the first to
the second WITHOUT MAKING AN EXTRA IN-MEMORY COPY.

My attempts to create an istringstream from the
chunk of data all seem to at least double the
amount of VM used. Here's a short program demonstrating
what I've tried. Is there any way to get "inside"
the istringstream and tell it to use the 'chunk'
directly, rather than insisting on making a copy?
PJSee the header <strstream>. It does exactly what you want,
PJand it's part of the C++ Standard (albeit a bit old
PJfashioned).

Thanks to Usenet, I now have two workable solutions.

Googling for strstream turns up lots of warnings that "strstream is
deprecated", with dire warnings that it may be removed from future
versions of the standard. OTOH, an istrstream does exactly what I
want, without any extra custom machinery ( struct membuf : public
streambuf ).

Other than simplicity and possible compatibility with future
standards, is there any reason to prefer one approach over the
other?

Cheers,
John Salmon
Dec 30 '06 #7
"John Salmon" <js*****@thesal mons.orgwrote in message
news:m3******** ****@river.fish net...
>>>>>"PJ" == P J Plauger <pj*@dinkumware .comwrites:

PJ"John Salmon" <js*****@thesal mons.orgwrote in message
PJnews:m3****** ******@river.fi shnet...
>>I'm working with two libraries, one written
in old school C, that returns a very large
chunk of data in the form of a C-style,
NUL-terminated string.

The other written in a more modern C++
is a parser for the chunk of bytes returned by
the first. It expects a reference to a
std::istrea m as its argument.

The chunk of data is very large.
I'd like to feed the output of the first to
the second WITHOUT MAKING AN EXTRA IN-MEMORY COPY.

My attempts to create an istringstream from the
chunk of data all seem to at least double the
amount of VM used. Here's a short program demonstrating
what I've tried. Is there any way to get "inside"
the istringstream and tell it to use the 'chunk'
directly, rather than insisting on making a copy?

PJSee the header <strstream>. It does exactly what you want,
PJand it's part of the C++ Standard (albeit a bit old
PJfashioned).

Thanks to Usenet, I now have two workable solutions.

Googling for strstream turns up lots of warnings that "strstream is
deprecated", with dire warnings that it may be removed from future
versions of the standard. OTOH, an istrstream does exactly what I
want, without any extra custom machinery ( struct membuf : public
streambuf ).

Other than simplicity and possible compatibility with future
standards, is there any reason to prefer one approach over the
other?
You should prefer strstream because:

1) it's exactly what you need

2) it's still part of the C++ Standard

3) there's no reason to believe it'll become nonstandard anytime
soon, despite the dire warnings

4) even if it does officially go away, there's not a sane vendor
who'll stop supporting it for the next decade

So what the hell.

P.J. Plauger
Dinkumware, Ltd.
http://www.dinkumware.com
Dec 30 '06 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
2329
by: matthurne | last post by:
I'm doing a chapter 12 exercise from Accelerated C++ ... writing a string-like class which stores its data in a low-level way. My class, called Str, uses a char array and length variable. I've gotten everything working that I want working so far, except for std::istream& operator>>(std::istream&, Str&) The way my Str class manages itself of course requires that the size of the char array to store is known when it is allocated. The...
3
5707
by: lpe540 | last post by:
Hi, I'm having trouble using istream to read in a file in its entirety on UNIX. I've written a dummy program that essencially reads in a file from stdin and writes it out to a file. When I cat a binary file through a unix pipe to the program (cat file | prog) everything works fine. However, when I feed data from a program, one that connects to an open socket so that multiple files can be processed, across a unix pipe (prog1 | prog) the...
13
3744
by: Randy | last post by:
Is there any way to do this? I've tried tellg() followed by seekg(), inserting the stream buffer to an ostringstream (ala os << is.rdbuf()), read(), and having no luck. The problem is, all of these methods EXTRACT the data at one point or another. The other problem is there appears to be NO WAY to get at the actual buffer pointer (char*) of the characters in the stream. There is a way to get the streambuf object associated with the...
5
2375
by: Jim Langston | last post by:
In one of my files I am outputting the value of a char into a human readable file. That is, char a = 123; std::ofstream CharFile( ("Players\\" + Name + ".char").c_str()); if ( CharFile.is_open() ) CharFile << (int) a; So the file has the character a stored as "123". That was the easy part, now comes the fun of reading it back into the char.
1
2153
by: mwebel | last post by:
Hi, My Module (B) needs to read from a istream (provided by another module A) and again the module (C) i use accepts only char*. So actually it accepts filenames. But i dont want to store the stream onto the disk and pass it again. I am also not allowed to change the API on the module i use. I can chang the way it works though. So i thought i'd cheat by storing the stream adress in the char and restoring it again once "inside" the code.
3
3384
by: KWienhold | last post by:
I'm currently writing an application (using Visual Studio 2003 SP1 and C#) that stores files and additional information in a single compound file using IStorage/IStream. Since files in a compound file aren't really useful to a user, I use the IStream::Read function together with a StreamWriter to extract single files from my compound document. When I first tested these functions everything seemed to work fine (and basically, it does),...
4
4476
by: =?Utf-8?B?Sm9obg==?= | last post by:
Hi all, I am developing website application in asp.net , visual C# and atl com. I am using atl com component in visual C# application. One of the function of com component interface returns IStream interface. I want to read data from that IStream interface. I am new to visual c#. I have written some code. But, it is returning whole data. It is returning some null characters.
4
8114
by: Ralf | last post by:
Hallo, I'm trying to call a COM function from C#. The function has a parameter from the type array of IStream* (IStream**). The COM component (out-proc-server (.exe)) has the following code: (C++) STDMETHODIMP CIntf::SendFiles(int noOfStreams, IUnknown** dataStreams) {
4
2383
by: james.lawton | last post by:
Hi, I'm having a problem that I can't diagnose. I'm creating istreams of unknown types, and want to manage them on the stack, co I'm passing around ownership-holding pointers. Usually, I would use std::auto_ptr<std::istream>, but it seems to be deallocating early, as the call to read(...) below breaks. I've condensed a test case. Using my own pointer works fine, but using auto_ptr does not (see USE_AUTO_PTR). I solved the issue by a...
0
9901
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9751
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
11024
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
10749
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10367
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7912
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7079
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5939
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
4152
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.