By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
458,053 Members | 922 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 458,053 IT Pros & Developers. It's quick & easy.

Standard Read and Write FILE functions to refer to memory

P: n/a
I have huge number of legacy code which use standard files functions.
I would like to pass a memory pointer rather than a FILE pointer.

I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.

I do not want to touch the legacy code. Does any one know of a library
to do this. I would appreciate the help.

Ben Beroukhim
Nov 14 '05 #1
Share this Question
Share on Google+
16 Replies


P: n/a
bb********@aws.com (ben beroukhim) writes:
I am trying to use FILEs in the code to refer to memory
buffers. Basically, I want to be able to use all the standard
read and write functions, but I want them to refer to memory
locations, rather than disk files.


There is no standard way to do this. Many implementations of the
C library offer extensions that allow it, however. You should
refer to the reference manual for your C implementation to see if
yours is one of them.
--
int main(void){char p[]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuv wxyz.\
\n",*q="kl BIcNBFr.NKEzjwCIxNJC";int i=sizeof p/2;char *strchr();int putchar(\
);while(*q){i+=strchr(p,*q++)-p;if(i>=(int)sizeof p)i-=sizeof p-1;putchar(p[i]\
);}return 0;}
Nov 14 '05 #2

P: n/a
On 30 Nov 2004 11:54:01 -0800
bb********@aws.com (ben beroukhim) wrote:
I have huge number of legacy code which use standard files functions.
I would like to pass a memory pointer rather than a FILE pointer.

I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.
Since the C file functions are file functions they operate on files.
I do not want to touch the legacy code. Does any one know of a library
to do this. I would appreciate the help.


No library can turn files in to anything other than files. Even if such
a thing were possible, it would still be off topic for this group since
we only discuss standard C.

<OT>
You could ask in a group dedicated to your system about RAM disks.
</OT>
--
Flash Gordon
Living in interesting times.
Although my email address says spam, it is real and I read it.
Nov 14 '05 #3

P: n/a
Flash Gordon <sp**@flash-gordon.me.uk> writes:
On 30 Nov 2004 11:54:01 -0800
bb********@aws.com (ben beroukhim) wrote:
I have huge number of legacy code which use standard files functions.
I would like to pass a memory pointer rather than a FILE pointer.

I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.


Since the C file functions are file functions they operate on files.


Sure, but what is a "file"?

There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.

In a sense, it's just like asking how to use FILEs to refer to files
stored on a hard drive (except that almost all C implementations
support such an interface).

This is entirely implementation-specific, of course. There's no way
to do it in standard C that's going to work on all implementations.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.
Nov 14 '05 #4

P: n/a
There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.
Hmmmm.... Not sure wether that's a good idea...
In a sense, it's just like asking how to use FILEs to refer to files
stored on a hard drive (except that almost all C implementations
support such an interface).

This is entirely implementation-specific, of course. There's no way
to do it in standard C that's going to work on all implementations.


Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Both could easily (and portably) be set using fcntl().

If they are called with a reference to the current location in the
file-buffer and a count of bytesto write (which isn't hard to manage), the
usual stdio suspects can be used to write data anywhere at an application
level.

Unbuffered FILE's call the same routines for each byte read or written, thus
behaving as a one byte buffer.

On the downside: ungetc() would pose a problem.

dandelion

Disclaimer: This idea has been nicked off C++. All the merit belongs there.

Nov 14 '05 #5

P: n/a
dandelion wrote:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

.... snip ...
Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Both could easily (and portably) be set using fcntl().


No it can't. One possible reason is that fcntl() is not standard.

--
Chuck F (cb********@yahoo.com) (cb********@worldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net> USE worldnet address!

Nov 14 '05 #6

P: n/a

"CBFalconer" <cb********@yahoo.com> wrote in message
news:41***************@yahoo.com...
dandelion wrote:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

... snip ...

Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Both could easily (and portably) be set using fcntl().


No it can't. One possible reason is that fcntl() is not standard.


Right. The rest of the idea isn't standard, either (but that's on purpose).
I got Posix confused with C standards again, it seems. Still, the idea of a
couple of callbacks for the purpose mentioned seems a good idea.
Nov 14 '05 #7

P: n/a
I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.


Just an idea -- use a temporary file and then mmap it in.

Jon
----
Learn to program using Linux assembly language
http://www.cafeshops.com/bartlettpublish.8640017
Nov 14 '05 #8

P: n/a
Jonathan Bartlett <jo*****@eskimo.com> wrote:
I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk


Just an idea -- use a temporary file and then mmap it in.


Not in ISO C.

Richard
Nov 14 '05 #9

P: n/a
In article <41***********************@dreader17.news.xs4all.n l>
dandelion <da*******@meadow.net> wrote:
Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.


I put this sort of thing, too, into the 4.3BSD stdio. In addition
to snprintf() (which did get into C99), I have "funopen":

FILE *funopen(cookie, readfn, writefn, seekfn, closefn)

where "cookie" is a "void *" (const-qualified in funopen but not
in the underlying functions), and the read, write, seek, and close
function parameters are pointers to functions that implement reading,
writing, seeking, and closing. Each function is optional; passing
NULL means "nothing to do for this". You cannot pass NULL for both
read and write functions -- using NULL for one of them makes it a
read-only or write-only stream as appropriate; using NULL for both
would make it a close-only stream. :-) Giving NULL as the seek
function makes it an unseekable stream (fseek() returns errors).

The cookie is simply passed back to your functions, giving you a
place to store data you want to associate with that particular stream,
e.g., the memory area for a "memory-stream", or window data
structure for a window-system output-message window, or compression
data for a zip/unzip stream, etc. Note that a compression stream
would typically sit atop a regular-file stream:

struct compression_data {
FILE *lower_file;
... rest of compression data ...
};

int compressor_write(void *cookie, const char *buf, int len) {
struct compression_data *cd = cookie;
... compress the data from buf[] using whatever algorithm ...
... put each character out via putc(c, cd->lower_file),
or fwrite(ptr, size, n, cd->lower_file) ...
return n_written_successfully;
}

but of course the "lower file" could be anything stdio handles,
including a memory stream, or an encryptor stream, or a binary-to-base64
mime-encoder stream, etc. It might well be reasonable to stack a
compressor above an encryptor that sits above a mime-encoder (and,
conversely, a mime-decoder below a decryptor below a decompressor).

For whatever reason, these "stackable" streams did not get into
C99.
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (4039.22'N, 11150.29'W) +1 801 277 2603
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.
Nov 14 '05 #10

P: n/a
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.


Hmmmm.... Not sure wether that's a good idea...

[...]

I wrote the above; you snipped the attribution.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.
Nov 14 '05 #11

P: n/a
Keith Thompson wrote:
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.


Hmmmm.... Not sure wether that's a good idea...

[...]

I wrote the above; you snipped the attribution.


Let me offer a theory: People who are incapable of properly
snipping and attributing quotes are very unlikely to be able to
cope with the vagaries of the C language. They show an appalling
lack of attention to detail.

Corollary: Any ugly looking article is not worth reading.

--
Chuck F (cb********@yahoo.com) (cb********@worldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net> USE worldnet address!
Nov 14 '05 #12

P: n/a

"Keith Thompson" <ks***@mib.org> wrote in message
news:ln************@nuthaus.mib.org...
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.


Hmmmm.... Not sure wether that's a good idea...

[...]

I wrote the above; you snipped the attribution.


Sorry.

Snip sig.
Nov 14 '05 #13

P: n/a

"CBFalconer" <cb********@yahoo.com> wrote in message
news:41***************@yahoo.com...
Keith Thompson wrote:
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

Hmmmm.... Not sure wether that's a good idea... [...]

I wrote the above; you snipped the attribution.


Let me offer a theory: People who are incapable of properly
snipping and attributing quotes are very unlikely to be able to
cope with the vagaries of the C language. They show an appalling
lack of attention to detail.


Here's another: People who write vindictive 'laws' are anal retentive.
Please look up the definition of the word 'Theory' before you postulate any,
especially accusing others of whowing lack of attention to detail.
Corollary: Any ugly looking article is not worth reading.


Unfortunately You don't know that before you read it. Anything of substance
to add or will you stick you your usual whining today?
Nov 14 '05 #14

P: n/a

"CBFalconer" <cb********@yahoo.com> wrote in message
news:41***************@yahoo.com...
Keith Thompson wrote:
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

Hmmmm.... Not sure wether that's a good idea... [...]

I wrote the above; you snipped the attribution.


Let me offer a theory: People who are incapable of properly
snipping and attributing quotes are very unlikely to be able to
cope with the vagaries of the C language. They show an appalling
lack of attention to detail.


Here's another: People who write vindictive 'laws' are anal retentive.
Corollary: Any ugly looking article is not worth reading.

Nov 14 '05 #15

P: n/a
CBFalconer wrote:
Corollary: Any ugly looking article is not worth reading.


I like it, Falconer's Corollary!

Brian
Nov 14 '05 #16

P: n/a
dandelion wrote:
Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Those are the sort of things you want if you're trying
specifically to read/write from/to strings. The more usual
hooks are general-purpose write and read functions (though doing
it that way, it's true, does not end up easily supporting the
additional goal of writing to a malloc'ed region of memory that
grows as needed).

Chris Torek replied: I put this sort of thing, too, into the 4.3BSD stdio...

FILE *funopen(cookie, readfn, writefn, seekfn, closefn)

where "cookie" is a "void *" and the... function parameters are
pointers to functions that implement reading, writing, seeking,
and closing.
It turns out that the GNU folks have implemented this, also,
in at least one of the versions of glibc. The recipe is slightly
different:

cookie_io_functions_t funcs = {readfn, writefn, seekfn, closefn};
FILE *fp = fopencookie(cookie, mode, funcs);

where mode is "r" or "w", as usual.

Since this is an extension, you have to define the macro
_GNU_SOURCE to enable it. Once you've done so, you also have
access to

FILE *fmemopen(void *str, size_t len, const char *mode)

which gives just the functionality the original poster was asking
for (i.e. without having to define any of your own auxiliary read
or write functions), and also

FILE *open_memstream(char **strp, size_t *lenp)

which writes to the malloc'ed region pointed to by *strp and
grows it as needed (updating the size *lenp as it does so).
I don't know if any of this is documented anywhere; I stumbled
across it only recently while looking for something else in GNU's
copy of <stdio.h>.

Anyway, if you're using Linux or some other glibc-using system,
this functionality may well be available to you already. (But
before the topicality police come down on me, I have to reiterate
that these extensions are not standard and that code which makes
use of them is therefore not portable.)
For whatever reason, these "stackable" streams did not get into C99.


And a pity it is, because it's insanely useful functionality,
and not difficult or expensive to implement. It seems like
practically every large program I've ever written has needed to
do this sort of thing (i.e. read or write interchangeably to a
file or a string) at some point; half the time I end up writing
my own little ad-hoc wrapper on top of stdio to let me do this,
which is doubly inefficient.

Steve Summit
sc*@eskimo.com
Nov 14 '05 #17

This discussion thread is closed

Replies have been disabled for this discussion.