472,958 Members | 2,697 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,958 software developers and data experts.

Standard Read and Write FILE functions to refer to memory

I have huge number of legacy code which use standard files functions.
I would like to pass a memory pointer rather than a FILE pointer.

I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.

I do not want to touch the legacy code. Does any one know of a library
to do this. I would appreciate the help.

Ben Beroukhim
Nov 14 '05 #1
16 3410
bb********@aws.com (ben beroukhim) writes:
I am trying to use FILEs in the code to refer to memory
buffers. Basically, I want to be able to use all the standard
read and write functions, but I want them to refer to memory
locations, rather than disk files.


There is no standard way to do this. Many implementations of the
C library offer extensions that allow it, however. You should
refer to the reference manual for your C implementation to see if
yours is one of them.
--
int main(void){char p[]="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuv wxyz.\
\n",*q="kl BIcNBFr.NKEzjwCIxNJC";int i=sizeof p/2;char *strchr();int putchar(\
);while(*q){i+=strchr(p,*q++)-p;if(i>=(int)sizeof p)i-=sizeof p-1;putchar(p[i]\
);}return 0;}
Nov 14 '05 #2
On 30 Nov 2004 11:54:01 -0800
bb********@aws.com (ben beroukhim) wrote:
I have huge number of legacy code which use standard files functions.
I would like to pass a memory pointer rather than a FILE pointer.

I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.
Since the C file functions are file functions they operate on files.
I do not want to touch the legacy code. Does any one know of a library
to do this. I would appreciate the help.


No library can turn files in to anything other than files. Even if such
a thing were possible, it would still be off topic for this group since
we only discuss standard C.

<OT>
You could ask in a group dedicated to your system about RAM disks.
</OT>
--
Flash Gordon
Living in interesting times.
Although my email address says spam, it is real and I read it.
Nov 14 '05 #3
Flash Gordon <sp**@flash-gordon.me.uk> writes:
On 30 Nov 2004 11:54:01 -0800
bb********@aws.com (ben beroukhim) wrote:
I have huge number of legacy code which use standard files functions.
I would like to pass a memory pointer rather than a FILE pointer.

I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.


Since the C file functions are file functions they operate on files.


Sure, but what is a "file"?

There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.

In a sense, it's just like asking how to use FILEs to refer to files
stored on a hard drive (except that almost all C implementations
support such an interface).

This is entirely implementation-specific, of course. There's no way
to do it in standard C that's going to work on all implementations.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.
Nov 14 '05 #4
There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.
Hmmmm.... Not sure wether that's a good idea...
In a sense, it's just like asking how to use FILEs to refer to files
stored on a hard drive (except that almost all C implementations
support such an interface).

This is entirely implementation-specific, of course. There's no way
to do it in standard C that's going to work on all implementations.


Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Both could easily (and portably) be set using fcntl().

If they are called with a reference to the current location in the
file-buffer and a count of bytesto write (which isn't hard to manage), the
usual stdio suspects can be used to write data anywhere at an application
level.

Unbuffered FILE's call the same routines for each byte read or written, thus
behaving as a one byte buffer.

On the downside: ungetc() would pose a problem.

dandelion

Disclaimer: This idea has been nicked off C++. All the merit belongs there.

Nov 14 '05 #5
dandelion wrote:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

.... snip ...
Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Both could easily (and portably) be set using fcntl().


No it can't. One possible reason is that fcntl() is not standard.

--
Chuck F (cb********@yahoo.com) (cb********@worldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net> USE worldnet address!

Nov 14 '05 #6

"CBFalconer" <cb********@yahoo.com> wrote in message
news:41***************@yahoo.com...
dandelion wrote:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

... snip ...

Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Both could easily (and portably) be set using fcntl().


No it can't. One possible reason is that fcntl() is not standard.


Right. The rest of the idea isn't standard, either (but that's on purpose).
I got Posix confused with C standards again, it seems. Still, the idea of a
couple of callbacks for the purpose mentioned seems a good idea.
Nov 14 '05 #7
I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.


Just an idea -- use a temporary file and then mmap it in.

Jon
----
Learn to program using Linux assembly language
http://www.cafeshops.com/bartlettpublish.8640017
Nov 14 '05 #8
Jonathan Bartlett <jo*****@eskimo.com> wrote:
I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk


Just an idea -- use a temporary file and then mmap it in.


Not in ISO C.

Richard
Nov 14 '05 #9
In article <41***********************@dreader17.news.xs4all.n l>
dandelion <da*******@meadow.net> wrote:
Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.


I put this sort of thing, too, into the 4.3BSD stdio. In addition
to snprintf() (which did get into C99), I have "funopen":

FILE *funopen(cookie, readfn, writefn, seekfn, closefn)

where "cookie" is a "void *" (const-qualified in funopen but not
in the underlying functions), and the read, write, seek, and close
function parameters are pointers to functions that implement reading,
writing, seeking, and closing. Each function is optional; passing
NULL means "nothing to do for this". You cannot pass NULL for both
read and write functions -- using NULL for one of them makes it a
read-only or write-only stream as appropriate; using NULL for both
would make it a close-only stream. :-) Giving NULL as the seek
function makes it an unseekable stream (fseek() returns errors).

The cookie is simply passed back to your functions, giving you a
place to store data you want to associate with that particular stream,
e.g., the memory area for a "memory-stream", or window data
structure for a window-system output-message window, or compression
data for a zip/unzip stream, etc. Note that a compression stream
would typically sit atop a regular-file stream:

struct compression_data {
FILE *lower_file;
... rest of compression data ...
};

int compressor_write(void *cookie, const char *buf, int len) {
struct compression_data *cd = cookie;
... compress the data from buf[] using whatever algorithm ...
... put each character out via putc(c, cd->lower_file),
or fwrite(ptr, size, n, cd->lower_file) ...
return n_written_successfully;
}

but of course the "lower file" could be anything stdio handles,
including a memory stream, or an encryptor stream, or a binary-to-base64
mime-encoder stream, etc. It might well be reasonable to stack a
compressor above an encryptor that sits above a mime-encoder (and,
conversely, a mime-decoder below a decryptor below a decompressor).

For whatever reason, these "stackable" streams did not get into
C99.
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (4039.22'N, 11150.29'W) +1 801 277 2603
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.
Nov 14 '05 #10
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.


Hmmmm.... Not sure wether that's a good idea...

[...]

I wrote the above; you snipped the attribution.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.
Nov 14 '05 #11
Keith Thompson wrote:
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.


Hmmmm.... Not sure wether that's a good idea...

[...]

I wrote the above; you snipped the attribution.


Let me offer a theory: People who are incapable of properly
snipping and attributing quotes are very unlikely to be able to
cope with the vagaries of the C language. They show an appalling
lack of attention to detail.

Corollary: Any ugly looking article is not worth reading.

--
Chuck F (cb********@yahoo.com) (cb********@worldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net> USE worldnet address!
Nov 14 '05 #12

"Keith Thompson" <ks***@mib.org> wrote in message
news:ln************@nuthaus.mib.org...
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.


Hmmmm.... Not sure wether that's a good idea...

[...]

I wrote the above; you snipped the attribution.


Sorry.

Snip sig.
Nov 14 '05 #13

"CBFalconer" <cb********@yahoo.com> wrote in message
news:41***************@yahoo.com...
Keith Thompson wrote:
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

Hmmmm.... Not sure wether that's a good idea... [...]

I wrote the above; you snipped the attribution.


Let me offer a theory: People who are incapable of properly
snipping and attributing quotes are very unlikely to be able to
cope with the vagaries of the C language. They show an appalling
lack of attention to detail.


Here's another: People who write vindictive 'laws' are anal retentive.
Please look up the definition of the word 'Theory' before you postulate any,
especially accusing others of whowing lack of attention to detail.
Corollary: Any ugly looking article is not worth reading.


Unfortunately You don't know that before you read it. Anything of substance
to add or will you stick you your usual whining today?
Nov 14 '05 #14

"CBFalconer" <cb********@yahoo.com> wrote in message
news:41***************@yahoo.com...
Keith Thompson wrote:
"dandelion" <da*******@meadow.net> writes:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

Hmmmm.... Not sure wether that's a good idea... [...]

I wrote the above; you snipped the attribution.


Let me offer a theory: People who are incapable of properly
snipping and attributing quotes are very unlikely to be able to
cope with the vagaries of the C language. They show an appalling
lack of attention to detail.


Here's another: People who write vindictive 'laws' are anal retentive.
Corollary: Any ugly looking article is not worth reading.

Nov 14 '05 #15
CBFalconer wrote:
Corollary: Any ugly looking article is not worth reading.


I like it, Falconer's Corollary!

Brian
Nov 14 '05 #16
dandelion wrote:
Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Those are the sort of things you want if you're trying
specifically to read/write from/to strings. The more usual
hooks are general-purpose write and read functions (though doing
it that way, it's true, does not end up easily supporting the
additional goal of writing to a malloc'ed region of memory that
grows as needed).

Chris Torek replied: I put this sort of thing, too, into the 4.3BSD stdio...

FILE *funopen(cookie, readfn, writefn, seekfn, closefn)

where "cookie" is a "void *" and the... function parameters are
pointers to functions that implement reading, writing, seeking,
and closing.
It turns out that the GNU folks have implemented this, also,
in at least one of the versions of glibc. The recipe is slightly
different:

cookie_io_functions_t funcs = {readfn, writefn, seekfn, closefn};
FILE *fp = fopencookie(cookie, mode, funcs);

where mode is "r" or "w", as usual.

Since this is an extension, you have to define the macro
_GNU_SOURCE to enable it. Once you've done so, you also have
access to

FILE *fmemopen(void *str, size_t len, const char *mode)

which gives just the functionality the original poster was asking
for (i.e. without having to define any of your own auxiliary read
or write functions), and also

FILE *open_memstream(char **strp, size_t *lenp)

which writes to the malloc'ed region pointed to by *strp and
grows it as needed (updating the size *lenp as it does so).
I don't know if any of this is documented anywhere; I stumbled
across it only recently while looking for something else in GNU's
copy of <stdio.h>.

Anyway, if you're using Linux or some other glibc-using system,
this functionality may well be available to you already. (But
before the topicality police come down on me, I have to reiterate
that these extensions are not standard and that code which makes
use of them is therefore not portable.)
For whatever reason, these "stackable" streams did not get into C99.


And a pity it is, because it's insanely useful functionality,
and not difficult or expensive to implement. It seems like
practically every large program I've ever written has needed to
do this sort of thing (i.e. read or write interchangeably to a
file or a string) at some point; half the time I end up writing
my own little ad-hoc wrapper on top of stdio to let me do this,
which is doubly inefficient.

Steve Summit
sc*@eskimo.com
Nov 14 '05 #17

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: Steven T. Hatton | last post by:
I haven't looked very closely, but from what I'm seeing, it looks like there are no buffered I/O streams in the Standard Library. There are stream buffers, but not buffered streams. I don't have...
43
by: Steven T. Hatton | last post by:
Now that I have a better grasp of the scope and capabilities of the C++ Standard Library, I understand that products such as Qt actually provide much of the same functionality through their own...
32
by: toolmaster | last post by:
Since many of the modern computer languages have built-in namespace features, I can't understand why not add this feature into standard C. I've heard many people complain of the lacking of...
15
by: Gan Quan | last post by:
I'm writing a c++ program that has many (100+) threads read/write files simultaneously. It works well if not considering the efficiency. The file i/o seems to be the bottleneck. This is my code...
20
by: J de Boyne Pollard | last post by:
MThe library functions which are included to allow process Mlaunch, forking, and termination, imply that it is both Mpossible and desirable for a process to fork itself. This is Ma fundamental...
270
by: jacob navia | last post by:
In my "Happy Christmas" message, I proposed a function to read a file into a RAM buffer and return that buffer or NULL if the file doesn't exist or some other error is found. It is interesting...
126
by: Dann Corbit | last post by:
Rather than create a new way of doing things: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2497.html why not just pick up ACE into the existing standard:...
20
by: Peter Olcott | last post by:
Is there any standard C++ way to determine the size of a file before it is read?
27
by: =?ISO-8859-1?Q?Tom=E1s_=D3_h=C9ilidhe?= | last post by:
I have a fully-portable C program (or at least I think I do). It works fine on Windows, but malfunctions on Linux. I suspect that there's something I don't know about the standard input stream...
0
by: lllomh | last post by:
Define the method first this.state = { buttonBackgroundColor: 'green', isBlinking: false, // A new status is added to identify whether the button is blinking or not } autoStart=()=>{
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 4 Oct 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
0
tracyyun
by: tracyyun | last post by:
Hello everyone, I have a question and would like some advice on network connectivity. I have one computer connected to my router via WiFi, but I have two other computers that I want to be able to...
2
by: giovanniandrean | last post by:
The energy model is structured as follows and uses excel sheets to give input data: 1-Utility.py contains all the functions needed to calculate the variables and other minor things (mentions...
1
by: Teri B | last post by:
Hi, I have created a sub-form Roles. In my course form the user selects the roles assigned to the course. 0ne-to-many. One course many roles. Then I created a report based on the Course form and...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 1 Nov 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM) Please note that the UK and Europe revert to winter time on...
3
by: nia12 | last post by:
Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...
0
NeoPa
by: NeoPa | last post by:
Introduction For this article I'll be focusing on the Report (clsReport) class. This simply handles making the calling Form invisible until all of the Reports opened by it have been closed, when it...
0
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.