looking for statefull snprintf

[dimka]

Hello

Here is my situation: I have a char buf[1024]; that is used as a buffer

for a log output. Normally, I do:

rc = snprintf( buf, sizeof(buf), some_format, some_args );

This makes the formatting my output and then I flush the buffer.
In 90% of cases this works well, exept the cases when the format or the

arguments require more then 1024 bytes of formatted input.
There is no easy way to re-run snprintf() form 'last' position.

Suppose that using of asprintf(), another buffer and other kinds of
malloc() is not an option. Suppose the buffer must be preprocessed
before it hits the actual io.

Is there any open/closed libraries that solve the problem? Are there
any discussions held on the subject? Please provide the references.

Thanks
--
Dima

[websnarf]

dimka wrote:

Here is my situation: I have a char buf[1024]; that is used as a buffer
for a log output.

You should immediately cringe whenever you find yourself in situations
where you are declaring large arrays like that, basically in hopes that
1024 is close enough to infinity.

[...] Normally, I do:

rc = snprintf( buf, sizeof(buf), some_format, some_args );

This makes the formatting my output and then I flush the buffer.
In 90% of cases this works well, exept the cases when the format or the
arguments require more then 1024 bytes of formatted input.
There is no easy way to re-run snprintf() form 'last' position.

Right, in fact you can't. The arguments are always considered formed
once at compile time. You cannot even process the same va_list more
than once (the standards committee added va_copy to C99, but C99 has
not been widely adopted).

Suppose that using of asprintf(), another buffer and other kinds of
malloc() is not an option. Suppose the buffer must be preprocessed
before it hits the actual io.

Is there any open/closed libraries that solve the problem? Are there
any discussions held on the subject? Please provide the references.

The regulars in this group *cannot* help you. They will cite the
standard, and cannot read well enough to see that you say specifically
that you cannot call malloc() or asprintf().

Ok, the only straight forward "in standard" solution that is possible
for you is to use vfprintf to write the output to a file, then read
back that file in chunks that you can then send to your real output.
Of course this involves the file system and is very likely to be highly
distasteful (what are you supposed to do when the disk is full?).

Another approach, of course, is to go find some open source snprintf's
out there and hack on them to make them do what you want. There are
numerous such snprintf packages out there. However, its also possible
to write your own. Either way, what you're going to want to do is to
change interface to something like:

long vpprintf (long (*output) (const char * fragment, int sz,
void * ctx),
void * ctx, const char * fmt, va_list arg);

The idea is that you would incrementally generate the formatted output
into fragments that you would then emit to the output() function
pointer in pieces (with the length passed in; this is required so that
you can output %c, with a value of '\0'). As a bonus, you would make
it so that if output() returned a negative number vpprintf would stop,
and re-return this value (to do some sort of error propogation.)
Otherwise you would return the total positive number of characters
"written" to the output callback function. Such a solution would
always use a finite amount of memory but be able to process an
arbitrarily long amount of formatted output.

Keep in mind that its not impossible to write this code from scratch
yourself. Primarily you have to break down the formatting and
arguments (via va_arg) yourself, but for the actual conversion you can
generally fall back on just plain old sprintf(). Obviously, you would
detect formatting widths beyond a certain size, and perform some
specially slicing and dicing and iterate it through a single reasonably
sized auto buffer. So there is not need to solve the difficult task of
converting floating point a to string or similar kinds of things.

If you write this code, you will basically be able to solve any problem
you ever have with C-style formatted output just using this interface.
What this says about the C standard committee (i.e., the fact that this
function or equivalent (i.e., allow for FILE *'s to wrap function
callbacks) doesn't already exist) I will leave for you to decide.

--
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/

[Ben Pfaff]

"dimka" <di************ @gmail.comwrite s:

Well, let me explain the problem again.
Lets assume that the buffer is empty, thus no need to flush it.
then

rc = snprintf( buffer, sizeof(buffer)-1, "%s", str );

Works pretty well. No buffer overrun, in the new standard
rc == strlen(str), in old standard rc == -1, thus I have a way
to check that there was the potential overrun.

The problem arises when strlen(str) sizeof(buffer).
In this situation the buffer will contain trunkated version
of str, I have the information that the string was
trunkated, but no way to say "hey, I have flushed the
buffer, please continue from the place where you
stopped". There must to be an alternative implementation
of snprintf() to do exactly what I want.

You still haven't explained why you can't just enlarge the buffer
and rerun snprintf in the enlarged buffer.
--
"I don't have C&V for that handy, but I've got Dan Pop."
--E. Gibbons

[CBFalconer]

dimka wrote:

>
This is funny :) Everyone can cite the standard. But this is EXACTLY
the code that does NOT work.

Don't top-post. Read the code again. Note that it first tests for
overflow, and if such would occur flushes the buffer and marks it
empty. The only assumption is that a single output string fits in
the buffer.

This design is intended to reuse a buffer, but mitigate the system
interruption to perform actual output.

--
"The power of the Executive to cast a man into prison without
formulating any charge known to the law, and particularly to
deny him the judgement of his peers, is in the highest degree
odious and is the foundation of all totalitarian government
whether Nazi or Communist." -- W. Churchill, Nov 21, 1943

[CBFalconer]

we******@gmail. com wrote:

dimka wrote:

>Here is my situation: I have a char buf[1024]; that is used as a
buffer for a log output.

.... snip ...

>
Right, in fact you can't. The arguments are always considered formed
once at compile time. You cannot even process the same va_list more
than once (the standards committee added va_copy to C99, but C99 has
not been widely adopted).

>Suppose that using of asprintf(), another buffer and other kinds of
malloc() is not an option. Suppose the buffer must be preprocessed
before it hits the actual io.

Is there any open/closed libraries that solve the problem? Are there
any discussions held on the subject? Please provide the references.

The regulars in this group *cannot* help you. They will cite the
standard, and cannot read well enough to see that you say specifically
that you cannot call malloc() or asprintf().

Ok, the only straight forward "in standard" solution that is possible
for you is to use vfprintf to write the output to a file, then read
back that file in chunks that you can then send to your real output.
Of course this involves the file system and is very likely to be highly
distasteful (what are you supposed to do when the disk is full?).

If you take the trouble to read the solution I posted, you will
find that there are no calls to malloc etc. involved, and that the
buffer is never overwritten. It only involves a few lines of code,
and no tortuous rebuilding of the wheel as you advise. The key is
that snprintf(NULL, 0, ....) returns the output length without any
actual output. Ben has pointed out that some libraries do not
adhere to the standard, but that is a QOI issue.

--
"The power of the Executive to cast a man into prison without
formulating any charge known to the law, and particularly to
deny him the judgement of his peers, is in the highest degree
odious and is the foundation of all totalitarian government
whether Nazi or Communist." -- W. Churchill, Nov 21, 1943

[websnarf]

CBFalconer wrote:

we******@gmail. com wrote:

dimka wrote:

Here is my situation: I have a char buf[1024]; that is used as a
buffer for a log output.

... snip ...

Right, in fact you can't. The arguments are always considered formed
once at compile time. You cannot even process the same va_list more
than once (the standards committee added va_copy to C99, but C99 has
not been widely adopted).

Suppose that using of asprintf(), another buffer and other kinds of
malloc() is not an option. Suppose the buffer must be preprocessed
before it hits the actual io.

Is there any open/closed libraries that solve the problem? Are there
any discussions held on the subject? Please provide the references.

The regulars in this group *cannot* help you. They will cite the
standard, and cannot read well enough to see that you say specifically
that you cannot call malloc() or asprintf().

Ok, the only straight forward "in standard" solution that is possible
for you is to use vfprintf to write the output to a file, then read
back that file in chunks that you can then send to your real output.
Of course this involves the file system and is very likely to be highly
distasteful (what are you supposed to do when the disk is full?).

If you take the trouble to read the solution I posted, you will
find that there are no calls to malloc etc. involved,

I *DID* read your post. It just plain doesn't work, as the OP was
immediately able to point out. But that's just par for the course with
you, so I didn't bother pointing it out directly.

[...] and that the buffer is never overwritten.

I don't even know what you mean here -- even *your* solution reuses the
buffer, but that is only step 1 in trying to solve this.

[...] It only involves a few lines of code,
and no tortuous rebuilding of the wheel as you advise.

And that's fine if the OP doesn't care to have a working solution. My
solution is only meant for the very narrow goal of correctly answering
the OP's question as asked. As to the much broader scope of irrelevant
answers that obviously don't work, I'll leave that to you.

[...] The key is
that snprintf(NULL, 0, ....) returns the output length without any
actual output. Ben has pointed out that some libraries do not
adhere to the standard, but that is a QOI issue.

*MANY* libraries do not adhere to the standard. Go look at the source
for asprintf, or in fact my bformat(a) functions in Bstrlib. We go to
very extreme lengths to tease out the results we want because of the
really poor state of standards compliance on that function.

QOI issue or not, that OP was not asking for a platform specific
solution (isn't that supposed to be OT in this newsgroup?) and was not
asking for a non-functional solution (which you and Ben seemed happy to
provide) either.

--
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/

[CBFalconer]

we******@gmail. com wrote:

CBFalconer wrote:

.... snip ...

>>
If you take the trouble to read the solution I posted, you will
find that there are no calls to malloc etc. involved,

I *DID* read your post. It just plain doesn't work, as the OP was
immediately able to point out. But that's just par for the course
with you, so I didn't bother pointing it out directly.

No you didn't read the code, or you are incapable of reading
standard C. From your cavalier attitude to standards I suspect the
latter. I even posted a portion of the standard to justify the
technique.

>

>[...] and that the buffer is never overwritten.

I don't even know what you mean here -- even *your* solution reuses the
buffer, but that is only step 1 in trying to solve this.

Here I meant that there never is any overflow. Poorly worded by
me. I guess you require very short monosyllabic phrases. Buffers
are normally intended to be filled and emptied. Believe it or not,
memory is a limited resource.

I apologize for using three and four syllable words, but I find it
hard to express myself without them. You can find most of them in
any dictionary.

--
"The power of the Executive to cast a man into prison without
formulating any charge known to the law, and particularly to
deny him the judgement of his peers, is in the highest degree
odious and is the foundation of all totalitarian government
whether Nazi or Communist." -- W. Churchill, Nov 21, 1943

[websnarf]

CBFalconer wrote:

we******@gmail. com wrote:

CBFalconer wrote:

... snip ...

If you take the trouble to read the solution I posted, you will
find that there are no calls to malloc etc. involved,

I *DID* read your post. It just plain doesn't work, as the OP was
immediately able to point out. But that's just par for the course
with you, so I didn't bother pointing it out directly.

No you didn't read the code, or you are incapable of reading
standard C. From your cavalier attitude to standards I suspect the
latter. I even posted a portion of the standard to justify the
technique.

*Sigh*. Ok, from the top:

1. templgh = snprintf(NULL, 0, format, whatever);
2. if (templgh + bufindex >= sizeof(buffer) {
3. flushbuf(buffer ); bufindex = 0;
4. }
5. bufindex += snprintf(buffer[bufindex],
sizeof(buffer) - 1 - bufindex,
format, whatever);

Line 1 has a well known problem; what snprintf outputs on different
systems is just inconsistent, so your test on line 2 just does not
correctly identify the incremental overflow case. You can claim this
is just a violation of the standard, so we will ignore your contempt
for real world systems for the moment.

The first argument of Line 5 *might* make sense if you've declared
buffer to be char buffer[<something>][1024] or something like that, but
then we are screwed back on Line 2 with the sizeof(buffer). Ok, so
just looking at the "units" of each parameter, it seems that you
probably meant &buffer[bufindex] -- I don't know how to proceed in
analyzing your code if I can't make this assumption.

Ok, as the OP points out, "whatever" can be a mass of stuff including a
single 8.5 megabyte string. Now if sizeof(buffer) < the length of the
contents of a formatted "whatever", then what the hell would is line 5
going to do? It will just truncate whatever it is you are trying to
sprintf down to sizeof(buffer)-1 characters. So that's just a wrong
result.

Furthermore format and "whatever" are a complex sequence of arguments
-- restarting them, and this sprintf somehow, just isn't going to cut
it; it can't be done. This is the crux of his problem, and this is
what he is asking. He made the point clearer in a follow-up post (I
got it the first time.) Your ignoring this problem is an interesting
way of trying to answer it, but that doesn't help the OP.

[...] and that the buffer is never overwritten.

I don't even know what you mean here -- even *your* solution reuses the
buffer, but that is only step 1 in trying to solve this.

Here I meant that there never is any overflow.

I see. Well as I point out above, that isn't the problem with your
solution anyways.

[...] Poorly worded by
me. I guess you require very short monosyllabic phrases.

Its not poorly worded -- its wrongly worded. I only require correct
sentences using correct words. I'm fine with you simply correcting
yourself after the fact here (I'm hardly immune to typos myself) but of
course, you had to push it.

--
Paul Hsieh
http://www.pobox.com/~qed/
http://bstring.sf.net/

[dimka]

There are few reasons:

The application must use minimum heap memory (application environment
is close
to embeded applications, sory but I cant disclose the exact details).
The malloc() is very
expensive. Also, it is very important to keep low rate of the memory
leaks, while allocating the buffer dynamically will require later
clean-up in all possible exception
handling situations. This will increase chanse of the memory leaks.

You still haven't explained why you can't just enlarge the buffer
and rerun snprintf in the enlarged buffer.

[dimka]

You are right. The original question was "does anybody know about open
source vpprintf() implementation? ". The problem is obvious and exists
since sprintf() was invented.
Thus I hope somebody already solved it.

Anyway, thanks for your answers guys!

Another approach, of course, is to go find some open source snprintf's
out there and hack on them to make them do what you want. There are
numerous such snprintf packages out there. However, its also possible
to write your own. Either way, what you're going to want to do is to
change interface to something like:

long vpprintf (long (*output) (const char * fragment, int sz,
void * ctx),
void * ctx, const char * fmt, va_list arg);

[Chris Torek]

[someone -- I cannot quite untangle the attributions at this point --
wrote]

>>>bufindex += snprintf(buffer[bufindex],
sizeof(buffer) - 1 - bufindex,
format, whatever);

This code is obviously wrong: the first argument needs to be
&buffer[bufindex] (or equivalent).

In article <11************ **********@i3g2 000cwc.googlegr oups.com>
dimka <di************ @gmail.comwrote :

>Lets assume that the buffer is empty, thus no need to flush it.
then

rc = snprintf( buffer, sizeof(buffer)-1, "%s", str );

Works pretty well. No buffer overrun, in the new standard
rc == strlen(str), in old standard rc == -1, thus I have a way
to check that there was the potential overrun.

The problem arises when strlen(str) sizeof(buffer).
In this situation the buffer will contain trunkated version
of str, I have the information that the string was
trunkated, but no way to say "hey, I have flushed the
buffer, please continue from the place where you
stopped". There must to be an alternative implementation
of snprintf() to do exactly what I want.

In the general case, this is too hard (MUCH too hard). Consider
what has to happen if, for instance, the format argument is not
just "%s", but instead, we have something like:

int aw, ap; /* width and precision for "a" */
double a; /* value of "a" */
int sw; /* width for "s" */
char *s; /* etc */
int k;
int x;
...
/* there is no need for -1 on sizeof buf; snprintf does that */
x = snprintf(buf, sizeof buf, "%*.*f %*.32s %-12d", aw, ap, a, sw, s, k);

It is possible that, e.g., the printf engine stopped somewhere in
the middle of the "%*.32s", having run out of room for padding or
having run out of the maximum of 32 characters in "s" (or both).
The only reasonable method is to start over, discarding the "unwanted"
characters.

Of course, if your format is always exactly "%s", you do not need
snprintf() at all! The "%s" format just copies a string unchanged,
and the length of the string is simply strlen(str). So given the
simplified version above:

size_t i, part, len;
const char *p;

len = strlen(str);
for (i = len, p = str; i 0; p += part, i -= part) {
/* find out how much of the string fits */
part = sizeof buffer - 1;
if (part len)
part = len;

/* put it in the buffer */
memcpy(buffer, p, part);
buffer[part] = 0;

... do something with buffer ...
/* or, do it directly with p[0]..p[part-1], eliminating the memcpy */
}

You only need snprintf() if the format is complicated, in which
case, you usually need the "restart from beginning" behavior.

If you really need something fancier, you can always write your
own (possibly limited) printf-like function(s) in Standard C. Or,
if you are willing to go beyond Standard C, you can:

- use the funopen() I put in the 4.4BSD stdio, or
- use the GNU equivalent of funopen()

to "open" a stdio stream with a user-supplied "buffer writing"
function, and then supply that stream to fprintf(). Of course
using such extensions, while convenient, gives up all kinds of
portability.
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) [phone]
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.