fclose then fopen equivalent for stdout?

[David Mathog]

A program of mine writes to a tape unit. Output can be either through
stdout or through a file opened with fopen(). When all the data is
transferred to tape the program needs to close the output stream so that
the tape driver will write a filemark on the tape. Otherwise multiple
clumps of data saved to tape would all appear to be one big file on the
tape.

When the tape unit device was explicitly opened with fopen()
that's possible: call fclose() and then for the next batch
of data fopen() the tape device again and write some more.

However when data is going through stdout like:

program /dev/nst0

is there an equivalent operation? Maybe something like this:

fputc(stdout,EOF);

or this

freopen(NULL,"wb",stdout);

?

Thanks,

David Mathog

[David Mathog]

David Mathog wrote:

fputc(stdout,EOF);

oops, that should have been:

fputc(EOF,stdout);

David Mathog

[Eric Sosman]

David Mathog wrote On 10/25/06 12:48,:

A program of mine writes to a tape unit. Output can be either through
stdout or through a file opened with fopen(). When all the data is
transferred to tape the program needs to close the output stream so that
the tape driver will write a filemark on the tape. Otherwise multiple
clumps of data saved to tape would all appear to be one big file on the
tape.

When the tape unit device was explicitly opened with fopen()
that's possible: call fclose() and then for the next batch
of data fopen() the tape device again and write some more.

However when data is going through stdout like:

program /dev/nst0

is there an equivalent operation? Maybe something like this:

fputc(stdout,EOF);

(Corrected to fputc(EOF, stdout) in a follow-up.)

No, this would merely write a character to the output.
The identity of exactly which character gets written is a
little fuzzy in that it depends on the value of the EOF
macro (usually -1, but could be another negative integer)
and on what you get when converting that value to a char
in the local encoding.

or this

freopen(NULL,"wb",stdout);

Undefined behavior: The first argument is supposed to
be a string (a string that names a file), but NULL is not
a string. It's much like trying fopen(NULL, "wb").

Here's a suggestion: You're trying to obtain the effect
of fclose(), right? Does any particular function spring to
mind as being likely to perform the operations of fclose()?
How about ... <<wait for it>... fclose()?

--
Er*********@sun.com

[David Mathog]

Eric Sosman wrote:

Here's a suggestion: You're trying to obtain the effect
of fclose(), right? Does any particular function spring to
mind as being likely to perform the operations of fclose()?
How about ... <<wait for it>... fclose()?

I'm not trying to get fclose, I'm trying to get

fclose()
fopen()

How then does the program CONTINUE to write to stdout
so that data gets to the tape device after having called
fclose(stdout)?

What parameters for fopen() reassociate stdout with the same
output stream following the fclose()?

Regards,

David mathog

[Harald van Dijk]

Eric Sosman wrote:

David Mathog wrote On 10/25/06 12:48,:

A program of mine writes to a tape unit. Output can be either through
stdout or through a file opened with fopen(). When all the data is
transferred to tape the program needs to close the output stream so that
the tape driver will write a filemark on the tape. Otherwise multiple
clumps of data saved to tape would all appear to be one big file on the
tape.

When the tape unit device was explicitly opened with fopen()
that's possible: call fclose() and then for the next batch
of data fopen() the tape device again and write some more.

However when data is going through stdout like:

program /dev/nst0

is there an equivalent operation? Maybe something like this:

fputc(stdout,EOF);

(Corrected to fputc(EOF, stdout) in a follow-up.)

No, this would merely write a character to the output.
The identity of exactly which character gets written is a
little fuzzy in that it depends on the value of the EOF
macro (usually -1, but could be another negative integer)
and on what you get when converting that value to a char
in the local encoding.

or this

freopen(NULL,"wb",stdout);

Undefined behavior: The first argument is supposed to
be a string (a string that names a file), but NULL is not
a string. It's much like trying fopen(NULL, "wb").

Actually, freopen() has special behaviour for a NULL filename. I don't
know enough about it to know if that special behaviour is useful here,
though.

Here's a suggestion: You're trying to obtain the effect
of fclose(), right? Does any particular function spring to
mind as being likely to perform the operations of fclose()?
How about ... <<wait for it>... fclose()?

So how do you write to stdout again after closing it?

[David Mathog]

Harald van Dijk wrote:

Eric Sosman wrote:

>David Mathog wrote On 10/25/06 12:48,:

>> freopen(NULL,"wb",stdout);

Undefined behavior: The first argument is supposed to
be a string (a string that names a file), but NULL is not
a string. It's much like trying fopen(NULL, "wb").

Actually, freopen() has special behaviour for a NULL filename. I don't
know enough about it to know if that special behaviour is useful here,
though.

I think probably not helpful. From what I can tell this syntax is
supposed to be used to switch the stream from binary to text (for
instance), but doesn't specify that the stream must be closed and
reopened. And it's C99. Perhaps one of the language gurus can
clarify that point.

Anyway, as far as I can tell C allows stdin,stdout,stderr to be closed,
but provides no way to open them again afterwards such that the
stream is connected to the previous source or destination.

Thanks,

David Mathog

[Christopher Benson-Manica]

David Mathog <ma****@caltech.eduwrote:

What parameters for fopen() reassociate stdout with the same
output stream following the fclose()?

It sounds like what you want to do is fclose(stdout) and then somehow
open it again; while not an exact match for your situation, FAQ 12.34
(http://c-faq.com/stdio/undofreopen.html) suggests strongly to me that
once you fclose(stdout) you are on your own getting it open again.
I don't believe you stated what your system setup is, but 12.34
suggests that you have some potential options if you're using Unix -
of course, those options would be best discussed on
comp.unix.programmer.

--
C. Benson Manica | I *should* know what I'm talking about - if I
cbmanica(at)gmail.com | don't, I need to know. Flames welcome.

[Jordan Abel]

2006-10-25 <eh**********@naig.caltech.edu>,
David Mathog wrote:

Eric Sosman wrote:

> Here's a suggestion: You're trying to obtain the effect
of fclose(), right? Does any particular function spring to
mind as being likely to perform the operations of fclose()?
How about ... <<wait for it>... fclose()?

I'm not trying to get fclose, I'm trying to get

fclose()
fopen()

How then does the program CONTINUE to write to stdout
so that data gets to the tape device after having called
fclose(stdout)?

I think his question is, basically. WHY do you need to fclose() in the
first place? What's the fclose() for, if not to have the stream be
permanently closed i.e. completely done with writing data to it?

[David Mathog]

Christopher Benson-Manica wrote:

David Mathog <ma****@caltech.eduwrote:

>What parameters for fopen() reassociate stdout with the same
output stream following the fclose()?

It sounds like what you want to do is fclose(stdout) and then somehow
open it again; while not an exact match for your situation, FAQ 12.34
(http://c-faq.com/stdio/undofreopen.html) suggests strongly to me that
once you fclose(stdout) you are on your own getting it open again.
I don't believe you stated what your system setup is, but 12.34
suggests that you have some potential options if you're using Unix -
of course, those options would be best discussed on
comp.unix.programmer.

Yes, that does suggest that what I'm after is not supported by the C
language standard. Actually I didn't want so much to fclose it as to
set EOF, but there was no other way to do it besides calling fclose().

It is a bit odd that ANSI C provides the functions clearerr()
and feof(), to clear and test the EOF status on a file, but
it does not provide conjugate seteof() or seterr() functions.
These would have allowed a couple of bits of out of band
communication to ride along on a binary data stream. That's
a pity because in the general unixy pipeline processing
of an arbitrary binary data stream

program1 | program2

there is currently no way to signal through that same pipe any other
information, in particular, EOF. Open another pipe and you must then
take pains to make sure the data streams stay in sync. The
safest way to process multiple independent
blocks of binary information with this pipe is to run it multiple times
with separate data, as opposed to running it once and stuffing EOF
states into the pipe in program1 with:

seteof(stdout)

which could then be caught in program2 which could then issue
a cleareof(stdin) to open up the pipe again.

Thanks,

David Mathog

[David Mathog]

Jordan Abel wrote:

2006-10-25 <eh**********@naig.caltech.edu>,
David Mathog wrote:

>I'm not trying to get fclose, I'm trying to get

fclose()
fopen()

I think his question is, basically. WHY do you need to fclose() in the
first place? What's the fclose() for, if not to have the stream be
permanently closed i.e. completely done with writing data to it?

See the first two paragraphs of the top post.

David Mathog

[Chris Torek]

In article <eh**********@naig.caltech.edu>
David Mathog <ma****@caltech.eduwrote:

>A program of mine writes to a tape unit. Output can be either through
stdout or through a file opened with fopen(). When all the data is
transferred to tape the program needs to close the output stream so that
the tape driver will write a filemark on the tape. Otherwise multiple
clumps of data saved to tape would all appear to be one big file on the
tape.

When the tape unit device was explicitly opened with fopen()
that's possible: call fclose() and then for the next batch
of data fopen() the tape device again and write some more.

However when data is going through stdout like:

program /dev/nst0

is there an equivalent operation?

Not only is this impossible in Standard C, it is not even possible
on most POSIX systems. You can *close* the standard output stream
with:

fclose(stdout);

but you can never guarantee to get it open again. That is, if this
fclose succeeds, stdout is dead. (The next fopen() call may in
fact re-use the old stdout "FILE *", effectively resurrecting it,
but it also might not.)

By cheating -- going directly to POSIX file descriptors -- you can
avoid "killing" the C library stdout stream, while closing the
underlying file descriptor. For instance, something like this:

fflush(stdout); /* pass output to underlying fd */
close(STDOUT_FILENO); /* POSIX-specific: close stdout */
fd = open(path, openmode, creatmode_opt); /* POSIX-specific */
if (fd != STDOUT_FILENO) ... uh oh ...

(which relies on STDOUT_FILENO being the first available one so
that open() uses it), but you will need to find the "path" (by
"magic"; see below). Moreover, with this method -- just as with
the fopen(), fclose(), fopen() again method -- there is no guarantee
that the same pathname refers to the same physical device twice in
a row (consider what happens if the super-user renames /dev/nst0
to /dev/nst0.old and creates a new /dev/nst0, for instance).

Your best best for "truly safe" operation, given the non-portability
of the whole thing, is to use a system-specific operation (probably
an ioctl) to write tape-marks, without ever closing the device.
If that is unavailable or impractical, your second-best-bet is to
require the path-name; "recovering" the device name (by using,
e.g., fts_open() on "/dev" and comparing "st_dev" IDs in "stat"
structures) is possible but tricky.
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) [phone]
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.

[Eric Sosman]

David Mathog wrote On 10/25/06 13:28,:

Eric Sosman wrote:

> Here's a suggestion: You're trying to obtain the effect
of fclose(), right? Does any particular function spring to
mind as being likely to perform the operations of fclose()?
How about ... <<wait for it>... fclose()?

I'm not trying to get fclose, I'm trying to get

fclose()
fopen()

Aha! Okay, sorry -- that wasn't clear to me.

How then does the program CONTINUE to write to stdout
so that data gets to the tape device after having called
fclose(stdout)?

What parameters for fopen() reassociate stdout with the same
output stream following the fclose()?

I don't think there's any way to do this in Standard C's
very simple model of I/O. In fact, I don't think what you're
doing with an explicit device name is guaranteed to work: C
only understands "files," not multi-file "volumes."

--
Er*********@sun.com

[Eric Sosman]

Harald van Dijk wrote On 10/25/06 13:29,:

Eric Sosman wrote:

>>David Mathog wrote On 10/25/06 12:48,:

>> freopen(NULL,"wb",stdout);

Undefined behavior: The first argument is supposed to
be a string (a string that names a file), but NULL is not
a string. It's much like trying fopen(NULL, "wb").

Actually, freopen() has special behaviour for a NULL filename. I don't
know enough about it to know if that special behaviour is useful here,
though.

Oh, drat! Serves me right for looking at a man page
instead of checking the Standard. The behavior is not
undefined, but implementation-defined .. and as far as
I can tell, it doesn't help the O.P.

> Here's a suggestion: You're trying to obtain the effect
of fclose(), right? Does any particular function spring to
mind as being likely to perform the operations of fclose()?
How about ... <<wait for it>... fclose()?

So how do you write to stdout again after closing it?

You don't. It's closed. (I had not realized that he
wanted to keep on writing until he counter-replied to my
reply. It was all there in the original message, if I'd
squinted at it hard enough -- not my day, evidently ...)

--
Er*********@sun.com

[CBFalconer]

David Mathog wrote:

Eric Sosman wrote:

> Here's a suggestion: You're trying to obtain the effect
of fclose(), right? Does any particular function spring to
mind as being likely to perform the operations of fclose()?
How about ... <<wait for it>... fclose()?

I'm not trying to get fclose, I'm trying to get

fclose()
fopen()

How then does the program CONTINUE to write to stdout
so that data gets to the tape device after having called
fclose(stdout)?

What parameters for fopen() reassociate stdout with the same
output stream following the fclose()?

No guaranteed way. Read the manuals for your system. Most tape
units have some sort of control channel that can be told "Now write
an inter-file gap". Since this is system specific it is off-topic
here, where we deal in portable C programming. I have no idea what
your system is and thus no idea what newsgroup would be
appropriate.

--
Chuck F (cbfalconer at maineline dot net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net>

[Serve Laurijssen]

"David Mathog" <ma****@caltech.eduwrote in message
news:eh**********@naig.caltech.edu...

>A program of mine writes to a tape unit. Output can be either through
stdout or through a file opened with fopen(). When all the data is
transferred to tape the program needs to close the output stream so that
the tape driver will write a filemark on the tape.

what part of fclose exactly triggers the filemark writing? You probably
tried it, but it might be as simple as fflush

[Rod Pemberton]

"David Mathog" <ma****@caltech.eduwrote in message
news:eh**********@naig.caltech.edu...

Eric Sosman wrote:

Here's a suggestion: You're trying to obtain the effect
of fclose(), right? Does any particular function spring to
mind as being likely to perform the operations of fclose()?
How about ... <<wait for it>... fclose()?

I'm not trying to get fclose, I'm trying to get

fclose()
fopen()

How then does the program CONTINUE to write to stdout
so that data gets to the tape device after having called
fclose(stdout)?

What parameters for fopen() reassociate stdout with the same
output stream following the fclose()?

I'm a bit unfamiliar with pipes, but, won't closing stdout from your program
also close the pipe? How do you intend to keep the pipe open through an
open and close of stdout? I suspect this is a second issue no one else has
mentioned.

I think your best bet is to not use a pipe at all, and replace the fclose();
fopen() sequence with a routine specific to the tape drive which writes the
control data, (as Falconer suggested).

Another option would be to have a single control program which spawn()'s or
exec()'s, etc., a separate program which writes one file at a time. The
command line for the spawn'd or exec'd program could use a pipe, if desired.
Rod Pemberton

[David Mathog]

Serve Laurijssen wrote:

"David Mathog" <ma****@caltech.eduwrote in message
news:eh**********@naig.caltech.edu...

>A program of mine writes to a tape unit. Output can be either through
stdout or through a file opened with fopen(). When all the data is
transferred to tape the program needs to close the output stream so that
the tape driver will write a filemark on the tape.

what part of fclose exactly triggers the filemark writing? You probably
tried it, but it might be as simple as fflush

When the linux st driver receives a close operation following a write it
(the st driver) writes the filemark. The fclose() and fflush() don't
write that mark themselves. Apparently the only way to write multiple
files to tape through stdout, and to place a filemark between them,
is to use ioctl(). Even that is tricky, as doing something like:

fwrite(buffer1,BUFSIZE,1,stdout);
ioctl( <appropriate parameters);
fwrite(buffer2,BUFSIZE,1,stdout);

might result in the filemarks being written before all of the buffer1
data gets to the tape drive.

Thanks all,

David Mathog

[Keith Thompson]

David Mathog <ma****@caltech.eduwrites:

Serve Laurijssen wrote:

[...]

>what part of fclose exactly triggers the filemark writing? You
probably tried it, but it might be as simple as fflush

When the linux st driver receives a close operation following a write it
(the st driver) writes the filemark. The fclose() and fflush() don't
write that mark themselves. Apparently the only way to write multiple
files to tape through stdout, and to place a filemark between them,
is to use ioctl(). Even that is tricky, as doing something like:

fwrite(buffer1,BUFSIZE,1,stdout);
ioctl( <appropriate parameters);
fwrite(buffer2,BUFSIZE,1,stdout);

might result in the filemarks being written before all of the buffer1
data gets to the tape drive.

<SEMI-OT>

Perhaps this would work:

fwrite(buffer1,BUFSIZE,1,stdout);
fflush(stdout);
ioctl( <appropriate parameters);
fwrite(buffer2,BUFSIZE,1,stdout);

</SEMI-OT>

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[Walter Roberson]

In article <eh**********@news2.zwoll1.ov.home.nl>,
Serve Laurijssen <se*@n.tkwrote:

>"David Mathog" <ma****@caltech.eduwrote in message
news:eh**********@naig.caltech.edu...

>>A program of mine writes to a tape unit. Output can be either through
stdout or through a file opened with fopen(). When all the data is
transferred to tape the program needs to close the output stream so that
the tape driver will write a filemark on the tape.

>what part of fclose exactly triggers the filemark writing? You probably
tried it, but it might be as simple as fflush

As others have noted, it is part of the device driver, not a standard
C function.

Someone referenced a Linux ioctl(); that's certainly a possibility on
appropriate versions of Linux.

As additional off-topic information about the difficulty of doing this:
On SCSI tape devices, the end-of-file marker or end-of-medium markers
are handled by specific operation codes. If your device driver does not
provide an ioctl() or equivilent, then you may have to issue the
SCSI-layer command yourself, if your system provides a generalized
SCSI device driver.

Also it might be worth keeping in mind that most Unix-type systems
provide a command "mt" with a "weof" option. But then there's the
difficulty that "mt" wants the name of a tape device on its command
line, and if you are using stdout then you don't know the tape device
name. Since you'd be presuming a unix-type system for this purpose,
you could fstat() to find the major and minor device numbers
associated with stdout, and iterate through the standard tape drive
name /dev subdirectory and hope for a match... and then mangle that
name to ensure that you include the appropriate name code for
non-rewind.

I wonder if the OP took into account that if the invoking user
redirected to an autorewind tape device (as is likely if they
hadn't been hit over the head several times with this issue),
that the fclose() equivilent that is proposed to be done would
trigger a tape rewind...
How would I handle it? Well, I would code the application to
take a tape device name as a parameter, and write to that instead
of stdout; perhaps I'd allow the semi-standard use of a single dash
to indicate standard output, but in that case I'd have the program
balk if it would have been expected to write several file marks.

No matter what you know about your own system, I don't think you
can reasonably be expected to be able to write multiple file marks
in a scenario such as

$ OurTapeBackups basedirectory | \
ssh -c operator@remotehost "dd of=/dev/tape"
--
"law -- it's a commodity"
-- Andrew Ryan (The Globe and Mail, 2005/11/26)

[CBFalconer]

Walter Roberson wrote:

>

.... snip ...

>
I wonder if the OP took into account that if the invoking user
redirected to an autorewind tape device (as is likely if they
hadn't been hit over the head several times with this issue),
that the fclose() equivilent that is proposed to be done would
trigger a tape rewind...

In which case he might be surprised at how little tape is needed
for the complete dump.

--
Chuck F (cbfalconer at maineline dot net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net>

[Jordan Abel]

2006-10-26 <eh*********@naig.caltech.edu>,
David Mathog wrote:

When the linux st driver receives a close operation following a write it
(the st driver) writes the filemark. The fclose() and fflush() don't
write that mark themselves. Apparently the only way to write multiple
files to tape through stdout, and to place a filemark between them,
is to use ioctl(). Even that is tricky, as doing something like:

fwrite(buffer1,BUFSIZE,1,stdout);

fflush(stdout) before mixing stdio and low-level

ioctl( <appropriate parameters);
fwrite(buffer2,BUFSIZE,1,stdout);

might result in the filemarks being written before all of the buffer1
data gets to the tape drive.

solved above.