Binary-mode i/o, width of char, endianness

[T Koster]

Hi group,

I'm having some difficulty figuring out the most portable way to read 24
bits from a file. This is related to a Base-64 encoding.

The file is opened in binary mode, and I'm using fread to read three
bytes from it. The question is though, where should fread put this? I
have considered two alternatives, but neither seem like a good idea:

In most cases, the width of a char is 8 bits, so an array of 3 chars
would suffice, but the width of a char is guaranteed to be only *at
least* 8 bits, so the actual number of chars required would be 24 /
CHAR_BIT, rounded up. Since you can't round in a constant integral
expression, 3 chars is a good safe buffer size because it's guaranteed
to be at least 24 bits. However, since I need to be able to divide
those 24 bits into four 6-bit numbers, indices into the char array
become more complicated as the 6-bit numbers do not fall evenly on the
(presumably) 8-bit boundaries that indexes in the array would give me.
If the width of a char is not 8 bits, then knowing which indices to look
at and shift/mask is even more difficult. As such, I thought of the
second option.

The second option is to allocate the input buffer as simply one int
object that is guaranteed to be at least 24 bits wide: the long int,
which even has 8 bytes to spare. fread can safely write 3 bytes of data
into a long int. I only have worries that because a long int is a
multi-byte integer, accessing various parts of it is dangerous due to
endianness considerations, or is endianness only relevant to the
represented *value* of the multi-byte integer as a whole? fread doesn't
care about that: it writes three bytes into the address of the long int,
starting at the lowest-positioned byte, but would the shifting/masking
be portable? For example a multi-byte integer constant 0x1234 has a
most-significant byte of value 0x12, but on a big-endian machine would
be stored on the *lowest* memory address of the space it takes up. As
such, the mask required to leave only the *lowest* 6 bits of a 32-bit
integer could be either 0x3F000000 or 0x0000003F depending on
endianness, right? Or are hexadecimal integer constants always stored
as-is? That is, the lowest byte is positioned last in an integer
constant instead of the least significant byte positioned last? This
seems counter-intuitive.

If neither of these options is good, is there another way?

Thanks in advance,
Thomas

[infobahn]

T Koster wrote:

Hi group,

I'm having some difficulty figuring out the most portable way to read 24
bits from a file. This is related to a Base-64 encoding.

The file is opened in binary mode, and I'm using fread to read three
bytes from it. The question is though, where should fread put this? I
have considered two alternatives, but neither seem like a good idea:

In most cases, the width of a char is 8 bits, so an array of 3 chars
would suffice, but the width of a char is guaranteed to be only *at
least* 8 bits, so the actual number of chars required would be 24 /
CHAR_BIT, rounded up. Since you can't round in a constant integral
expression, 3 chars is a good safe buffer size because it's guaranteed
to be at least 24 bits.
To store BITS bits, you need at least (BITS + CHAR_BIT - 1) / CHAR_BIT
bytes. If BITS is constant:

#define BITS 24

then:

unsigned char buf[(BITS + CHAR_BIT - 1) / CHAR_BIT] = {0};

is legal.
However, since I need to be able to divide
those 24 bits into four 6-bit numbers, indices into the char array
become more complicated as the 6-bit numbers do not fall evenly on the
(presumably) 8-bit boundaries that indexes in the array would give me.
So you need to mask and shift. If we assume that each octet of data
is stored in a separate byte, then this isn't as hard as it sounds.

/* 1. get bits 7 through 2 of first octet */
num[0] = (buf[0] & 0xFC) >> 2;
/* 2. get bits 1 and 0 of first octet, and bits 7 through 4 of
second octet */
num[1] = ((buf[0] & 0x03) << 6) | ((buf[1] & 0xF0) >> 4);

etc.
If the width of a char is not 8 bits, then knowing which indices to look
at and shift/mask is even more difficult.
See above if they're spread out, with 8 value bits to each byte
(the remaining bits being unused). If they're packed in, you just
have to be a little clever with CHAR_BIT. Once you start to analyse
this problem, you'll see that it isn't as hard as it sounds.
As such, I thought of the
second option.

The second option is to allocate the input buffer as simply one int
object that is guaranteed to be at least 24 bits wide: the long int,
which even has 8 bytes to spare.
Well, at least 8 *bits* to spare. :-)
fread can safely write 3 bytes of data
into a long int.

Not necessarily. On platforms such as the kind you are worrying about
(CHAR_BIT > 8), long int may well be fewer than four bytes wide!

Consider a platform with 11-bit bytes. On such a platform, long ints
may only occupy 3 bytes. On (perhaps more common) platforms with
16-bit or 32-bit bytes, long int may be only 2 bytes, or even 1 byte.

I would stick to unsigned char for this project. Long ints will
multiply your headaches, divide your attention, add to your
worries, and subtract from your understanding (modulo their
day-to-day uses, obviously).

[T Koster]

infobahn wrote:

T Koster wrote:

I'm having some difficulty figuring out the most portable way to read 24
bits from a file. This is related to a Base-64 encoding.

The file is opened in binary mode, and I'm using fread to read three
bytes from it. The question is though, where should fread put this? I
have considered two alternatives, but neither seem like a good idea:

In most cases, the width of a char is 8 bits, so an array of 3 chars
would suffice, but the width of a char is guaranteed to be only *at
least* 8 bits, so the actual number of chars required would be 24 /
CHAR_BIT, rounded up. Since you can't round in a constant integral
expression, 3 chars is a good safe buffer size because it's guaranteed
to be at least 24 bits.
To store BITS bits, you need at least (BITS + CHAR_BIT - 1) / CHAR_BIT
bytes. If BITS is constant:

#define BITS 24

then:

unsigned char buf[(BITS + CHAR_BIT - 1) / CHAR_BIT] = {0};

is legal.

Ahh, good idea.

However, since I need to be able to divide
those 24 bits into four 6-bit numbers, indices into the char array
become more complicated as the 6-bit numbers do not fall evenly on the
(presumably ) 8-bit boundaries that indexes in the array would give me.

So you need to mask and shift. If we assume that each octet of data
is stored in a separate byte, then this isn't as hard as it sounds.

/* 1. get bits 7 through 2 of first octet */
num[0] = (buf[0] & 0xFC) >> 2;
/* 2. get bits 1 and 0 of first octet, and bits 7 through 4 of
second octet */
num[1] = ((buf[0] & 0x03) << 6) | ((buf[1] & 0xF0) >> 4);

etc.

If the width of a char is not 8 bits, then knowing which indices to look
at and shift/mask is even more difficult.

See above if they're spread out, with 8 value bits to each byte
(the remaining bits being unused). If they're packed in, you just
have to be a little clever with CHAR_BIT. Once you start to analyse
this problem, you'll see that it isn't as hard as it sounds.

We seem to be using the term 'byte' with different meanings...see below.

As such, I thought of the
second option.

The second option is to allocate the input buffer as simply one int
object that is guaranteed to be at least 24 bits wide: the long int,
which even has 8 bytes to spare.

Well, at least 8 *bits* to spare. :-)

Certainly :)

fread can safely write 3 bytes of data
into a long int.

Not necessarily. On platforms such as the kind you are worrying about
(CHAR_BIT > 8), long int may well be fewer than four bytes wide!

Consider a platform with 11-bit bytes. On such a platform, long ints
may only occupy 3 bytes. On (perhaps more common) platforms with
16-bit or 32-bit bytes, long int may be only 2 bytes, or even 1 byte.

Hmmm, this appears to be becoming a question of terminology. I thought
that by definition, one byte is eight bits wide. I'm not using the C
type 'char' interchangably with 'an int that is one _byte_ big'. When I
consider that CHAR_BIT may be greater than 8, I mean exactly that, and
not that a byte of storage on this platform has more than eight bits,
since I thought that was nonsense. That is, a char may occupy more than
one byte of storage, but a byte is still an 8-bit byte. Calling fread
and asking for three bytes implies that 24 bits will be read,
irrespective of platform, correct? As such, a long int, being
guaranteed to have at least 32 bits, is guaranteed to occupy at least
four bytes of storage, which is why I say that fread can safely store
three bytes (24 bits by definition) in a long int. Correct me if I'm
wrong here.
I would stick to unsigned char for this project. Long ints will
multiply your headaches, divide your attention, add to your
worries, and subtract from your understanding (modulo their
day-to-day uses, obviously).

Thanks,
Thomas.

[infobahn]

T Koster wrote:

infobahn wrote:

T Koster wrote:

fread can safely write 3 bytes of data
into a long int.
Not necessarily. On platforms such as the kind you are worrying about
(CHAR_BIT > 8), long int may well be fewer than four bytes wide!
>
Consider a platform with 11-bit bytes. On such a platform, long ints
may only occupy 3 bytes. On (perhaps more common) platforms with
16-bit or 32-bit bytes, long int may be only 2 bytes, or even 1 byte.

Hmmm, this appears to be becoming a question of terminology. I thought
that by definition, one byte is eight bits wide.

Not in comp.lang.c, it isn't, because ISO C recognises that it
simply isn't true on some platforms.
I'm not using the C
type 'char' interchangably with 'an int that is one _byte_ big'. When I
consider that CHAR_BIT may be greater than 8, I mean exactly that, and
not that a byte of storage on this platform has more than eight bits,
since I thought that was nonsense.
Many people think that, and many people are wrong. If CHAR_BIT is
greater than 8, it is because bytes are greater than 8 bits wide
for that implementation.
That is, a char may occupy more than
one byte of storage, but a byte is still an 8-bit byte.
In C, by definition, a char is exactly one byte in size.
sizeof(char) always yields 1 as its value. But chars can
be wider than 8 bits. If they are, then so are bytes.
Calling fread
and asking for three bytes implies that 24 bits will be read,
irrespective of platform, correct?
Nope. Consider a typical modern DSP. You might get anything from
48 to 96 bits! (Maybe even more, nowadays.)
As such, a long int, being
guaranteed to have at least 32 bits, is guaranteed to occupy at least
four bytes of storage, which is why I say that fread can safely store
three bytes (24 bits by definition) in a long int. Correct me if I'm
wrong here.

You can certainly guarantee to get 24 bits into a long int, yes.
But you might not need three bytes to do it in, and if you read
three bytes you might end up with more than you can chew.

(Pun definitely intended.)

[Lew Pitcher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

T Koster wrote:

infobahn wrote: [snip]

Consider a platform with 11-bit bytes. On such a platform, long ints
may only occupy 3 bytes. On (perhaps more common) platforms with
16-bit or 32-bit bytes, long int may be only 2 bytes, or even 1 byte.

Hmmm, this appears to be becoming a question of terminology.

Yes. This is one of those difference in terms that catches a lot of people.
I thought
that by definition, one byte is eight bits wide.
Not in the C language, although your observation holds true in many
other areas of computing.

Here, a byte is as wide as a byte is wide, and that is 8 bits /or more/.
I'm not using the C
type 'char' interchangably with 'an int that is one _byte_ big'.
A C char data type is, by definition, one byte big. A byte, however, has
no defined size, other than a minimum size of 8 bits. A byte (and hence
a char) could be 9 bits wide, or 32 bits wide, or even 64 bits wide,
depending on the implementation.
When I
consider that CHAR_BIT may be greater than 8, I mean exactly that,
As do we
and
not that a byte of storage on this platform has more than eight bits,
There may be a difference between the machine's minimum unit of
addressable memory and the minimum unit of addressable memory that the C
'virtual machine' recognizes. Here, in comp.lang.c, the C 'virtual
machine' interpretation wins, and a byte is CHAR_BIT bits long, no
matter how the physical machine implements it's load and store
instructions. Think of it this way, in C, you can't get any smaller a
single unit of memory than a char, so a char /is/ the minimum unit of
addressable memory.
since I thought that was nonsense.
I'll let this pass :-)
That is, a char may occupy more than
one byte of storage, but a byte is still an 8-bit byte.
Difference in terms. A byte is CHAR_BIT bits long, and that can be more
than 8 bits.
Calling fread
and asking for three bytes implies that 24 bits will be read,
irrespective of platform, correct?
Nope. Calling fread(,3,1,) (one element, 3 char in length) will read one
block of 3 C char elements. If each C char element is 24 bits long
(CHAR_BIT == 24), then you get 72 bits worth of data in one fread().
As such, a long int, being
guaranteed to have at least 32 bits, is guaranteed to occupy at least
four bytes of storage,
No, is guaranteed to occupy at least CHAR_BIT * sizeof(long int) bits of
storage, or sizeof(long int) bytes of storage.
which is why I say that fread can safely store
three bytes (24 bits by definition) in a long int.
Correct me if I'm wrong here.

See above.
- --

Lew Pitcher, IT Specialist, Enterprise Data Systems
Enterprise Technology Solutions, TD Bank Financial Group

(Opinions expressed here are my own, not my employer's)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (MingW32)

iD8DBQFCJHGHagV FX4UWr64RAvZUAJ 9kwiCwxKiEY5RJI fHSwpMc75u+zQCf bDbx
Ezd/+ESWGlZovKGmMNo/DzQ=
=2v7/
-----END PGP SIGNATURE-----

[Keith Thompson]

infobahn <in******@btint ernet.com> writes:

T Koster wrote:

[...]

Hmmm, this appears to be becoming a question of terminology. I thought
that by definition, one byte is eight bits wide.

Not in comp.lang.c, it isn't, because ISO C recognises that it
simply isn't true on some platforms.

I'm not using the C
type 'char' interchangably with 'an int that is one _byte_ big'. When I
consider that CHAR_BIT may be greater than 8, I mean exactly that, and
not that a byte of storage on this platform has more than eight bits,
since I thought that was nonsense.

Many people think that, and many people are wrong. If CHAR_BIT is
greater than 8, it is because bytes are greater than 8 bits wide
for that implementation.

That's not quite the way I'd put it. The fact that a "byte" is
defined to be CHAR_BIT bits isn't necessarily something imposed by the
underlying platform; it's a choice that can be made by the author of
the C implementation. It's about the definition of the word "byte",
not necessarily about any inherent attribute of the platform.

For example, I've worked on systems where the natural fundamental unit
of storage is 64 bits, but the C implementation defines CHAR_BIT==8.
It has to jump through some hoops to make this work; the advantage is
compatibility with other systems.

The C standard could have defined the term "byte" as a unit of storage
consisting of 8 bits.

The end result is the same, though. Out in the real world a "byte" is
almost always 8 bits ("octet" is an unambiguous term for this). In
comp.lang.c, a "byte" is always exactly CHAR_BIT bits, and maximally
portable code must not assume that CHAR_BIT==8.

--
Keith Thompson (The_Other_Keit h) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.

[infobahn]

Keith Thompson wrote:

infobahn <in******@btint ernet.com> writes:

If CHAR_BIT is
greater than 8, it is because bytes are greater than 8 bits wide
for that implementation.
That's not quite the way I'd put it. The fact that a "byte" is
defined to be CHAR_BIT bits isn't necessarily something imposed by the
underlying platform; it's a choice that can be made by the author of
the C implementation.

That's why I said "implementation ", not "platform".
It's about the definition of the word "byte",
not necessarily about any inherent attribute of the platform.
That's why I said "implementation ", not "platform".
For example, I've worked on systems where the natural fundamental unit
of storage is 64 bits, but the C implementation defines CHAR_BIT==8.
It has to jump through some hoops to make this work; the advantage is
compatibility with other systems.

Cray, by any chance?

[T Koster]

infobahn wrote:

T Koster wrote:

infobahn wrote:

T Koster wrote:
fread can safely write 3 bytes of data
into a long int.

Not necessarily. On platforms such as the kind you are worrying about
(CHAR_BIT > 8), long int may well be fewer than four bytes wide!

Consider a platform with 11-bit bytes. On such a platform, long ints
may only occupy 3 bytes. On (perhaps more common) platforms with
16-bit or 32-bit bytes, long int may be only 2 bytes, or even 1 byte.

Hmmm, this appears to be becoming a question of terminology. I thought
that by definition, one byte is eight bits wide.

Not in comp.lang.c, it isn't, because ISO C recognises that it
simply isn't true on some platforms.

I'm not using the C
type 'char' interchangably with 'an int that is one _byte_ big'. When I
consider that CHAR_BIT may be greater than 8, I mean exactly that, and
not that a byte of storage on this platform has more than eight bits,
since I thought that was nonsense.

Many people think that, and many people are wrong. If CHAR_BIT is
greater than 8, it is because bytes are greater than 8 bits wide
for that implementation.

That is, a char may occupy more than
one byte of storage, but a byte is still an 8-bit byte.

In C, by definition, a char is exactly one byte in size.
sizeof(char) always yields 1 as its value. But chars can
be wider than 8 bits. If they are, then so are bytes.

Calling fread
and asking for three bytes implies that 24 bits will be read,
irrespectiv e of platform, correct?

Nope. Consider a typical modern DSP. You might get anything from
48 to 96 bits! (Maybe even more, nowadays.)

As such, a long int, being
guaranteed to have at least 32 bits, is guaranteed to occupy at least
four bytes of storage, which is why I say that fread can safely store
three bytes (24 bits by definition) in a long int. Correct me if I'm
wrong here.

You can certainly guarantee to get 24 bits into a long int, yes.
But you might not need three bytes to do it in, and if you read
three bytes you might end up with more than you can chew.

(Pun definitely intended.)

Okay okay, so it is impossible to portably tell fread to read exactly 24
bits of data, since you can only ask it to give you an integer number of
bytes, however big they may be on the particular implementation.

As such, should I allocate a larger buffer, of a fixed number of bytes
(regardless of how wide a byte is), and get fread to read in that fixed
number of bytes, and then do the shifting/masking as appropriate
depending on CHAR_BIT, reading more data as needed?

Also, will I need to generate the masks as expressions in terms of
CHAR_BIT, rather than a hexadecimal integer constant? The mask 0xFC
(for the first six bits) defines only 8 bits worth of a char, which we
know can be wider than 8 bits.
Would something like
x[0] = (buf[0] >> (CHAR_BIT - 6)) & 63;
be more appropriate? What about the next six bits? With an 8-bit char,
four bits will have to come from buf[1], but with a 12-bit-or-more char
buf[0] has all we need. In general, I guess since a char is at least
eight bits wide, we only need to examine at most two chars. I'm just
trying to figure out some sort of expression for x[n] (any nth 6-bit
number contained in the 'stream' of chars). I'll have to think about
this one.

Thanks,
Thomas

[Walter Roberson]

In article <S7************ ********@news-server.bigpond. net.au>,
T Koster <re************ @use.net> wrote:
:Okay okay, so it is impossible to portably tell fread to read exactly 24
:bits of data, since you can only ask it to give you an integer number of
:bytes, however big they may be on the particular implementation.

:As such, should I allocate a larger buffer, of a fixed number of bytes
:(regardless of how wide a byte is), and get fread to read in that fixed
:number of bytes, and then do the shifting/masking as appropriate
:depending on CHAR_BIT, reading more data as needed?

You could find the lowest common multiple of 24 and CHAR_BIT
(e.g., 72 for CHAR_BIT==9, 120 for CHAR_BIT==10, 282 for CHAR_BIT==11,
24 for CHAR_BIT==12). Divide that by CHAR_BIT and you get the number
of bytes you need to read at a time in order to be able to unpack
on exact boundaries.
:I'm just
:trying to figure out some sort of expression for x[n] (any nth 6-bit
:number contained in the 'stream' of chars).

if ( (n * 6) % CHAR_BIT < 6 ) then you need to span bytes.
--
Beware of bugs in the above code; I have only proved it correct,
not tried it. -- Donald Knuth

[Keith Thompson]

T Koster <re************ @use.net> writes:
[...]

Okay okay, so it is impossible to portably tell fread to read exactly
24 bits of data, since you can only ask it to give you an integer
number of bytes, however big they may be on the particular
implementation.

As such, should I allocate a larger buffer, of a fixed number of bytes
(regardless of how wide a byte is), and get fread to read in that
fixed number of bytes, and then do the shifting/masking as appropriate
depending on CHAR_BIT, reading more data as needed?

[...]

I wonder how much effort this is worth. I don't know of any hosted C
implementations with CHAR_BIT != 8 (and freestanding implementations
don't necessarily support fread).

If you want absolute portability, you need to write your code so it
works on systems with CHAR_BIT != 8. But then it's not clear what a
data file would look like after being copied to such a system.
Whatever mechanism is used to copy the file will need to deal with the
difference in byte sizes, and there are a number of ways it could do
so. If CHAR_BIT % 8 == 0, it could pack multiple octets into each
byte. If not, it could either treat the file as a stream of bits
(with some octets crossing byte boundaries), or it could just store
each octet in a byte and zero out the high-order bits.

It might be sufficient to put the following code in an application
header file:

#include <limits.h>
#if CHAR_BIT != 8
#error This application is supported only on systems with CHAR_BIT==8
#endif

On the other hand, if this is intended as an exercise in theoretical
manic portability, have fun!

--
Keith Thompson (The_Other_Keit h) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.