converting char to float (reading binary data from file)

[itdevries]

Hi,
I'm trying to convert some char data I read from a binary file (using
ifstream) to a float type. I've managed to convert the int types but
now I need to do the float types as well but it doesn't seem to work.
The code below is what I'm trying to use. Anyone see any obvious
errors? or have any hints/pointers?
regards,
Igor

float floatRead = 0;

UINT32* ptr = (UINT32 *) (&floatRead);

int offset = 0;

for (int ii=startInd; ii<=endInd; ii++){
*ptr |= (charBuf[ii] << offset);
offset += 8;
};

[itdevries]

On May 21, 11:29*am, Michael DOUBEZ <[e-mail]:

itdevries a écrit :

I'm trying to convert some char data I read from a binary file (using
ifstream) to a float type. I've managed to convert the int types but
now I need to do the float types as well but it doesn't seem to work.
The code below is what I'm trying to use. Anyone see any obvious
errors? or have any hints/pointers?

I suppose charBuf is of the type char[] ? In this case 'charBuf[ii] <<
offset' is 0 as soon as offset>=8 so you get only 0s.

float floatRead = 0;

UINT32* * *ptr *= (UINT32 *) (&floatRead);

int offset * = 0;

for (int ii=startInd; ii<=endInd; ii++){
* *ptr |= (charBuf[ii] << offset);
* offset *+= 8;
};

Your use of bitwise operator looks clumsy unless you have some logic to
handle different byte ordering.

What's wrong with
memcpy(&floatRead,charBuf+startInd,sizeof(floatRea d));
?

--
Michael

thanks, that works... however, I don't understand what's wrong with my
original code. any ideas?
Igor

[itdevries]

On May 21, 12:23*pm, Michael DOUBEZ <[e-mail]:

itdevries a écrit :

On May 21, 11:29 am, Michael DOUBEZ <[e-mail]:

itdevries a écrit :
float floatRead = 0;
UINT32* * *ptr *= (UINT32 *) (&floatRead);
int offset * = 0;
for (int ii=startInd; ii<=endInd; ii++){
* *ptr |= (charBuf[ii] << offset);
* offset *+= 8;
};

I don't understand what's wrong with my
original code. any ideas?

Unrolling your loop:
*ptr |= (charBuf[startInd+0] << *0);
*ptr |= (charBuf[startInd+1] << *8);
*ptr |= (charBuf[startInd+2] << 16);
*ptr |= (charBuf[startInd+3] << 24);

Becomes (as I have mentionned):
*ptr |= charBuf[startInd];
*ptr |= 0;
*ptr |= 0;
*ptr |= 0;

Because you shift-left a char more times than its size in bits (usually
8) so it becomes 0.

Example:
char x=0xBF;
assert( (x<< 8) == 0);
assert( (x<<12) == 0);
assert( (x<<42) == 0);

--
Michael

Thanks so much for taking the time to respond. I understand the logic
you're using and it was one of the initial concerns I had with the
code, however it seemed to work fine for int types (maybe by
coincidence) so I thought it would work for float types as well. One
thing I don't understand however is that when I step through the loop
with the debugger I see the value of the float change even though from
step 2 I thought I'd be doing "*ptr |= 0" which I thought shouldn't
alter the value of the float. that lead me to the conclusion that the
bitwise shift worked differently from what I expected.
Igor

[James Kanze]

On 21 mai, 11:29, Michael DOUBEZ <[e-mail]:

itdevries a écrit :

I'm trying to convert some char data I read from a binary file (using
ifstream) to a float type. I've managed to convert the int types but
now I need to do the float types as well but it doesn't seem to work.
The code below is what I'm trying to use. Anyone see any obvious
errors? or have any hints/pointers?

I suppose charBuf is of the type char[] ? In this case 'charBuf[ii] <<
offset' is 0 as soon as offset>=8 so you get only 0s.

float floatRead = 0;

UINT32* ptr = (UINT32 *) (&floatRead);

int offset = 0;

for (int ii=startInd; ii<=endInd; ii++){
*ptr |= (charBuf[ii] << offset);
offset += 8;
};

Your use of bitwise operator looks clumsy unless you have some logic to
handle different byte ordering.

What's wrong with
memcpy(&floatRead,charBuf+startInd,sizeof(floatRea d));
?

--
Michael

[James Kanze]

On 21 mai, 10:46, itdevries <[e-mail]:

Hi,
I'm trying to convert some char data I read from a binary file (using
ifstream) to a float type. I've managed to convert the int types but
now I need to do the float types as well but it doesn't seem to work.
The code below is what I'm trying to use. Anyone see any obvious
errors? or have any hints/pointers?
regards,
Igor

float floatRead = 0;

UINT32* ptr = (UINT32 *) (&floatRead);

int offset = 0;

for (int ii=startInd; ii<=endInd; ii++){
*ptr |= (charBuf[ii] << offset);
offset += 8;

};

[James Kanze]

On May 26, 8:15 pm, [e-mail] wrote:

On May 22, 1:58 am, James Kanze <[e-mail]:

[...]

In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. Here is a copy
of a file that is involved:

Note however:

[...]

// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. It makes no attempt to
// convert to any canonical form.

// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON

Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options. If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.

In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[gpderetta]

On May 27, 10:04*am, James Kanze <[e-mail]:

On May 26, 8:15 pm, [e-mail] wrote:

On May 22, 1:58 am, James Kanze <[e-mail]:

* * [...]

In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. *Here is a copy
of a file that is involved:

Note however:

* * [...]

// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. *It makes no attempt to
// convert to any canonical form.
// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON

Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options. *If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.

In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.

Modulo what is guaranteed by the compiler/platform ABI, I guess.

In particular, the Boost.Serialization binary format is primarily used
by Boost.MPI (which obviously is a wrapper around MPI) for inter
process communication. I think that the idea is that the MPI layer
will take care of marshaling between peers and thus resolve any
representation difference. I think that in practice most (but not all)
MPI implementations just assume that peers use the same layout format
(i.e. same CPU/compiler/OS) and just network copy bytes back and
forward.

In a sense the distributed program is a logical single run of the same
program even if in practice are different processes running on
different machines, so your observation is still valid

--
Giovanni P. Deretta

[James Kanze]

On May 27, 12:07 pm, gpderetta <[e-mail]:

On May 27, 10:04 am, James Kanze <[e-mail]:

On May 26, 8:15 pm, [e-mail] wrote:

On May 22, 1:58 am, James Kanze <[e-mail]:

[...]

In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. Here is a copy
of a file that is involved:

Note however:

[...]

// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. It makes no attempt to
// convert to any canonical form.
// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON

Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options. If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.

In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.

Modulo what is guaranteed by the compiler/platform ABI, I guess.

Supposing you can trust them to be stable:-). In actual
practice, I've seen plenty of size changes, and I've seen long
and the floating point types change their representation, just
between different versions of the compiler. Not to mention
changes in padding which, at least in some cases depend on
compiler options. (For that matter, on most of the machines I
use, the size of a long depends on compiler options. And is the
sort of option that someone is likely to change in the makefile,
because e.g. they suddenly have to deal with big files.)

In particular, the Boost.Serialization binary format is
primarily used by Boost.MPI (which obviously is a wrapper
around MPI) for inter process communication. I think that the
idea is that the MPI layer will take care of marshaling
between peers and thus resolve any representation difference.
I think that in practice most (but not all) MPI
implementations just assume that peers use the same layout
format (i.e. same CPU/compiler/OS) and just network copy bytes
back and forward.

In a sense the distributed program is a logical single run of
the same program even if in practice are different processes
running on different machines, so your observation is still
valid

If the programs are not running on different machines, what's
the point of marshalling. Just put the objects in shared
memory. Marshalling is only necessary if the data is to be used
in a different place or time (networking or persistency). And a
different place or time means a different machine (sooner or
later, in the case of time).

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[gpderetta]

On May 28, 10:30*am, James Kanze <[e-mail]:

On May 27, 12:07 pm, gpderetta <[e-mail]:

On May 27, 10:04 am, James Kanze <[e-mail]:

On May 26, 8:15 pm, [e-mail] wrote:
On May 22, 1:58 am, James Kanze <[e-mail]:
* * [...]
In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. *Here is a copy
of a file that is involved:
Note however:
* * [...]
// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. *It makes no attemptto
// convert to any canonical form.
// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON
Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options. *If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.
In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.

Modulo what is guaranteed by the compiler/platform ABI, I guess.

Supposing you can trust them to be stable:-). *In actual
practice, I've seen plenty of size changes, and I've seen long
and the floating point types change their representation, just
between different versions of the compiler. *Not to mention
changes in padding which, at least in some cases depend on
compiler options. *(For that matter, on most of the machines I
use, the size of a long depends on compiler options. *And is the
sort of option that someone is likely to change in the makefile,
because e.g. they suddenly have to deal with big files.)

The size of long or that of off_t?

>

In particular, the Boost.Serialization binary format is
primarily used by Boost.MPI (which obviously is a wrapper
around MPI) for inter process communication. I think that the
idea is that the MPI layer will take care of marshaling
between peers and thus resolve any representation difference.
I think that in practice most (but not all) MPI
implementations just assume that peers use the same layout
format (i.e. same CPU/compiler/OS) and just network copy bytes
back and forward.
In a sense the distributed program is a logical single run of
the same program even if in practice are different processes
running on different machines, so your observation is still
valid

If the programs are not running on different machines, what's
the point of marshalling. *Just put the objects in shared
memory. *Marshalling is only necessary if the data is to be used
in a different place or time (networking or persistency). *And a
different place or time means a different machine (sooner or
later, in the case of time).

Well, MPI programs runs on large clusters of, usually, homogeneous
machines, connected via LAN. The same program will spawn
multiple copies of itself on every machine in the cluster, and every
copy communicates via message passing.
So you have one logical program which is partitioned on multiple
machines. I guess that most MPI implementations do not bother (in fact
I do not even know if it is required by the standard) to convert
messages to a machine agnostic format before sending it to another
peer.

--
Giovanni P. Deretta

[James Kanze]

On May 28, 12:11 pm, gpderetta <[e-mail]:

On May 28, 10:30 am, James Kanze <[e-mail]:

On May 27, 12:07 pm, gpderetta <[e-mail]:

On May 27, 10:04 am, James Kanze <[e-mail]:
On May 26, 8:15 pm, [e-mail] wrote:
On May 22, 1:58 am, James Kanze <[e-mail]:
[...]
In Boost 1.35 they've added an optimization to take advantage of
contiguous collections of primitive data types. Here is a copy
of a file that is involved:
Note however:
[...]
// archives stored as native binary - this should be the fastest way
// to archive the state of a group of obects. It makes no attemptto
// convert to any canonical form.
// IN GENERAL, ARCHIVES CREATED WITH THIS CLASS WILL NOT BE READABLE
// ON PLATFORM APART FROM THE ONE THEY ARE CREATE ON
Where "same platform" here means compiled on the same hardware,
using the same version of the same compiler, and the same
compiler options. If you ever recompile your executable with a
more recent version of the compiler, or with different options,
you may no longer be able to read the data.
In sum, it's an acceptable solution for temporary files within a
single run of the executable, but not for much else.
Modulo what is guaranteed by the compiler/platform ABI, I guess.

Supposing you can trust them to be stable:-). In actual
practice, I've seen plenty of size changes, and I've seen long
and the floating point types change their representation, just
between different versions of the compiler. Not to mention
changes in padding which, at least in some cases depend on
compiler options. (For that matter, on most of the machines I
use, the size of a long depends on compiler options. And is the
sort of option that someone is likely to change in the makefile,
because e.g. they suddenly have to deal with big files.)

The size of long or that of off_t?

No matter. The point is that they have to compile with
different options, and suddenly long has changed its size.

In particular, the Boost.Serialization binary format is
primarily used by Boost.MPI (which obviously is a wrapper
around MPI) for inter process communication. I think that
the idea is that the MPI layer will take care of
marshaling between peers and thus resolve any
representation difference. I think that in practice most
(but not all) MPI implementations just assume that peers
use the same layout format (i.e. same CPU/compiler/OS) and
just network copy bytes back and forward. In a sense the
distributed program is a logical single run of the same
program even if in practice are different processes
running on different machines, so your observation is
still valid

If the programs are not running on different machines,
what's the point of marshalling. Just put the objects in
shared memory. Marshalling is only necessary if the data is
to be used in a different place or time (networking or
persistency). And a different place or time means a
different machine (sooner or later, in the case of time).

Well, MPI programs runs on large clusters of, usually,
homogeneous machines, connected via LAN.

That's original. I don't think I've ever seen a cluster of
machines where every system in the cluster was identical. At
the very least, you'll have different versions of Sparc, or PC.
Some of which are 32 bit, and others 64. The cluster may start
out homogeneous, but one of the machines breaks down, and is
replaced with a newer model...

The real question, however, doesn't concern just the machines.
If all of the machines are running a single executable, loaded
from the same shared disk, it will probably work. If not, then
sooner or later, some of the machines will have different
compiles of the program, which may or may not be binary
compatible. In practice, the old rule always holds: identical
copies aren't. (Remember, binary compatibility can be lost just
by changing options, or using a newer version of the compiler.)

The same program will spawn multiple copies of itself on every
machine in the cluster, and every copy communicates via
message passing. So you have one logical program which is
partitioned on multiple machines. I guess that most MPI
implementations do not bother (in fact I do not even know if
it is required by the standard) to convert messages to a
machine agnostic format before sending it to another peer.

Well, I don't know much about that context. In my work, we have
a hetrogeneous network, with PC's under Windows as clients, and
either PC's under Linux or Sparcs under Solaris as servers (and
high level clients). And that more or less corresponds to what
I've seen elswhere as well.

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[gpderetta]

On May 28, 10:06 pm, James Kanze <[e-mail]:

On May 28, 12:11 pm, gpderetta <[e-mail]:

On May 28, 10:30 am, James Kanze <[e-mail]:

On May 27, 12:07 pm, gpderetta <[e-mail]:
In particular, the Boost.Serialization binary format is
primarily used by Boost.MPI (which obviously is a wrapper
around MPI) for inter process communication. I think that
the idea is that the MPI layer will take care of
marshaling between peers and thus resolve any
representation difference. I think that in practice most
(but not all) MPI implementations just assume that peers
use the same layout format (i.e. same CPU/compiler/OS) and
just network copy bytes back and forward. In a sense the
distributed program is a logical single run of the same
program even if in practice are different processes
running on different machines, so your observation is
still valid
If the programs are not running on different machines,
what's the point of marshalling. Just put the objects in
shared memory. Marshalling is only necessary if the data is
to be used in a different place or time (networking or
persistency). And a different place or time means a
different machine (sooner or later, in the case of time).

Well, MPI programs runs on large clusters of, usually,
homogeneous machines, connected via LAN.

That's original. I don't think I've ever seen a cluster of
machines where every system in the cluster was identical.

I think that for MPI it is common. Some vendors even sell shrink
wrapped clusters in a box (something like a big closet with thousands
of different computers-on-a-board, each running a different OS image).
Even custom built MPI clusters are fairly homogeneous (i.e. at least
same architecture and OS version).

I think that you work mostly on services applications, while MPI is
more common on high performance computing.

[...]
The real question, however, doesn't concern just the machines.
If all of the machines are running a single executable, loaded
from the same shared disk, it will probably work. If not, then
sooner or later, some of the machines will have different
compiles of the program, which may or may not be binary
compatible. In practice, the old rule always holds: identical
copies aren't. (Remember, binary compatibility can be lost just
by changing options, or using a newer version of the compiler.)

Yep, one need to be careful, but at least with the compiler I use,
options that change the ABI are explicitly documented as such.
Probably a much bigger problem are differences in third party
libraries between machines (i.e. do not expect the layout of objects
you do not control to stay stable).

The same program will spawn multiple copies of itself on every
machine in the cluster, and every copy communicates via
message passing. So you have one logical program which is
partitioned on multiple machines. I guess that most MPI
implementations do not bother (in fact I do not even know if
it is required by the standard) to convert messages to a
machine agnostic format before sending it to another peer.

Well, I don't know much about that context. In my work, we have
a hetrogeneous network, with PC's under Windows as clients, and
either PC's under Linux or Sparcs under Solaris as servers (and
high level clients). And that more or less corresponds to what
I've seen elswhere as well.

Where I work, clusters are composed of hundreds of very different
machines, but all use the same architecture and exact same OS version
(so that we can copy binaries around and not have to worry about
library incompatibilities). We do not use MPI though, but have an in-
house communication framework which does take care of marshaling in a
(mostly) system agnostic format.

--
Giovanni P. Deretta

[James Kanze]

On May 30, 6:38 pm, gpderetta <[e-mail]:

On May 28, 10:06 pm, James Kanze <[e-mail]:

On May 28, 12:11 pm, gpderetta <[e-mail]:

On May 28, 10:30 am, James Kanze <[e-mail]:
On May 27, 12:07 pm, gpderetta <[e-mail]:
In particular, the Boost.Serialization binary format is
primarily used by Boost.MPI (which obviously is a wrapper
around MPI) for inter process communication. I think that
the idea is that the MPI layer will take care of
marshaling between peers and thus resolve any
representation difference. I think that in practice most
(but not all) MPI implementations just assume that peers
use the same layout format (i.e. same CPU/compiler/OS) and
just network copy bytes back and forward. In a sense the
distributed program is a logical single run of the same
program even if in practice are different processes
running on different machines, so your observation is
still valid
If the programs are not running on different machines,
what's the point of marshalling. Just put the objects in
shared memory. Marshalling is only necessary if the data is
to be used in a different place or time (networking or
persistency). And a different place or time means a
different machine (sooner or later, in the case of time).
Well, MPI programs runs on large clusters of, usually,
homogeneous machines, connected via LAN.

That's original. I don't think I've ever seen a cluster of
machines where every system in the cluster was identical.

I think that for MPI it is common. Some vendors even sell
shrink wrapped clusters in a box (something like a big closet
with thousands of different computers-on-a-board, each running
a different OS image). Even custom built MPI clusters are
fairly homogeneous (i.e. at least same architecture and OS
version).

I think that you work mostly on services applications, while
MPI is more common on high performance computing.

I realized that much, but I wasn't aware that it was that common
even on high performance computing. The high performance
computing solutions I've seen have mostly involved a lot of
CPU's using the same memory, so marshalling wasn't an issue.
(But I'm not much of an expert in the domain, and I've not seen
that many systems, so what I've seen doesn't mean much.)

[...]
The real question, however, doesn't concern just the machines.
If all of the machines are running a single executable, loaded
from the same shared disk, it will probably work. If not, then
sooner or later, some of the machines will have different
compiles of the program, which may or may not be binary
compatible. In practice, the old rule always holds: identical
copies aren't. (Remember, binary compatibility can be lost just
by changing options, or using a newer version of the compiler.)

Yep, one need to be careful, but at least with the compiler I use,
options that change the ABI are explicitly documented as such.

Lucky guy:-). For the most part, what the options actually do
is well documented, and if you understand a bit about what it
means at the hardware level, you can figure out which ones are
safe, and which aren't. But it's far from explicit.

Note that this can be a problem just trying to statically link
libraries; you don't need marshalling at all to get into
trouble. (Or rather: you don't want to have to marshall every
time you pass an std::vector to a function in another module.)

Probably a much bigger problem are differences in third party
libraries between machines (i.e. do not expect the layout of
objects you do not control to stay stable).

That's another problem entirely, and affects linking more than
marshalling. The problem is that compilers may change
representation between versions, etc.

The same program will spawn multiple copies of itself on every
machine in the cluster, and every copy communicates via
message passing. So you have one logical program which is
partitioned on multiple machines. I guess that most MPI
implementations do not bother (in fact I do not even know if
it is required by the standard) to convert messages to a
machine agnostic format before sending it to another peer.

Well, I don't know much about that context. In my work, we have
a hetrogeneous network, with PC's under Windows as clients, and
either PC's under Linux or Sparcs under Solaris as servers (and
high level clients). And that more or less corresponds to what
I've seen elswhere as well.

Where I work, clusters are composed of hundreds of very
different machines, but all use the same architecture and
exact same OS version (so that we can copy binaries around and
not have to worry about library incompatibilities). We do not
use MPI though, but have an in- house communication framework
which does take care of marshaling in a (mostly) system
agnostic format.

Yes. We do something more or less like this for the clients:
they're all PC's under Windows, and we use a lowest common
denominator which should work for all Windows systems. Our
machines are geographically distributed, however, so
realistically, ensuring exactly the same version of the OS,
isn't possible.

For the servers, economic considerations result in a decision to
move from Solaris on Sparc to Linux on PC, at least for all but
the most critical systems. Similarly, economic considerations
mean that the entire park won't be upgraded at the same instant.
Are you saying that if a decision comes to upgrade the
architecture, you change all of the machines in a cluster at
once? (But maybe... I can imagine that all of the machines in a
cluster still cost less than one supercomputer. And if you were
using a supercomputer, and wanted to upgrade, you'd change it
all at once. I guess it's just a different mindset.)

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]