storing floating point data with using float or double

[Steve]

Hi,

I've developed a testing application that stores formatted results in a
database. Recently it was requested that I add the ability to generate
graphs from the raw, un formatted test results (100,000+ float values)

I don't intend to store all of the 100,000 datapoints, but rather a subset
of say 250. Due to the volume of testing that we need and the volume of
results stored, I need to be VERY careful with data size and keep things as
lean and small as possible.

Given the example data: (-5.2485, 5.9875, -1.2548, 7.5842)
Those samples represent our data value boundaries, they won't get any bigger
than that, they are signed and they are floating point.

I don't need the precision of a float, the above values would be fine to be
stored as:
-5.29
5.99
-1.25
7.58

A couple of ideas I have come up with is to try compressing the array of 250
floats and storing the compressed binary in the DB BLOB field. I also
though about breaking the values into an sbyte and a byte like this:
-5.29

represented as:
sbyte l = -5;
byte r = 29;

This would save me to bytes from using a float. If I then compressed this
"byte data" I could save even more space.

Anyway, I have never done any of this before and wanted to invite you guys
to offer any suggestions or past experiences you might of had with doing
something like this.

As you can see, I'm not real sure what I'm doing ;0)

Thanks for reading,
Steve

[Steve]

Subject was supposed to be "withOUT using float or double" - sorry
"Steve" <sk**@skle.com> wrote in message
news:uN*************@TK2MSFTNGP05.phx.gbl...

Hi,

I've developed a testing application that stores formatted results in a
database. Recently it was requested that I add the ability to generate
graphs from the raw, un formatted test results (100,000+ float values)

I don't intend to store all of the 100,000 datapoints, but rather a subset
of say 250. Due to the volume of testing that we need and the volume of
results stored, I need to be VERY careful with data size and keep things
as lean and small as possible.

Given the example data: (-5.2485, 5.9875, -1.2548, 7.5842)
Those samples represent our data value boundaries, they won't get any
bigger than that, they are signed and they are floating point.

I don't need the precision of a float, the above values would be fine to
be stored as:
-5.29
5.99
-1.25
7.58

A couple of ideas I have come up with is to try compressing the array of
250 floats and storing the compressed binary in the DB BLOB field. I also
though about breaking the values into an sbyte and a byte like this:
-5.29

represented as:
sbyte l = -5;
byte r = 29;

This would save me to bytes from using a float. If I then compressed this
"byte data" I could save even more space.

Anyway, I have never done any of this before and wanted to invite you guys
to offer any suggestions or past experiences you might of had with doing
something like this.

As you can see, I'm not real sure what I'm doing ;0)

Thanks for reading,
Steve

[Göran Andersson]

You can store the values as fixed point values in shorts (Int16). You
would be able to get a precision of three decimals if you'd like.

Multiply the values by 1000 and convert to short:

-5249
5988
1255
7584

The numbers you could store this way would range from -32.768 to 32.767.

Eat my shorts. ;)

Steve wrote:

Hi,

I've developed a testing application that stores formatted results in a
database. Recently it was requested that I add the ability to generate
graphs from the raw, un formatted test results (100,000+ float values)

I don't intend to store all of the 100,000 datapoints, but rather a subset
of say 250. Due to the volume of testing that we need and the volume of
results stored, I need to be VERY careful with data size and keep things as
lean and small as possible.

Given the example data: (-5.2485, 5.9875, -1.2548, 7.5842)
Those samples represent our data value boundaries, they won't get any bigger
than that, they are signed and they are floating point.

I don't need the precision of a float, the above values would be fine to be
stored as:
-5.29
5.99
-1.25
7.58

A couple of ideas I have come up with is to try compressing the array of 250
floats and storing the compressed binary in the DB BLOB field. I also
though about breaking the values into an sbyte and a byte like this:
-5.29

represented as:
sbyte l = -5;
byte r = 29;

This would save me to bytes from using a float. If I then compressed this
"byte data" I could save even more space.

Anyway, I have never done any of this before and wanted to invite you guys
to offer any suggestions or past experiences you might of had with doing
something like this.

As you can see, I'm not real sure what I'm doing ;0)

Thanks for reading,
Steve

[Greg Young]

If your data posted is typical of your data ... why not just store

(int16) data * 100

then on the way out just divide by 100 ? :)

Cheers,

Greg
"Steve" <sk**@skle.com> wrote in message
news:uN*************@TK2MSFTNGP05.phx.gbl...

Hi,

I've developed a testing application that stores formatted results in a
database. Recently it was requested that I add the ability to generate
graphs from the raw, un formatted test results (100,000+ float values)

I don't intend to store all of the 100,000 datapoints, but rather a subset
of say 250. Due to the volume of testing that we need and the volume of
results stored, I need to be VERY careful with data size and keep things
as lean and small as possible.

Given the example data: (-5.2485, 5.9875, -1.2548, 7.5842)
Those samples represent our data value boundaries, they won't get any
bigger than that, they are signed and they are floating point.

I don't need the precision of a float, the above values would be fine to
be stored as:
-5.29
5.99
-1.25
7.58

A couple of ideas I have come up with is to try compressing the array of
250 floats and storing the compressed binary in the DB BLOB field. I also
though about breaking the values into an sbyte and a byte like this:
-5.29

represented as:
sbyte l = -5;
byte r = 29;

This would save me to bytes from using a float. If I then compressed this
"byte data" I could save even more space.

Anyway, I have never done any of this before and wanted to invite you guys
to offer any suggestions or past experiences you might of had with doing
something like this.

As you can see, I'm not real sure what I'm doing ;0)

Thanks for reading,
Steve

[Steve]

Oh man, I need a rock to crawl under and hide!
Thanks for the tip, jeez.. I should know that, shouldn't I? I think I was
over complicating and skipped past simple and easy. I hate that.

Have a great weekend (and maybe Cinco de Mayo, depending where you live)

"Greg Young" <Dr*************@hotmail.com> wrote in message
news:%2****************@TK2MSFTNGP03.phx.gbl...

If your data posted is typical of your data ... why not just store

(int16) data * 100

then on the way out just divide by 100 ? :)

Cheers,

Greg
"Steve" <sk**@skle.com> wrote in message
news:uN*************@TK2MSFTNGP05.phx.gbl...

Hi,

I've developed a testing application that stores formatted results in a
database. Recently it was requested that I add the ability to generate
graphs from the raw, un formatted test results (100,000+ float values)

I don't intend to store all of the 100,000 datapoints, but rather a
subset of say 250. Due to the volume of testing that we need and the
volume of results stored, I need to be VERY careful with data size and
keep things as lean and small as possible.

Given the example data: (-5.2485, 5.9875, -1.2548, 7.5842)
Those samples represent our data value boundaries, they won't get any
bigger than that, they are signed and they are floating point.

I don't need the precision of a float, the above values would be fine to
be stored as:
-5.29
5.99
-1.25
7.58

A couple of ideas I have come up with is to try compressing the array of
250 floats and storing the compressed binary in the DB BLOB field. I
also though about breaking the values into an sbyte and a byte like this:
-5.29

represented as:
sbyte l = -5;
byte r = 29;

This would save me to bytes from using a float. If I then compressed
this "byte data" I could save even more space.

Anyway, I have never done any of this before and wanted to invite you
guys to offer any suggestions or past experiences you might of had with
doing something like this.

As you can see, I'm not real sure what I'm doing ;0)

Thanks for reading,
Steve

[Steve]

lol.. it's not often you can say "Eat my shorts" and have it actually be
relevant to the issue, good one ;0)
thanks for the suggestion, this is what I will do.
Have a great weekend!

"Göran Andersson" <gu***@guffa.com> wrote in message
news:%2****************@TK2MSFTNGP03.phx.gbl...

You can store the values as fixed point values in shorts (Int16). You
would be able to get a precision of three decimals if you'd like.

Multiply the values by 1000 and convert to short:

-5249
5988
1255
7584

The numbers you could store this way would range from -32.768 to 32.767.

Eat my shorts. ;)

Steve wrote:

Hi,

I've developed a testing application that stores formatted results in a
database. Recently it was requested that I add the ability to generate
graphs from the raw, un formatted test results (100,000+ float values)

I don't intend to store all of the 100,000 datapoints, but rather a
subset of say 250. Due to the volume of testing that we need and the
volume of results stored, I need to be VERY careful with data size and
keep things as lean and small as possible.

Given the example data: (-5.2485, 5.9875, -1.2548, 7.5842)
Those samples represent our data value boundaries, they won't get any
bigger than that, they are signed and they are floating point.

I don't need the precision of a float, the above values would be fine to
be stored as:
-5.29
5.99
-1.25
7.58

A couple of ideas I have come up with is to try compressing the array of
250 floats and storing the compressed binary in the DB BLOB field. I
also though about breaking the values into an sbyte and a byte like this:
-5.29

represented as:
sbyte l = -5;
byte r = 29;

This would save me to bytes from using a float. If I then compressed
this "byte data" I could save even more space.

Anyway, I have never done any of this before and wanted to invite you
guys to offer any suggestions or past experiences you might of had with
doing something like this.

As you can see, I'm not real sure what I'm doing ;0)

Thanks for reading,
Steve

[V]

Hi Steve,

If it is really only a storage problem, then you could probably invest
some time and implement any of the compression algorithms available in
C#, and then maybe simply concatenate your numbers into a string and
then compress them using such an algorithm (huffman or arithmetic
compression), and then simply store the compressed string in a varchar
field.

For your dataset size, it will be really fast to do the
compress/decompress and the size of the resultant (compressed) varchar
will also be pretty small.

Regards,
Vaibhav
www.nagarro.com