How to write array of doubles to stream without using a loop?

[John Dumais]

Hello,

I have been trying to figure out how to write an array of doubles (in
this specific case) to a binary stream without using a loop. What I
have been doing is...

foreach(double d in TraceData)
{
instanceOfBinaryWriter.Write(d);
}

The loop is introducing overhead I don't want. In unmanaged c++, I
would just do something like...

fwrite(buffer, sizeof(double), numArrayElements, fileStream)

Is their an equivalent using c#?

Thanks,

[Sean Hederman]

Surely fwrite would have a loop inside it? If it does, then surely it
doesn't matter if the loop occurs in your code or in library code?

"John Dumais" <no****************@agilent.com> wrote in message
news:11**************@cswreg.cos.agilent.com...

Hello,

I have been trying to figure out how to write an array of doubles (in this
specific case) to a binary stream without using a loop. What I have been
doing is...

foreach(double d in TraceData)
{
instanceOfBinaryWriter.Write(d);
}

The loop is introducing overhead I don't want. In unmanaged c++, I would
just do something like...

fwrite(buffer, sizeof(double), numArrayElements, fileStream)

Is their an equivalent using c#?

Thanks,

[Nicholas Paldino [.NET/C# MVP]]

John,

You won't really be able to do that. It would be easier if you had
direct memory access, but unfortunately, you don't. The only thing that you
could do is do a conversion to a byte array, but you know that will
introduce overhead as well (that routine has to loop through as well to do
the conversion).

The only other option you would have is if your method can use unsafe
code. That way, you could cast the double array to a double pointer, and
then cast that to a pointer of a byte, and then pass the byte to the method
without having to loop through to get a byte array.

Hope this helps.

--
- Nicholas Paldino [.NET/C# MVP]
- mv*@spam.guard.caspershouse.com

"John Dumais" <no****************@agilent.com> wrote in message
news:11**************@cswreg.cos.agilent.com...

Hello,

I have been trying to figure out how to write an array of doubles (in this
specific case) to a binary stream without using a loop. What I have been
doing is...

foreach(double d in TraceData)
{
instanceOfBinaryWriter.Write(d);
}

The loop is introducing overhead I don't want. In unmanaged c++, I would
just do something like...

fwrite(buffer, sizeof(double), numArrayElements, fileStream)

Is their an equivalent using c#?

Thanks,

[AMoose]

> You won't really be able to do that. It would be easier if you had

direct memory access, but unfortunately, you don't. The only thing that you could do is do a conversion to a byte array, but you know that will
introduce overhead as well (that routine has to loop through as well to do
the conversion).
Take a look at Buffer.BlockCopy. It only works with primitive types. But I
remember looking at the rotor source code and its an internal call and does
a memcopy , memove, mem something but not a for loop. The docs say "This
class provides better performance for manipulating primitive types than
similar methods in the System.Array class.". Here is a sample that did work.

Dim Ds() As Integer = {1, 2, 3}
Dim I() As Byte = Array.CreateInstance(GetType(Byte), Buffer.ByteLength(Ds))
Buffer.BlockCopy(Ds, 0, I, 0, Buffer.ByteLength(Ds))

"Nicholas Paldino [.NET/C# MVP]" <mv*@spam.guard.caspershouse.com> wrote in
message news:ew****************@TK2MSFTNGP14.phx.gbl... John,

You won't really be able to do that. It would be easier if you had
direct memory access, but unfortunately, you don't. The only thing that you could do is do a conversion to a byte array, but you know that will
introduce overhead as well (that routine has to loop through as well to do
the conversion).

The only other option you would have is if your method can use unsafe
code. That way, you could cast the double array to a double pointer, and
then cast that to a pointer of a byte, and then pass the byte to the method without having to loop through to get a byte array.

Hope this helps.

--
- Nicholas Paldino [.NET/C# MVP]
- mv*@spam.guard.caspershouse.com

"John Dumais" <no****************@agilent.com> wrote in message
news:11**************@cswreg.cos.agilent.com...

Hello,

I have been trying to figure out how to write an array of doubles (in this specific case) to a binary stream without using a loop. What I have been doing is...

foreach(double d in TraceData)
{
instanceOfBinaryWriter.Write(d);
}

The loop is introducing overhead I don't want. In unmanaged c++, I would just do something like...

fwrite(buffer, sizeof(double), numArrayElements, fileStream)

Is their an equivalent using c#?

Thanks,

[Jon Skeet [C# MVP]]

John Dumais <no****************@agilent.com> wrote:

I have been trying to figure out how to write an array of doubles (in
this specific case) to a binary stream without using a loop. What I
have been doing is...

foreach(double d in TraceData)
{
instanceOfBinaryWriter.Write(d);
}

The loop is introducing overhead I don't want.

Have you measured the overhead? Is it definitely causing a problem?
There's a tendency to assume that things like this are bottlenecks
without experimentation - that may not be the case here, but it's worth
knowing before we get into much riskier code.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[John Dumais]

Jon Skeet [C# MVP] wrote:

Have you measured the overhead? Is it definitely causing a problem?
There's a tendency to assume that things like this are bottlenecks
without experimentation - that may not be the case here, but it's worth
knowing before we get into much riskier code.

Not specifically in C#, because I can't get an apples-to-apples
comparison, but my experience has been that most run-time
libraries work better when you supply them with one hunk of data
as opposed to supplying many little hunks in a loop. The attached
example will illustrate. It's in C++, because I can't come up
with a reasonable comarison in C#. Sorry in advance -- I wrote
this illustration on my Linux machine. I don't have a Windows
PC at home. I ran the program 10 times in a loop and got the
following results.
Using loop
Elapsed time: 2 seconds, 223613 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 725197 micro-seconds
Using loop
Elapsed time: 3 seconds, 256561 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 277398 micro-seconds
Using loop
Elapsed time: 2 seconds, 224807 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 45266 micro-seconds
Using loop
Elapsed time: 2 seconds, 215541 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 231558 micro-seconds
Using loop
Elapsed time: 2 seconds, 221532 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 278577 micro-seconds
Using loop
Elapsed time: 2 seconds, 220584 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 11936 micro-seconds
Using loop
Elapsed time: 2 seconds, 215263 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 28349 micro-seconds
Using loop
Elapsed time: 2 seconds, 223527 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 273473 micro-seconds
Using loop
Elapsed time: 2 seconds, 226312 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 7940 micro-seconds
Using loop
Elapsed time: 2 seconds, 217676 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 23728 micro-seconds

The program...
#include <cstdio>
#include <cstdlib>
#include <sys/time.h>

typedef void(*WriterFuncPtr)(FILE*, double*, size_t);

void timeIt(WriterFuncPtr funcPtr, FILE *fp, double *data,
size_t numDataPoints)
{
timeval startTime = {0, 0};
timeval endTime = {0, 0};
timeval elapsedTime = {0, 0};

if( ! gettimeofday(&startTime, 0)){
funcPtr(fp, data, numDataPoints);

if( ! gettimeofday(&endTime, 0)){
if(startTime.tv_usec > endTime.tv_usec){
endTime.tv_usec += 1000000;
endTime.tv_sec--;
}

elapsedTime.tv_usec = endTime.tv_usec - startTime.tv_usec;
elapsedTime.tv_sec = endTime.tv_sec - startTime.tv_sec;

printf("Elapsed time: %ld seconds, %ld micro-seconds\n",
elapsedTime.tv_sec, elapsedTime.tv_usec);
}
}

}

void checkFileContents(FILE *fp)
{
double data = 0;

size_t numRead = 0;

do{
numRead = fread(&data, sizeof(data), 1, fp);
printf("%lf\n", data);
} while(numRead > 0);
}

void writeDataUsingLoop(FILE *fp, double *data, size_t numDataPoints)
{
for(size_t i = 0; i < numDataPoints; ++i){
(void)fwrite(data++, sizeof(double), 1, fp);
}
}

void writeDataInOneHunk(FILE *fp, double *data, size_t numDataPoints)
{
(void)fwrite(data, sizeof(double), numDataPoints, fp);
}

int main(void)
{
FILE *fp = fopen("data", "w");
if(fp){
const size_t numDataPoints = 8 * 1024 * 1024;
double *data = (double*)malloc(numDataPoints * sizeof(double));
if(data){
for(size_t i = 0; i < numDataPoints; ++i){
data[i] = i;
}

printf("Using loop\n");
timeIt(writeDataUsingLoop, fp, data, numDataPoints);

fclose(fp);

fp = fopen("data", "w");
if(fp){
printf("Writing in a single hunk\n");
timeIt(writeDataInOneHunk, fp, data, numDataPoints);

fclose(fp);
fp = 0;
}

free(data);
}
}

/*
fp = fopen("data", "r");
if(fp){
checkFileContents(fp);

fclose(fp);
fp = 0;
}
*/

return 0;
}

Thanks,