Faster way to write in a file

[LilacSkin]

Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

Tk.

[santosh]

LilacSkin wrote:

Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

[LilacSkin]

On 13 fév, 13:47, santosh <[e-mail]:

LilacSkin wrote:

Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

It's not working !

[santosh]

LilacSkin wrote:

On 13 fév, 13:47, santosh <[e-mail]:

>LilacSkin wrote:

Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I
don't know how to use it.
Can you help me, please ?

size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

It's not working !

How exactly? Did you check that the elements parameter is correct? What
is the value that rc has after the fwrite() call? Is pFilec pointing to
a valid stream for which you have appropriate permissions?

[Mark Bluemel]

santosh wrote:

LilacSkin wrote:

>Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

[Morris Dovey]

Mark Bluemel wrote:

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

A special-purpose double -text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

http://groups.google.com/group/comp....5f74496633fe3a

(mind the wrap)

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USA
http://www.iedu.com/DeSoto

[Mark Bluemel]

Morris Dovey wrote:

Mark Bluemel wrote:

>How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

My point was that the OP had clearly been trying to produce a textual
representation.

From context, I would not have expected the OP to understand why
Santosh's suggestion was not compatible with his/her original goal.

[Malcolm McLean]

"LilacSkin" <lp******@iseb.frwrote in message news

Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

Replace %lli with %llx and measure the speedup.
Is hexadecimal output acceptable to you?

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

[LilacSkin]

On 13 fév, 15:13, Morris Dovey <[e-mail]:

Mark Bluemel wrote:

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

A special-purpose double -text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

http://groups.google.com/group/comp....d/thread/eb329...

(mind the wrap)

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USAhttp://www.iedu.com/DeSoto

Your idea is to use your "convert" function and to fwrite the
converted buffer ?
If yes, can I cast an long long in double to directly use it ?

Thanks !

[LilacSkin]

On 13 fév, 19:51, LilacSkin <[e-mail]:

On 13 fév, 15:13, Morris Dovey <[e-mail]:

Mark Bluemel wrote:

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

A special-purpose double -text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

http://groups.google.com/group/comp....d/thread/eb329...

(mind the wrap)

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USAhttp://www.iedu.com/DeSoto

Your idea is to use your "convert" function and to fwrite the
converted buffer ?
If yes, can I cast an long long in double to directly use it ?

Thanks !

In fact the most important thing is to write faster.
I have 1,6 Mbyte/s of raw data to write during a couple of days.
Because of the amount of data, I can't reformat their after.
The data are stored in a char buffer and I need to put in a file as a
signed long long.
The size of the buzzer is 8 Mbyte, refreshed every 5 sec.

I think the best way is to format them and use a fwrite or _write
function .

[Morris Dovey]

LilacSkin wrote:

>
On 13 fév, 15:13, Morris Dovey <[e-mail]:

Mark Bluemel wrote:

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

A special-purpose double -text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

http://groups.google.com/group/comp....d/thread/eb329...

(mind the wrap)

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USAhttp://www.iedu.com/DeSoto

Your idea is to use your "convert" function and to fwrite the
converted buffer ?
If yes, can I cast an long long in double to directly use it ?

You probably won't want to use _that_ convert function because it
does more than you want (double -string) when you're really
wanting (long long -string).

Just throw away the part that deals with the fractional part of
the double - and if all of your values are positive, throw away
the sign logic, too. That won't leave very much - and it
shouldn't waste many cycles.

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USA
http://www.iedu.com/DeSoto

[LilacSkin]

#include <stdio.h>

void convert(long long v,char *s,int sz,int dp)
{ char *p = s + sz;
long long x = 1LL;
int sign = v < 0.0;

if (sign) v = -v;
while (dp--) x *= 10;
x = x * v;

*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));

while (p >= s) *p-- = '0';

if (sign) *s = '-';

}

int main(void)
{ long long test = -123456789012345;
char buffer[64];

convert(test,buffer,25,5);
printf("'%s'\n",buffer);
// then fwrite(buffer...)

return 0;

}

[Morris Dovey]

LilacSkin wrote:

#include <stdio.h>

void convert(long long v,char *s,int sz,int dp)
{ char *p = s + sz;
long long x = 1LL;
int sign = v < 0.0;

if (sign) v = -v;
while (dp--) x *= 10;
x = x * v;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = '0';
if (sign) *s = '-';
}

Pretty close. Since you're not working with a double, you can
dispense with the fractional part logic and shorten to something
like:

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

--
Morris Dovey
DeSoto Solar
DeSoto, Iowa USA
http://www.iedu.com/DeSoto

[Ben Bacarisse]

LilacSkin <lp******@iseb.frwrites:

#include <stdio.h>

void convert(long long v,char *s,int sz,int dp)
{ char *p = s + sz;
long long x = 1LL;
int sign = v < 0.0;

if (sign) v = -v;
while (dp--) x *= 10;
x = x * v;

*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));

while (p >= s) *p-- = '0';

if (sign) *s = '-';

}

int main(void)
{ long long test = -123456789012345;
char buffer[64];

convert(test,buffer,25,5);
printf("'%s'\n",buffer);
// then fwrite(buffer...)

return 0;

}

You may want to re-consider. Numbers are usually converted to a
decimal string representation so the we (humans) can read them. If
your program is producing such a volume of data that it is hard to get
converted output fast enough, will the resulting data ever be read by
a person? I doubt it. You'd need a program to scan just one second's
worth.

It might pay to store the data as native binary numbers. That is
probably what was being suggested when someone said use "fwrite".

Of course, if the output *has* to be processed by an existing program
that needs decimal input, then the conversion has to happen some time,
but it could be done later, at leisure, so to speak. It might even be
possible to process into decimal only the portion you are interested in.
Maybe the binary data could be indexed for rapid access to specific
parts. Without knowing the ultimate fate of this data, it is hard to
be more specific, but I suggest you look beyond your current headache
of not being able to use fprintf because it seems too slow.

--
Ben.

[Mark Bluemel]

LilacSkin wrote:

I have 1,6 Mbyte/s of raw data to write during a couple of days.
Because of the amount of data, I can't reformat their after.
The data are stored in a char buffer and I need to put in a file as a
signed long long.

So why were you even trying to use printf, which is for formatted
output?

If you have to write to the file as signed long long, then fwrite is
appropriate.

You really could help us to help you if you explained the requirement
better.

[LilacSkin]

On 14 fév, 09:59, Mark Bluemel <[e-mail]:

LilacSkin wrote:

I have 1,6 Mbyte/s of raw data to write during a couple of days.
Because of the amount of data, I can't reformat their after.
The data are stored in a char buffer and I need to put in a file as a
signed long long.

So why were you even trying to use printf, which is for formatted
output?

If you have to write to the file as signed long long, then fwrite is
appropriate.

You really could help us to help you if you explained the requirement
better.

I need to write the data in text file !
Morris, I try your code, the problem is, it puts a space character
before a positive number.

[LilacSkin]

The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :
#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}

int main(void)
{
time_t start_time;
time_t end_time;
double diff_time = 0;

int i;
int num_digit;
char buffer[64];
long long test = -1844674407370955161;
FILE * pFile = NULL;
pFile = fopen( "data_test_write.txt", "w" );

time ( &start_time );

for (i=0;i<16000000;i++)
{
num_digit=digits(test);
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
if (i % 16 == 15) fputc ( (int) '\n' , pFile );
else fputc ( (int) '\t' , pFile );
}
time ( &end_time );
diff_time = difftime ( end_time, start_time );
printf("Duration : %lf sec\n", diff_time);
printf("Num of samples (16*64bit + /t and /n) : %i\n", i/16);

return 0;
}

The problem is, there is a space character before a positive number.
NEEDED: 1 line each 80 us
REACHED: 1 line each 36 us

1

[LilacSkin]

The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :
#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}

int main(void)
{
time_t start_time;
time_t end_time;
double diff_time = 0;

int i;
int num_digit;
char buffer[64];
long long test = -1844674407370955161;
FILE * pFile = NULL;
pFile = fopen( "data_test_write.txt", "w" );

time ( &start_time );

for (i=0;i<16000000;i++)
{
num_digit=digits(test);
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
if (i % 16 == 15) fputc ( (int) '\n' , pFile );
else fputc ( (int) '\t' , pFile );
}
time ( &end_time );
diff_time = difftime ( end_time, start_time );
printf("Duration : %lf sec\n", diff_time);
printf("Num of samples (16*64bit + /t and /n) : %i\n", i/16);

return 0;
}

The problem is, there is a space character before a positive number.
NEEDED: 1 line each 80 us
REACHED: 1 line each 36 us

1

[LilacSkin]

The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :
#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}

int main(void)
{
time_t start_time;
time_t end_time;
double diff_time = 0;

int i;
int num_digit;
char buffer[64];
long long test = -1844674407370955161;
FILE * pFile = NULL;
pFile = fopen( "data_test_write.txt", "w" );

time ( &start_time );

for (i=0;i<16000000;i++)
{
num_digit=digits(test);
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
if (i % 16 == 15) fputc ( (int) '\n' , pFile );
else fputc ( (int) '\t' , pFile );
}
time ( &end_time );
diff_time = difftime ( end_time, start_time );
printf("Duration : %lf sec\n", diff_time);
printf("Num of samples (16*64bit + /t and /n) : %i\n", i/16);

return 0;
}

The problem is, there is a space character before a positive number.
NEEDED: 1 line each 80 us
REACHED: 1 line each 36 us

1

[Mark Bluemel]

LilacSkin wrote:

The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :
#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}

int main(void)
{
time_t start_time;
time_t end_time;
double diff_time = 0;

int i;
int num_digit;
char buffer[64];
long long test = -1844674407370955161;
FILE * pFile = NULL;
pFile = fopen( "data_test_write.txt", "w" );

time ( &start_time );

for (i=0;i<16000000;i++)
{
num_digit=digits(test);
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
if (i % 16 == 15) fputc ( (int) '\n' , pFile );
else fputc ( (int) '\t' , pFile );
}
time ( &end_time );
diff_time = difftime ( end_time, start_time );
printf("Duration : %lf sec\n", diff_time);
printf("Num of samples (16*64bit + /t and /n) : %i\n", i/16);

return 0;
}

The problem is, there is a space character before a positive number.

Easy fix :-

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
else *s = '+';
}

if you don't want a sign character at all, then a crude approach could
be :-
if (test < 0) fwrite (buffer , 1 , num_digit, pFile );
else fwrite (buffer + 1, 1, num_digit-1, pFile);

More elegant solutions are obviously possible, but for quick simple
hacks, these may work.

[Ben Bacarisse]

LilacSkin <lp******@iseb.frwrites:

The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :
#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}

<snip main>

>
The problem is, there is a space character before a positive number.

NEEDED: 1 line each 80 us
REACHED: 1 line each 36 us

Form a quick test here, it seems faster (almost twice the speed) to
reverse the string after getting the digits:

int convert2(long long x, char *s)
{
char *p = s;
int nd, sign = x < 0;
if (sign) x = -x;
do {
*p++ = '0' + (x % 10);
x /= 10;
} while (x);
if (sign) *p++ = '-';
*p = 0;
nd = p - s;
while (--p s) {
char c = *s;
*s++ = *p;
*p = c;
}
return nd;
}

You get the digit count almost for free that way.

--
Ben.

[Keith Thompson]

Army1987 <ar******@NOSPAM.itwrites:

LilacSkin wrote:

[...]

> convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );

You can use fprintf with "lld":
fprintf(pFile, "%lld", test);
Its implementer probably has used the fastest possible routine to convert
a number to a decimal representation on your system.

[...]

True, but printf also has to parse the format string. A hand-written
conversion routine could easily beat printf's performance because of
this.

--
Keith Thompson (The_Other_Keith) <ks***@mib.org>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Joe Wright]

Keith Thompson wrote:

Army1987 <ar******@NOSPAM.itwrites:

>LilacSkin wrote:

[...]

>> convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );

You can use fprintf with "lld":
fprintf(pFile, "%lld", test);
Its implementer probably has used the fastest possible routine to convert
a number to a decimal representation on your system.

[...]

True, but printf also has to parse the format string. A hand-written
conversion routine could easily beat printf's performance because of
this.

I'd think that file I/O would take so much time that the performance of
conversion routines would be imperceptible.

--
Joe Wright
"Everything should be made as simple as possible, but not simpler."
--- Albert Einstein ---

[Keith Thompson]

Joe Wright <jo********@comcast.netwrites:

Keith Thompson wrote:

>Army1987 <ar******@NOSPAM.itwrites:

>>LilacSkin wrote:

[...]

>>> convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
You can use fprintf with "lld":
fprintf(pFile, "%lld", test);
Its implementer probably has used the fastest possible routine to convert
a number to a decimal representation on your system.

[...]

True, but printf also has to parse the format string. A hand-written
conversion routine could easily beat printf's performance because of
this.

I'd think that file I/O would take so much time that the performance
of conversion routines would be imperceptible.

Possibly. The answer, of course, is to perform manual optimizations
if and only if the performance is measurably poor and the optimization
significantly improves it.

Note also that output is usually buffered, so a given fprintf call
might not result in any physical I/O at all. Or you might be using
sprintf rather than fprintf, or the target file might be on a RAM
disk, or ....

--
Keith Thompson (The_Other_Keith) <ks***@mib.org>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"