Converting strings to numbers - a question of speed

[zexpe]

I have an extremely cpu/data intensive piece of code that makes heavy
use of the following function:

void convertToDouble(const std::string& in, double& out)
{
out = atof(in.c_str());
}

I would really like to get away from using any old C-style functions.
So, I modified the above function to make it follow the C++ convention:

void convertToDouble(const std::string& in, double& out)
{
std::stringstream ss(in);
ss >> out;
}

However, my test code that previously took 30s to run, now takes 45s
(linux gcc 4.0.2 with -03 option). That's a heavy price to pay for
adopting the C++ convention. So, my question is... Is there an
*efficient* way to convert a string to a number (e.g. double) without
requiring the use of old C libraries?

[John Harrison]

zexpe wrote:

I have an extremely cpu/data intensive piece of code that makes heavy
use of the following function:

void convertToDouble(const std::string& in, double& out)
{
out = atof(in.c_str());
}

I would really like to get away from using any old C-style functions.
So, I modified the above function to make it follow the C++ convention:

void convertToDouble(const std::string& in, double& out)
{
std::stringstream ss(in);
ss >> out;
}

However, my test code that previously took 30s to run, now takes 45s
(linux gcc 4.0.2 with -03 option). That's a heavy price to pay for
adopting the C++ convention. So, my question is... Is there an
*efficient* way to convert a string to a number (e.g. double) without
requiring the use of old C libraries?

C is legal C++ (mostly). If the C version works better, why not use it?
C libraries aren't old, they're part of modern C++.

john

[Puppet_Sock]

zexpe wrote:

I have an extremely cpu/data intensive piece of code that makes heavy
use of the following function:

void convertToDouble(const std::string& in, double& out)
{
out = atof(in.c_str());
}

I would really like to get away from using any old C-style functions.
So, I modified the above function to make it follow the C++ convention:

void convertToDouble(const std::string& in, double& out)
{
std::stringstream ss(in);
ss >> out;
}

However, my test code that previously took 30s to run, now takes 45s
(linux gcc 4.0.2 with -03 option). That's a heavy price to pay for
adopting the C++ convention. So, my question is... Is there an
*efficient* way to convert a string to a number (e.g. double) without
requiring the use of old C libraries?

Hey, you could roll your own. <evil grin>

But seriously, what was the objection to using atof()? It's part of the
language for a reason. stringstream does plenty of stuff other than
just what atof does, so you get to pay for the overhead.

Also, you might be able to speed up some stuff about your stringstream
use, depending on exactly what your compiler is doing. For example,
what about making your variable ss static? You might check a few
examples of ways of doing that. Maybe the ctor call for strinstream is
where a lot of the extra time is going. But measure it before you
decide.
You also need to be more aware of initializing it as required on each
call instead of depending on the ctor to prepare it for you.
Socks

[deane_gavin]

John Harrison wrote:

zexpe wrote:

<snip code comparing atof and stringstream>

However, my test code that previously took 30s to run, now takes 45s
(linux gcc 4.0.2 with -03 option). That's a heavy price to pay for
adopting the C++ convention. So, my question is... Is there an
*efficient* way to convert a string to a number (e.g. double) without
requiring the use of old C libraries?

C is legal C++ (mostly). If the C version works better, why not use it?
C libraries aren't old, they're part of modern C++.

I may be wrong, but doesn't atof suffer from the same problem as atoi -
namely that it's impossible to distinguish correctly converting zero
from an error case. I believe strtod would be a better C library
solution.

Gavin Deane

[wittempj]

A more efficient approach using the same approach you have can be found
here:
http://www.parashift.com/c++-faq-lit....html#faq-39.2

[Greg]

Puppet_Sock wrote:

zexpe wrote:

I have an extremely cpu/data intensive piece of code that makes heavy
use of the following function:

void convertToDouble(const std::string& in, double& out)
{
out = atof(in.c_str());
}

I would really like to get away from using any old C-style functions.
So, I modified the above function to make it follow the C++ convention:

void convertToDouble(const std::string& in, double& out)
{
std::stringstream ss(in);
ss >> out;
}

However, my test code that previously took 30s to run, now takes 45s
(linux gcc 4.0.2 with -03 option). That's a heavy price to pay for
adopting the C++ convention. So, my question is... Is there an
*efficient* way to convert a string to a number (e.g. double) without
requiring the use of old C libraries?

Hey, you could roll your own. <evil grin>

But seriously, what was the objection to using atof()? It's part of the
language for a reason. stringstream does plenty of stuff other than
just what atof does, so you get to pay for the overhead.

atof() has been "deprecated" which means that it is on the way out of
the standard and should not be used in new code. It is also neither
thread-safe nor async canceable on most systems.

strtod() is the recommended replacement. Instead of calling atof like
this

#include <cstdlib>
char * nptr; // pointer to string of the number
...

double num = std::atof(nptr);

just replace it with a call to strtod():

double num = std::strtod(nptr, NULL);

Greg

[zexpe]

wi******@hotmail.com wrote:

A more efficient approach using the same approach you have can be found
here:
http://www.parashift.com/c++-faq-lit....html#faq-39.2

Is that not the same solution?

How does it differ/improve the efficiency of the conversation?

[zexpe]

Puppet_Sock wrote:

Also, you might be able to speed up some stuff about your stringstream
use, depending on exactly what your compiler is doing. For example,
what about making your variable ss static? You might check a few
examples of ways of doing that. Maybe the ctor call for strinstream is
where a lot of the extra time is going. But measure it before you
decide.
You also need to be more aware of initializing it as required on each
call instead of depending on the ctor to prepare it for you.

Yep. If ss were static, you'd need to clear the stream each time. Can't
see it helping.

Thanks,
Ross

[zexpe]

Greg wrote:

just replace it with a call to strtod():

double num = std::strtod(nptr, NULL);

Thanks for the tip. I'll give it a go.

My main objection to the C-way, is having to use the old <cstdlib>
library, when I'm beginning to view C++ as a complete replacement for
C. I'd rather work with C++ objects, e.g. strings, and never have to
worry about pointers, char strings etc. forever more! However, I'm
beginning to see that C++ alternatives are not always as fast as the
original C versions!

Ross

[wittempj]

Well - it uses std::istringstream (specialised for reading) and the
function is inline, although I didn't do myself a measurement these two
differences are supposed to increase performance. But the stringstream
approach will give an overhead comparted to using a function like
mentioned on the other posts

[John Harrison]

zexpe wrote:

Greg wrote:

just replace it with a call to strtod():

double num = std::strtod(nptr, NULL);

Thanks for the tip. I'll give it a go.

My main objection to the C-way, is having to use the old <cstdlib>
library, when I'm beginning to view C++ as a complete replacement for
C. I'd rather work with C++ objects, e.g. strings, and never have to
worry about pointers, char strings etc. forever more! However, I'm
beginning to see that C++ alternatives are not always as fast as the
original C versions!

Ross

It's very likely that the C++ library (strings, vectors etc.) uses the C
library internally. There's nothing wrong with you doing the same. Use
pointers but hide them in classes where they can't do any damage to the
rest of your program.

john

[Walter Bright]

"zexpe" <us****@zexpe.freeserve.co.uk> wrote in message
news:11**********************@g47g2000cwa.googlegr oups.com...

So, my question is... Is there an
*efficient* way to convert a string to a number (e.g. double) without
requiring the use of old C libraries?

Doing string to double conversion is a non-trivial, tricky process. A lot of
effort has typically gone into making the C versions (atof, strtod, ecvt,
etc.) fast and accurate. All the C++ conversions I've seen eventually wound
up being shells around the C code.

The non-C features of C++ bring nothing to the table as far as implementing
a fast, accurate string to double conversion routine, so there's no reason
to reimplement the C versions. Even the D programming language relies on
shelling the C float conversion routines.

That said, if you or anyone else can figure out a significantly better way
to do it, I'm certainly interested.

-Walter Bright
www.digitalmars.com C, C++, D programming language compilers

[Pete Becker]

Greg wrote:

atof() has been "deprecated" which means that it is on the way out of
the standard and should not be used in new code. It is also neither
thread-safe nor async canceable on most systems.

atof has not been deprecated by the standards committee. One compiler
vendor has decided to discourage the use of atof, among several others.
The merits of this decision are unclear.

--

Pete Becker
Dinkumware, Ltd. (http://www.dinkumware.com)

[Greg Comeau]

In article <11**********************@g47g2000cwa.googlegroups .com>,
zexpe <us****@zexpe.freeserve.co.uk> wrote:

I have an extremely cpu/data intensive piece of code that makes heavy
use of the following function:

void convertToDouble(const std::string& in, double& out)
{
out = atof(in.c_str());
}

I would really like to get away from using any old C-style functions.
So, I modified the above function to make it follow the C++ convention:

void convertToDouble(const std::string& in, double& out)
{
std::stringstream ss(in);
ss >> out;
}

However, my test code that previously took 30s to run, now takes 45s
(linux gcc 4.0.2 with -03 option). That's a heavy price to pay for
adopting the C++ convention. So, my question is... Is there an
*efficient* way to convert a string to a number (e.g. double) without
requiring the use of old C libraries?

It may be that you really do need to make this task faster,
but I'd for one firstly be concerned about the safety of it
even working at all. For instance, using atof() is not the
preferred function. Anyway, have a look at some of the choices at

http://www.comeaucomputing.com/techtalk/#atoi

I have not done any timing comparisions, but some combo may
be able to meet safety and efficiency satisfactorily.
--
Greg Comeau / Celebrating 20 years of Comeauity!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

[zexpe]

wi******@hotmail.com wrote:

Well - it uses std::istringstream (specialised for reading) and the
function is inline, although I didn't do myself a measurement these two
differences are supposed to increase performance. But the stringstream
approach will give an overhead comparted to using a function like
mentioned on the other posts

Thanks for your suggestion. Although explicitly declaring the function
inline (in my implementation it was already implicitly inline) and
using std::istringstream did improve the performance it was by just a
mere 0.3s of the 15s I lost by switching to a stringstream
implementation.

I guess the point I'm trying to make is that it is a pity many C++
guides, such as the C++ FAQ, boldly encourage us to use C++
methodologies in favour of the traditional C styles, without always
mentioning the performance penalty. I understand this penalty is
negligible in the majority of scenarios, but for the few such as mine,
it would be helpful if the performance penalties were more clearly
highlighted. Especially in those guides that also make a point of
claiming that C++ can still be as efficient as C! ;-)

Thanks,
Ross

[zexpe]

Greg Comeau wrote:

It may be that you really do need to make this task faster,
but I'd for one firstly be concerned about the safety of it
even working at all. For instance, using atof() is not the
preferred function. Anyway, have a look at some of the choices at

http://www.comeaucomputing.com/techtalk/#atoi

I have not done any timing comparisions, but some combo may
be able to meet safety and efficiency satisfactorily.

Thanks for the pointer. That's a very thorough discussion, albeit with
no mention of performance differences. My code typically takes 6 hours
to execute, so the difference between 6 and 9 hours execution time is
enough to force me away from the stringstream method!

I'll be using strtod() from now on. It's just as fast, and safer. For
the record here's the relative computation times of each method for my
test case, scaled to the atof() benchmark:

atof() 1.00
strtod() 1.00
istringstream 1.54
stringstream 1.56

Cheers,
Ross

[zexpe]

zexpe wrote:

atof() 1.00
strtod() 1.00
istringstream 1.54
stringstream 1.56

Just tried the boost::lexical_cast<double>() method too, and it's even
slower:

boost::lexical_cast<double>() 1.79

There's a price to paid for simplicity!

Ross

[Greg Comeau]

In article <11*********************@f14g2000cwb.googlegroups. com>,
zexpe <us****@zexpe.freeserve.co.uk> wrote:

zexpe wrote:

atof() 1.00
strtod() 1.00
istringstream 1.54
stringstream 1.56

Just tried the boost::lexical_cast<double>() method too, and it's even
slower:

boost::lexical_cast<double>() 1.79

There's a price to paid for simplicity!

I'm not convinced of that, but then I'm not unconvinced either :)

BTW, since this one operation is so important to you (why again?),
I'm surprised you are not pursuing tweaks of the different ways
and/or even trying to get strtod faster (if that is possible).
I mean, assuming 1.0 truly is the current best, since it matters
so much, why settle for it?
--
Greg Comeau / Celebrating 20 years of Comeauity!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

[Roland Pibinger]

On Wed, 16 Nov 2005 13:00:40 -0800, "Walter Bright"
<wa****@nospamm-digitalmars.com> wrote:

Doing string to double conversion is a non-trivial, tricky process. A lot of
effort has typically gone into making the C versions (atof, strtod, ecvt,
etc.) fast and accurate. All the C++ conversions I've seen eventually wound
up being shells around the C code.

The non-C features of C++ bring nothing to the table as far as implementing
a fast, accurate string to double conversion routine, so there's no reason
to reimplement the C versions. Even the D programming language relies on
shelling the C float conversion routines.

Which interface would you prefer for a C++ 'shell' str2long which is:
- convenient
- does appropriate error handling?

a function interface? E.g.

long str2long (const std::string& str) throw (std::runtime_error);//
(1)
long str2long (const std::string& str) throw(); // (2)
bool str2long (const std::string& str, long& out) throw(); // (3)
std::pair<bool,long> str2long (const std::string& str) throw();// (4)
....?

(1) throws exception in case of conversion error
(2) returns default value (== 0) in case of error
(3) returns true/false in case of error and result in out-parameter
(4) returns true/false in .first, result in .second

(2) cannot distinguish if return 0 indicates an error or not; Is
therfore an additional isLong (const std::string& str) function
necessary (a la IsNumeric())?

Or do you prefer a class interface? E.g.

class Converter {
public:
long str2long (const std::string& str) throw();
bool isOk() throw(); // (5)
// ...
};

(5) true if previous conversion has been ok

Best regards,
Roland Pibinger

[Roland Pibinger]

On 17 Nov 2005 08:37:56 -0800, "zexpe" <us****@zexpe.freeserve.co.uk>
wrote:

I'll be using strtod() from now on. It's just as fast, and safer. For
the record here's the relative computation times of each method for my
test case, scaled to the atof() benchmark:

atof() 1.00
strtod() 1.00
istringstream 1.54
stringstream 1.56

The stringstream times are very fast. Do you reuse the same
stringstream object all the time? Or do you use a Standard library
which implements SSO for std::string?

Best wishes,
Roland Pibinger

[MIckey Mathieson]

This is the fastest way to convert String to Double.
double d;
char[10] s;

// must make sure s in a valid double or conversion will not work!

memcpy(&d, s, 8); // assuming that double are 8 bytes long.

Mickey

"Roland Pibinger" <rp*****@yahoo.com> wrote in message
news:43**************@news.utanet.at...

On 17 Nov 2005 08:37:56 -0800, "zexpe" <us****@zexpe.freeserve.co.uk>
wrote:

I'll be using strtod() from now on. It's just as fast, and safer. For
the record here's the relative computation times of each method for my
test case, scaled to the atof() benchmark:

atof() 1.00
strtod() 1.00
istringstream 1.54
stringstream 1.56

The stringstream times are very fast. Do you reuse the same
stringstream object all the time? Or do you use a Standard library
which implements SSO for std::string?

Best wishes,
Roland Pibinger

[Old Wolf]

zexpe wrote:

zexpe wrote:

atof() 1.00
strtod() 1.00
istringstream 1.54
stringstream 1.56

Just tried the boost::lexical_cast<double>() method too,
and it's even slower:

boost::lexical_cast<double>() 1.79

I consider the boost one to be the least simple: it can
throw an exception.

[zexpe]

Roland Pibinger wrote:

Which interface would you prefer for a C++ 'shell' str2long which is:
- convenient
- does appropriate error handling?

[ ... ]

The question is of implementation rather than interface. Sure, you
could write a variety of nice C++ interfaces that wrap underlying C
library calls. What we are discussing is more to do with how does one
implement a string to number conversion function without relying upon
any of the traditional C libraries, such as <cstdlib>. The only
solution is to use stringstreams which are not as efficient as the
<cstdlib> functions for the specific task required. Therefore, if speed
is required, then one must ultimately use the <cstdlib> library
functions directly (or within your own home-brewed C++ interface
wrapper, such as the ones you have described).

Ross

[zexpe]

Roland Pibinger wrote:

I'll be using strtod() from now on. It's just as fast, and safer. For
the record here's the relative computation times of each method for my
test case, scaled to the atof() benchmark:

atof() 1.00
strtod() 1.00.
istringstream 1.54
stringstream 1.56

The stringstream times are very fast. Do you reuse the same
stringstream object all the time? Or do you use a Standard library
which implements SSO for std::string?

Perhaps I should have restated more clearly here that these are not
true clear-cut scientific benchmark comparisons between each of the
methods. They simply denote by how much my program has slowed down by
changing the underlying implementation of its convertToDouble()
function. I'm just illustrating the overall impact of such a change in
my Real World program that is performing other calculations as well
which of course will have a systematic affect upon these "benchmarks".

I realise these benchmarks are terribly useful to anyone else, they
just illustrate a point. My implementation, btw, is as stated earlier
in this thread.

Ross

[zexpe]

Greg Comeau wrote:

atof() 1.00
strtod() 1.00
istringstream 1.54
stringstream 1.56
Just tried the boost::lexical_cast<double>() method too, and it's even
slower:

boost::lexical_cast<double>() 1.79

I'm not convinced of that, but then I'm not unconvinced either :)

See my last post for comments about the "benchmarks". Regarding the
boost method, it's slowness may be due to something specific about my
installation - I've never used boost before.
BTW, since this one operation is so important to you (why again?),
I'm surprised you are not pursuing tweaks of the different ways
and/or even trying to get strtod faster (if that is possible).
I mean, assuming 1.0 truly is the current best, since it matters
so much, why settle for it?

You are, of course, right. When it comes to optimisation it's best to
look at the bigger picture. I'm working on a code base that I've
inherited, and if I ever have the time, I believe it would be made much
faster by minimising the dependence on string to number operations as
much as possible. However, the design of the code at present is
strongly based on these conversions, and being a rather large program,
would take a long time to completely redesign. So, I'd rather find a
"quick fix" for the time being, and would like to be convinced that I'm
using both the most efficient technique *and* if possible the most
approved modern C++ technique. The point of this thread is a discussion
of the best, standard technique. If the code is to be completely
redesigned in the future then there's not much point trying to do
anything as ambitious as create my own string to number conversion
function.

If you're interested in the background... the code is manipulating
approximately 10 terabytes of data in an astronomical database.
However, this test is just working on 50,000 records.

Thanks for all your comments everyone!

Ross

[zexpe]

Old Wolf wrote:

I consider the boost one to be the least simple: it can
throw an exception.

I meant simple in terms of writing C++-style code and it's readability.
So, in that sense, it's just being compared to the stringstream
technique.

Ross

[Walter Bright]

"zexpe" <us****@zexpe.freeserve.co.uk> wrote in message
news:11**********************@z14g2000cwz.googlegr oups.com...

If you're interested in the background... the code is manipulating
approximately 10 terabytes of data in an astronomical database.
However, this test is just working on 50,000 records.

If you have control over the generation of the data, consider having it
generated in %a format rather than %g. It's convertible back to floating
point much faster, and avoids creeping roundoff errors.

Also, if profiling shows that strtod() really is the bottleneck, it can be
productive to take the source code for it and convert it to hand-tuned
assembler.

Walter Bright
www.digitalmars.com C, C++, D programming language compilers

[Earl Purple]

You would not want to make the istringstream static because that would
be non thread-safe. But you
might want an object that has its own istringstream and re-use the
object within the same thread for
multiple converts.

You could make your class a template, and also have a variety of ways
of handling a failed convert.
(Note, for floating point types you can always return a NaN).

Beware that if you reuse a stringstream you must reset it whenever an
error has occurred before you
can ask it to convert another string.

[Vladimir]

zexpe wrote:

would take a long time to completely redesign. So, I'd rather find a
"quick fix" for the time being, and would like to be convinced that I'm

Are you aware that atof/strtod take thousands of CPU cycles
(more than 3000 on my config)? They are quite universal to handle
many possible cases which may be not needed for you. You can
examine how your data is formatted and take great advantage of it.

For example, if strings are like "434.312827", simple code would
suffice:

double convert(char * s)
{
double result = 0.0;

// convert integer part
while(*s >= '0' && *s <= '9') result = result * 10 + (*s++ - '0');

// skip dot
s++;

// convert fractional part
double f = 0.1;
while(*s >= '0' && *s <= '9') result += (*s++ - '0') * f, f *= 0.1;

return result;
}

This runs at around 100 cycles here which gives a whopping factor of
30!
You can make it safe and even faster after examining your data.

--
Vladimir

[Greg Comeau]

In article <11*********************@f14g2000cwb.googlegroups. com>,
Vladimir <vl*********@yahoo.com> wrote:

zexpe wrote:

would take a long time to completely redesign. So, I'd rather find a
"quick fix" for the time being, and would like to be convinced that I'm

Are you aware that atof/strtod take thousands of CPU cycles
(more than 3000 on my config)? They are quite universal to handle
many possible cases which may be not needed for you. You can
examine how your data is formatted and take great advantage of it.

For example, if strings are like "434.312827", simple code would
suffice:

double convert(char * s)
{
double result = 0.0;

// convert integer part
while(*s >= '0' && *s <= '9') result = result * 10 + (*s++ - '0');

// skip dot
s++;

// convert fractional part
double f = 0.1;
while(*s >= '0' && *s <= '9') result += (*s++ - '0') * f, f *= 0.1;

return result;
}

This runs at around 100 cycles here which gives a whopping factor of
30!
You can make it safe and even faster after examining your data.

zexpe, this, and Walter's point is my point: Your "complaint" is
that stringstreams were too burdensome on an X hour program run,
but frankly although the difference between 1.5 second and 1 second
is 50%, either (a) the time of this one operations does not really
matter on your program or (b) it does matter whereas if it were
me I would look at how to make it better -- imagine you could
get it to 0.50 or less, etc. As Walter pointed out, find out
first and foremost if this is really your bottleneck, then look
at the stragegies Walter and Vladimir mentioned, and any others,
otherwise, we don't understand your premise.
--
Greg Comeau / Celebrating 20 years of Comeauity!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

[zexpe]

Greg Comeau wrote:

zexpe, this, and Walter's point is my point: Your "complaint" is
that stringstreams were too burdensome on an X hour program run,
but frankly although the difference between 1.5 second and 1 second
is 50%, either (a) the time of this one operations does not really
matter on your program or (b) it does matter whereas if it were
me I would look at how to make it better -- imagine you could
get it to 0.50 or less, etc. As Walter pointed out, find out
first and foremost if this is really your bottleneck, then look
at the stragegies Walter and Vladimir mentioned, and any others,
otherwise, we don't understand your premise.

Sorry for confusing you Greg. I think I ended up confusing myself!

The thread started following my attempt to upgrade a piece of C++ code
I'd inherited, as an experiment in "modernising" old code design. These
days we are recommended to use STL containers and algorithms, and
abandon old C-style practises wherever possible. So I ditched the
<cstdlib> atof() in favour of the stringstream technique, but
discovered it made my program run 50% more slowly than before. Clearly
this is too high a price to pay for just "modernisation". Being
aggrieved at having been given bad advice (well, incomplete advice)
about the benefits of the stringstream technique, I posted a message to
find out if there was an equivalent modern C++ to my simple
stringstream approach that did not have a performance penalty. Clearly
there is not.

I had not originally intended to be optimising the code (I *was* happy
with its original execution time and there are more important things to
be done). I was merely modernising the code as a learning exercise for
myself. However, this exercise has highlighted the fact that my program
spends a significant proprotion of its time converting strings to
numbers. I think a lot of these conversions are strictly not necessary
and simply a product of poor overall code design. So, when I have the
time I'll redesign the code, trying to remove as many of these
conversions as possible (I believe the original author wasn't very
mindful of the 1000s of CPU cycles a string to number conversion
consumes). If these conversions remain the most time consuming part of
the program, then I shall investigate the strategies of Walter and
Vladimir.

Thanks for your time everyone.

Ross