Is there any new C++ compiler standard conformance test result?

Tao Wang wrote:[color=blue]
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi,
>
> I saw cuj's conformance roundup, but the result is quite old. I think
> many people like me want to know newer c++ standard conformance test
> result. Especially, I want to know the compiler conformance of vc7.1,
> vc8.0, gcc 4.x and intel c++, and also each implementation of STL. The
> world is moving on, right? :)[/color]

Well, it's always possible that cuj will publish another survey. In the
meantime, a fairly good barometer of compiler compliance would probably
be the Boost Release summary: (http://tinyurl.com/7aq2t - be patient,
page takes awhile to load), The information though has to be mined from
these build reports. Figuring out the limitations of particular
compiler requires sifting through the failures and error messages to
understand the problem.

Alternately, much can be learned from most compilers' documentation -
although again it is often necessary to consult extensive release notes
for the latest information. Compiler vendors for competitive reasons
may not always wish to highlight areas that put them at a competitive
disadvantage vis-a-vis other compilers. I'm not suggesting that there
is anything untoward in doing so, and indeed compiler makers tend to be
remarkably forthcoming about the features and limitations of their
products, at least in my limited experience.

And lastly I would note that the best conformance suites are likely to
be commercial. When I worked for a C embedded systems compiler company,
each release of the compiler was tested against the Plum Hall
validation suite - a series of tortuous tests designed to thoroughly
vet the compiler's ANSI C's conformance. Needless to say, the results
of a run were used to fix bugs in the compiler before its release; so
the company had every reason to keep these results private.
Furthermore, Plum Hall put a lot of work and a lot of expertise into
writing their test suites and, given the small size of the market for C
and C++ test suites in general, they are probably too expensive for
anyone other than a compiler maker to license.

Greg

In article <8Ewbf.10445$Hj2.2369@news-server.bigpond.net.au>,
Tao Wang <dancefire@gmail.com> wrote:[color=blue]
>I saw cuj's conformance roundup, but the result is quite old. I think
>many people like me want to know newer c++ standard conformance test
>result. Especially, I want to know the compiler conformance of vc7.1,
>vc8.0, gcc 4.x and intel c++, and also each implementation of STL. The
>world is moving on, right? :)[/color]

I have prodded a number of publications to do said, but
as with any ongoing project, that can take considerable work,
and so it never materializes. Even when smaller scopes,
and different themes to consider, are mentioned.
--
Greg Comeau / Celebrating 20 years of Comeauity!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

In article <1131323358.519297.154360@f14g2000cwb.googlegroups. com>,
Greg <greghe@pacbell.net> wrote:[color=blue]
>a fairly good barometer of compiler compliance would probably
>be the Boost Release summary....[/color]

The Boost folks themselves disagree with such a statement,
so we should too.
--
Greg Comeau / Celebrating 20 years of Comeauity!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

Greg Comeau wrote:[color=blue]
> In article <1131323358.519297.154360@f14g2000cwb.googlegroups. com>,
> Greg <greghe@pacbell.net> wrote:[color=green]
> >a fairly good barometer of compiler compliance would probably
> >be the Boost Release summary....[/color]
>
> The Boost folks themselves disagree with such a statement,
> so we should too.[/color]

Granted, the purpose of the Release pages is not to test compiler
compliance, nor do the Boost libraries provide any type of systematic
test coverage. Furthermore the compiler is not the only tool used in a
build, so a failure may be a problem with the linker, for example. As I
tried to communicate in my original post, the information here is quite
raw. To find data that would help answer the poster's question would
require carefully sifting through all of these results and would
necessarily entail discarding a lot of irrelevant information.

Nevertheless, it would be difficult to assemble another set of C++
source files on one's own that would make use of so many advanced
language features as the do the Boost libraries. So it must be possible
to learn something about a compiler by having it build this set of
libraries. How useful that information may be is another matter.
Certainly, it's unlikely that one's own code would look much like a
Boost library, so it is valid to argue that any information gleaned
from the Boost build results is of academic interest. In strictly
practical terms, I doubt it makes much difference to the average C++
programmer whether:


boost::wave::cpplexer::lex_input_interface<boost::wave::cpplexer::lex_token<T2>>
*boost::wave::cpplexer::new_lexer_gen<T1, T2>::new_lexer(
const T1 &,
const T1 &,
const T2 &,
boost::wave::language_support)
[with T1=char *,
T2=boost::wave::util::file_position<boost::wave::util::
flex_string<char, std::char_traits<char>,
std::allocator<char>,
boost::wave::util::CowString<boost::wave::util::
AllocatorStringStorage<char,std::allocator<char>>, char
*>>>

is multiply-defined or not.

But with the maturity of today's C++ compilers, the issue of language
compliance really has been pushed to the more esoteric language
constructs. The days when compiling "hello, world" in order to
reasonably assess a C++ compiler have passed. Granted there are easier
ways to answer the question than sifting through reams of build logs,
but if those methods are not readily available, then answering the
question can still be done I believe, but it would require a lot of
work.

Greg

In article <1131349390.677056.326030@g49g2000cwa.googlegroups. com>,
Greg <greghe@pacbell.net> wrote:[color=blue]
>Greg Comeau wrote:[color=green]
>> In article <1131323358.519297.154360@f14g2000cwb.googlegroups. com>,
>> Greg <greghe@pacbell.net> wrote:[color=darkred]
>> >a fairly good barometer of compiler compliance would probably
>> >be the Boost Release summary....[/color]
>>
>> The Boost folks themselves disagree with such a statement,
>> so we should too.[/color]
>
>Granted, the purpose of the Release pages is not to test compiler
>compliance, nor do the Boost libraries provide any type of systematic
>test coverage. Furthermore the compiler is not the only tool used in a
>build, so a failure may be a problem with the linker, for example.[/color]

Yes, that can be so. But...
[color=blue]
>As I
>tried to communicate in my original post, the information here is quite
>raw. To find data that would help answer the poster's question would
>require carefully sifting through all of these results and would
>necessarily entail discarding a lot of irrelevant information.[/color]

.... you did communicate that originally, and it's a leap that
does not exist.
[color=blue]
>Nevertheless, it would be difficult to assemble another set of C++
>source files on one's own that would make use of so many advanced
>language features as the do the Boost libraries.[/color]

Probably, but that's a very different issue.
[color=blue]
>So it must be possible
>to learn something about a compiler by having it build this set of
>libraries.[/color]

Sure, but again, a different issue.
[color=blue]
>How useful that information may be is another matter.[/color]

Exactly.
[color=blue]
>The days when compiling "hello, world" in order to
>reasonably assess a C++ compiler have passed.[/color]

Even though that drags in "tons" of stuff,
that could never reasonable access a C++ compiler.
What it did was give a quick and gentle indication
if there was even a fighting chance in that direction.
[color=blue]
>Granted there are easier
>ways to answer the question than sifting through reams of build logs,
>but if those methods are not readily available, then answering the
>question can still be done I believe, but it would require a lot of
>work.[/color]

The Boost folks themselves disagree with such a statement,
so we should too.
--
Greg Comeau / Celebrating 20 years of Comeauity!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greg Comeau wrote:[color=blue]
> In article <1131349390.677056.326030@g49g2000cwa.googlegroups. com>,
> Greg <greghe@pacbell.net> wrote:
>[color=green]
>>Greg Comeau wrote:
>>[color=darkred]
>>>In article <1131323358.519297.154360@f14g2000cwb.googlegroups. com>,
>>>Greg <greghe@pacbell.net> wrote:
>>>
>>>>a fairly good barometer of compiler compliance would probably
>>>>be the Boost Release summary....
>>>
>>>The Boost folks themselves disagree with such a statement,
>>>so we should too.[/color]
>>
>>Granted, the purpose of the Release pages is not to test compiler
>>compliance, nor do the Boost libraries provide any type of systematic
>>test coverage. Furthermore the compiler is not the only tool used in a
>>build, so a failure may be a problem with the linker, for example.[/color]
>
>
> Yes, that can be so. But...
>
>[color=green]
>>As I
>>tried to communicate in my original post, the information here is quite
>>raw. To find data that would help answer the poster's question would
>>require carefully sifting through all of these results and would
>>necessarily entail discarding a lot of irrelevant information.[/color]
>
>
> ... you did communicate that originally, and it's a leap that
> does not exist.
>
>[color=green]
>>Nevertheless, it would be difficult to assemble another set of C++
>>source files on one's own that would make use of so many advanced
>>language features as the do the Boost libraries.[/color]
>
>
> Probably, but that's a very different issue.
>
>[color=green]
>>So it must be possible
>>to learn something about a compiler by having it build this set of
>>libraries.[/color]
>
>
> Sure, but again, a different issue.
>
>[color=green]
>>How useful that information may be is another matter.[/color]
>
>
> Exactly.
>
>[color=green]
>>The days when compiling "hello, world" in order to
>>reasonably assess a C++ compiler have passed.[/color]
>
>
> Even though that drags in "tons" of stuff,
> that could never reasonable access a C++ compiler.
> What it did was give a quick and gentle indication
> if there was even a fighting chance in that direction.
>
>[color=green]
>>Granted there are easier
>>ways to answer the question than sifting through reams of build logs,
>>but if those methods are not readily available, then answering the
>>question can still be done I believe, but it would require a lot of
>>work.[/color]
>
>
> The Boost folks themselves disagree with such a statement,
> so we should too.[/color]

Thanks Greg and Greg Comeau for answers.

Boost compiler release page is good for overview of the compiler but not
good enough I think.

In cuj's conformance roundup, Herb Sutter used Dinkumware, Perennial and
Plum Hall's test case tools for help to get the result. I think they are
more objective to the truth. I prefer the style of Perennial and Plum
Hall, they try to make a whole list of C++ feature according to the C++
Standard Document.

If there is a test case set provided by C++ Standard committee, it
should make developer much easizer to have more clear concept of current
using compiler ability. What it can do, what is cannot do.

But today, it's not open, maybe only linguist know the real different
between each compiler or library. For user or normal developer ,it's not
clear enough. Only way to know the limits of compiler is I do something
which I really sure it should correct, and it doesn't work. or it worked
on some other compiler, but this time it doesn't work. It will comfuse
some beginners to learn the modern features of C++.

If we have a open c++ standard conformance test tools, it will much
clear to know the current using compiler, and will help us choosing
between the compiler.

Thanks.

Dancefire
- --
CCNA
http://www.dancefires.com/
http://blog.csdn.net/dancefire/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFDb0+eRS5AkKgtcCcRAn5YAKCw2t8+CfKngfEa522wxW lqnSZ/xACfYSoV
D5n+Ky9uAMKUoVqS7GPLTRk=
=dV4s
-----END PGP SIGNATURE-----

"Tao Wang" <dancefire@gmail.com> wrote in message
news:JcIbf.10984$Hj2.5783@news-server.bigpond.net.au...
[color=blue]
> Thanks Greg and Greg Comeau for answers.
>
> Boost compiler release page is good for overview of the compiler but not
> good enough I think.
>
> In cuj's conformance roundup, Herb Sutter used Dinkumware, Perennial and
> Plum Hall's test case tools for help to get the result. I think they are
> more objective to the truth. I prefer the style of Perennial and Plum
> Hall, they try to make a whole list of C++ feature according to the C++
> Standard Document.[/color]

Agreed. It's difficult to get any published results of Plum Hall and
Perennial tests, but you can see the results of our library tests at
our web site. They're nearly three years old now, and we're about to
update them, but the results have changed remarkably little. STLport
and Rogue Wave have been stagnant for years, while libstdc++ is
improving slowly. Only the Dinkumware C99 and C++ libraries score
anywhere near 100 per cent on conformance. (We fail none of our own
tests, of course, and just a handful of Perennial C++ tests -- because
of compiler bugs.) Since the other C++ libraries depends on existing
C libraries, they fare particularly poorly on tests for the C
enhancements required by the C++ Standard.

As for the C++ language, it should be well known by now that gcc and
VC++ V7.1/V8.0 score high on conformance, except that neither implements
separate compilation of templates. Only the Edison Design Group front
end does, and it gets essentially perfect scores on the validation
suites. You'll find the EDG front end in quite a few commercial compilers
out there.
[color=blue]
> If there is a test case set provided by C++ Standard committee, it
> should make developer much easizer to have more clear concept of current
> using compiler ability. What it can do, what is cannot do.[/color]

There are a gazillion reasons why the C and C++ committees will never
issue validation suites. The sheer amount of work involved is daunting,
and the politics is nearly insurmountable.
[color=blue]
> But today, it's not open, maybe only linguist know the real different
> between each compiler or library. For user or normal developer ,it's not
> clear enough. Only way to know the limits of compiler is I do something
> which I really sure it should correct, and it doesn't work. or it worked
> on some other compiler, but this time it doesn't work. It will comfuse
> some beginners to learn the modern features of C++.[/color]

That is indeed one of the drawbacks of a complex language.
[color=blue]
> If we have a open c++ standard conformance test tools, it will much
> clear to know the current using compiler, and will help us choosing
> between the compiler.[/color]

There *are* test suites available for things like gcc, libstdc++,
and STLport. Unfortunately, they're more in the nature of regression
tests for their respective products. Try them on a different
compiler or library and you'll get all sorts of false failures
because of dialect issues. They're also a bit thin and spotty, IMO.

P.J. Plauger
Dinkumware, Ltd.
http://www.dinkumware.com