A question about ## vs. juxtaposition

[Dan W.]

I've been acting as messenger the past few days between the BOOST and
Digital Mars peoples, and they can't seem to come to an agreement
about the semantics of using ## vs. juxtaposition. The problem arose
when I was trying to compile a boost example program with the DM
compiler, and the name of a file which was put together by a set of
macros, ended up as,
....\...\list10 .cpp
vs.
....\...\list10 .cpp

(notice the space before the period)

The macro where the problem occurs is in a file called
"...boost\mpl\l ist.hpp", where it goes:

# define MPL_AUX_LIST_HE ADER
BOOST_PP_CAT(li st,BOOST_MPL_LI MIT_LIST_SIZE). hpp /**/

which I was able to work-around by adding the ##, such as:
......,BOOST_MP L_LIMIT_LIST_SI ZE)##.hpp /**/

I reported the problem to boost mailing list, and got this reply:

The compilers are wrong to insert a space, and a concatenation here is undefined
behavior. A better way to fix it would be to try this:

#define MPL_AUX_LIST_HE ADER \
MPL_AUX_LIST_HE ADER_I(BOOST_PP _CAT(list,BOOST _MPL_LIMIT_LIST _SIZE)) \
/**/
#define MPL_AUX_LIST_HE ADER_I(name) name.
After posting the above at the DM forum, I got this reply:
I don't see how that derives from the spec. The only token concatention
allowed by the spec is with ##, extra whitespace should be quite irrelevant.
## is defined as token concatenation, so I don't see how that is undefined.
I suggested a compromise: just get rid of the space, and let a popular
bug live, if that's what it is, but DM insist:
The separator inserted by dmc is to make the preprocessor work right, it
isn't easilly removed. I don't really understand why boost seems to want to
rely on the 'juxtaposition-equals-concatenation' kludge, the ## operator was
added to Standard C specifically to move away from that practice.

Would someone here be so kind to clarify this question for me, and
better yet, if possible, settle it by pasting the section of the
Standard that pertains.

Thanks in advance.
dan

[Dan W.]

Sorry, I made an error when copying and pasting to my original post.
The last line of the first quoted block, where it says,

#define MPL_AUX_LIST_HE ADER_I(name) name.
should actually have said
#define MPL_AUX_LIST_HE ADER_I(name) name.hpp

Anyhow, I got a more detailed reply from the person from boost that
answered the first time, I'll just paste it below. It seems pretty
comprehensive to me, but then again, who am I? I'd welcome any
further observations.
---------------------------------------------------- Even if 'concatenation' per-se is not called for, and against
the Standard, could it be that the "." (dot) relieves the
preprocessor from
responsibility for
adding a space at the end of the preceding string (since the
dot already
acts as
a kind of 'separator'..)?
No. The preprocessor does not "insert spaces" *ever*. At this point
in translation, the preprocessor is operating on preprocessing tokens,
not characters. There is a big difference between a lack of
whitespace and concatenation. The first simply has adjacent
preprocessing tokens, while the second forms a new preprocessing
token. E.g.

#define ID(x) x

#define MACRO(a, b) ID(a)b

MACRO(+,+)

results in two immediately adjacent '+' preprocessing tokens. There
is no intervening whitespace. Whether or not whitespace exists is
irrelevant for all purposes *except* stringizing and the creation of
an <h-char-sequence>.

A preprocessor that does text stream -> text stream must insert
whitespace in order to avoid the errant retokenization that would
occur when the result gets reprocessed by some other tool (such as a C
or C++ compiler). However, that is just a hack to make it work
similarly in the presence of retokenization which does not exist in
the phases of translation.
I just find it hilarious how the boost libraries work with so
many compilers, but only need dozens of ## in many files to
work with DM. I wouldn't be surprised at all that they'd be
all wrong; --won't be the first time that everybody is wrong,
but this bug may be just about ready for acceptance by
ANSI/ISO/whatever... ;-)
I wish that arbitrary token-pasting was well-defined. However, the
example given doesn't even make sense (per se). The reason is that
token-pasting occurs prior to rescanning, so a construction like this:

#define A(x) B(x) ## .h
#define B(x) x

The period (.) gets concatenated to right parenthesis before the
expansion of B(x). Even if arbitrary token-pasting was well-defined,
the argument 'x' could contain any amount of whitespace, and cause the
construction to not work properly:

#define EMPTY()

A(file EMPTY()) // file .h

In other words, there are only certain points in which whitespace is
removed or when whitespace is condensed to only a single whitespace.
This is not one of them. As I said before, however, this kind of
problem only occurs during stringizing and during the creation of a
header-name preprocessing token of the form <h-char-sequence>.

Further, there is only one sure-fire way to guarantee that no
whitespace exists and that is to concatenate to a placemarker
preprocessing token ala C99:

#define NO_LEADING(x) NO_LEADING_I(, x)
#define NO_LEADING_I(p, x) p ## x

#define NO_TRAILING(x) NO_TRAILING_I(, x)
#define NO_TRAILING_I(p , x) x ## p

#define NO_LEADING_AND_ TRAILING(x) \
NO_LEADING(NO_T RAILING(x)) \
/**/

....but that is not currently well-defined in C++ as it is in C99.
--------------------------------------------------------------
------------------------

>The separator inserted by dmc is to make the preprocessor work right, it >isn't easilly removed. I don't really
understand why boost seems to want to >rely on the
'juxtaposition-equals-concatenation' kludge, the ## operator
was >added to Standard C specifically to move away from that
practice.

Juxtaposition is not concatenation, and a preprocessor that is
operating at the character level rather than the preprocessing token
level at this point in translation has to jump through hoops to mimic
the behavior the actual phases of translation. This is not a kludge
on Boost's side, this is a preprocessor implementation kludge
revolving around textual representation at a phase of translation
where it doesn't exist.
--------------------------------------------------------------
------------------------
Maybe if someone could paste the section of the Standard
dealing with this,
I'd much appreciate it.
Yours.
dan

There is no section of the standard that *ever* says whitespace should
be inserted. There are only places where it says whitespace should be
removed or adjacent whitespace should be condensed.

Regards,
Paul Mensonides
----------------------------------------------------