generic way to access C++ libs?

[Gabriel Zachmann]

Is there any generic way to use C++ libraries from within Python.

I seem to recall that there are tools to generate wrappers for C-libraries
semi-automatically.

But those were still way too cumbersome, IMHO.

What I would like to have is some module (or whatever), with which I can
say "load this C++ library", and then, "create that C++ object" or "call
method x of C++ object y".

Without doing anything else (such as recompiling the library or generating
wrappers).

I agree that templates could pose a major problem, so I would be happy if
it worked with pre-instantiated templates.

Is there anything?
Gab.

--
/-------------------------------------------------------------------------\
| There are works which wait, |
| and which one does not understand for a long time; [...] |
| for the question often arrives a terribly long time after the answer. |
| (Oscar Wilde) |
+-------------------------------------------------------------------------+
| za**@cs.uni-bonn.de __@/' www.gabrielzachmann.org |
\-------------------------------------------------------------------------/

[Diez B. Roggisch]

> What I would like to have is some module (or whatever), with which I can

say "load this C++ library", and then, "create that C++ object" or "call
method x of C++ object y".
C++ has no concept of runtime type information (except from saying "this is
of class X). This sort of information is known as "reflection" in java or
python, and that allows for late time bindings to those languages. In other
words: you can create on the fly wrappers for them.

But c++ lacking this means that you need to feed the specification of
objecst - the header files - to some generator. WHich is exactly what
happens in the wrapper generators like swig and sip. I never toyed around
with boost, but I don't believe they made the impossible possible.

So to answer your question: Its not possible. There are other reasons as
well: C++ defines no binary layout of objects, the result is that a lib
complied with different compilers (gcc, intel, msvc) results in
incompatible binaries. So even if one would ship the lib with the header
files: to actually generate the wrappers, the appropriate compliler has to
be used. And as you rarely find compilers of different kinds on end user
machines, you'll have to do it on the developers machine.
Without doing anything else (such as recompiling the library or generating
wrappers).

As I just said: not doable.

Third thing is that differences in language design make it necessary to
invest developer time in creating wrappings: In c++ you can pass arguments
by a pointer to them, allowing to modify them in the callers stackframe (or
whereever they live). As this is not possible in python, you need to work
around this, usually by returning a tuple of modified values in addition to
the result of the method/function itself.

So there are strong limits on automatically generationg wrappers - a
minmimum of work has to be done.
--
Regards,

Diez B. Roggisch

[Gabriel Zachmann]

C++ has no concept of runtime type information (except from saying "this
is
of class X). This sort of information is known as "reflection" in java
or
let's assume we have a well-populatyed symbol table in the lib
(which is usually the case, or, at least, not too hard a restriction).
So to answer your question: Its not possible. There are other reasons as
well: C++ defines no binary layout of objects, the result is that a lib
complied with different compilers (gcc, intel, msvc) results in
incompatible binaries. So even if one would ship the lib with the header

that's not quite true.
actually, each platform (wintel, linux, ...) has a pretty well-defined
object file format (ELF under unix/linux, for instance).
current icc/linux and gcc/g++ work pretty well together in most cases, and
icc/windows and cl, too.

It is well understood that the envisioned python module would have to be
platform-specific.
Best regards,
gabriel.
--
/-------------------------------------------------------------------------\
| There are works which wait, |
| and which one does not understand for a long time; [...] |
| for the question often arrives a terribly long time after the answer. |
| (Oscar Wilde) |
+-------------------------------------------------------------------------+
| za**@cs.uni-bonn.de __@/' www.gabrielzachmann.org |
\-------------------------------------------------------------------------/

[Diez B. Roggisch]

> let's assume we have a well-populatyed symbol table in the lib

(which is usually the case, or, at least, not too hard a restriction).
So what? The names are mangled - and each one according to its compilers own
rules. Even if symbols themselves are defined: there is no data type
structure layout for structs and classes themselves stored in the symbol
table.
that's not quite true.
actually, each platform (wintel, linux, ...) has a pretty well-defined
object file format (ELF under unix/linux, for instance).
I did not talk about binary executable formats, but the memory layout of C++
objects. The c++ standard doesn't define where e.g. the vtable of an
objects virtual method resides - or even if virtual methods have to be
implemented by a vtable at first place.

A c++ object created by g++ is total garbage passed to a VC lib that appears
to
current icc/linux and gcc/g++ work pretty well together in most cases, and
icc/windows and cl, too.

No, they don't - not for C++ code. Google for name mangling and the reasons
why every compiler uses its own scheme. Intel claims that there is binary
compatibility between them compiler and gcc, but thats only true for
certain compiler versions - which usage you have no control of in your
planned scenario.

I suggest you first delve somewhat more on the subject of c++ code
generation and difficulties observed by those trying to develop libraries
for c++ (libs that are shipped to customer/users not for their own projects
of course). Thats one major reason why there are only few c++ libs out
there - the pains e.g. trolltech has to go through to not break binary
compatibility between different versions can be observed here:

http://developer.kde.org/documentati...atibility.html
--
Regards,

Diez B. Roggisch

[Diez B. Roggisch]

> A c++ object created by g++ is total garbage passed to a VC lib that

appears to

That sentence should be
A c++ object created by g++ is total garbage passed to a VC lib that
appears to use the same objects denoted by some headerfile.
--
Regards,

Diez B. Roggisch

[Skip Montanaro]

Diez> A c++ object created by g++ is total garbage passed to a VC lib
Diez> that appears to use the same objects denoted by some headerfile.

Sure, but the name mangling schemes are certainly well-defined. The GNU
c++filt program on my Mac understands the following formats according to its
--help output:

none,auto,gnu,lucid,arm,hp,edg,gnu-v3,java,gnat

I don't know what most of them are, but I guess c++filt does. I imagine
something like ctypes could be trained to know how to decipher the
signatures as well. There's still the problem of templates.

Skip

[Neal Becker]

Gabriel Zachmann wrote:

Is there any generic way to use C++ libraries from within Python.

Looked at boost::python?

[Diez B. Roggisch]

> Sure, but the name mangling schemes are certainly well-defined. The GNU

c++filt program on my Mac understands the following formats according to
its --help output:

none,auto,gnu,lucid,arm,hp,edg,gnu-v3,java,gnat

I don't know what most of them are, but I guess c++filt does. I imagine
something like ctypes could be trained to know how to decipher the
signatures as well. There's still the problem of templates.

The name mangling is not the important part - memory layout of the objects
is. As setting members is made due to offsets to the objects address in
memory, one has to know exactly in which order declared and possibly
inherited members are layed out. And as this is not part of the c++
standard, every compiler does it as it suits it.
--
Regards,

Diez B. Roggisch

[Neil Hodgson]

Diez B. Roggisch:

The name mangling is not the important part - memory layout of the objects
is. As setting members is made due to offsets to the objects address in
memory, one has to know exactly in which order declared and possibly
inherited members are layed out. And as this is not part of the c++
standard, every compiler does it as it suits it.

The layout is also modified by various compiler options. I believe that,
with symbolic debugging information, this feature could be implemented but
it would be a large amount of work.

Neil

[Jacek Generowicz]

Gabriel Zachmann <za**@cs.uni-bonn.de> writes:

Is there any generic way to use C++ libraries from within Python. Without doing anything else (such as recompiling the library or generating
wrappers).

Bit of a tall order, don't you think?

What would be so cumbersome about invoking a single program which
requires the location of the library, the location of its headers, and
which gives you a Python module wrapping the library in return ?

[Alex Martelli]

Jacek Generowicz <ja**************@cern.ch> wrote:

Gabriel Zachmann <za**@cs.uni-bonn.de> writes:

Is there any generic way to use C++ libraries from within Python.

Without doing anything else (such as recompiling the library or generating
wrappers).

Bit of a tall order, don't you think?

Well, ctypes does that for C libraries (as long as they're
DLL/so/dynlib/...), it's not immediately obvious that using C++
libraries is an order of magnitude harder (though probably true).

What would be so cumbersome about invoking a single program which
requires the location of the library, the location of its headers, and
which gives you a Python module wrapping the library in return ?

Without a C/C++ compiler around, you mean? Most Python users these days
don't have one (as they use Python on Windows)...
Alex

[Jacek Generowicz]

al*****@yahoo.com (Alex Martelli) writes:

Jacek Generowicz <ja**************@cern.ch> wrote:

Gabriel Zachmann <za**@cs.uni-bonn.de> writes:

Is there any generic way to use C++ libraries from within Python.

Without doing anything else (such as recompiling the library or
generating wrappers).

Bit of a tall order, don't you think?

Well, ctypes does that for C libraries (as long as they're
DLL/so/dynlib/...), it's not immediately obvious that using C++
libraries is an order of magnitude harder (though probably true).

Maybe not _immediately_ obvious, but obvious after a few minutes
thought :-)

What would be so cumbersome about invoking a single program which
requires the location of the library, the location of its headers, and
which gives you a Python module wrapping the library in return ?

Without a C/C++ compiler around, you mean? Most Python users these days
don't have one (as they use Python on Windows)...

Good point. I hadn't though of this one. In my environment the users
are expected to have at least one C++ compiler, and are even expected
to use it on a regular basis.

[Alex Martelli]

Jacek Generowicz <ja**************@cern.ch> wrote:
...

> Is there any generic way to use C++ libraries from within Python.

> Without doing anything else (such as recompiling the library or
> generating wrappers).

Bit of a tall order, don't you think?

Well, ctypes does that for C libraries (as long as they're
DLL/so/dynlib/...), it's not immediately obvious that using C++
libraries is an order of magnitude harder (though probably true).

Maybe not _immediately_ obvious, but obvious after a few minutes
thought :-)

To somebody with a good grasp of the current state of C++ technology,
maybe. Somebody who might just like to using existing dynlib/&c which
happen to be oriented to C++ rather than C might quite reasonably not
find the distinction obvious, IMHO.

Indeed, I suspect ctypes could be extended to do some of the requested
task, if one focused on a single, specific C++ compiler.

What would be so cumbersome about invoking a single program which
requires the location of the library, the location of its headers, and
which gives you a Python module wrapping the library in return ?

Without a C/C++ compiler around, you mean? Most Python users these days
don't have one (as they use Python on Windows)...

Good point. I hadn't though of this one. In my environment the users
are expected to have at least one C++ compiler, and are even expected
to use it on a regular basis.

Ah, yes, a definitely atypical environment. Anyway, if my guess is
correct that the demand for such a 'c++types' is really burning only on
Windows, then maybe it could be made for MS VC++7.1 specifically. But,
it IS just a guess. ctypes does require some understanding of some C
concepts, for example, even though it does not require access to a C
compiler; it also requires specific coding to a certain dynlib's
interfaces. I think Boost Python, if all needed tools were present,
might be able to do a more automatic job of producing the wrapper, maybe
requiring from the Python-level user even less C++ knowledge than that
hypothetical c++types might...
Alex

[Gabriel Zachmann]

> The name mangling is not the important part - memory layout of the objects

is. As setting members is made due to offsets to the objects address in
memory, one has to know exactly in which order declared and possibly

given the header file of the c++ lib, it should be possible to determine
that at run-time.

Or, one could give the header file to the generic python wrapper module and
tell the module also, with which compiler the c++ lib was compiled.

Gab.

--
/-------------------------------------------------------------------------\
| There are works which wait, |
| and which one does not understand for a long time; [...] |
| for the question often arrives a terribly long time after the answer. |
| (Oscar Wilde) |
+-------------------------------------------------------------------------+
| za**@cs.uni-bonn.de __@/' www.gabrielzachmann.org |
\-------------------------------------------------------------------------/

[Gabriel Zachmann]

> The layout is also modified by various compiler options.

really? that would mean that c++ libs themselves are not binary compatible
among each other?

(You are not talking of that insane issue with M$'s libs when compiled with
or without debug info, are you?)

I believe that,
with symbolic debugging information, this feature could be implemented but
it would be a large amount of work.

i can believe that. but it would tremendously foster Python's spreading.

cheers,
gab.

--
/-------------------------------------------------------------------------\
| There are works which wait, |
| and which one does not understand for a long time; [...] |
| for the question often arrives a terribly long time after the answer. |
| (Oscar Wilde) |
+-------------------------------------------------------------------------+
| za**@cs.uni-bonn.de __@/' www.gabrielzachmann.org |
\-------------------------------------------------------------------------/

[Diez B. Roggisch]

> given the header file of the c++ lib, it should be possible to determine

that at run-time.
Or, one could give the header file to the generic python wrapper module
and tell the module also, with which compiler the c++ lib was compiled.

For a certain compiler, that might work. But usually, a shipped c++ library
hasn't its header files attached to it.

And as I said before: Not everything in c++ allows for direct translation.
Go take a look at sip or swig, and how to write wrappers for them - in
theory, its only copying the header file. In practice, some amount of extra
work has to be done.

And while it might be possible to make the dyn-wrapper know the internals of
certain compilers, keeping track of these is a tedious and errorprone task
- so why not instead let the compilers do that work? And voila, you've got
your average wrapper generator.
--
Regards,

Diez B. Roggisch

[Gabriel Zachmann]

> Looked at boost::python?

Thanks a lot!

That's a very neat tool (like everything from Boost ;-) ),
and pretty close to what I was envisioning,
except that one still has to sort of manually transform header files into
BOOST_PYTHON_MODULE declarations ...

Cheers,
Gab.
--
/-------------------------------------------------------------------------\
| There are works which wait, |
| and which one does not understand for a long time; [...] |
| for the question often arrives a terribly long time after the answer. |
| (Oscar Wilde) |
+-------------------------------------------------------------------------+
| za**@cs.uni-bonn.de __@/' www.gabrielzachmann.org |
\-------------------------------------------------------------------------/

[Jacek Generowicz]

al*****@yahoo.com (Alex Martelli) writes:

Jacek Generowicz <ja**************@cern.ch> wrote:
...

Maybe not _immediately_ obvious, but obvious after a few minutes
thought :-)
To somebody with a good grasp of the current state of C++ technology,
maybe. Somebody who might just like to using existing dynlib/&c which
happen to be oriented to C++ rather than C might quite reasonably not
find the distinction obvious, IMHO.

I Absolutely agree ... which is why I put the smiley there.
Anyway, if my guess is correct that the demand for such a 'c++types'
is really burning only on Windows,
What's behind this guess ?
I think Boost Python, if all needed tools were present, might be
able to do a more automatic job of producing the wrapper,

Hmmm. Boost does refuse to make any assumptions, and therefore
requires human intervention in quite a few cases. Maybe default
assumptions could be built into Pyste. Maybe they already have; it's a
while since I looked at Pyste.

And I also suspect that, in practice, there will be a fairly low limit
on the size of library that Boost can wrap, because of compile-time
memory consumption. ISTRT wrapping about 20-30 methods taken from
about 5 classes used up half a Gb of RAM. So you probably don't want
the throw Boost at a whole library indiscriminantly.

Having said that, I recall that was some utility being developed to
split the compilation into small submodules, as an attempt to manage
the memory explosion.

[Alex Martelli]

Jacek Generowicz <ja**************@cern.ch> wrote:
...

Anyway, if my guess is correct that the demand for such a 'c++types'
is really burning only on Windows,
What's behind this guess ?

The fact that Windows is the only widespread OS where getting a compiler
isn't as easy as installing it, for free, off the OS media. Admittedly
many Mac users don't even bother to install the compilers and IDEs that
Apple packs on the MacOSX media, but then these are people not at all
interested in developing programs -- they wouldn't program using the
hypothetical c++types either.

I think Boost Python, if all needed tools were present, might be
able to do a more automatic job of producing the wrapper,

Hmmm. Boost does refuse to make any assumptions, and therefore
requires human intervention in quite a few cases. Maybe default
assumptions could be built into Pyste. Maybe they already have; it's a
while since I looked at Pyste.

Yeah, pyste is what I had in mind, I had just forgotten the name (it had
been quite a while for me, too).
And I also suspect that, in practice, there will be a fairly low limit
on the size of library that Boost can wrap, because of compile-time
memory consumption. ISTRT wrapping about 20-30 methods taken from
about 5 classes used up half a Gb of RAM. So you probably don't want
the throw Boost at a whole library indiscriminantly.
Wow -- THAT bad?! Eeek.
Having said that, I recall that was some utility being developed to
split the compilation into small submodules, as an attempt to manage
the memory explosion.

Makes sense, I guess.
Alex

[Jacek Generowicz]

al*****@yahoo.com (Alex Martelli) writes:

Jacek Generowicz <ja**************@cern.ch> wrote:
...

Anyway, if my guess is correct that the demand for such a 'c++types'
is really burning only on Windows,

What's behind this guess ?

The fact that Windows is the only widespread OS where getting a compiler
isn't as easy as installing it, for free, off the OS media.

Ah, indeed. Once again my unusual environment stops me from seeing the
blatantly obvious :-)

And I also suspect that, in practice, there will be a fairly low
limit on the size of library that Boost can wrap, because of
compile-time memory consumption. ISTRT wrapping about 20-30
methods taken from about 5 classes used up half a Gb of RAM. So
you probably don't want the throw Boost at a whole library
indiscriminantly.

Wow -- THAT bad?! Eeek.

I went back and checked.

38 methods (4 classes)
2 non-member functions.
4 instantiations of std::vector, with just one method each.

Compiling this with 256Mb RAM made the machine unusable for about 10
mins.

Upgrading to 768Mb RAM, made it not need to swap ... but it still
wasn't a breeze.

Sorry, can't be bothered to check exactly how much memory it does use.

[Jacek Generowicz]

Gabriel Zachmann <za**@cs.uni-bonn.de> writes:

Looked at boost::python?

Thanks a lot!

That's a very neat tool (like everything from Boost ;-) ),
and pretty close to what I was envisioning,
except that one still has to sort of manually transform header files into
BOOST_PYTHON_MODULE declarations ...

Then check out Pyste ... which should come with Boost itself.

[Thomas Heller]

al*****@yahoo.com (Alex Martelli) writes:

Jacek Generowicz <ja**************@cern.ch> wrote:
...

> > > Is there any generic way to use C++ libraries from within Python.
> >
> > > Without doing anything else (such as recompiling the library or
> > > generating wrappers).
> >
> > Bit of a tall order, don't you think?
>
> Well, ctypes does that for C libraries (as long as they're
> DLL/so/dynlib/...), it's not immediately obvious that using C++
> libraries is an order of magnitude harder (though probably true).

Maybe not _immediately_ obvious, but obvious after a few minutes
thought :-)

To somebody with a good grasp of the current state of C++ technology,
maybe. Somebody who might just like to using existing dynlib/&c which
happen to be oriented to C++ rather than C might quite reasonably not
find the distinction obvious, IMHO.

Indeed, I suspect ctypes could be extended to do some of the requested
task, if one focused on a single, specific C++ compiler.

I guess that MSVC uses the same binary layout for C++ objects as for COM
objects, so it should be possible. I don't know if the name mangling rules
are documented somewhere.

But I have no idea how inline definitions of member functions (I'm not
sure that's the correct term - I mean code defined in the header files)
should be converted to Python code.

Thomas

[Diez B. Roggisch]

> I guess that MSVC uses the same binary layout for C++ objects as for COM

objects, so it should be possible. I don't know if the name mangling
rules are documented somewhere.
COM has no binary layout of objects - the whole purpose of COM is to _not_
access objects by memory location, but instead by interfaces and thus
functions. Even what you perceive as attributes/properties are set/get
functions. Conveniently, COM objects can be wrapped on the fly be the
python win32 extensions.
But I have no idea how inline definitions of member functions (I'm not
sure that's the correct term - I mean code defined in the header files)
should be converted to Python code.

While inlining is an optimization technique that allows for copying method
code directly into the callee's code, they will still be exposed
explicitely as function as otherwise even linking between C++ libs won't
work.

--
Regards,

Diez B. Roggisch

[Thomas Heller]

"Diez B. Roggisch" <de*********@web.de> writes:

I guess that MSVC uses the same binary layout for C++ objects as for COM
objects, so it should be possible. I don't know if the name mangling
rules are documented somewhere.
COM has no binary layout of objects - the whole purpose of COM is to _not_
access objects by memory location, but instead by interfaces and thus
functions. Even what you perceive as attributes/properties are set/get
functions.

Sorry, I simply used the wrong terminus - I meant interfaces, of course.
Conveniently, COM objects can be wrapped on the fly be the
python win32 extensions.
Yes, but only Dispatch interfaces. ctypes is able to wrap native
interfaces by constructing the vtable at runtime, by using a manually
written or generated interface definition.

But I have no idea how inline definitions of member functions (I'm not
sure that's the correct term - I mean code defined in the header files)
should be converted to Python code.

While inlining is an optimization technique that allows for copying method
code directly into the callee's code, they will still be exposed
explicitely as function as otherwise even linking between C++ libs won't
work.

I mean, for example, code like this (from MS' GdiPlusTypes.h):

class CharacterRange
{
public:
CharacterRange(
INT first,
INT length
) :
First (first),
Length (length)
{}

CharacterRange() : First(0), Length(0)
{}

CharacterRange & operator = (const CharacterRange &rhs)
{
First = rhs.First;
Length = rhs.Length;
return *this;
}

INT First;
INT Length;
};

Thomas

[Diez B. Roggisch]

>> Conveniently, COM objects can be wrapped on the fly be the

python win32 extensions.
Yes, but only Dispatch interfaces. ctypes is able to wrap native
interfaces by constructing the vtable at runtime, by using a manually
written or generated interface definition.

COM is espacially designed to allow for this - language interoparability. So
its not to surprising that its easily wrapped. But one can draw _no_
conclusions from that about C++ - as COM is a binary standard defined in
terms of C. The fact that VStudio integrates it nicely with C++ has nothing
to do with that.
I mean, for example, code like this (from MS' GdiPlusTypes.h):

class CharacterRange <snip>
};

I know, I meant that code too.
--
Regards,

Diez B. Roggisch

[Damjan]

>> that's not quite true.

actually, each platform (wintel, linux, ...) has a pretty well-defined
object file format (ELF under unix/linux, for instance).

I did not talk about binary executable formats, but the memory layout of
C++ objects. The c++ standard doesn't define where e.g. the vtable of an
objects virtual method resides - or even if virtual methods have to be
implemented by a vtable at first place.

There's a C++ ABI standard on Linux now (maybe it covers other Unixish
platforms too). GCC-3.x supports it and also the Intel compiler supports
it. That's how you can now use (link) libraries compiled with some gcc-3.x
compiler to a program compiled with Intel's compilers or vice versa.

Though, some of the early gcc-3.0 compilers may not be 100% C++ ABI
compliant. There were some bugs there.
--
damjan

[Diez B. Roggisch]

> There's a C++ ABI standard on Linux now (maybe it covers other Unixish

platforms too). GCC-3.x supports it and also the Intel compiler supports
it. That's how you can now use (link) libraries compiled with some gcc-3.x
compiler to a program compiled with Intel's compilers or vice versa.

Though, some of the early gcc-3.0 compilers may not be 100% C++ ABI
compliant. There were some bugs there.

Interesting to hear that they work on these shortcomings of c++. From what
google told me, it looks as if this stuff is defined for Itanium processors
only - do you know if its also used for x86 systems?
--
Regards,

Diez B. Roggisch

[Jacek Generowicz]

"Diez B. Roggisch" <de*********@web.de> writes:

There's a C++ ABI standard on Linux now (maybe it covers other Unixish
platforms too). GCC-3.x supports it and also the Intel compiler supports
it. That's how you can now use (link) libraries compiled with some gcc-3.x
compiler to a program compiled with Intel's compilers or vice versa.

Though, some of the early gcc-3.0 compilers may not be 100% C++ ABI
compliant. There were some bugs there.

Interesting to hear that they work on these shortcomings of c++. From what
google told me, it looks as if this stuff is defined for Itanium processors
only - do you know if its also used for x86 systems?

Yes, I use it in GCC on x86, G4 and G5. It should work wherever GCC
works.

[Jacek Generowicz]

"Diez B. Roggisch" <de*********@web.de> writes:

But I have no idea how inline definitions of member functions (I'm not
sure that's the correct term - I mean code defined in the header files)
should be converted to Python code.

Member functions which are _defined_ (as opposed to merely declared)
within the class, are implicitly marked "inline". And the "inline"
keyword is a hint to the compiler that you suggest that the function
be inlined; it's a hind which the compiler is allowed to ignore.
While inlining is an optimization technique that allows for copying method
code directly into the callee's code, they will still be exposed
explicitely as function as otherwise even linking between C++ libs won't
work.

I don't think so. Given that the _definition_ of the function is in
the header, and the header is needed to compile any code which might
want to link with the library, the compiler has all it needs, without
a need for a corresponding function to exist in the library.

[Neil Hodgson]

Gabriel Zachmann:

The layout is also modified by various compiler options.

really? that would mean that c++ libs themselves are
not binary compatible among each other?

I was thinking in terms of Microsoft C++ options like the vtordisp
setting used to handle virtual inheritance (/vd0 or /vd1), pointer to member
representation (/vm*), and the old favourite struct alignment (/Zp*).
Borland adds options for allowing small enumerations to take only one byte
(-b-), zero size empty base classes (-Ve), zero size empty members (-Vx),
and a layout compatible with old compiler versions (-Vl).

Much of the time C++ code only has to be compatible with the other
modules it is delivered with. So long as all the modules use the same
compiler flags, this will be achieved.

Neil

[Diez B. Roggisch]

> I don't think so. Given that the _definition_ of the function is in

the header, and the header is needed to compile any code which might
want to link with the library, the compiler has all it needs, without
a need for a corresponding function to exist in the library.

It has been a while, but I think what happened when including such a header
on several locations that produced .o-files was that a duplicate symbol
problem arised on linking time. Why would there be a symbol at all, when
the code got "pasted" into the callee's code?

--
Regards,

Diez B. Roggisch

[Jacek Generowicz]

"Diez B. Roggisch" <de************@web.de> writes:

I don't think so. Given that the _definition_ of the function is in
the header, and the header is needed to compile any code which might
want to link with the library, the compiler has all it needs, without
a need for a corresponding function to exist in the library.
It has been a while, but I think what happened when including such a header
on several locations that produced .o-files was that a duplicate symbol
problem arised on linking time.

I find this hard to believe, unless we are talking about a buggy
compiler or linker. Remember that in the case of templates (until the
EDG people implemented the mythical "export" keyword, relatively
recently) the _only_ way to use templates in more than one translation
unit was to include the full _definitions_ of everything in the
template, in the header.

That's why C++ has the One Definition Rule (ODR). Here's what
Stroustrup has to say about the ODR in _The C++ Programming Language_
3rd edition (p203):

A given class, enumeration, and template, etc., must be defined
exactly once in a program. From a practical point of view, this
means that there must be exactly one definition of, say, a class
ersiding in a single file somewhere. Unfortunately, the language
rule cannot be that simple. [...] Consequently the rule in the
standard that says that there must be a unique definition of a
class, template, etc., is phrased in a somewhat more compliated and
subtle manner. This rule is commonly referred to as "the
one-definiton rule," the ODR. That is, two definitions of a class,
template, or inline function are accepted as examples of the same
unique definition if and only if

[1] they appear in different translation units
[2] they are token-for-token identical
[3] the meanings of those tokens are the same in both translation
units.

[...]

The intent of the ODR is to allow inclusion of a class definiton in
different tranlation units from a common source file.

Why would there be a symbol at all, when the code got "pasted" into
the callee's code?

Remember that "inline" (and the implicit inline you generate by
_defining_ the method in the class itself) is merely a hint to the
compiler, which the compiler is perfectly entitled to ignore. One
situation in which the compiler is _guaranteed_ to ignore it is if you
also declare the method to be virtual, for example. In general, it may
choose to inlne some of the calls while not others.

I don't think that declaring a function to be inline makes any
guarantess about whether you will find a symbol for that function in
the object code. But the ODR guarantees that sorting this mess out is
the compiler's problem, not yours.

[Diez B. Roggisch]

> Remember that "inline" (and the implicit inline you generate by

_defining_ the method in the class itself) is merely a hint to the
compiler, which the compiler is perfectly entitled to ignore. One
situation in which the compiler is _guaranteed_ to ignore it is if you
also declare the method to be virtual, for example. In general, it may
choose to inlne some of the calls while not others.

I don't think that declaring a function to be inline makes any
guarantess about whether you will find a symbol for that function in
the object code. But the ODR guarantees that sorting this mess out is
the compiler's problem, not yours.

You are right - I just created a test-project with two object files that
stem from .cc-files where the same class with an inlined function "bar" is
included. What happens is that nm shows me that the function "bar" is
defined weak - according to nm (and what I understand about that) this
means that in presence of another symbol of that name, it will be ignored.
So both files contain a definition for bar, but the linker is able to sort
it out.

--
Regards,

Diez B. Roggisch

[Jorgen Grahn]

On Tue, 09 Nov 2004 10:24:01 GMT, Neil Hodgson <nh******@bigpond.net.au> wrote:

Gabriel Zachmann:

> The layout is also modified by various compiler options.

really? that would mean that c++ libs themselves are
not binary compatible among each other?

I was thinking in terms of Microsoft C++ options like the vtordisp
setting used to handle virtual inheritance (/vd0 or /vd1), pointer to member
representation (/vm*), and the old favourite struct alignment (/Zp*).
Borland adds options for allowing small enumerations to take only one byte
(-b-), zero size empty base classes (-Ve), zero size empty members (-Vx),
and a layout compatible with old compiler versions (-Vl).

Much of the time C++ code only has to be compatible with the other
modules it is delivered with.

But most of the time it has to be compatible with the standard library.

Even if these flags exist, I think it's very rare to encounter more than one
layout, for one specific compiler/compiler version/architecture.

/Jorgen

--
// Jorgen Grahn <jgrahn@ Ph'nglui mglw'nafh Cthulhu
\X/ algonet.se> R'lyeh wgah'nagl fhtagn!

[Neil Hodgson]

Neil Hodgson:

# Much of the time C++ code only has to be compatible with the other
# modules it is delivered with.
Jorgen Grahn:

But most of the time it has to be compatible with the standard library.

Even if these flags exist, I think it's very rare to encounter more than one layout, for one specific compiler/compiler version/architecture.

The most commonly changed layout option is structure packing. Many C/C++
projects decide upon a particular packing option often just to be compatible
with files from the initial version or other applications and libraries.

The C++ standard library data types (vector, basic_string, etc.) are
mostly defined in header files which are compiled with the client code and
so use the same options. C system libraries like standard I/O and platform
libraries like Win32 use explicit compiler instructions to ensure they can
deal with client option choice. The Win32 headers use #pragma pack(n) to
ensure that client code will compile assuming particular packing of system
structures no matter how the client code is packing its own structures.

A C++ specific example is the use of the __declspec(novtable) compiler
directive (indirectly through the AFX_NOVTABLE macro) on many of the
interfaces in Microsoft's widely used ATL to remove code that references the
vtable and hence (normally) remove the vtable. A generic C++ library
accessor that opens a library based on ATL may have to know that some
classes that would be expected to have vtables will not actually have them.

Neil

[Jorgen Grahn]

On Thu, 11 Nov 2004 12:07:57 GMT, Neil Hodgson <nh******@bigpond.net.au> wrote:

Neil Hodgson:

# Much of the time C++ code only has to be compatible with the other
# modules it is delivered with.
Jorgen Grahn:

But most of the time it has to be compatible with the standard library.

Even if these flags exist, I think it's very rare to encounter more than

one

layout, for one specific compiler/compiler version/architecture.

The most commonly changed layout option is structure packing. Many C/C++
projects decide upon a particular packing option often just to be compatible
with files from the initial version or other applications and libraries.

[...] C system libraries like standard I/O and platform
libraries like Win32 use explicit compiler instructions to ensure they can
deal with client option choice. The Win32 headers use #pragma pack(n) to
ensure that client code will compile assuming particular packing of system
structures no matter how the client code is packing its own structures.

....

I see. Then this seems to me to be a Windows culture issue which I've missed
because I'm a Unix person. The GNU and Solaris libcs (and all other Unix
shared libraries that I'm aware of) do not do this. Same thing with
AmigaDOS, as I recall it. Access these from code compiled with non-standard
packing and you're toast.

(Some Unix include files (things in /usr/include/netinet/, for example) have
structs with elaborate padding so that they can match e.g. an IP header
perfectly. This places a restriction on compiler writers, of course.)

That also explains to me why so many embedded projects I've been in have
messed with structure packing - the architects were Windows people.

One reason this is rare in the non-Windows world is (I suppose) that there's
often a huge run-time penalty for accessing misaligned words on non-x86
processors. The default layout is almost always the best possible, unless
you are low on memory.

/Jörgen

--
// Jorgen Grahn <jgrahn@ Ph'nglui mglw'nafh Cthulhu
\X/ algonet.se> R'lyeh wgah'nagl fhtagn!