Embedding Python at the OS level - build problems

[Phil Frost]

Greetings all. I'm attempting to embed a python interpreter at a very
low level in an OS I am writing. Currently I'm stuck with build issues.
Firstly, as there is no working C compiler in our OS, I must cross
compile Python. I encountered errors when building Parser/pgen; I got
around this by building it nativly and copying it. Hopefully this won't
get me later...

My current problem (and there are sure to be more ;) ) is when building
Modules/posixmodule.c. I've already done a great deal of hacking; I
manually commented the inclusion of stropts.h as it was getting it from
/usr/include which is not part of the target machine's include files.
That was probably not a good idea, but hey...

Now, I get:

/home/indigo/dev/uuu/uuu/lib/c/bin-i386/diet gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -c ./Modules/posixmodule.c -o Modules/posixmodule.o
Modules/posixmodule.c: In function `posix_openpty':
Modules/posixmodule.c:2623: warning: implicit declaration of function `ioctl'
Modules/posixmodule.c:2623: error: `I_PUSH' undeclared (first use in this function)
Modules/posixmodule.c:2623: error: (Each undeclared identifier is reported only once
Modules/posixmodule.c:2623: error: for each function it appears in.)
Modules/posixmodule.c: In function `posix_setgroups':
Modules/posixmodule.c:4527: error: `MAX_GROUPS' undeclared (first use in this function)

I really have a swarm of questions. Among them are:

- is there a better way to cross compile python?
- is there anything I can do about the above error?

But most importantly:

- is the posix module really required? At the moment, I'm interested in
only getting a minimal Python working, so anything that can be
disabled, should be. What things can I do to eliminate all features,
modules, and whatever that isn't required to build libpython2.3.a?

[Ryan Paul]

On Tue, 25 May 2004 20:34:14 -0400, Phil Frost wrote:

Greetings all. I'm attempting to embed a python interpreter at a very
low level in an OS I am writing. Currently I'm stuck with build issues.
Firstly, as there is no working C compiler in our OS, I must cross
compile Python. I encountered errors when building Parser/pgen; I got
around this by building it nativly and copying it. Hopefully this won't
get me later...

My current problem (and there are sure to be more ;) ) is when building
Modules/posixmodule.c. I've already done a great deal of hacking; I
manually commented the inclusion of stropts.h as it was getting it from
/usr/include which is not part of the target machine's include files.
That was probably not a good idea, but hey...

Now, I get:

/home/indigo/dev/uuu/uuu/lib/c/bin-i386/diet gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -c ./Modules/posixmodule.c -o Modules/posixmodule.o
Modules/posixmodule.c: In function `posix_openpty':
Modules/posixmodule.c:2623: warning: implicit declaration of function `ioctl'
Modules/posixmodule.c:2623: error: `I_PUSH' undeclared (first use in this function)
Modules/posixmodule.c:2623: error: (Each undeclared identifier is reported only once
Modules/posixmodule.c:2623: error: for each function it appears in.)
Modules/posixmodule.c: In function `posix_setgroups':
Modules/posixmodule.c:4527: error: `MAX_GROUPS' undeclared (first use in this function)

I really have a swarm of questions. Among them are:

- is there a better way to cross compile python?
- is there anything I can do about the above error?

But most importantly:

- is the posix module really required? At the moment, I'm interested in
only getting a minimal Python working, so anything that can be
disabled, should be. What things can I do to eliminate all features,
modules, and whatever that isn't required to build libpython2.3.a?

interesting project. what are the benefits of integrating python into an
operating system? I could understand making kernel modules for services
that you would otherwise run as a daemon, but what is the motivation for
such low level integration of a high level interpreter?

[Phil Frost]

On Wed, May 26, 2004 at 01:09:38AM +0000, Ryan Paul wrote:

On Tue, 25 May 2004 20:34:14 -0400, Phil Frost wrote:

Greetings all. I'm attempting to embed a python interpreter at a very
low level in an OS I am writing.

[snip lotts of problems]

interesting project. what are the benefits of integrating python into an
operating system? I could understand making kernel modules for services
that you would otherwise run as a daemon, but what is the motivation for
such low level integration of a high level interpreter?

Well, it's an interesting story...
The project started entirely in x86 assembly, and we got something like
modern MS-DOS done like that. We had fun playing with the hardware, and
breaking floppy drives by seeking them too far, and it was very fast,
but we looked back and realized we hadn't really written anything new,
so we started a rewrite.

We have produced tons of ideas now, and some of them we think are even
new. One side effect of this is that we are often rewriting code to fit
new ideas, which really becomes a pain in assembly, because slight
changes in function might require now eight registers, rather than the
usually availible 7, and well, now you have to rewrite the whole thing
to accommodate an additional parameter.

Of course the usual solution is "use C!", but I hate C. It's almost as
much of a pain as assembly, but it doesn't allow anything near the
freedom or the satisfaction of working directly with the hardware. It's
my belief that there are two enjoyable aspects to programming: the joy
of creation, and the satisfaction of solving a puzzle. C removes most of
the puzzle aspect because no longer must one allocate registers and
optimize. It also deemphisizes the creation aspect by requiring the
programer to manage all the tedious aspects of computing - memory
management, string manipulation, basic data structures, etc. Very little
time is spent creating something new in C.

Consequently, it was decided that a language should be assembly, and
failing that it should remain a joy to use. We considered many languages
that were not C, such as D, Ada, Ocaml, and Python, as well as dozens of
languages too academic to remain in my memory. D and Ada both fall short
of the goal of a joyful language. Ocaml was definately cool, especially
since it maintains extremely good performance. However, it's not as
accessible as Python.

Of course, I realize Python isn't exactly erm...fast. It is my belief
that this is a matter of the way python is run, and not a problem with
the language itself. Current projects such as Pyrex and Psyco
demonstrate that Python can be made faster, and I think this trend will
continue. Furthermore, in the worst case we can always write Python
modules in C, or assembly (wheee!) where speed is important, and take
comfort in knowing that the rest of our code is maintainble, allowing us
to focus on making an OS and not cursing at C.

If you are interested, our webpage is http://unununium.org/. I hear that
the page looks very professional all the time, but I must say at this
point it's mostly fluff. There really isn't much of an OS there, and
most of our ideas havn't been documented. Time and money have been in
very short supply recently, so the project has been nearly idle for
longer than I'd like to admit. However, some day...

--
Phil Frost

[Donn Cave]

Quoth Phil Frost <in****@bitglue.com>:
....
| I really have a swarm of questions. Among them are:
|
| - is there a better way to cross compile python?
| - is there anything I can do about the above error?

I have never cross compiled anything, but I find it kind of
troubling to hear that the compiler was finding include files
in /usr/include. If they're missing in the target, it should
just fail to find them.

posixmodule is naturally one of the more difficult things to
build on a new platform. If your problems are no worse than
I_PUSH is missing, you are in fat city. socketmodule may also
be difficult, and if you have your own platform specific threading
API that will take some work.

| But most importantly:
|
| - is the posix module really required? At the moment, I'm interested in
| only getting a minimal Python working, so anything that can be
| disabled, should be. What things can I do to eliminate all features,
| modules, and whatever that isn't required to build libpython2.3.a?

I don't think posixmodule is strictly required, but that brings
up another can of worms. Python builds itself. You build it
with posixmodule (cf. Modules/Setup) so it can run setup.py.
This is, of course, to run on your build host. But of course
you need a to end up with different binary to run on your target
host. I shouldn't worry, I'm sure people do this all the time,
it just sounds kind of hairy to me.

I know I'm not much help, but you should be able to find some
good technical discussion of minimal Python builds in the Google
archives of comp.lang.python. It has definitely been done.

Donn Cave, do**@drizzle.com