2.4.2 on AIX fails compiling _codecs_cn.c

[Paul Watson]

Any ideas why ./Modules/cjkcodecs/_codecs_cn.c fails to compile? It
appears that the CODEC_STATELESS macro is concatenating 'hz' with a
number and text.
building '_codecs_cn' extension
cc -DNDEBUG -O -I. -I/home/pwatson/src/python/Python-2.4.2/./Include
-I/home/pwatson/src/python/Python-2.4.2/Include
-I/home/pwatson/src/python/Python-2.4.2 -c
/home/pwatson/src/python/Python-2.4.2/Modules/cjkcodecs/_codecs_cn.c -o
build/temp.aix-4.3-2.4/_codecs_cn.o
"/home/pwatson/src/python/Python-2.4.2/Modules/cjkcodecs/_codecs_cn.c",
line 431.3: 1506-206 (S) Suffix of integer constant 100_encode is not valid.
"/home/pwatson/src/python/Python-2.4.2/Modules/cjkcodecs/_codecs_cn.c",
line 431.3: 1506-196 (W) Initialization between types "int(*)(uni on
{...}*,const void*,const unsigned short**,unsigne d long,unsigned
char**,unsigned long,int)" and "int" is not allowed.

[Martin v. Löwis]

Paul Watson wrote:

Any ideas why ./Modules/cjkcodecs/_codecs_cn.c fails to compile? It
appears that the CODEC_STATELESS macro is concatenating 'hz' with a
number and text.

More likely, hz is already defined to be 100, then forming 100_encode.

It would be best if you could find out what AIX header file defines
hz to be 100, and whether there is any way to supress that definition.

Regards,
Martin

[Paul Watson]

Martin v. Löwis wrote:

Paul Watson wrote:

Any ideas why ./Modules/cjkcodecs/_codecs_cn.c fails to compile? It
appears that the CODEC_STATELESS macro is concatenating 'hz' with a
number and text.

More likely, hz is already defined to be 100, then forming 100_encode.

It would be best if you could find out what AIX header file defines
hz to be 100, and whether there is any way to supress that definition.

Regards,
Martin

This is on AIX 4.3.3

$ grep -i _hz $(find . -name m_param.h)
#define _HZ 100 /* ticks per second of the clock */
#define __hz HZ /* Berkeley uses lower case hz */
#define HZ _HZ
#define hz __hz

$ cc_r 2>&1|head -1
VisualAge C++ Professional / C for AIX Compiler, Version 5

[Paul Watson]

Martin v. Löwis wrote:

Paul Watson wrote:

Any ideas why ./Modules/cjkcodecs/_codecs_cn.c fails to compile? It
appears that the CODEC_STATELESS macro is concatenating 'hz' with a
number and text.

More likely, hz is already defined to be 100, then forming 100_encode.

It would be best if you could find out what AIX header file defines
hz to be 100, and whether there is any way to supress that definition.

Regards,
Martin

Here are the /usr/include/*.h files that include sys/m_param.h

$ grep sys/m_param $(find . -name "*.h")
../sys/pthdebug.h:#inc lude <sys/m_param.h> /* _NGPRS, _NFPRS */
../sys/context.h:#incl ude <sys/m_param.h>
../sys/mstsave.h:#incl ude <sys/m_param.h> /* for machine dependent
defines*/
../sys/param.h:#includ e <sys/m_param.h>
../sys/proc.h:#include <sys/m_param.h>
../sys/sched.h:#includ e <sys/m_param.h>

Can we #undef _ALL_SOURCE for _codecs_cn.c compilation?

There is a description in sys/context.h that seems to suggest that it is
an internal vs. external definition issue.

+44 /*
+45 * XPG4.2 requires structures and structure elements to be
defined such
+46 * that they do not pollute the namespace. _ALL_SOURCE
contains the
+47 * kernel version, while not _ALL_SOURCE contains the sanitized
versions.
+48 */

[Martin v. Löwis]

Paul Watson wrote:

This is on AIX 4.3.3

$ grep -i _hz $(find . -name m_param.h)
#define _HZ 100 /* ticks per second of the clock */
#define __hz HZ /* Berkeley uses lower case hz */
#define HZ _HZ
#define hz __hz

I expected to see something like this. However: in the file containing
the #define hz: Is there any #ifdef around it that could be
disabled/enabled?

Regards,
Martin

[Martin v. Löwis]

Paul Watson wrote:

Can we #undef _ALL_SOURCE for _codecs_cn.c compilation?

Where does _ALL_SOURCE come from? Why is it defined?
What is its effect on hz?

Regards,
Martin

[Paul Watson]

Martin v. Löwis wrote:

Paul Watson wrote:

Can we #undef _ALL_SOURCE for _codecs_cn.c compilation?

Where does _ALL_SOURCE come from? Why is it defined?
What is its effect on hz?

Regards,
Martin

Martin v. Löwis wrote: Paul Watson wrote:

Can we #undef _ALL_SOURCE for _codecs_cn.c compilation?

Where does _ALL_SOURCE come from? Why is it defined?
What is its effect on hz?

Regards,
Martin

It appears that _ALL_SOURCE gets defined in the /usr/include/standards.h
file. If we could #define _ANSI_C_SOURCE or _POSIX_SOURCE, it appears
that it would eleminate _ALL_SOURCE.

$ cat t.c
#include <stdio.h>
#include <stdlib.h>

int main()
{
printf("hello, world\n");

#ifdef _ALL_SOURCE
printf("hello, again\n");
#endif

exit(0);
}

$ cc_r t.c

$ ./a.out
hello, world
hello, again

[Martin v. Löwis]

Paul Watson wrote:

It appears that _ALL_SOURCE gets defined in the /usr/include/standards.h
file. If we could #define _ANSI_C_SOURCE or _POSIX_SOURCE, it appears
that it would eleminate _ALL_SOURCE.

Ah, ok - this should be easy enough. Python would normally define
_POSIX_SOURCE (through _XOPEN_SOURCE), but configure(.in) has this
block:

# On AIX 4 and 5.1, mbstate_t is defined only when _XOPEN_SOURCE ==
500 but
# used in wcsnrtombs() and mbsnrtowcs() even if _XOPEN_SOURCE is not
defined
# or has another value. By not (re)defining it, the defaults come in
place.
AIX/4)
define_xopen_so urce=no;;
AIX/5)
if test `uname -r` -eq 1; then
define_xopen_so urce=no
fi
;;

which causes _XOPEN_SOURCE (and subsequently probably _POSIX_SOURCE) not
to be defined. What AIX version are you using? Can you try removing
the fragment from configure(.in), rerun configure, verify that
_XOPEN_SOURCE is defined in pyconfig.h, and then try building again?

If this works, this might be a solution. Otherwise, we need to put
something like this into _codecs_cn.c:

#ifdef hz
/* On AIX version such-and-such, hz is defined because _ALL_SOURCE is
defined, this in turn is defined because _XOPEN_SOURCE is not.
As _XOPEN_SOURCE cannot be enabled (see configure.in), we just
work around by removing the hz definition again. */
#undef hz
#endif

Regards,
Martin

[Paul Watson]

Martin v. Löwis wrote:

Paul Watson wrote:

It appears that _ALL_SOURCE gets defined in the
/usr/include/standards.h file. If we could #define _ANSI_C_SOURCE or
_POSIX_SOURCE, it appears that it would eleminate _ALL_SOURCE.

Ah, ok - this should be easy enough. Python would normally define
_POSIX_SOURCE (through _XOPEN_SOURCE), but configure(.in) has this
block:

# On AIX 4 and 5.1, mbstate_t is defined only when _XOPEN_SOURCE ==
500 but
# used in wcsnrtombs() and mbsnrtowcs() even if _XOPEN_SOURCE is not
defined
# or has another value. By not (re)defining it, the defaults come in
place.
AIX/4)
define_xopen_so urce=no;;
AIX/5)
if test `uname -r` -eq 1; then
define_xopen_so urce=no
fi
;;

which causes _XOPEN_SOURCE (and subsequently probably _POSIX_SOURCE) not
to be defined. What AIX version are you using? Can you try removing
the fragment from configure(.in), rerun configure, verify that
_XOPEN_SOURCE is defined in pyconfig.h, and then try building again?

If this works, this might be a solution. Otherwise, we need to put
something like this into _codecs_cn.c:

#ifdef hz
/* On AIX version such-and-such, hz is defined because _ALL_SOURCE is
defined, this in turn is defined because _XOPEN_SOURCE is not.
As _XOPEN_SOURCE cannot be enabled (see configure.in), we just
work around by removing the hz definition again. */
#undef hz
#endif

Regards,
Martin

Commenting out the section in configure(.in) did not cause it to work.
It still ended up with '100_encode' complaint.

Using '#undef hz' in ./Modules/cjkcodecs/_codecs_cn.c does cause it to
compile. However, it will not pass ./Lib/test/test_codecsenco ding_cn.py
or ./Lib/test/test_codecmaps_ cn.py. The others (_hk, _jp, _kr, and _tw)
do pass the test.

I also note that compiles occurring after the complaint about not
finding Tcl/Tk do not appear to get the OPT= setting I specified on the
'make' command line. It starts with compilation of structmodule.c and
includes the _codecs_??.c files. Does this have any significance? Is
it possible that there are other settings not being used?

$ python test_codecencod ings_cn.py
test_chunkcodin g (__main__.Test_ GB2312) ... ERROR
test_customrepl ace (__main__.Test_ GB2312) ... ERROR
test_errorhandl e (__main__.Test_ GB2312) ... ERROR
test_streamread er (__main__.Test_ GB2312) ... ERROR
test_streamwrit er (__main__.Test_ GB2312) ... ERROR
test_xmlcharref replace (__main__.Test_ GB2312) ... ERROR
test_chunkcodin g (__main__.Test_ GBK) ... ERROR
test_customrepl ace (__main__.Test_ GBK) ... ERROR
test_errorhandl e (__main__.Test_ GBK) ... ERROR
test_streamread er (__main__.Test_ GBK) ... ERROR
test_streamwrit er (__main__.Test_ GBK) ... ERROR
test_xmlcharref replace (__main__.Test_ GBK) ... ERROR
test_chunkcodin g (__main__.Test_ GB18030) ... ERROR
test_customrepl ace (__main__.Test_ GB18030) ... ERROR
test_errorhandl e (__main__.Test_ GB18030) ... ERROR
test_streamread er (__main__.Test_ GB18030) ... ERROR
test_streamwrit er (__main__.Test_ GB18030) ... ERROR
test_xmlcharref replace (__main__.Test_ GB18030) ... ERROR

=============== =============== =============== =============== ==========
ERROR: test_chunkcodin g (__main__.Test_ GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb2312

=============== =============== =============== =============== ==========
ERROR: test_customrepl ace (__main__.Test_ GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb2312

=============== =============== =============== =============== ==========
ERROR: test_errorhandl e (__main__.Test_ GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb2312

=============== =============== =============== =============== ==========
ERROR: test_streamread er (__main__.Test_ GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb2312

=============== =============== =============== =============== ==========
ERROR: test_streamwrit er (__main__.Test_ GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb2312

=============== =============== =============== =============== ==========
ERROR: test_xmlcharref replace (__main__.Test_ GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb2312

=============== =============== =============== =============== ==========
ERROR: test_chunkcodin g (__main__.Test_ GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gbk

=============== =============== =============== =============== ==========
ERROR: test_customrepl ace (__main__.Test_ GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gbk

=============== =============== =============== =============== ==========
ERROR: test_errorhandl e (__main__.Test_ GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gbk

=============== =============== =============== =============== ==========
ERROR: test_streamread er (__main__.Test_ GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gbk

=============== =============== =============== =============== ==========
ERROR: test_streamwrit er (__main__.Test_ GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gbk

=============== =============== =============== =============== ==========
ERROR: test_xmlcharref replace (__main__.Test_ GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gbk

=============== =============== =============== =============== ==========
ERROR: test_chunkcodin g (__main__.Test_ GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb18030

=============== =============== =============== =============== ==========
ERROR: test_customrepl ace (__main__.Test_ GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb18030

=============== =============== =============== =============== ==========
ERROR: test_errorhandl e (__main__.Test_ GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb18030

=============== =============== =============== =============== ==========
ERROR: test_streamread er (__main__.Test_ GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb18030

=============== =============== =============== =============== ==========
ERROR: test_streamwrit er (__main__.Test_ GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb18030

=============== =============== =============== =============== ==========
ERROR: test_xmlcharref replace (__main__.Test_ GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytec odec_support.py ",
line 27, in setUp
self.codec = codecs.lookup(s elf.encoding)
LookupError: unknown encoding: gb18030

----------------------------------------------------------------------
Ran 18 tests in 0.015s

FAILED (errors=18)
Traceback (most recent call last):
File "test_codecenco dings_cn.py", line 60, in ?
test_main()
File "test_codecenco dings_cn.py", line 57, in test_main
test_support.ru n_suite(suite)
File "/home/pwatson/usr/lib/python2.4/test/test_support.py ", line
274, in run_suite
raise TestFailed(msg)
test.test_suppo rt.TestFailed: errors occurred; run in verbose mode for
details
10:38 pwatson [ kbs80:/home/pwatson/src/python/Python-2.4.2/Lib/test ] 672
$ python test_codecencod ings_hk.py
test_chunkcodin g (__main__.Test_ Big5HKSCS) ... ok
test_customrepl ace (__main__.Test_ Big5HKSCS) ... ok
test_errorhandl e (__main__.Test_ Big5HKSCS) ... ok
test_streamread er (__main__.Test_ Big5HKSCS) ... ok
test_streamwrit er (__main__.Test_ Big5HKSCS) ... ok
test_xmlcharref replace (__main__.Test_ Big5HKSCS) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.063s

OK

[Martin v. Löwis]

Paul Watson wrote:

I also note that compiles occurring after the complaint about not
finding Tcl/Tk do not appear to get the OPT= setting I specified on the
'make' command line. It starts with compilation of structmodule.c and
includes the _codecs_??.c files. Does this have any significance? Is
it possible that there are other settings not being used?

At this point, I'm giving up. I would need access to an AIX and more
time than I have to be of further help.

Good luck,
Martin