2.4.2 on AIX fails compiling _codecs_cn.c

[Paul Watson]

Any ideas why ./Modules/cjkcodecs/_codecs_cn.c fails to compile? It
appears that the CODEC_STATELESS macro is concatenating 'hz' with a
number and text.
building '_codecs_cn' extension
cc -DNDEBUG -O -I. -I/home/pwatson/src/python/Python-2.4.2/./Include
-I/home/pwatson/src/python/Python-2.4.2/Include
-I/home/pwatson/src/python/Python-2.4.2 -c
/home/pwatson/src/python/Python-2.4.2/Modules/cjkcodecs/_codecs_cn.c -o
build/temp.aix-4.3-2.4/_codecs_cn.o
"/home/pwatson/src/python/Python-2.4.2/Modules/cjkcodecs/_codecs_cn.c",
line 431.3: 1506-206 (S) Suffix of integer constant 100_encode is not valid.
"/home/pwatson/src/python/Python-2.4.2/Modules/cjkcodecs/_codecs_cn.c",
line 431.3: 1506-196 (W) Initialization between types "int(*)(union
{...}*,const void*,const unsigned short**,unsigned long,unsigned
char**,unsigned long,int)" and "int" is not allowed.

[Martin v. Löwis]

Paul Watson wrote:

Any ideas why ./Modules/cjkcodecs/_codecs_cn.c fails to compile? It
appears that the CODEC_STATELESS macro is concatenating 'hz' with a
number and text.

More likely, hz is already defined to be 100, then forming 100_encode.

It would be best if you could find out what AIX header file defines
hz to be 100, and whether there is any way to supress that definition.

Regards,
Martin

[Paul Watson]

Martin v. Löwis wrote:

Paul Watson wrote:

Any ideas why ./Modules/cjkcodecs/_codecs_cn.c fails to compile? It
appears that the CODEC_STATELESS macro is concatenating 'hz' with a
number and text.

More likely, hz is already defined to be 100, then forming 100_encode.

It would be best if you could find out what AIX header file defines
hz to be 100, and whether there is any way to supress that definition.

Regards,
Martin

This is on AIX 4.3.3

$ grep -i _hz $(find . -name m_param.h)
#define _HZ 100 /* ticks per second of the clock */
#define __hz HZ /* Berkeley uses lower case hz */
#define HZ _HZ
#define hz __hz

$ cc_r 2>&1|head -1
VisualAge C++ Professional / C for AIX Compiler, Version 5

[Paul Watson]

Martin v. Löwis wrote:

Paul Watson wrote:

Any ideas why ./Modules/cjkcodecs/_codecs_cn.c fails to compile? It
appears that the CODEC_STATELESS macro is concatenating 'hz' with a
number and text.

More likely, hz is already defined to be 100, then forming 100_encode.

It would be best if you could find out what AIX header file defines
hz to be 100, and whether there is any way to supress that definition.

Regards,
Martin

Here are the /usr/include/*.h files that include sys/m_param.h

$ grep sys/m_param $(find . -name "*.h")
../sys/pthdebug.h:#include <sys/m_param.h> /* _NGPRS, _NFPRS */
../sys/context.h:#include <sys/m_param.h>
../sys/mstsave.h:#include <sys/m_param.h> /* for machine dependent
defines*/
../sys/param.h:#include <sys/m_param.h>
../sys/proc.h:#include <sys/m_param.h>
../sys/sched.h:#include <sys/m_param.h>

Can we #undef _ALL_SOURCE for _codecs_cn.c compilation?

There is a description in sys/context.h that seems to suggest that it is
an internal vs. external definition issue.

+44 /*
+45 * XPG4.2 requires structures and structure elements to be
defined such
+46 * that they do not pollute the namespace. _ALL_SOURCE
contains the
+47 * kernel version, while not _ALL_SOURCE contains the sanitized
versions.
+48 */

[Martin v. Löwis]

Paul Watson wrote:

This is on AIX 4.3.3

$ grep -i _hz $(find . -name m_param.h)
#define _HZ 100 /* ticks per second of the clock */
#define __hz HZ /* Berkeley uses lower case hz */
#define HZ _HZ
#define hz __hz

I expected to see something like this. However: in the file containing
the #define hz: Is there any #ifdef around it that could be
disabled/enabled?

Regards,
Martin

[Martin v. Löwis]

Paul Watson wrote:

Can we #undef _ALL_SOURCE for _codecs_cn.c compilation?

Where does _ALL_SOURCE come from? Why is it defined?
What is its effect on hz?

Regards,
Martin

[Paul Watson]

Martin v. Löwis wrote:

Paul Watson wrote:

Can we #undef _ALL_SOURCE for _codecs_cn.c compilation?

Where does _ALL_SOURCE come from? Why is it defined?
What is its effect on hz?

Regards,
Martin

Martin v. Löwis wrote: Paul Watson wrote:

Can we #undef _ALL_SOURCE for _codecs_cn.c compilation?

Where does _ALL_SOURCE come from? Why is it defined?
What is its effect on hz?

Regards,
Martin

It appears that _ALL_SOURCE gets defined in the /usr/include/standards.h
file. If we could #define _ANSI_C_SOURCE or _POSIX_SOURCE, it appears
that it would eleminate _ALL_SOURCE.

$ cat t.c
#include <stdio.h>
#include <stdlib.h>

int main()
{
printf("hello, world\n");

#ifdef _ALL_SOURCE
printf("hello, again\n");
#endif

exit(0);
}

$ cc_r t.c

$ ./a.out
hello, world
hello, again

[Martin v. Löwis]

Paul Watson wrote:

It appears that _ALL_SOURCE gets defined in the /usr/include/standards.h
file. If we could #define _ANSI_C_SOURCE or _POSIX_SOURCE, it appears
that it would eleminate _ALL_SOURCE.

Ah, ok - this should be easy enough. Python would normally define
_POSIX_SOURCE (through _XOPEN_SOURCE), but configure(.in) has this
block:

# On AIX 4 and 5.1, mbstate_t is defined only when _XOPEN_SOURCE ==
500 but
# used in wcsnrtombs() and mbsnrtowcs() even if _XOPEN_SOURCE is not
defined
# or has another value. By not (re)defining it, the defaults come in
place.
AIX/4)
define_xopen_source=no;;
AIX/5)
if test `uname -r` -eq 1; then
define_xopen_source=no
fi
;;

which causes _XOPEN_SOURCE (and subsequently probably _POSIX_SOURCE) not
to be defined. What AIX version are you using? Can you try removing
the fragment from configure(.in), rerun configure, verify that
_XOPEN_SOURCE is defined in pyconfig.h, and then try building again?

If this works, this might be a solution. Otherwise, we need to put
something like this into _codecs_cn.c:

#ifdef hz
/* On AIX version such-and-such, hz is defined because _ALL_SOURCE is
defined, this in turn is defined because _XOPEN_SOURCE is not.
As _XOPEN_SOURCE cannot be enabled (see configure.in), we just
work around by removing the hz definition again. */
#undef hz
#endif

Regards,
Martin

[Paul Watson]

Martin v. Löwis wrote:

Paul Watson wrote:

It appears that _ALL_SOURCE gets defined in the
/usr/include/standards.h file. If we could #define _ANSI_C_SOURCE or
_POSIX_SOURCE, it appears that it would eleminate _ALL_SOURCE.

Ah, ok - this should be easy enough. Python would normally define
_POSIX_SOURCE (through _XOPEN_SOURCE), but configure(.in) has this
block:

# On AIX 4 and 5.1, mbstate_t is defined only when _XOPEN_SOURCE ==
500 but
# used in wcsnrtombs() and mbsnrtowcs() even if _XOPEN_SOURCE is not
defined
# or has another value. By not (re)defining it, the defaults come in
place.
AIX/4)
define_xopen_source=no;;
AIX/5)
if test `uname -r` -eq 1; then
define_xopen_source=no
fi
;;

which causes _XOPEN_SOURCE (and subsequently probably _POSIX_SOURCE) not
to be defined. What AIX version are you using? Can you try removing
the fragment from configure(.in), rerun configure, verify that
_XOPEN_SOURCE is defined in pyconfig.h, and then try building again?

If this works, this might be a solution. Otherwise, we need to put
something like this into _codecs_cn.c:

#ifdef hz
/* On AIX version such-and-such, hz is defined because _ALL_SOURCE is
defined, this in turn is defined because _XOPEN_SOURCE is not.
As _XOPEN_SOURCE cannot be enabled (see configure.in), we just
work around by removing the hz definition again. */
#undef hz
#endif

Regards,
Martin

Commenting out the section in configure(.in) did not cause it to work.
It still ended up with '100_encode' complaint.

Using '#undef hz' in ./Modules/cjkcodecs/_codecs_cn.c does cause it to
compile. However, it will not pass ./Lib/test/test_codecsencoding_cn.py
or ./Lib/test/test_codecmaps_cn.py. The others (_hk, _jp, _kr, and _tw)
do pass the test.

I also note that compiles occurring after the complaint about not
finding Tcl/Tk do not appear to get the OPT= setting I specified on the
'make' command line. It starts with compilation of structmodule.c and
includes the _codecs_??.c files. Does this have any significance? Is
it possible that there are other settings not being used?

$ python test_codecencodings_cn.py
test_chunkcoding (__main__.Test_GB2312) ... ERROR
test_customreplace (__main__.Test_GB2312) ... ERROR
test_errorhandle (__main__.Test_GB2312) ... ERROR
test_streamreader (__main__.Test_GB2312) ... ERROR
test_streamwriter (__main__.Test_GB2312) ... ERROR
test_xmlcharrefreplace (__main__.Test_GB2312) ... ERROR
test_chunkcoding (__main__.Test_GBK) ... ERROR
test_customreplace (__main__.Test_GBK) ... ERROR
test_errorhandle (__main__.Test_GBK) ... ERROR
test_streamreader (__main__.Test_GBK) ... ERROR
test_streamwriter (__main__.Test_GBK) ... ERROR
test_xmlcharrefreplace (__main__.Test_GBK) ... ERROR
test_chunkcoding (__main__.Test_GB18030) ... ERROR
test_customreplace (__main__.Test_GB18030) ... ERROR
test_errorhandle (__main__.Test_GB18030) ... ERROR
test_streamreader (__main__.Test_GB18030) ... ERROR
test_streamwriter (__main__.Test_GB18030) ... ERROR
test_xmlcharrefreplace (__main__.Test_GB18030) ... ERROR

================================================== ====================
ERROR: test_chunkcoding (__main__.Test_GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb2312

================================================== ====================
ERROR: test_customreplace (__main__.Test_GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb2312

================================================== ====================
ERROR: test_errorhandle (__main__.Test_GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb2312

================================================== ====================
ERROR: test_streamreader (__main__.Test_GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb2312

================================================== ====================
ERROR: test_streamwriter (__main__.Test_GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb2312

================================================== ====================
ERROR: test_xmlcharrefreplace (__main__.Test_GB2312)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb2312

================================================== ====================
ERROR: test_chunkcoding (__main__.Test_GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gbk

================================================== ====================
ERROR: test_customreplace (__main__.Test_GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gbk

================================================== ====================
ERROR: test_errorhandle (__main__.Test_GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gbk

================================================== ====================
ERROR: test_streamreader (__main__.Test_GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gbk

================================================== ====================
ERROR: test_streamwriter (__main__.Test_GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gbk

================================================== ====================
ERROR: test_xmlcharrefreplace (__main__.Test_GBK)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gbk

================================================== ====================
ERROR: test_chunkcoding (__main__.Test_GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb18030

================================================== ====================
ERROR: test_customreplace (__main__.Test_GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb18030

================================================== ====================
ERROR: test_errorhandle (__main__.Test_GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb18030

================================================== ====================
ERROR: test_streamreader (__main__.Test_GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb18030

================================================== ====================
ERROR: test_streamwriter (__main__.Test_GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb18030

================================================== ====================
ERROR: test_xmlcharrefreplace (__main__.Test_GB18030)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pwatson/usr/lib/python2.4/test/test_multibytecodec_support.py",
line 27, in setUp
self.codec = codecs.lookup(self.encoding)
LookupError: unknown encoding: gb18030

----------------------------------------------------------------------
Ran 18 tests in 0.015s

FAILED (errors=18)
Traceback (most recent call last):
File "test_codecencodings_cn.py", line 60, in ?
test_main()
File "test_codecencodings_cn.py", line 57, in test_main
test_support.run_suite(suite)
File "/home/pwatson/usr/lib/python2.4/test/test_support.py", line
274, in run_suite
raise TestFailed(msg)
test.test_support.TestFailed: errors occurred; run in verbose mode for
details
10:38 pwatson [ kbs80:/home/pwatson/src/python/Python-2.4.2/Lib/test ] 672
$ python test_codecencodings_hk.py
test_chunkcoding (__main__.Test_Big5HKSCS) ... ok
test_customreplace (__main__.Test_Big5HKSCS) ... ok
test_errorhandle (__main__.Test_Big5HKSCS) ... ok
test_streamreader (__main__.Test_Big5HKSCS) ... ok
test_streamwriter (__main__.Test_Big5HKSCS) ... ok
test_xmlcharrefreplace (__main__.Test_Big5HKSCS) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.063s

OK

[Martin v. Löwis]

Paul Watson wrote:

I also note that compiles occurring after the complaint about not
finding Tcl/Tk do not appear to get the OPT= setting I specified on the
'make' command line. It starts with compilation of structmodule.c and
includes the _codecs_??.c files. Does this have any significance? Is
it possible that there are other settings not being used?

At this point, I'm giving up. I would need access to an AIX and more
time than I have to be of further help.

Good luck,
Martin