473,287 Members | 1,582 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

Too Many if Statements?

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

Feb 7 '06 #1
39 6776
slogging_away wrote:
Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?


Nope - but I never saw any code block with 1150+ tests in it neither :-/

Smells like a design problem anyway. Have you considered refactoring
your code ?
--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
Feb 7 '06 #2
slogging_away wrote:
Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?


I can't say I've run into it before but your description makes me think
there are other approaches you could take. Does splitting the large
number of if statements into separate functions help at all?
If you're checking some text to see if a number of static strings are
contained therein, you could put the test strings into a list and loop
through...something like this (untested):

tests = [
'looking for this string',
'and this one',
'am i in the text'
] # etc...

for test in tests:
if test not in text:
raise LookupError

Feb 7 '06 #3
I don't consider myself to be a seasoned programmer so if you mean
redesigning the script to make the checks and therefore reduce the
number of 'if' statements, I'm not sure if that can be done. The
script needs to make numerous checks for the existence of particular
strings within the configuration file. It also uses 'if' statements to
determine what type of file is being examined, etc.. If an error is
encounterd it writes warning messages to a master file. I guess what I
am trying to say is that in order to make the many checks on the
configuration files I do not know of any other way than to check for
the existance of particular statements, (strings), and then report on
those if they are incorrect or missing - hence at least one 'if'
statement for every check.

I appreciate the feedback though!

Feb 7 '06 #4
Ah! I see what you are saying snoe, (and most likely what bruno at
modulix was recommending). That technique should provide a workaround
to the direct 'if' approach currently used and also offer some
modularity to the logic as well.

Thank you for pointing me in the right direction - I'll give it a go.

Feb 7 '06 #5
slogging_away wrote:
I don't consider myself to be a seasoned programmer
nor do I.
so if you mean
redesigning the script to make the checks and therefore reduce the
number of 'if' statements, I'm not sure if that can be done.
I strongly doubt it could *not* be done !-)
The
script needs to make numerous checks for the existence of particular
strings within the configuration file. It also uses 'if' statements to
determine what type of file is being examined, etc.. If an error is
encounterd it writes warning messages to a master file.
Yeps, that's pretty common with this kind of scripts. I recently had a
script doing thousands of regexp substitutions, image resizing, file
moves, database inserts etc, and of course a fair amount of logging. And
I can tell you that there many few "if" in this code.
I guess what I
am trying to say is that in order to make the many checks on the
configuration files I do not know of any other way than to check for
the existance of particular statements, (strings), and then report on
those if they are incorrect or missing - hence at least one 'if'
statement for every check.
Suppose you have to match a line against a list of regexp and log if it
doesn't match. You could of course repeat the whole code for each
regexp, ie:

if not re.match(r'a/regexp/here', line):
log('a first message')

if not re.match(r'another/regexp/here', line):
log('another message')

(... 150 regexps later ...)

if not re.match(r'150/regexps/later', line):
log('pfww, getting tired of copy/pasting')

etc...

But you could also factor much of it:

def checkMatch(line, regexp, msg):
if not re.match(regexp, line):
log(msg)

then have a list of regexps/messages pairs and:
for exp, msg in regexps:
checkMatch(line, exp, msg)

And now, you can add as many thousands regexps you want, you still have
one (and only one) if in the code (well, in this snippet at least...).
I appreciate the feedback though!


You're welcome !-)

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
Feb 7 '06 #6
In article <11**********************@z14g2000cwz.googlegroups .com>,
slogging_away <hi***@yahoo.com> wrote:
Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?


I generated files with 10000, 25000, and 50000 simple if statements and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000 segfaulted
and died. My system has plenty of memory and it isn't obvious to me why python
should be so bothered about this. I'm not sure why I can have 10x the number of
if statements that cause you trouble. There might be some overall limitation
on the number of statements in a file.

Alan
--
Defendit numerus
Feb 7 '06 #7
Try the state(s) pattern!

"slogging_away" <hi***@yahoo.com> wrote:
Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

--
Regards,
Casey
Feb 7 '06 #8
Alan Morgan wrote:
slogging_away wrote:
Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?

I generated files with 10000, 25000, and 50000 simple if statements and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000 segfaulted
and died. My system has plenty of memory and it isn't obvious to me why python
should be so bothered about this. I'm not sure why I can have 10x the number of
if statements that cause you trouble. There might be some overall limitation
on the number of statements in a file.


I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.

Code that does *not* demo the error on my systems:

#! /usr/bin/env python

lines = ["""#! /usr/bin/env python

import random
c = 0
n = random.randrange(10L**10)
"""]

for i in range(100000):
lines.append('if n % random.randrange(2, 1000) == 0: c += 1')
lines.append('print c')
lines.append('##############')
progtext = '\n'.join(lines)

f = file('manyifs.py', 'w')
f.write(progtext)
f.close()

exec progtext

--
--Bryan
Feb 7 '06 #9
Hmmm - good responses all around. Thank you all for your valued
feedback.

Perhaps it's too may 'if' statements under the for XXX in range(x,x,x)
statement as most of the 'if' statements appear there. It could be
something entirely else. I'm afraid its a bug with Python, (if I try
and run it several times it keeps going to the IDLE console prompt and
it eventually crashes out of Python entirely).

Some useful suggestions were provided in terms of better design so that
may be my route at this point. Thanks again for all of your help!

Feb 7 '06 #10

"Bryan Olson" <fa*********@nowhere.org> wrote in message
news:GU*****************@newssvr25.news.prodigy.ne t...
I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.


The OP did not specify whether all of his if-tests were sequential as in
your test or if some were nested. I vaguely remember there being an indent
limit (40??).

tjr

Feb 7 '06 #11
In article <GU*****************@newssvr25.news.prodigy.net> ,
Bryan Olson <fa*********@nowhere.org> wrote:
Alan Morgan wrote:
slogging_away wrote:
Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

I generated files with 10000, 25000, and 50000 simple if statements and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000 segfaulted
and died. My system has plenty of memory and it isn't obvious to me why python
should be so bothered about this. I'm not sure why I can have 10x the number of
if statements that cause you trouble. There might be some overall limitation
on the number of statements in a file.


I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.


Mine was a simple

#!/usr/local/bin/python

zot=24999
if zot == 0:
print "It's 0"

if zot == 1:
print "It's 1"

.....

if zot == 24999:
print "It's 24999"

generated (I'm ashamed to admit) by a perl script. Is there any good reason why
it is failing? I'd prefer a "Too many silly walks in your program. Reduce!" to
a crash. I could experiment with putting the matching 'if' at the beginning
rather than at the end, but I'm not sure what that would tell me.

Alan
--
Defendit numerus
Feb 7 '06 #12
Terry Reedy wrote:
"Bryan Olson" wrote:
I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.


The OP did not specify whether all of his if-tests were sequential as in
your test or if some were nested. I vaguely remember there being an indent
limit (40??).


A 40-level indent limit is reasonable (I may get around to looking
it up and/or testing it), but if we believe 'slogging' and Alan, what
we have here is an outright Python bug. They did not report a message
about static code being too deeply nested, nor about dynamic calls
exceeding a recursion limit. They reported incorrect behavior.

My favorite joke goes:

Patient: Doctor...doctor -- it hurts when I do that! (With 'that'
being some movement that is unusual but normally innocuous.)

Doctor: Don't do that.

Ah...cracks me up. Funny on so many levels.
--
--Bryan
Feb 7 '06 #13

slogging_away wrote:
Hmmm - good responses all around. Thank you all for your valued
feedback.

Perhaps it's too may 'if' statements under the for XXX in range(x,x,x)
Have you tried xrange() instead of range()?
statement as most of the 'if' statements appear there. It could be
something entirely else. I'm afraid its a bug with Python, (if I try
and run it several times it keeps going to the IDLE console prompt and
it eventually crashes out of Python entirely).

Some useful suggestions were provided in terms of better design so that
may be my route at this point. Thanks again for all of your help!


Feb 7 '06 #14
Alan Morgan wrote:

generated (I'm ashamed to admit) by a perl script. Is there any good reason why
it is failing? I'd prefer a "Too many silly walks in your program. Reduce!" to
a crash.


Everyone,

Please file a bug report anytime you make Python crash!

http://sourceforge.net/tracker/?grou...70&atid=105470

If you aren't willing to deal with SF, at least send me source code
(not a description) that makes python crash.

n

Feb 8 '06 #15
This is because Python has a hidden mechanism to detect programs
generated by Perl scripts, and make them crash with no explanation

Pierre

Feb 8 '06 #16
Bryan Olson wrote:
Alan Morgan wrote:
slogging_away wrote:
Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

Due to the number of checks perfromed by the script on the text files,
(over 500), there are quite a few 'if' statements in the script, (over
1150). It seems that it is at the point that when I add any additional
'if' statements the script will not run. No error is produced - it
just returns to the python prompt much the same as when a successful
'Check Module' command is selected. If I delete some other 'if'
statements the new ones work so it appears that it has hit a limit on
the number of 'if' statements. This has stunted any further checks for
the script to make on the text files.

Hs anyone ever run into this sort of thing?


I generated files with 10000, 25000, and 50000 simple if statements
and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000
segfaulted
and died. My system has plenty of memory and it isn't obvious to me
why python
should be so bothered about this. I'm not sure why I can have 10x the
number of
if statements that cause you trouble. There might be some overall
limitation
on the number of statements in a file.

I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.


I tried that code. It runs fine.

However, the following gives a SystemError with only 2500 elif's.

#!/usr/bin/env python
lines = [ 'n = -1','m = 0','if n < 0:',' m = 2*n',]
for i in range(2500):
lines.append('elif n == %i:' % i)
lines.append(' m = 2*n')
prog = '\n'.join(lines)
progfile = file('if.py','w')
progfile.writelines(prog)
progfile.close()
exec prog

Traceback (most recent call last):
File "iftest.py", line 10, in ?
exec prog
SystemError: com_backpatch: offset too large

I tried this with Python 2.3.3 and 2.3.4 (Linux) and both fail.
Feb 8 '06 #17
Pierre Quentel wrote:
This is because Python has a hidden mechanism to detect programs
generated by Perl scripts, and make them crash with no explanation


KEYBOARD !

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
Feb 8 '06 #18

bruno at modulix wrote:
[...]
Suppose you have to match a line against a list of regexp and log if it
doesn't match. You could of course repeat the whole code for each
regexp, ie:

if not re.match(r'a/regexp/here', line):
log('a first message')

if not re.match(r'another/regexp/here', line):
log('another message')

(... 150 regexps later ...)

if not re.match(r'150/regexps/later', line):
log('pfww, getting tired of copy/pasting')

etc...

But you could also factor much of it:

def checkMatch(line, regexp, msg):
if not re.match(regexp, line):
log(msg)

then have a list of regexps/messages pairs and:
for exp, msg in regexps:
checkMatch(line, exp, msg)

And now, you can add as many thousands regexps you want, you still have
one (and only one) if in the code (well, in this snippet at least...).


If your checks are this complicated, I think you should consider
writing a parser for your configuration file. If you use a parser
generator it's not that difficult. Moreover a lexical analyzer could be
enough if your syntax is simple. I found Dave Beazley's PLY reasonably
easy to use: http://www.dabeaz.com/ply/

Cheers,
Nicola Musatti

Feb 8 '06 #19
In article <11**********************@g14g2000cwa.googlegroups .com>,
Pierre Quentel <qu************@wanadoo.fr> wrote:
This is because Python has a hidden mechanism to detect programs
generated by Perl scripts, and make them crash with no explanation


In my case it turned out to be python having a hidden method to detect when
you are using an ancient version of python. Retesting with a newer version
didn't find any problems.

Alan
--
Defendit numerus
Feb 8 '06 #20
I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.


I tried that code. It runs fine.

However, the following gives a SystemError with only 2500 elif's.

...

I tried this with Python 2.3.3 and 2.3.4 (Linux) and both fail.


Just tried it with 2.4.2 On FreeBSD 6.0 and I get the same result:

Traceback (most recent call last):
File "buggyif.py", line 10, in ?
exec prog
SystemError: com_backpatch: offset too large

Curtis

Feb 8 '06 #21
"ra************@gmail.com" <ra************@gmail.com> writes:
SystemError: com_backpatch: offset too large


Yeah, that sounds like there's some 16-bit fields in the bytecode format.
Feb 8 '06 #22
Juho Schultz wrote:

However, the following gives a SystemError with only 2500 elif's.
SystemErrors should never occur, if you see one it's a bug.

[valid program which demonstrates a python bug]
Traceback (most recent call last):
File "iftest.py", line 10, in ?
exec prog
SystemError: com_backpatch: offset too large

I tried this with Python 2.3.3 and 2.3.4 (Linux) and both fail.


Yup, 2.4 fails too. Unfortunately, this looks like a bugger to fix in
2.4. So I doubt it will be fixed for old versions of python. The good
news is that it's fixed for 2.5.

n

Feb 9 '06 #23
Terry Reedy wrote:
The OP did not specify whether all of his if-tests were sequential as in
your test or if some were nested. I vaguely remember there being an indent
limit (40??).
Most of the if statements are nested. Almost all of them fall under a
central 'for xxx in range(x,x,x)', (this is the statement that checks
thorugh each of the saved configuration files). Under that 'for'
statment are the bulk of the 'if' statements - some nested and some not
- some also fall under other 'for' statements. The indent level does
not exceed 10..

Delaney, Timothy (Tim) wrote:
I'm pretty sure the OP has hit the python script line limit (32767?).


The script is 4903 lines long.

Slightly off topic; I am just a Network Engineer that can write some
code that accomplishes what I need to get done. I'm learning something
new everyday but I am really blown away by the responses to this
thread. I could not buy support this good. Thanks for your responses.

Feb 9 '06 #24
Alan Morgan <am*****@xenon.Stanford.EDU> wrote:
In article <GU*****************@newssvr25.news.prodigy.net> ,
Bryan Olson <fa*********@nowhere.org> wrote:
Alan Morgan wrote:
slogging_away wrote:

Hi - I'm running Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310
32 bit (Intel)] on win32, and have a script that makes numerous checks
on text files, (configuration files), so discrepancies can be reported.
The script works fine but it appears that I may have hit a wall with
'if' statements.

I generated files with 10000, 25000, and 50000 simple if statements and ran
them. 10000 was okay, 25000 gave a bizarre internal error, and 50000 segfaulted
and died. My system has plenty of memory and it isn't obvious to me why python
should be so bothered about this. I'm not sure why I can have 10x the number of
if statements that cause you trouble. There might be some overall limitation
on the number of statements in a file.


I made a script with 100,000 if's, (code below) and it appears
to work on a couple systems, including Python 2.4.2 on Win32-XP.
So at first cut, it doesn't seem to be just the if-count that
triggers the bug.


Mine was a simple

#!/usr/local/bin/python

zot=24999
if zot == 0:
print "It's 0"

if zot == 1:
print "It's 1"

....

if zot == 24999:
print "It's 24999"

generated (I'm ashamed to admit) by a perl script. Is there any good
reason why it is failing? I'd prefer a "Too many silly walks in your
program. Reduce!" to a crash. I could experiment with putting the
matching 'if' at the beginning rather than at the end, but I'm not
sure what that would tell me.

Here[1] it works with 400000 (with 500000 it starts swapping too much)
"if"-statements generated by

==========================================
#!/usr/bin/env python

print """#!/usr/bin/env python

zot=24999
"""

for i in range(0, 400000):
print """
if zot == %d:
print "It's %d"
"""%(i,i)

============================================
[1] Python 2.4.2 (#2, Sep 30 2005, 21:19:01)
[GCC 4.0.2 20050808 (prerelease) (Ubuntu 4.0.1-4ubuntu8)]

Florian
--
No no no! In maths things are usually named after Euler, or the first
person to discover them after Euler.
[Steven D'Aprano in <pa****************************@REMOVETHIScyber.co m.au>]
Feb 9 '06 #25

slogging_away wrote:
Terry Reedy wrote:
The OP did not specify whether all of his if-tests were sequential as in
your test or if some were nested. I vaguely remember there being an indent
limit (40??).


Most of the if statements are nested. Almost all of them fall under a
central 'for xxx in range(x,x,x)', (this is the statement that checks
thorugh each of the saved configuration files). Under that 'for'
statment are the bulk of the 'if' statements - some nested and some not
- some also fall under other 'for' statements. The indent level does
not exceed 10..


Has anyone considered that this may be part of the issue? If he is
stepping through a range this is not just X if statements but n * x
where n is the number of loops. Possibly some variables that are not
getting freed between loops? (my guess would be that it is related to
logging) Anyways, no expert here, just wanted to point that out.

Feb 9 '06 #26
It appears it may not be a 'if' statment limitation at all. This is
because I added another 800 element array in which to store the various
error messages generated when a configuration file error is deteceted
based on their severity level. The simple addition of the array caused
the same symptom stated in the initial posting. This additional array
was removed and the script performs as expected. Adding it back in
cause it to not run - no error message - just a return to the >>> in
the IDLE console window much as if I had executed the 'Check Module'
command.

At this point I guess I'll find another way work around this issue via
some of the previously suggested methods , etc. It appears to be a bug
as far as I can tell.

Feb 9 '06 #27
slogging_away wrote:
It appears it may not be a 'if' statment limitation at all. This is
because I added another 800 element array
Looks like a memory problem then...
in which to store the various
error messages generated when a configuration file error is deteceted
based on their severity level.


Why storing error messages ? Why don't you just write'em out (be it to
stdout or to a file) ?

(snip)

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
Feb 10 '06 #28
slogging_away wrote:
Adding it back in
cause it to not run - no error message - just a return to the >>> in
the IDLE console window much as if I had executed the 'Check Module'
command.


What happens if you run it from the command line instead
of IDLE? After all, it might be some problem in IDLE involved
here. Even if it doesn't work correctly outside IDLE, I was
thinking that IDLE might swallow some kind of error message.
Feb 10 '06 #29
bruno at modulix wrote:
Looks like a memory problem then...
The system I am using has 2GB of memory, (unless you are syaing the
memory is faulty).
Why storing error messages ? Why don't you just write'em out (be it to
stdout or to a file) ?


I guess I could do that, (write them to a file as they are discovered).
Right now the error messages are stored in the array and then the the
array is scanned via, a for loop and the error messages are written to
several files in different formats based on the severity of the errors,
(on a per device basis, a per severity basis, etc.). This keeps the
write statements to a minimum and in a central location of the script
instead of having several statements for each individual error message
spread throughout the script, (three write statements per error message
at over 500 error messages would be a significant change).

Not to complain but if I can't use arrays then thats a pretty
significant limitation.

Feb 10 '06 #30
slogging_away wrote:
bruno at modulix wrote:
Why storing error messages ? Why don't you just write'em out (be it to
stdout or to a file) ?

I guess I could do that, (write them to a file as they are discovered).
Right now the error messages are stored in the array and then the the
array is scanned via, a for loop and the error messages are written to
several files in different formats based on the severity of the errors,
(on a per device basis, a per severity basis, etc.). This keeps the
write statements to a minimum and in a central location of the script
instead of having several statements for each individual error message
spread throughout the script, (three write statements per error message
at over 500 error messages would be a significant change).


Sounds like you might like the logging module. A single log entry can be
written to multiple destinations based on level or source of the entry.
The distribution of log entries is controlled by the logging
configuration which can be stored in a text file if you like.

Kent
Feb 10 '06 #31
Magnus Lycka wrote:
What happens if you run it from the command line instead
of IDLE? After all, it might be some problem in IDLE involved
here. Even if it doesn't work correctly outside IDLE, I was
thinking that IDLE might swallow some kind of error message.


Excellent suggestion, (behold the power of the command line!). I ran
two saved versions of the script that had produced the symptom
originally described. The fist seemed to be caused by too many 'if'
statements, the second by adding another array, but both came up with
the same system error at the command prompt level shown here:

SystemError: com_backpatch: offset too large

This error is not produced with the IDLE console but is seen only when
executing from the command line. I'll search around and see if I can
determine what this means and a possible fix. Thanks for the
suggestion!

Feb 10 '06 #32
On Fri, 10 Feb 2006 06:50:25 -0800, slogging_away wrote:
Excellent suggestion, (behold the power of the command line!). I ran
two saved versions of the script that had produced the symptom
originally described. The fist seemed to be caused by too many 'if'
statements, the second by adding another array, but both came up with
the same system error at the command prompt level shown here:

SystemError: com_backpatch: offset too large


Is this a Python error or a shell error?

If it is a Python error, perhaps you would like to post the entire
traceback?

--
Steven.

Feb 10 '06 #33
Steven D'Aprano wrote:
On Fri, 10 Feb 2006 06:50:25 -0800, slogging_away wrote:

Excellent suggestion, (behold the power of the command line!). I ran
two saved versions of the script that had produced the symptom
originally described. The fist seemed to be caused by too many 'if'
statements, the second by adding another array, but both came up with
the same system error at the command prompt level shown here:

SystemError: com_backpatch: offset too large

Is this a Python error or a shell error?

If it is a Python error, perhaps you would like to post the entire
traceback?


I would believe CamelCaseErrorMessages are produced by Python.

The message is exactly the same I reported with the 2500 elifs. I fooled
around a bit with this, and it seems that also for/while blocks
containing more than ~4860 lines give this error.

slogging_away claims his script is about 4900 lines, most of that in a
for loop, so my bet is he has trouble with the same bug.
Feb 10 '06 #34

"slogging_away" <hi***@yahoo.com> wrote in message
news:11**********************@g14g2000cwa.googlegr oups.com...
SystemError: com_backpatch: offset too large


This message is generated in the backpatch function in Python/compile.c in
the source tree. (See below: sorry, tab indents did not survive cut and
paste operation.) As the comment says, and the code shows, it patches in a
2-byte jump offset. The error is raised if the offset is not 0 after
beinging divided by 256 twice (ie, by 65536). This code is called from any
function that compiles a construct that potentially jumps: for, while,
if/elif, and short-circuiting logic expressions.

To me, this message indicates not a bug but a mismatch between code demand
and interpreter capability. The solution is to reduce the demand. In the
present case, changing

for line in file:
<extra-long body>

to

def checkline(line):
<extra-long body>
for line in file:
checkline(line)

might be sufficient. If not, break up the <extra-lone body> into multiple
pieces.

Terry Jan Reedy

static void
com_backpatch(struct compiling *c, int anchor)
{
unsigned char *code = (unsigned char *) PyString_AS_STRING(c->c_code);
int target = c->c_nexti;
int dist;
int prev;
for (;;) {
/* Make the JUMP instruction at anchor point to target */
prev = code[anchor] + (code[anchor+1] << 8);
dist = target - (anchor+2);
code[anchor] = dist & 0xff;
dist >>= 8;
code[anchor+1] = dist;
dist >>= 8;
if (dist) {
com_error(c, PyExc_SystemError,
"com_backpatch: offset too large");
break;
}
if (!prev)
break;
anchor -= prev;
}
}


Feb 11 '06 #35
slogging_away wrote:
Magnus Lycka wrote:

What happens if you run it from the command line instead
of IDLE? After all, it might be some problem in IDLE involved
here. Even if it doesn't work correctly outside IDLE, I was
thinking that IDLE might swallow some kind of error message.

Excellent suggestion, (behold the power of the command line!). I ran
two saved versions of the script that had produced the symptom
originally described. The fist seemed to be caused by too many 'if'
statements, the second by adding another array, but both came up with
the same system error at the command prompt level shown here:

SystemError: com_backpatch: offset too large

This error is not produced with the IDLE console but is seen only when
executing from the command line. I'll search around and see if I can
determine what this means and a possible fix. Thanks for the
suggestion!

Now we have seen the real error message it does appear as though this is
an error in the interpreter triggered by an unusually long source
program. See

http://mail.python.org/pipermail/pyt...er/249270.html

for a previous discussion. In that email Diez Roggisch says:

"""I can't believe that a program with 10000 lines only adding strings
can't be written in a more concise way, which would most probably solve
your problem. So I suggest you post some parts of your code so that we
can have a look at it and suggest a solution."""

The same applies here.

Clearly it would be a good idea to remove whatever problem is causing
the error, but in the meantime is there some way your code could be
written more concisely? Or even split up into multiple modules?

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

Feb 11 '06 #36

"Steve Holden" <st***@holdenweb.com> wrote in message
news:ds**********@sea.gmane.org...
Clearly it would be a good idea to remove whatever problem is causing
the error,


The problem (see my post of the com_backpatch code) is writing a compound
statement (here a for loop) with a body so large as to require a jump of
more than 64K bytes in the compiled bytecode (ie, from the test at the top
of the loop to the code that follows after the loop). Until the jump limit
is raised (likely a long wait ;-), the OP must factor some of the code out
of the loop.

Terry Jan Reedy

Feb 11 '06 #37
Now that I know the root cause of the problem, I can write around it as
suggested by Steve Holden and Terry Reedy, (and others). In fact, it
has helped me in a way as I am thinking not in terms of the easiest
solution, (read; the first one that comes to mind), but more effcient
and cleaner ways to write a section of code to accomplish the same
objective. The key was identifying the root cause which was provided by
the error message seen only at the command line level and by
contibutors to this post.

Thanks again for everyone's suggestions and expertise.

Feb 12 '06 #38
On Sat, 11 Feb 2006 15:40:49 -0500, "Terry Reedy" <tj*****@udel.edu> wrote:

"Steve Holden" <st***@holdenweb.com> wrote in message
news:ds**********@sea.gmane.org...
Clearly it would be a good idea to remove whatever problem is causing
the error,


The problem (see my post of the com_backpatch code) is writing a compound
statement (here a for loop) with a body so large as to require a jump of
more than 64K bytes in the compiled bytecode (ie, from the test at the top
of the loop to the code that follows after the loop). Until the jump limit
is raised (likely a long wait ;-), the OP must factor some of the code out
of the loop.

Easy example:
def test(n): ... while True:
... try: co = compile('if x:\n'+ n*' a=1\n','','exec')
... except Exception,e: break
... n += 1
... print 'Stopped at n=%s due to %s: %s'%(n, e.__class__.__name__,e)
...

get an idea of where to start that:
import dis
n=3
dis.dis( compile('if x:\n'+ n*' a=1\n','','exec')) 1 0 LOAD_NAME 0 (x)
3 JUMP_IF_FALSE 22 (to 28)
6 POP_TOP

2 7 LOAD_CONST 0 (1)
10 STORE_NAME 1 (a)

3 13 LOAD_CONST 0 (1)
16 STORE_NAME 1 (a)

4 19 LOAD_CONST 0 (1)
22 STORE_NAME 1 (a)
25 JUMP_FORWARD 1 (to 29)
28 POP_TOP
29 LOAD_CONST 1 (None)

32 RETURN_VALUE (2**16-7)/(13-7) 10921

back off 1
test(10920) Stopped at n=10922 due to SystemError: com_backpatch: offset too large

Decided to test the exact 65536 jump with code chunks of 16 byte-codes and one
chunk at the end to make 16 with the last JUMP_FORWARD.
n=4095
dis.dis( compile('if x:\n'+ n*' a=1+2,4 \n'+' x=0,x','','exec')) Traceback (most recent call last):
File "<stdin>", line 1, in ?
SystemError: com_backpatch: offset too large n=4094
dis.dis( compile('if x:\n'+ n*' a=1+2,4 \n'+' x=0,x','','exec'))

1 0 LOAD_NAME 0 (x)
3 JUMP_IF_FALSE 65520 (to 65526)
6 POP_TOP

2 7 LOAD_CONST 0 (1)
10 LOAD_CONST 1 (2)
13 BINARY_ADD
14 LOAD_CONST 2 (4)
17 BUILD_TUPLE 2
20 STORE_NAME 1 (a)

3 23 LOAD_CONST 0 (1)

So the corner case of 2**16 is ok. Believe it or not, I once discovered a compiler
error based on optimizing a 2**16-involving loop condition as if it were 0 and the
loop didn't execute! IIRC, I bumped into it processing an array of records with a stride of
exactly 2**n and it thought it could calculate a 16-bit number of strides for end of loop.
No good for arraysize/stridesize==2**16 ;-)

If the OP really HAD to, he could always (untested) break

if cond:
too
large
else:
two
large,also
into
if cond:
too
else:
two
if cond:
large
else:
large,also

but that reads gawdawfully. (So, I imagine, does about any code hitting the offset limit ;-)

If it's a matter of too many elifs, the OP could break that more readably, e.g. (untested)

done=True
if cond:
bla
elif c2:
bla 2
...
elif c456:
bla456
else:
done=False

more, done = not done,True
if more and c457:
bla c457
elif c458:
bla458
...
elif c1012:
bla1012
else:
done = False

more, done = not done,True
... etc

But the OP should find another approach I think,
because this is still disgusting code ;-)

Regards,
Bengt Richter
Feb 12 '06 #39
slogging_away wrote:
bruno at modulix wrote:

Looks like a memory problem then...

The system I am using has 2GB of memory, (unless you are syaing the
memory is faulty).


Nope, just that I don't know of any system with unlimited memory. BTW,
having 2GB of ram does not mean there are 2GB available for your program.

(And BTW, this is just one of the *possible* reasons of the problem
you've discovered. May be quite unrelated as well...)
Why storing error messages ? Why don't you just write'em out (be it to
stdout or to a file) ?


I guess I could do that, (write them to a file as they are discovered).


Well, this is how most programs do.
Right now the error messages are stored in the array and then the the
array is scanned via, a for loop and the error messages are written to
several files in different formats based on the severity of the errors,
(on a per device basis, a per severity basis, etc.). This keeps the
write statements to a minimum and in a central location of the script
instead of having several statements for each individual error message
spread throughout the script, (three write statements per error message
at over 500 error messages would be a significant change).
This is a design problem, not an implementation problem. Just factor out
the part that do the error message dispatch into a function, then call
this function instead of storing the message. Or even better, use an
existing logging library...
Not to complain but if I can't use arrays then thats a pretty
significant limitation.


Please understand that available memory is a limitation of the *system*
- you'd get the same limitation with *every* language ever (well, if you
know of a language that expand free ram ad infinitum, please let us know !-)
--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
Feb 13 '06 #40

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Bart Nessux | last post by:
Should an if statement have a corresponding else statement? Or, is it OK to have an if statement by itself. For completeness, it seems the two should be together, but from experience I know that a...
9
by: Jaime Wyant | last post by:
I know I've seen this somewhere, but can't seem to google it. Is there a way to use an alternate statement separator, other than the default ';'? jw
1
by: Tom D | last post by:
I'm rewriting a database interface that our company currently has. Currently it's using the Pear::DB interface, but we found that that was introducing a bit too much overhead. I'm rewriting the...
0
by: Fuzzyman | last post by:
Hello all, The following is a copy of a blog entry. It's asking a question about future statements and the built in compile function. I'd appreciate any pointers or comments about possible...
20
by: Neroku | last post by:
Hello, i would like to know what the serious definition of statements and expressions is: i know an expression are evaluated to a value, i.e: 1 == 2 5+7 foo( 1,2) and a statement is...
2
by: ojorus | last post by:
Hi! Some questions regarding the mysqli-extension (php5) 1) Prepared statements: If I understand things right, prepared statements will give better performance if you make several similar...
3
by: Dmitri | last post by:
Hello! I have a developer that is playing around with some SQL statements using VB.NET. He has a test table in a SQL 2000 database, and he has about 2000 generated INSERT statements. When the...
0
by: Gary Herron | last post by:
Ohad Frand wrote: There is no way you can consider 'elif', 'else', 'except', and 'from' statements. However, as someone pointed out, the kwlist from the keyword module is the closest thing we...
0
by: Ohad Frand | last post by:
Hi Thanks a lot for your reply I think the main uses for it is to study the language and to see that I didn't miss anything else or that something is changed from one version to another. The...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.