473,385 Members | 1,620 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,385 software developers and data experts.

namespace/dictionary quandry


I have been delegated to produce a tool that has
python embedded in it. The desire is to have a
command line interface that inherits all the python
scripting functionality so people can use the tool
either with a flat commandline entry like you would
a debugger or with a script using scoping, loops and
conditional.

I am going from knowing nothing about python to
almost nothing so the learning curve is rather
steep at this point.

I have a dummy program that inherits code.InteractiveConsole
functionality. I can grab each input line and scan it for
commands that match my tool's syntax ("attach") and if it
does, I change the line to reflect a python legal function
call associated with that commond (DoAttach) and from there
do what I want with the line. The actual input lines are
executed with a self.push() call.

The problem I have is in timely symbol resolution. I really
want to run eval() on each of the arguments at the time
the DoAttach() function is executed, but don't know how
and where from does one get the dictionary with the proper
name space. If I run eval at the time of initial reading
of the input it works for flat first level input, but not
for nested scoping:

cli.py
bosco=5
if 1:
attach bosco
bosco=7
attach bosco

attach bosco

This will result in:
bosco=5
if 1: ... attach bosco
... bosco=7
... attach bosco
...
DoAttach: ['5']
DoAttach: ['5'] <--- WRONG (at least, not what I want) attach bosco

DoAttach: ['7']

How does one export the correct dictionary?

Attached is the toy program that creates this output.

Keep in mind that I am probably aproaching this all
wrong. Any relevant suggestions would be most appreciated,

Jack

*****************
File cli.py:
*****************
#!/usr/bin/env python

import myparse

cli = myparse.CLI(globals())
cli.interact()
*****************
File myparse.py:
*****************
import code
import re
import string
import sys

def DoAttach(args):
print "DoAttach:", args
pass
class CLI(code.InteractiveConsole):
"""Simple test of a Python interpreter augmented with custom
commands."""

commands = { \
"attach" : "DoAttach"
}

def __init__(self, locals = None):

# Call super-class initializer
code.InteractiveConsole.__init__(self, locals, "<console>")

# Compile regular expression for finding commmands
self.regexp = re.compile('[a-z]*')
def interact(self):

my_dictionary = self.locals

# Set the primary and secondary prompts
sys.ps1 = ">>> "
sys.ps2 = "... "

# Input Loop
is_more = 0
bosco = 0
while 1:
try :
# Display the appropriate prompt
if not sys.stdin.isatty():
prompt = ""
elif is_more:
prompt = sys.ps2
else:
prompt = sys.ps1

# Read the next line of input
#self.write("interact 1\n")
line = self.raw_input(prompt)

# TODO: add logging of input line here...

# Process complete lines
if 1:
line = self.process(line)

# Push incomplete lines onto input stack
if line or is_more:
is_more = self.push(line)

# Handle CTRL-C
except KeyboardInterrupt:
self.write("\nKeyboardInterrupt\n")
is_more = 0
self.resetbuffer()

# Handle CTRL-D
except EOFError:
self.write("\n")
is_more = 0
self.resetbuffer()
raise SystemExit
def process(parent, line):

# Attempt to match line against our command regular expression

temp_line = string.lstrip(line)
len_1 = len(line)
len_2 = len(temp_line)
white_spaces = len_1-len_2
if white_spaces is not 0:
front_padding = line[0:white_spaces]

match = parent.regexp.match(temp_line)
if match is not None:

# Extract the command and argument strings
cmd_string = match.group()
arg_string = string.lstrip(temp_line[match.end():])

# Find the function for this command in the command
dictionary
function = parent.commands.get(cmd_string)

if function is not None:

# Split argument string into individual arguments
args = string.split(arg_string)

# Recursively evaluate all arguments
i = 0
while i < len(args):
try:

# Grab value and position of this argument
unevaluated, pos, i = args[i], i, i + 1

# Have Python attempt evaluation of the argument

evaluated = eval(unevaluated, parent.locals)

# Did the evaluation produce a "new" result?
if str(evaluated) != str(unevaluated):

# Place the evaluation in the argument list
args = args[:pos] + \
string.split(str(evaluated)) + \
args[pos + 1:]

# Go back to the beginning of the argument
list
i = 0

except (SyntaxError, NameError):
pass

# Convert to Python function-call syntax for this
command
line = "myparse." + function + "(" + str(args) + ")"
if white_spaces is not 0:
line = front_padding + line

# Return the line to be processed by Python
return line


Jul 18 '05 #1
8 1878
Jack Carter wrote:
This will result in:
>>> bosco=5
>>> if 1: ... attach bosco
... bosco=7
... attach bosco
...
DoAttach: ['5']
DoAttach: ['5'] <--- WRONG (at least, not what I want) >>> attach bosco

DoAttach: ['7']


The suite

attach bosco
bosco = 7
attach bosco

is only executed after the final empty line. Therefore any access to the
right dictionary in the process() method will give you the current value
bosco==5 which is not what you want.
You are in effect translating the suite into (simplified)

DoAttach(5)
bosco = 7
DoAttach(5) # the previous line has not yet been executed

I think the easiest way to get the desired effect ("late binding") is like
so:

def process(self, line):
temp_line = line.lstrip()
front_padding = line[:len(line)-len(temp_line)]

match = self.regexp.match(temp_line)
if match:
line = front_padding + "myparse.DoAttach(bosco)"

return line

I. e. change the generated line to contain variable names instead of values
and leave the resolution to python. (You hint you don't wont that either.
Why?)

By the way, I wasn't able to run your code - the indentation is seriously
messed up. I recommend you ensure a 4-space indentation in all code you
currently have before you move on. That will spare you a lot of trouble
later on.

Another minor issue: 'value is 0' may or may not work depending on the
python implementation ('value is 1000' won't work even in current CPython).
With 'value == 0' you are on the safe side.

Peter

Jul 18 '05 #2
Peter,

I guess I just don't understand.

Basically this is to go into a tool that accepts
commandline arguments, one at a time, but will also
allow scripting for testing purposes. Thus the desire
to leverage off python for the commandline interface,
but not for the whole program.

The input from the user will be either from the console or
read from a command file. A simple example of what could
be entered is:

bosco=5
if 1:
print bosco
attach bosco 9
bosco=7
print bosco
attach bosco 9

The result I would expect would be for my DoAttach()
routine to receive the python evaluated value of each
of the arguments leaving the ones that it doesn't understand
alone. Whether this happens automagically or by hand I don't
care as long as I get what right value.

In the above trying to follow your advice, remembering that
at this stage of the game I am probably missing the point, this
is the result I expect:

johmar % demo.py
bosco=5
if 1: .... print bosco
.... attach bosco 9
.... bosco=7
.... print bosco
.... attach bosco 9
....
5
DoAttach: [5, 9]
7
DoAttach: [7, 9]

This is what I get:

johmar % demo.py bosco=5
if 1:

.... print bosco
.... attach bosco 9
.... bosco=7
.... print bosco
.... attach bosco 9
....
5
DoAttach: ['bosco', '9']
7
DoAttach: ['bosco', '9']

Now I realize that this is probably due to the fact that
I have the lines:

if white_spaces:
line = front_padding + "myparse." + function + "(" +
str(args) + ")"
else :
line = "myparse." + function + "(" + str(args) + ")"

which put make the arguments strings, but that is because I don't
know how to appropriately pack the "line" for later parsing. Maybe
that is the crux of my problem. Remember, there will be many commands
for my tool and the arguments will be variable length and this code
will not know what variables the gentle use would use.

Here is the simplified code with hopefully the tabs expanded
base on your earlier input. Hopefully you'll see the obvious
error of my way and point it out.

Thanks ever so much,

Jack

**********************************
demo.py
**********************************
#!/usr/bin/env python

import myparse

cli = myparse.CLI(globals())
cli.interact()

**********************************
myparse.py
**********************************
import code
import re
import string
import sys

################################################## ##############################
#
# DoAttach
#
# Dummy function that I will eventually use to do
# real stuff.
#
################################################## ##############################
def DoAttach(args):

print "DoAttach:", args
pass
class CLI(code.InteractiveConsole):
"""Simple test of a Python interpreter augmented with custom commands."""

commands = { \
"attach" : "DoAttach"
}

def __init__(self, locals = None):

# Call super-class initializer
code.InteractiveConsole.__init__(self, locals, "<console>")

# Compile regular expression for finding commmands
self.regexp = re.compile('[a-z]*')
################################################## ################
#
# interact
#
# This will read and process input lines from within
# my main application as though on a python commandline.
#
################################################## ################
def interact(self):

# Set the primary and secondary prompts
sys.ps1 = ">>> "
sys.ps2 = "... "

# Input Loop
is_more = 0
bosco = 0
while 1:
try :
# Display the appropriate prompt
if not sys.stdin.isatty():
prompt = ""
elif is_more:
prompt = sys.ps2
else:
prompt = sys.ps1

# Read the next line of input
#self.write("interact 1\n")
line = self.raw_input(prompt)

# TODO: add logging of input line here...

# Process complete lines
if 1:
line = self.process(line)

# Push incomplete lines onto input stack
if line or is_more:
is_more = self.push(line)

# Handle CTRL-C
except KeyboardInterrupt:
self.write("\nKeyboardInterrupt\n")
is_more = 0
self.resetbuffer()

# Handle CTRL-D
except EOFError:
self.write("\n")
is_more = 0
self.resetbuffer()
raise SystemExit

################################################## ################
#
# process
#
# This will determine if the input command is either
# from my application's command language or a python
# construct.
#
################################################## ################
def process(parent, line):

# Attempt to match line against our command regular expression

temp_line = string.lstrip(line)
len_1 = len(line)
len_2 = len(temp_line)

white_spaces = len_1-len_2
if white_spaces:
front_padding = line[0:white_spaces]

match = parent.regexp.match(temp_line)
if match is not None:

#parent.write("process 1\n")
# Extract the command and argument strings
cmd_string = match.group()
arg_string = string.lstrip(temp_line[match.end():])

# Find the function for this command in the command dictionary
function = parent.commands.get(cmd_string)

if function is not None:

# Split argument string into individual arguments
args = string.split(arg_string)

# Convert to Python function-call syntax for this command
if white_spaces:
line = front_padding + "myparse." + function + "(" +
str(args) + ")"
else :
line = "myparse." + function + "(" + str(args) + ")"

# Return the line to be processed by Python
return line

Jul 18 '05 #3
Peter,

Actually although your solution is a good one, it won't
really help me because of the nature of our command language.

We have commands in the form of:

result = <command_name> <filename> -e <experiment_name>

I currently scan ahead for <command_name> and could do so for
the rest to prevent them from being evaluated by python by
packing them with quotes and then unpacking them later. The
problem lies in the case where the name being used in the
commandline may or may not be a formal name such as a filename
or may be a variable that will evaluate into a filename.

This will be used in a list of filenames or a list of experiment
names iterated through a for loop.
for name in (['file1','file2','file3']): .... expClose name
....
expClose: ['file1']
expClose: ['file2']
expClose: ['file3']

But when the user tries to use a formal name that is
not through a python variable he/she hits an undefine
name error:
expClose file4

Traceback (most recent call last):
File "<console>", line 1, in ?
NameError: name 'file4' is not defined

If I prescan the <filename> and pack it with quotes I
lose the python evaluation of the name.

The same problem will occur with all my other option arguments
for which there are many.

The solution it would seem would be to do the evaluation
later within the called function. That way I could assume
that all failed eval()'ed names are literals meant for my
command and not a python variable/name.

If this makes sense, the problem I need to solve is how to
deliver the correct namespace dictionary to the called function
so I can invoke eval with it.

Does this make sense?

Regards,

Jack
Jul 18 '05 #4
Jack Carter <jc*****@johmar.engr.sgi.com> wrote:
...
The solution it would seem would be to do the evaluation
later within the called function. That way I could assume
that all failed eval()'ed names are literals meant for my
command and not a python variable/name.

If this makes sense, the problem I need to solve is how to
deliver the correct namespace dictionary to the called function
so I can invoke eval with it.

Does this make sense?


Not very, but then I didn't follow the previous LONG posts on this
thread, so I'll just answer this specific question and hope it helps.
I'll assume the known variable-names are in some dictionary (such as a
locals() or globals() or vars(something)):

class WeirdNamespace:
def __init__(self, d): self.d = d
def __getitem__(self, n): return self.d.get(n,repr(n))

voila: if n is a key in dict d, this returns the corresponding value,
otherwise it returns n suitably quoted. Just pass to eval a
WeirdNamespace(d) rather than the bare d.

It appears to me your user interface is courting trouble: if I mispell
'variablename' as 'varaiblename' I end up creating a file I didn't mean
to rather than getting a clean error about unknown variable names. But
I'll assume you know your users better than I do and that they _do_
really desire with all their hearts this unholy confusion between
variables and constants...
Alex
Jul 18 '05 #5
On Sep 22, 7:18pm, Alex Martelli wrote:
Subject: Re: namespace/dictionary quandry
Jack Carter <jc*****@johmar.engr.sgi.com> wrote:
...
The solution it would seem would be to do the evaluation
later within the called function. That way I could assume
that all failed eval()'ed names are literals meant for my
command and not a python variable/name.

If this makes sense, the problem I need to solve is how to
deliver the correct namespace dictionary to the called function
so I can invoke eval with it.

Does this make sense?
Not very, but then I didn't follow the previous LONG posts on this


Alex,

The length was really taken up with a testcase to make
the problem less hand wavey.
thread, so I'll just answer this specific question and hope it helps.
I'll assume the known variable-names are in some dictionary (such as a
locals() or globals() or vars(something)):

class WeirdNamespace:
def __init__(self, d): self.d = d
def __getitem__(self, n): return self.d.get(n,repr(n))
So, how and or where does this fit in with my example?
Does both the call to the function where I want to do
the eval() and self.push(line) command have to be in the
same namespace and or file for this to work?

voila: if n is a key in dict d, this returns the corresponding value,
otherwise it returns n suitably quoted. Just pass to eval a
WeirdNamespace(d) rather than the bare d.

It appears to me your user interface is courting trouble: if I mispell
'variablename' as 'varaiblename' I end up creating a file I didn't mean
to rather than getting a clean error about unknown variable names. But
I'll assume you know your users better than I do and that they _do_
really desire with all their hearts this unholy confusion between
variables and constants...


This is driven by customers that who want both a type-in command
language and a variant using all the power of python for test
scripting purposes. If you misspell something and there is the
possibility of it screwing something up that we can't tell either
in the parser or downstream in the backend of the tool, well that's
life in the big city.

Thanks,

Jack
Jul 18 '05 #6
Jack Carter wrote:
class WeirdNamespace:
def __init__(self, d): self.d = d
def __getitem__(self, n): return self.d.get(n,repr(n))


So, how and or where does this fit in with my example?
Does both the call to the function where I want to do
the eval() and self.push(line) command have to be in the
same namespace and or file for this to work?


I use a slight variation of Alex' suggestion:

class Locals(dict):
def __getitem__(self, name):
try:
return dict.__getitem__(self, name)
except KeyError:
return name

class CLI(code.InteractiveConsole):
"""Simple test of a Python interpreter augmented with custom
commands."""

commands = { \
"attach" : "DoAttach"
}

def __init__(self, locals = None):

# Call super-class initializer
code.InteractiveConsole.__init__(self, locals, "<console>")
self.locals = Locals(self.locals)

# Compile regular expression for finding commmands
self.regexp = re.compile('[a-z]*')

[The rest of the code is the same as in my previous post]

Now try it:
for name in "abc":

fed to the snake: for name in "abc":
.... attach name
fed to the snake: myparse.DoAttach([name])
.... attach noname
fed to the snake: myparse.DoAttach([noname])
....
fed to the snake:
DoAttach: ['a']
DoAttach: ['noname']
DoAttach: ['b']
DoAttach: ['noname']
DoAttach: ['c']
DoAttach: ['noname']

Unfortunately this will only work with Python 2.4.

Here's a solution that might work for 2.3:

[Does not require the above modifications]

def process(self, line):
indent = line.lstrip()
# Attempt to match line against our command regular expression

temp_line = string.lstrip(line)
len_1 = len(line)
len_2 = len(temp_line)

white_spaces = len_1-len_2
if white_spaces:
front_padding = line[0:white_spaces]

match = self.regexp.match(temp_line)
if match is not None:

#self.write("process 1\n")
# Extract the command and argument strings
cmd_string = match.group()
arg_string = string.lstrip(temp_line[match.end():])

# Find the function for this command in the command dictionary
function = self.commands.get(cmd_string)

if function is not None:

args = parseArgs(arg_string)
for arg in args:
if arg not in self.locals:
self.locals[arg] = arg
line = makePythonCall("myparse." + function, args)
if white_spaces:
line = front_padding + line
print "fed to the snake:", line

# Return the line to be processed by Python
return line

I just ensure that all arguments not already in self.locals are added with
their name as their value. (I also renamed 'parent' to 'self' - I could not
stand it any longer :-)

Note that I don't particularly like both hacks and would rather use plain
old python functions with standard python syntax instead of your custom
language.
Peter


Jul 18 '05 #7
Jack Carter <jc*****@johmar.engr.sgi.com> wrote:
...
later within the called function. That way I could assume
that all failed eval()'ed names are literals meant for my
command and not a python variable/name.
...
The length was really taken up with a testcase to make
the problem less hand wavey.


If you can't make a shorter example, I'm not gonna make time to study
that one, sorry.

I'll assume the known variable-names are in some dictionary (such as a
locals() or globals() or vars(something)):

class WeirdNamespace:
def __init__(self, d): self.d = d
def __getitem__(self, n): return self.d.get(n,repr(n))


So, how and or where does this fit in with my example?
Does both the call to the function where I want to do
the eval() and self.push(line) command have to be in the
same namespace and or file for this to work?


All it does is implement what you asked: make a mapping that assumes all
unknown variable names are to be taken as literals, starting from a
mapping of all 'known variable names' to their values.

I'll assume you know your users better than I do and that they _do_
really desire with all their hearts this unholy confusion between
variables and constants...


This is driven by customers that who want both a type-in command
language and a variant using all the power of python for test
scripting purposes. If you misspell something and there is the
possibility of it screwing something up that we can't tell either
in the parser or downstream in the backend of the tool, well that's
life in the big city.


I suspect the first time your users want to use a file named 'for'
they'll scream bloody murder, but hopefully that won't happen before
they've paid you enough for you to retire in comfort to a nice
Carribbean island without extradition treaties, so, cheer up.
Alex
Jul 18 '05 #8
On Sep 22, 8:48pm, Peter Otten wrote:
Subject: Re: namespace/dictionary quandry I just ensure that all arguments not already in self.locals are added with
their name as their value. (I also renamed 'parent' to 'self' - I could not
stand it any longer :-)
Novice error on my part. Thanks for the correction.

Note that I don't particularly like both hacks and would rather use plain
old python functions with standard python syntax instead of your custom
language.
It makes the programmers job easier, but not the customer
who may just want a non-gui debugger commandline session tool.
That customer doesn't want to know anything about python or
function calls even though underneath the covers that's what
they are getting. At the same time, the testers want the full
scripting power of python.

To make life even more fun, there is a C++ front end and C++
back end that deal with many processes over many nodes. Some
of the commands are syncronous and return a value that feed
into the python name space and others are asyncronous that
may implode somewhere in the future and need to bring the
whole mess to some sane state to handle the problem

I guess it is my own private hell to work on.

Your change seems to work great!

Thanks,

Jack

Peter

Jul 18 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: Doug Rosser | last post by:
I'm writing a fairly complicated test framework and keeping configuration data inside ini files that are parsed at runtime by the ConfigParser module. For example, there would be a section...
8
by: Elaine Jackson | last post by:
I would like to be able to write a function f, which will live in a module M, and will call a function g, such that, when f is imported from M into the interpreter, and invoked there, its...
13
by: David Rysdam | last post by:
Getting no answer yesterday, I've done some investigation and I obviously don't understand how python namespaces work. Here's a test program: #!/usr/bin/python b = 2 def sumWithGlobal(a):...
8
by: Bo Peng | last post by:
Dear list, I am writing a Python extension module that needs a way to expose pieces of a big C array to python. Currently, I am using NumPy like the following: PyObject* res =...
31
by: Bo Peng | last post by:
Dear list, I have many dictionaries with the same set of keys and I would like to write a function to calculate something based on these values. For example, I have a = {'x':1, 'y':2} b =...
27
by: Ron Adam | last post by:
Hi, I found the following to be a useful way to access arguments after they are passed to a function that collects them with **kwds. class namespace(dict): def __getattr__(self, name): return...
4
by: Petr Jakes | last post by:
In my code I have relatively wide dictionary definition (about 100 rows). I would like to put it in to the different file (module) because of the main code readability (say the name of the file...
0
by: David | last post by:
Hi, I'm working on a project where we're juggling with two potential implementations. In the two scenarios, we create objects in the base namespace. These objects are interdependent, in the sense...
22
by: Luna Moon | last post by:
I am reading the book "C++ Annotations", and here is a quote from the book: Namespaces can be defined without a name. Such a namespace is anonymous and it restricts the visibility of the...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.