eval to dict problems NEWB going crazy !

[manstey]

Hi,

I have a text file called a.txt:

# comments
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 5), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 7 ), ('parse', {'pos': u'np', 'gen': u'm'})]

I read it using this:

filAnsMorph = codecs.open('a. txt', 'r', 'utf-8') # Initialise input
file
dicAnsMorph = {}
for line in filAnsMorph:
if line[0] != '#': # Get rid of comment lines
x = eval(line)
dicAnsMorph[x[0][1]] = x[1][1] # recid is key, parse dict is
value

But it crashes every time on x = eval(line). Why is this? If I change
a.txt to:

# comments
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

it works fine. Why doesn't it work with multiple lines? it's driving me
crazy!

Thanks,
Matthew

[Bruno Desthuilliers]

manstey wrote:

Hi,

I have a text file called a.txt:

# comments
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 5), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 7 ), ('parse', {'pos': u'np', 'gen': u'm'})]

I read it using this:

filAnsMorph = codecs.open('a. txt', 'r', 'utf-8') # Initialise input
file
dicAnsMorph = {}
for line in filAnsMorph:
if line[0] != '#': # Get rid of comment lines
x = eval(line)
dicAnsMorph[x[0][1]] = x[1][1] # recid is key, parse dict is
value

But it crashes every time on x = eval(line). Why is this? If I change
a.txt to:

# comments
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

it works fine. Why doesn't it work with multiple lines? it's driving me
crazy!

try with:
x = eval(line.strip ('\n'))
--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom. gro'.split('@')])"

[manstey]

That doesn't work. I just get an error:

x = eval(line.strip ('\n'))
File "<string>", line 1
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

SyntaxError: unexpected EOF while parsing
any other ideas?

Bruno Desthuilliers wrote:

manstey wrote:

Hi,

I have a text file called a.txt:

# comments
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 5), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 7 ), ('parse', {'pos': u'np', 'gen': u'm'})]

I read it using this:

filAnsMorph = codecs.open('a. txt', 'r', 'utf-8') # Initialise input
file
dicAnsMorph = {}
for line in filAnsMorph:
if line[0] != '#': # Get rid of comment lines
x = eval(line)
dicAnsMorph[x[0][1]] = x[1][1] # recid is key, parse dict is
value

But it crashes every time on x = eval(line). Why is this? If I change
a.txt to:

# comments
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

it works fine. Why doesn't it work with multiple lines? it's driving me
crazy!

try with:
x = eval(line.strip ('\n'))
--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom. gro'.split('@')])"

[Eric Deveaud]

manstey wrote:

That doesn't work. I just get an error:

x = eval(line.strip ('\n'))
File "<string>", line 1
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

SyntaxError: unexpected EOF while parsing

is the last line of your file empty ??

what with

for line in filAnsMorph:
# remove any trailing and leading whitespace includes removing \n
line = line.strip()
# Get rid of comment lines
if line.startswith ('#'):
continue
# Get rid of blank line
if line == '':
continue
#do the job
x = eval(line)
NB by default strip() removes leading and trailing characters from the target
string. with whitspace defined as whitespace = '\t\n\x0b\x0c\r '

Eric

[Fredrik Lundh]

"manstey" <ma*****@csu.ed u.auwrote:

That doesn't work. I just get an error:

x = eval(line.strip ('\n'))
File "<string>", line 1
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

SyntaxError: unexpected EOF while parsing

any other ideas?

hint 1:

>>eval("[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]\n")

[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

>>eval("[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]")

[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

hint 2:

>>eval("")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<string>", line 0

^
SyntaxError: unexpected EOF while parsing

>>eval("\n")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<string>", line 1

^
SyntaxError: unexpected EOF while parsing

hint 3: adding a "print" statement *before* the offending line is often a good way
to figure out why something's not working. "repr()" is also a useful thing:

if line[0] != '#': # Get rid of comment lines
print repr(line) # DEBUG: let's see what we're trying to evaluate
x = eval(line)
dicAnsMorph[x[0][1]] = x[1][1] # recid is key, parse dict is

</F>

[Roel Schroeven]

manstey schreef:

Hi,

I have a text file called a.txt:

# comments
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 5), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 7 ), ('parse', {'pos': u'np', 'gen': u'm'})]

I read it using this:

filAnsMorph = codecs.open('a. txt', 'r', 'utf-8') # Initialise input
file
dicAnsMorph = {}
for line in filAnsMorph:
if line[0] != '#': # Get rid of comment lines
x = eval(line)
dicAnsMorph[x[0][1]] = x[1][1] # recid is key, parse dict is
value

But it crashes every time on x = eval(line). Why is this? If I change
a.txt to:

# comments
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

it works fine. Why doesn't it work with multiple lines? it's driving me
crazy!

It looks like it's because of the trailing newline. When you read a file
like that, the newline at the end of each line is still in line. You can
strip it e.g. with rstrip, like so:

x = eval(line.rstri p('\n'))

--
If I have been able to see further, it was only because I stood
on the shoulders of giants. -- Isaac Newton

Roel Schroeven

[Fredrik Lundh]

hint 1:

hint 1b:

>>eval("[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]")

[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

>>eval("[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]\n")

[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]

>>eval("[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]\r\n")

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<string>", line 1
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]
^
SyntaxError: invalid syntax

</F>

[Steven D'Aprano]

On Thu, 06 Jul 2006 03:34:32 -0700, manstey wrote:

Hi,

I have a text file called a.txt:

# comments
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 5), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 7 ), ('parse', {'pos': u'np', 'gen': u'm'})]

I read it using this:

filAnsMorph = codecs.open('a. txt', 'r', 'utf-8') # Initialise input
file
dicAnsMorph = {}
for line in filAnsMorph:
if line[0] != '#': # Get rid of comment lines
x = eval(line)
dicAnsMorph[x[0][1]] = x[1][1] # recid is key, parse dict is
value

But it crashes every time on x = eval(line). Why is this?

Some people have incorrectly suggested the solution is to remove the
newline from the end of the line. Others have already pointed out one
possible solution.

I'd like to ask, why are you using eval in the first place?

The problem with eval is that it is simultaneously too finicky and too
powerful. It is finicky -- it has problems with lines ending with a
carriage return, empty lines, and probably other things. But it is also
too powerful. Your program wants a specific piece of data, but eval
will accept any string which is a valid Python expression. eval is quite
capable of giving you a dictionary, or an int, or just about anything --
and, depending on your code, you might not find out for a long time,
leading to hard-to-debug bugs.

Is your data under your control? Could some malicious person inject data
into your file a.txt? If so, you should be aware of the security
implications:

# comment
[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 5), ('parse', {'pos': u'np', 'gen': u'm'})]
# line injected by a malicious user
"__import__('os ').system('echo if I were bad I could do worse')"
[('recId', 7 ), ('parse', {'pos': u'np', 'gen': u'm'})]

Now, if the malicious user can only damage their own system, maybe you
don't care -- but the security hole is there. Are you sure that no
malicious third party, given *only* write permission to the file a.txt,
could compromise your entire system?

Personally, I would never use eval on any string I didn't write myself. If
I was thinking about evaluating a user-string, I would always write a
function to parse the string and accept only the specific sort of data I
expected. In your case, a quick-and-dirty untested function might be:

def parse(s):
"""Parse string s, and return a two-item list like this:

[tuple(string, integer), tuple(string, dict(string: string)]
"""

def parse_tuple(s):
"""Parse a tuple with two items exactly."""
s = s.strip()
assert s.startswith("( ")
assert s.endswith(")")
a, b = s[1:-1].split(",")
return (a.strip(), b.strip())

def parse_dict(s):
"""Parse a dict with two items exactly."""
s = s.strip()
assert s.startswith("{ ")
assert s.endswith("}")
a, b = s[1:-1].split(",")
key1, value1 = a.strip().split (":")
key2, value2 = b.strip().split (":")
return {key1.strip(): value1.strip(), key2.strip(): value2.strip()}

def parse_list(s):
"""Parse a list with two items exactly."""
s = s.strip()
assert s.startswith("[")
assert s.endswith("]")
a, b = s[1:-1].split(",")
return [a.strip(), b.strip()]

# Expected format is something like:
# [tuple(string, integer), tuple(string, dict(string: string)]
L = parse_list(s)
T0 = parse_tuple(L[0])
T1 = parse_tuple(L[1])
T0 = (T0[0], int(T0[1]))
T1 = (T1[0], parse_dict(T1[1]))
return [T0, T1]
That's a bit more work than eval, but I believe it is worth it.

--
Steven

[Ant]

[('recId', 3), ('parse', {'pos': u'np', 'gen': u'm'})]
[('recId', 5), ('parse', {'pos': u'np', 'gen': u'm'})]
# line injected by a malicious user
"__import__('os ').system('echo if I were bad I could do worse')"
[('recId', 7 ), ('parse', {'pos': u'np', 'gen': u'm'})]

I'm curious, if you disabled import, could you make eval safe?

For example:

>>eval("__impor t__('os').syste m('echo if I were bad I could do worse')")

if I were bad I could do worse
0

>>eval("__impor t__('os').syste m('echo if I were bad I could do worse')", {'__import__': lambda x:None})

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<string>", line 0, in ?
AttributeError: 'NoneType' object has no attribute 'system'

So, it seems to be possible to disable access to imports, but is this
enough? Are there other ways to access modules, or do damage via
built-in commands?

It seems that there must be a way to use eval safely, as there are
plenty of apps that embed python as a scripting language - and what's
the point of an eval function if impossible to use safely, and you have
to write your own Python parser!!

[Fredrik Lundh]

Ant wrote:

It seems that there must be a way to use eval safely, as there are
plenty of apps that embed python as a scripting language - and what's
the point of an eval function if impossible to use safely, and you have
to write your own Python parser!!

embedding python != accepting scripts from anywhere.

</F>