Code: Rolling a Container Into a String

[Kamilche]

I want to convert a dict into string form, then back again. After
discovering that eval is insecure, I wrote some code to roll a Python
object, dict, tuple, or list into a string. I've posted it below. Does
anyone know an easier way to accomplish this? Essentially, I want to
avoid doing an 'eval' on a string to get it back into dict form... but
still allow nested structures. (My current code doesn't handle nested
structures.)

I conked out before writing the 'unroll' code. I'm going to go watch
some boob tube with my husband, instead, and leave that code for
another day. If you know of a better way, or feel like writing the
'unroll' code and posting it, by all means, do so! :-D I'll check
Google newsgroups before I tackle the job, to see if some kind soul
took pity on me.

--Kamilche

import types

SimpleTypes = [types.BooleanTy pe, types.FloatType , types.IntType, \
types.LongType, types.NoneType, types.StringTyp e]

_dictdelim1 = "{"
_dictdelim2 = "}"
_listdelim1 = "["
_listdelim2 = "]"
_tupledelim1 = "("
_tupledelim2 = ")"

def roll(item):
"Return a string representation of an object, dict, tuple, or
list."
return _roll2(item, [], {})
def unroll(s):
"Unrolls a string back into a dict, tuple, or list."
if type(s) != types.StringTyp e:
raise Exception("You may only pass strings to this function!")
err = "Error occurred when parsing " + s + "!"
state = 0
container = None
for c in s:
if c == _dictdelim1:
lookfor = _dictdelim2
elif c == _listdelim1:
lookfor = _listdelim2
elif c == _tupledelim1:
lookfor = _tupledelim2
else:
raise Exception(err)
state = 1

def _quoted(s):
' Return a stringized value'
if type(s) != types.StringTyp e:
return str(s)
else:
l = []
s = s.replace("'", "\'")
l.append("'")
l.append(s)
l.append("'")
return ''.join(l)

def _roll2(d, lst, r):
' Function that does the work.'
# Start of _roll2
t = type(d)
if t == types.DictType:
theid = id(d)
if theid in r:
raise Exception("Recu rsion detected! Stopping now.")
r[theid] = theid
cnt = 0
lst.append(_dic tdelim1)
for key in d.keys():
if key[0] != '_':
lst.append(_quo ted(key))
lst.append(': ')
t = type(d[key])
if t in SimpleTypes:
lst.append(_quo ted(d[key]))
else:
_roll2(d[key], lst, r)
lst.append(", ")
cnt += 1
if cnt > 0:
del lst[-1]
lst.append(_dic tdelim2)
elif t in (types.ListType , types.TupleType ):
theid = id(d)
if theid in r:
raise Exception("Recu rsion detected! Stopping now.")
r[theid] = theid
cnt = 0
if t == types.ListType:
lst.append(_lis tdelim1)
else:
lst.append(_tup ledelim1)
for item in d:
if type(item) in SimpleTypes:
lst.append(_quo ted(item))
else:
_roll2(item, lst, r)
lst.append(", ")
cnt += 1
if cnt > 0:
del lst[-1]
if t == types.ListType:
lst.append(_lis tdelim2)
else:
lst.append(_tup ledelim2)
elif hasattr(d, '__dict__'):
_roll2(d.__dict __, lst, r)
else:
raise Exception("Unha ndled type " + str(t) + \
"! You may only pass dicts, tuples, lists, and
" + \
"objects with a __dict__ to this function!")
return ''.join(lst)


class simple:
pass

def main():
l = ['List1', 'List2', 'List3']
d = {'Dict1': 'd1', 'Dict2': 'd2', 'list': l}
t = ('Tuple1', d, 'Tuple2')

print "It handles dicts, lists, and tuples."
print roll(t), "\n"

o = simple()
o.name = 'the name'
o.password = 'the password'
o.list = ['ol1', 'ol2']
o.dict = {'od1': None, 'od2': 2}
o.tuple = ('tuple1', 'tuple2')
o.float = 1.5
o.long = 123456789012345 67890
o.bool = True
o.int = 1

print "It handles objects."
print roll(o), "\n"

print "It won't roll attributes whose name starts with '_'"
o._recursion = o.tuple
print roll(o), "\n"

print "It will raise an exception if it detects recursion."
print "This next one will cause recursion."
o.recursion = o.tuple
print roll(o), "\n"

print "You can't roll simple types."
print roll('a'), "\n"

if __name__ == "__main__":
main()

[Terry Reedy]

"Kamilche" <kl*******@home .com> wrote in message
news:88******** *************** ***@posting.goo gle.com...

I want to convert a dict into string form, then back again. After
discovering that eval is insecure,
With arbitrary code from an arbitrary source, yes.
If you *know* that you are eval-ing your own safe strings, then no problem.
I wrote some code to roll a Python
object, dict, tuple, or list into a string.

repr(object) already does that for you. Why duplicate the work?

You only need custom a eval function, which might check that string is safe
(no function calls, no list comps) and then eval, or which might do parsing
and construction itself.

Terry J. Reedy

[David Fraser]

Terry Reedy wrote:

"Kamilche" <kl*******@home .com> wrote in message
news:88******** *************** ***@posting.goo gle.com...

I want to convert a dict into string form, then back again. After
discovering that eval is insecure,

With arbitrary code from an arbitrary source, yes.
If you *know* that you are eval-ing your own safe strings, then no problem.

I wrote some code to roll a Python
object, dict, tuple, or list into a string.

repr(object) already does that for you. Why duplicate the work?

You only need custom a eval function, which might check that string is safe
(no function calls, no list comps) and then eval, or which might do parsing
and construction itself.

Terry J. Reedy

Or use the pprint module which does nice pretty-printing

David

[Paul McGuire]

"Kamilche" <kl*******@home .com> wrote in message
news:88******** *************** ***@posting.goo gle.com...

I want to convert a dict into string form, then back again. After
discovering that eval is insecure, I wrote some code to roll a Python
object, dict, tuple, or list into a string. I've posted it below. Does
anyone know an easier way to accomplish this? Essentially, I want to
avoid doing an 'eval' on a string to get it back into dict form... but
still allow nested structures. (My current code doesn't handle nested
structures.)

unroll is just repr().

Here is a pyparsing routine to "roll up" your data structures, a mere 60
lines or so. It's fairly tolerant of some odd cases, and fully handles
nested data. (Extension to include remaining items, such as boolean data
and scientific notation, is left as an exercise for the reader.) I hope
this is fairly easy to follow - I've dropped in a few comments.

For those of you who've been living in a cave, you can download pyparsing at
http://pyparsing.sourceforge.net.

-- Paul
from pyparsing import Word, ZeroOrMore, OneOrMore, Suppress, Forward, \
quotedString,nu ms,Combine,Opti onal,delimitedL ist,Group

# create a dictionary of ugly data, complete with nested lists, tuples
# and dictionaries, even imaginary numbers!
d1 = {}
d1['a'] = [1,2,3,[4,5,6]]
d1['b'] = (7,8,(9,10),'a' ,"",'')
d1['c'] = { 'aa' : 1, 'bb' : "lskdj'slkd jf", 'cc':1.232, 'dd':(('z',),) }
d1[('d','e')] = 5+10j

print repr(d1)

testdata = repr(d1)
"""
looks like this:
{'a': [1, 2, 3, [4, 5, 6]], ('d', 'e'): (5+10j), 'c': {'aa': 1, 'cc': 1.232,
'dd': (('z',),), 'bb': "lskdj'slkdjf"} , 'b': (7, 8, (9, 10), 'a', '', '')}
"""

#define low-level data elements
intNum = Word( nums+"+-", nums )
realNum = Combine(intNum + "." + Optional(Word(n ums)))
number = realNum | intNum
imagNum = Combine( "(" + number + "+" + number + "j" + ")" )

item = Forward() # set up for recursive grammar definition
tupleDef = Suppress("(") + ( delimitedList( item ) ^
( item + Suppress(",") ) ) + Suppress(")")
listDef = Suppress("[") + delimitedList( item ) + Suppress("]")
keyDef = tupleDef | quotedString | imagNum | number
keyVal = Group( keyDef + Suppress(":") + item )
dictDef = Suppress("{") + delimitedList( keyVal ) + Suppress("}")

item << ( quotedString | number | imagNum |
tupleDef | listDef | dictDef )

# define low-level conversion routines
intNum.setParse Action( lambda s,loc,toks: int(toks[0]) )
realNum.setPars eAction( lambda s,loc,toks: float(toks[0]) )
imagNum.setPars eAction( lambda s,loc,toks: eval(toks[0]) ) # no built-in to
convert imaginaries?

# strip leading and trailing character from parsed quoted string
quotedString.se tParseAction( lambda s,loc,toks: toks[0][1:-1] )

# define list-to-list/tuple/dict routines
evalTuple = lambda s,loc,toks: [ tuple(toks) ]
evalList = lambda s,loc,toks: [ toks.asList() ]
evalDict = lambda s,loc,toks: [ dict([tuple(kv) for kv in toks]) ]

tupleDef.setPar seAction( evalTuple )
listDef.setPars eAction( evalList )
dictDef.setPars eAction( evalDict )

# first element of returned tokens list is the reconstructed list/tuple/dict
results = item.parseStrin g( testdata )[0]
print results

if repr(results) == repr(d1):
print "Eureka!"
else:
print "Compare results for mismatch"
print repr(results)
print repr(d1)

[Pierre-Frédéric Caillaud]

Use YAML

import yaml

then from your code:

yaml.dump( whatever ) =>

then yaml.loadstring (str)...

It handles objects.
{'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
1.5, 'list': ['ol1', 'ol2'], 'long': 123456789012345 67890,
'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

--- !!__main__.simp le # instanciates class for you
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 123456789012345 67890
name: the name
password: the password
tuple:
- tuple1
- tuple2

It won't roll attributes whose name starts with '_'
{'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
1.5, 'list': ['ol1', 'ol2'], 'long': 123456789012345 67890,
'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

--- !!__main__.simp le
_recursion: # yaml does not handle recursion (I have an old version)
- tuple1
- tuple2
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 123456789012345 67890
name: the name
password: the password
tuple:
- tuple1
- tuple2

It will raise an exception if it detects recursion.
This next one will cause recursion.
--- !!__main__.simp le
_recursion:
- tuple1
- tuple2
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 123456789012345 67890
name: the name
password: the password
recursion:
- tuple1
- tuple2
tuple:
- tuple1
- tuple2

[Oliver Pieper]

> I want to convert a dict into string form, then back again.

Since you probably don't want to do this just for the joy of
converting something to string an back ...

Maybe the pickle (and cPickle) module does what you are looking for.

Oliver

[Kamilche]

Pierre-Frédéric Caillaud <pe****@free.fr > wrote in message news:<opr941cav 01v4ijd@musicbo x>...

Use YAML

It looked interesting, so I downloaded it... and was confronted with
dozens of files, and the need to compile before use... when I was
looking for a simple cross-platform 2 function solution that didn't
take any DLL's. Dang.

Well, it's a new day, maybe I'll be inspired.

[Pierre-Frédéric Caillaud]

did you download syck or the pure python yaml parser ?
on Linux the pure python module is just a matter of typing "emerge sync"
but I don't know about Syck...

On 25 Jun 2004 13:36:31 -0700, Kamilche <kl*******@home .com> wrote:

Pierre-Frédéric Caillaud <pe****@free.fr > wrote in message
news:<opr941cav 01v4ijd@musicbo x>...

Use YAML

It looked interesting, so I downloaded it... and was confronted with
dozens of files, and the need to compile before use... when I was
looking for a simple cross-platform 2 function solution that didn't
take any DLL's. Dang.

Well, it's a new day, maybe I'll be inspired.

--
Using Opera's revolutionary e-mail client: http://www.opera.com/m2/