471,066 Members | 1,588 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,066 software developers and data experts.

Code: Rolling a Container Into a String

I want to convert a dict into string form, then back again. After
discovering that eval is insecure, I wrote some code to roll a Python
object, dict, tuple, or list into a string. I've posted it below. Does
anyone know an easier way to accomplish this? Essentially, I want to
avoid doing an 'eval' on a string to get it back into dict form... but
still allow nested structures. (My current code doesn't handle nested
structures.)

I conked out before writing the 'unroll' code. I'm going to go watch
some boob tube with my husband, instead, and leave that code for
another day. If you know of a better way, or feel like writing the
'unroll' code and posting it, by all means, do so! :-D I'll check
Google newsgroups before I tackle the job, to see if some kind soul
took pity on me.

--Kamilche

import types

SimpleTypes = [types.BooleanType, types.FloatType, types.IntType, \
types.LongType, types.NoneType, types.StringType]

_dictdelim1 = "{"
_dictdelim2 = "}"
_listdelim1 = "["
_listdelim2 = "]"
_tupledelim1 = "("
_tupledelim2 = ")"

def roll(item):
"Return a string representation of an object, dict, tuple, or
list."
return _roll2(item, [], {})
def unroll(s):
"Unrolls a string back into a dict, tuple, or list."
if type(s) != types.StringType:
raise Exception("You may only pass strings to this function!")
err = "Error occurred when parsing " + s + "!"
state = 0
container = None
for c in s:
if c == _dictdelim1:
lookfor = _dictdelim2
elif c == _listdelim1:
lookfor = _listdelim2
elif c == _tupledelim1:
lookfor = _tupledelim2
else:
raise Exception(err)
state = 1

def _quoted(s):
' Return a stringized value'
if type(s) != types.StringType:
return str(s)
else:
l = []
s = s.replace("'", "\'")
l.append("'")
l.append(s)
l.append("'")
return ''.join(l)

def _roll2(d, lst, r):
' Function that does the work.'
# Start of _roll2
t = type(d)
if t == types.DictType:
theid = id(d)
if theid in r:
raise Exception("Recursion detected! Stopping now.")
r[theid] = theid
cnt = 0
lst.append(_dictdelim1)
for key in d.keys():
if key[0] != '_':
lst.append(_quoted(key))
lst.append(': ')
t = type(d[key])
if t in SimpleTypes:
lst.append(_quoted(d[key]))
else:
_roll2(d[key], lst, r)
lst.append(", ")
cnt += 1
if cnt > 0:
del lst[-1]
lst.append(_dictdelim2)
elif t in (types.ListType, types.TupleType):
theid = id(d)
if theid in r:
raise Exception("Recursion detected! Stopping now.")
r[theid] = theid
cnt = 0
if t == types.ListType:
lst.append(_listdelim1)
else:
lst.append(_tupledelim1)
for item in d:
if type(item) in SimpleTypes:
lst.append(_quoted(item))
else:
_roll2(item, lst, r)
lst.append(", ")
cnt += 1
if cnt > 0:
del lst[-1]
if t == types.ListType:
lst.append(_listdelim2)
else:
lst.append(_tupledelim2)
elif hasattr(d, '__dict__'):
_roll2(d.__dict__, lst, r)
else:
raise Exception("Unhandled type " + str(t) + \
"! You may only pass dicts, tuples, lists, and
" + \
"objects with a __dict__ to this function!")
return ''.join(lst)


class simple:
pass

def main():
l = ['List1', 'List2', 'List3']
d = {'Dict1': 'd1', 'Dict2': 'd2', 'list': l}
t = ('Tuple1', d, 'Tuple2')

print "It handles dicts, lists, and tuples."
print roll(t), "\n"

o = simple()
o.name = 'the name'
o.password = 'the password'
o.list = ['ol1', 'ol2']
o.dict = {'od1': None, 'od2': 2}
o.tuple = ('tuple1', 'tuple2')
o.float = 1.5
o.long = 12345678901234567890
o.bool = True
o.int = 1

print "It handles objects."
print roll(o), "\n"

print "It won't roll attributes whose name starts with '_'"
o._recursion = o.tuple
print roll(o), "\n"

print "It will raise an exception if it detects recursion."
print "This next one will cause recursion."
o.recursion = o.tuple
print roll(o), "\n"

print "You can't roll simple types."
print roll('a'), "\n"

if __name__ == "__main__":
main()
Jul 18 '05 #1
7 1555

"Kamilche" <kl*******@home.com> wrote in message
news:88**************************@posting.google.c om...
I want to convert a dict into string form, then back again. After
discovering that eval is insecure,
With arbitrary code from an arbitrary source, yes.
If you *know* that you are eval-ing your own safe strings, then no problem.
I wrote some code to roll a Python
object, dict, tuple, or list into a string.


repr(object) already does that for you. Why duplicate the work?

You only need custom a eval function, which might check that string is safe
(no function calls, no list comps) and then eval, or which might do parsing
and construction itself.

Terry J. Reedy
Jul 18 '05 #2
Terry Reedy wrote:
"Kamilche" <kl*******@home.com> wrote in message
news:88**************************@posting.google.c om...
I want to convert a dict into string form, then back again. After
discovering that eval is insecure,

With arbitrary code from an arbitrary source, yes.
If you *know* that you are eval-ing your own safe strings, then no problem.

I wrote some code to roll a Python
object, dict, tuple, or list into a string.

repr(object) already does that for you. Why duplicate the work?

You only need custom a eval function, which might check that string is safe
(no function calls, no list comps) and then eval, or which might do parsing
and construction itself.

Terry J. Reedy

Or use the pprint module which does nice pretty-printing

David
Jul 18 '05 #3
"Kamilche" <kl*******@home.com> wrote in message
news:88**************************@posting.google.c om...
I want to convert a dict into string form, then back again. After
discovering that eval is insecure, I wrote some code to roll a Python
object, dict, tuple, or list into a string. I've posted it below. Does
anyone know an easier way to accomplish this? Essentially, I want to
avoid doing an 'eval' on a string to get it back into dict form... but
still allow nested structures. (My current code doesn't handle nested
structures.)

unroll is just repr().

Here is a pyparsing routine to "roll up" your data structures, a mere 60
lines or so. It's fairly tolerant of some odd cases, and fully handles
nested data. (Extension to include remaining items, such as boolean data
and scientific notation, is left as an exercise for the reader.) I hope
this is fairly easy to follow - I've dropped in a few comments.

For those of you who've been living in a cave, you can download pyparsing at
http://pyparsing.sourceforge.net.

-- Paul
from pyparsing import Word, ZeroOrMore, OneOrMore, Suppress, Forward, \
quotedString,nums,Combine,Optional,delimitedList,G roup

# create a dictionary of ugly data, complete with nested lists, tuples
# and dictionaries, even imaginary numbers!
d1 = {}
d1['a'] = [1,2,3,[4,5,6]]
d1['b'] = (7,8,(9,10),'a',"",'')
d1['c'] = { 'aa' : 1, 'bb' : "lskdj'slkdjf", 'cc':1.232, 'dd':(('z',),) }
d1[('d','e')] = 5+10j

print repr(d1)

testdata = repr(d1)
"""
looks like this:
{'a': [1, 2, 3, [4, 5, 6]], ('d', 'e'): (5+10j), 'c': {'aa': 1, 'cc': 1.232,
'dd': (('z',),), 'bb': "lskdj'slkdjf"}, 'b': (7, 8, (9, 10), 'a', '', '')}
"""

#define low-level data elements
intNum = Word( nums+"+-", nums )
realNum = Combine(intNum + "." + Optional(Word(nums)))
number = realNum | intNum
imagNum = Combine( "(" + number + "+" + number + "j" + ")" )

item = Forward() # set up for recursive grammar definition
tupleDef = Suppress("(") + ( delimitedList( item ) ^
( item + Suppress(",") ) ) + Suppress(")")
listDef = Suppress("[") + delimitedList( item ) + Suppress("]")
keyDef = tupleDef | quotedString | imagNum | number
keyVal = Group( keyDef + Suppress(":") + item )
dictDef = Suppress("{") + delimitedList( keyVal ) + Suppress("}")

item << ( quotedString | number | imagNum |
tupleDef | listDef | dictDef )

# define low-level conversion routines
intNum.setParseAction( lambda s,loc,toks: int(toks[0]) )
realNum.setParseAction( lambda s,loc,toks: float(toks[0]) )
imagNum.setParseAction( lambda s,loc,toks: eval(toks[0]) ) # no built-in to
convert imaginaries?

# strip leading and trailing character from parsed quoted string
quotedString.setParseAction( lambda s,loc,toks: toks[0][1:-1] )

# define list-to-list/tuple/dict routines
evalTuple = lambda s,loc,toks: [ tuple(toks) ]
evalList = lambda s,loc,toks: [ toks.asList() ]
evalDict = lambda s,loc,toks: [ dict([tuple(kv) for kv in toks]) ]

tupleDef.setParseAction( evalTuple )
listDef.setParseAction( evalList )
dictDef.setParseAction( evalDict )

# first element of returned tokens list is the reconstructed list/tuple/dict
results = item.parseString( testdata )[0]
print results

if repr(results) == repr(d1):
print "Eureka!"
else:
print "Compare results for mismatch"
print repr(results)
print repr(d1)
Jul 18 '05 #4

Use YAML

import yaml

then from your code:

yaml.dump( whatever ) =>

then yaml.loadstring(str)...

It handles objects.
{'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
1.5, 'list': ['ol1', 'ol2'], 'long': 12345678901234567890,
'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

--- !!__main__.simple # instanciates class for you
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 12345678901234567890
name: the name
password: the password
tuple:
- tuple1
- tuple2

It won't roll attributes whose name starts with '_'
{'name': 'the name', 'tuple': ('tuple1', 'tuple2'), 'int': 1, 'float':
1.5, 'list': ['ol1', 'ol2'], 'long': 12345678901234567890,
'dict': {'od1': None, 'od2': 2}, 'bool': True, 'password': 'the password'}

--- !!__main__.simple
_recursion: # yaml does not handle recursion (I have an old version)
- tuple1
- tuple2
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 12345678901234567890
name: the name
password: the password
tuple:
- tuple1
- tuple2

It will raise an exception if it detects recursion.
This next one will cause recursion.
--- !!__main__.simple
_recursion:
- tuple1
- tuple2
bool: True
dict:
od1: ~
od2: 2
float: 1.5
int: 1
list:
- ol1
- ol2
long: 12345678901234567890
name: the name
password: the password
recursion:
- tuple1
- tuple2
tuple:
- tuple1
- tuple2
Jul 18 '05 #5
> I want to convert a dict into string form, then back again.

Since you probably don't want to do this just for the joy of
converting something to string an back ...

Maybe the pickle (and cPickle) module does what you are looking for.

Oliver
Jul 18 '05 #6
Pierre-Frédéric Caillaud <pe****@free.fr> wrote in message news:<opr941cav01v4ijd@musicbox>...
Use YAML

It looked interesting, so I downloaded it... and was confronted with
dozens of files, and the need to compile before use... when I was
looking for a simple cross-platform 2 function solution that didn't
take any DLL's. Dang.

Well, it's a new day, maybe I'll be inspired.
Jul 18 '05 #7

did you download syck or the pure python yaml parser ?
on Linux the pure python module is just a matter of typing "emerge sync"
but I don't know about Syck...

On 25 Jun 2004 13:36:31 -0700, Kamilche <kl*******@home.com> wrote:
Pierre-Frédéric Caillaud <pe****@free.fr> wrote in message
news:<opr941cav01v4ijd@musicbox>...
Use YAML

It looked interesting, so I downloaded it... and was confronted with
dozens of files, and the need to compile before use... when I was
looking for a simple cross-platform 2 function solution that didn't
take any DLL's. Dang.

Well, it's a new day, maybe I'll be inspired.


--
Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
Jul 18 '05 #8

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

2 posts views Thread by Maitre Bart | last post: by
6 posts views Thread by Dan McCollick | last post: by
1 post views Thread by Neo Geshel | last post: by
2 posts views Thread by MrCrool | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.