By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
464,388 Members | 1,186 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 464,388 IT Pros & Developers. It's quick & easy.

How to split a string containing nested commas-separated substrings

P: n/a
Hello,

I'd like to split a string by commas, but only at the "top level" so
to speak. An element can be a comma-less substring, or a
quoted string, or a substring which looks like a function call.
If some element contains commas, I don't want to split it.

Examples:

'foo, bar, baz' ='foo' 'bar' 'baz'
'foo, "bar, baz", blurf' ='foo' 'bar, baz' 'blurf'
'foo, bar(baz, blurf), mumble' ='foo' 'bar(baz, blurf)' 'mumble'

Can someone suggest a suitable regular expression or other
method to split such strings?

Thank you very much for your help.

Robert
Jun 27 '08 #1
Share this Question
Share on Google+
5 Replies

P: n/a
On Jun 18, 10:19 am, Robert Dodier <robert.dod...@gmail.comwrote:
Hello,

I'd like to split a string by commas, but only at the "top level" so
to speak. An element can be a comma-less substring, or a
quoted string, or a substring which looks like a function call.
If some element contains commas, I don't want to split it.

Examples:

'foo, bar, baz' ='foo' 'bar' 'baz'
'foo, "bar, baz", blurf' ='foo' 'bar, baz' 'blurf'
'foo, bar(baz, blurf), mumble' ='foo' 'bar(baz, blurf)' 'mumble'

Can someone suggest a suitable regular expression or other
method to split such strings?

Thank you very much for your help.

Robert
You might look at the shlex module. It doesn't get you 100%, but its
close:
>>shlex.split('foo, bar, baz')
['foo,', 'bar,', 'baz']
>>shlex.split( 'foo, "bar, baz", blurf')
['foo,', 'bar, baz,', 'blurf']
>>shlex.split('foo, bar(baz, blurf), mumble')
['foo,', 'bar(baz,', 'blurf),', 'mumble']

Using a RE will be tricky, especially if it is possible to have
recursive nesting (which by definition REs can't handle). For a real
general purpose solution you will need to create a custom parser.
There are a couple modules out there that can help you with that.

pyparsing is one: http://pyparsing.wikispaces.com/

Matt
Jun 27 '08 #2

P: n/a
Hi,

Le Wednesday 18 June 2008 19:19:57 Robert Dodier, vous avez écrit*:
Hello,

I'd like to split a string by commas, but only at the "top level" so
to speak. An element can be a comma-less substring, or a
quoted string, or a substring which looks like a function call.
If some element contains commas, I don't want to split it.

Examples:

'foo, bar, baz' ='foo' 'bar' 'baz'
'foo, "bar, baz", blurf' ='foo' 'bar, baz' 'blurf'
'foo, bar(baz, blurf), mumble' ='foo' 'bar(baz, blurf)' 'mumble'

Can someone suggest a suitable regular expression or other
method to split such strings?
I'd do something like this (note that it doesn't check for quote/parenthesis
mismatch and removes _all_ the quotes) :

def mysplit (string) :
pardepth = 0
quote = False
ret = ['']

for car in string :

if car == '(' : pardepth += 1
elif car == ')' : pardepth -= 1
elif car in ('"', "'") :
quote = not quote
car = '' # just if you don't want to keep the quotes

if car in ', ' and not (pardepth or quote) :
if ret[-1] != '' : ret.append('')
else :
ret[-1] += car

return ret

# test
for s in ('foo, bar, baz',
'foo, "bar, baz", blurf',
'foo, bar(baz, blurf), mumble') :
print "'%s' ='%s'" % (s, mysplit(s))

# result
'foo, bar, baz' ='['foo', 'bar', 'baz']'
'foo, "bar, baz", blurf' ='['foo', 'bar, baz', 'blurf']'
'foo, bar(baz, blurf), mumble' ='['foo', 'bar(baz, blurf)', 'mumble']'
--
Cédric Lucantis
Jun 27 '08 #3

P: n/a
On Jun 18, 12:19*pm, Robert Dodier <robert.dod...@gmail.comwrote:
Hello,

I'd like to split a string by commas, but only at the "top level" so
to speak. An element can be a comma-less substring, or a
quoted string, or a substring which looks like a function call.
If some element contains commas, I don't want to split it.

Examples:

'foo, bar, baz' ='foo' 'bar' 'baz'
'foo, "bar, baz", blurf' ='foo' 'bar, baz' 'blurf'
'foo, bar(baz, blurf), mumble' ='foo' 'bar(baz, blurf)' 'mumble'

Can someone suggest a suitable regular expression or other
method to split such strings?

Thank you very much for your help.

Robert
tests = """\
foo, bar, baz
foo, "bar, baz", blurf
foo, bar(baz, blurf), mumble""".splitlines()
from pyparsing import Word, alphas, alphanums, Optional, \
Group, delimitedList, quotedString

ident = Word(alphas+"_",alphanums+"_")
func_call = Group(ident + "(" + Optional(Group(delimitedList(ident)))
+ ")")

listItem = func_call | ident | quotedString

for t in tests:
print delimitedList(listItem).parseString(t).asList()
Prints:

['foo', 'bar', 'baz']
['foo', '"bar, baz"', 'blurf']
['foo', ['bar', '(', ['baz', 'blurf'], ')'], 'mumble']
-- Paul
Jun 27 '08 #4

P: n/a
On Jun 18, 10:54 am, Matimus <mccre...@gmail.comwrote:
On Jun 18, 10:19 am, Robert Dodier <robert.dod...@gmail.comwrote:
Hello,
I'd like to split a string by commas, but only at the "top level" so
to speak. An element can be a comma-less substring, or a
quoted string, or a substring which looks like a function call.
If some element contains commas, I don't want to split it.
Examples:
'foo, bar, baz' ='foo' 'bar' 'baz'
'foo, "bar, baz", blurf' ='foo' 'bar, baz' 'blurf'
'foo, bar(baz, blurf), mumble' ='foo' 'bar(baz, blurf)' 'mumble'
Can someone suggest a suitable regular expression or other
method to split such strings?
Thank you very much for your help.
Robert

You might look at the shlex module. It doesn't get you 100%, but its
close:
>shlex.split('foo, bar, baz')

['foo,', 'bar,', 'baz']>>shlex.split( 'foo, "bar, baz", blurf')

['foo,', 'bar, baz,', 'blurf']>>shlex.split('foo, bar(baz, blurf), mumble')

['foo,', 'bar(baz,', 'blurf),', 'mumble']

Using a RE will be tricky, especially if it is possible to have
recursive nesting (which by definition REs can't handle). For a real
general purpose solution you will need to create a custom parser.
There are a couple modules out there that can help you with that.

pyparsing is one:http://pyparsing.wikispaces.com/

Matt
Following up to my own post, Here is a working example that uses the
built-in _ast module. I posted something similar the other day. This
uses pythons own internal parser to do it for you. It works in this
case because, at least from what you have posted, your syntax doesn't
violate python syntax.

Expand|Select|Wrap|Line Numbers
  1. import _ast
  2.  
  3. def eval_tuple(text):
  4. """ Evaluate a string representing a tuple of strings, names and
  5. calls,
  6. returns a tuple of strings.
  7. """
  8.  
  9. ast = compile(text, "<string>", 'eval', _ast.PyCF_ONLY_AST)
  10. return _traverse(ast.body)
  11.  
  12. def _traverse(ast):
  13. """ Traverse the AST returning string representations of tuples
  14. strings
  15. names and calls.
  16. """
  17. if isinstance(ast, _ast.Tuple):
  18. return tuple(_traverse(el) for el in ast.elts)
  19. elif isinstance(ast, _ast.Str):
  20. return ast.s
  21. elif isinstance(ast, _ast.Name):
  22. return ast.id
  23. elif isinstance(ast, _ast.Call):
  24. name = ast.func.id
  25. args = [_traverse(x) for x in ast.args]
  26. return "%s(%s)"%(name, ", ".join(args))
  27. raise SyntaxError()
  28.  
  29. examples = [
  30. ('foo, bar, baz', ('foo', 'bar', 'baz')),
  31. ('foo, "bar, baz", blurf', ('foo', 'bar, baz', 'blurf')),
  32. ('foo, bar(baz, blurf), mumble', ('foo', 'bar(baz, blurf)',
  33. 'mumble')),
  34. ]
  35.  
  36. def test():
  37. for text, expected in examples:
  38. print "trying %r =%r"%(text, expected)
  39. result = eval_tuple(text)
  40. if result == expected:
  41. print "PASS"
  42. else:
  43. print "FAIL, GOT: %r"%result
  44.  
  45. if __name__ == "__main__":
  46. test()
  47.  
Matt
Jun 27 '08 #5

P: n/a
I have actually highlighted a small neat recipe for doing such
unpacking, that I use for parsing arbitrary parameters in Evoque
Templating. I never needed to handle "callable" parameters though, as
you do in your 3rd string example, so the little "unpack_symbol"
recipe I have publiched earlier does not handle it... anyhow, what I
referring to are:

Evoque Templating: http://evoque.gizmojo.org/
Code highlight: http://gizmojo.org/code/unpack_symbol/

However, a little variation of the aboverecipe can do what you are
looking for, in a rather cute way. The difference is to make the
original recipe handle "callable strings", and I achieve this by
modifying the recipe like so:
class callable_str(str):
def __call__(s, *args):
return s+str(args)

class _UnpackGlobals(dict):
def __getitem__(self, name):
return callable_str(name)

def unpack_symbol(symbol, globals=_UnpackGlobals()):
""" If compound symbol (list, tuple, nested) unpack to atomic
symbols """
return eval(symbol, globals, None)
Now, calling unpack_symbol() on each sample string gives the following
tuple of strings:
>>unpack_symbol('foo, bar, baz')
('foo', 'bar', 'baz')
>>unpack_symbol('foo, "bar, baz", blurf')
('foo', 'bar, baz', 'blurf')
>>unpack_symbol('foo, bar(baz, blurf), mumble')
('foo', "bar('baz', 'blurf')", 'mumble')
>>>

Mario Ruggier
Jun 27 '08 #6

This discussion thread is closed

Replies have been disabled for this discussion.