By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
426,115 Members | 919 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 426,115 IT Pros & Developers. It's quick & easy.

Design mini-lanugage for data input

P: n/a
This is an entry I just added to ASPN. It is a somewhat novel technique I
have employed quite successfully in my code. I repost it here for more
explosure and discussions.

http://aspn.activestate.com/ASPN/Coo.../Recipe/475158

wy
------------------------------------------------------------------------
Title: Design mini-lanugage for data input
Description:

Many programs need a set of initial data. For ease of use and flexibility,
design a mini-language for your input data. Use Python's superb text
handling capability to parse and build the data structure from the input
text.

Source: Text Source
# this is an example to demonstrate the programming technique

DATA = """
# data souce: http://www.mongabay.com/igapo/world_...ics_by_pop.htm
# Country / Captial / Area [sq. km] / 2002 Population Estimate
China / Beijing / 9,596,960 / 1,284,303,705
India / New Delhi / 3,287,590 / 1,045,845,226
United States / Washington DC / 9,629,091 / 280,562,489
Indonesia / Jakarta / 1,919,440 / 231,328,092
Russia / Moscow / 17,075,200 / 144,978,573
"""

def initData():
""" parse and return a country list of (name, captial, area,
population) """

countries = []
for line in DATA.splitlines():

# filter out blank lines/comment lines
line = line.strip()
if not line or line.startswith('#'):
continue

# 4 fields separated by '/'
parts = map(string.strip, line.split('/'))
country, captial, area, population = parts

# remove commas in numbers
area = int(area.replace(',',''))
population = int(population.replace(',',''))

countries.append((country, captial, area, population))

return countries
def findLargestCountry(countries):
# your algorithm here
def main():
countries = initData()
print findLargestCountry(countries)
Discussion:

Problem
-------

Many programs need a set of initial data. The simplest way is to construct
Python data structure directly as shown below. This is often not ideal.
Algorithm and data structure tend to change. Python program statements is
likely differ literally from its data source, which might be text pulled
from web pages or other place. This means a great deal of effort is often
needed to format and maintain the input as Python statements.

This is a sample program that initialize some geographical data.

# map of country -> (captial, area, population)
COUNTRIES = {}
COUNTRIES['China'] = ('Beijing', 9596960, 1284303705)
COUNTRIES['India'] = ('New Delhi', 3287590, 1045845226)
COUNTRIES['United States'] = ('Washington DC', 9629091, 280562489)
COUNTRIES['Indonesia'] = ('Jakarta', 1919440, 231328092)
COUNTRIES['Russia'] = ('Moscow', 17075200, 144978573)
Mini-language
-------------

A more flexible approach is to define a mini-lanugage to describe the
data. This can be as simple as formatting data into a multiple-line string.

1. Define the data format in text. It should mirror the data source and
designed for ease for human editing.

2. Define the data structure.

3. Write glue code to parse the input data and initialize the data
structure.

In the example above we use one line for each record. Each record has four
fields, Country, captial, area and population, separated by slashes. One
of the immediate benefit is that we no longer need to type so many quotes
for every string literal. This concise data format is much easiler to read
and edit than Python statements.

The parser simply break down the input text using splitlines() and then
loop through them line by line. It is useful to account for some extra
white space so that it is more flexible for human editor. In this case the
numbers (area, population) from the data source contains commas. Rather
than manually edit them out, they are copied as is into the text as is.
Then they are parsed into integer using

area = int(area.replace(',',''))

Slash is chosen as the separator (rather than the more common comma)
because it does not otherwise appear in the data. A record is parsed into
field using

line.split('/')

Don't forget to remove extra white space using string.strip()

Finally it built a data structure of list of country record as tuple of
(country, captial, area, population). It is just as easy to turn them into
objects or any other data structure as desired.

The mini-language technique can be refined to represent more complex, more
structured input. It makes transformation and maintenance of input data
much easier.
Mar 21 '06 #1
Share this Question
Share on Google+
3 Replies


P: n/a
Hmm,
Do you know about JSON and YAML?
http://en.wikipedia.org/wiki/JSON
http://en.wikipedia.org/wiki/YAML

They have the advantage of being maintained by a group of people and
being available for a number of languages. (as well as NOT being XML
:-)

- Cheers, Paddy.
--
http://paddy3118.blogspot.com/

Mar 21 '06 #2

P: n/a
Yes. But they have different motivations.

The mini-language concept is to design an input format that is convenient
for human editor and that is close to the semi-structured data source. I
think the benefit from ease of editing and flexibility would justify
writing a little parsing code.

JSON is mainly designed for data exchange between programs. You can hand
edit JSON data (as well as XML or Python statement) but it is not the most
convenient.

Just consider you don't have to enter two quotes for every string object
is almost liberating. These quotes are only artifacts for structured data
format. The idea to design a format convenient for human and let code to
parse and built the data structure.

wy
Hmm,
Do you know about JSON and YAML?
http://en.wikipedia.org/wiki/JSON
http://en.wikipedia.org/wiki/YAML

They have the advantage of being maintained by a group of people and
being available for a number of languages. (as well as NOT being XML
:-)

- Cheers, Paddy.
--
http://paddy3118.blogspot.com/


Mar 21 '06 #3

P: n/a
P.S. Also it is a 'mini-language' because it is an ad-hoc design that is
good enough and can be easily implemented for a given problem. This is
oppose to a general purpose solution like XML that is one translation from
the original data format and carries too much baggages.
Just consider you don't have to enter two quotes for every string object
is almost liberating. These quotes are only artifacts for structured
data format. The idea to design a format convenient for human and let
code to parse and built the data structure.

wy

Mar 21 '06 #4

This discussion thread is closed

Replies have been disabled for this discussion.