473,326 Members | 2,182 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

why use special config formats?

hey

i've been seeing lots of config-file-readers for python. be it
ConfigObj (http://www.voidspace.org.uk/python/configobj.html) or the
like. seems like a trend to me.
i came to this conclusion a long time ago: YOU DON'T NEED CONFIG FILES
FOR PYTHON. why re-invent stuff and parse text by yourself, why the
interpreter can do it for you? and anyway, i find this a very ugly
format:
http://www.voidspace.org.uk/python/c...ig-file-format

there are two use cases for configuration: static vs. dynamic
configuration.

for the most common case, static configuration, you just have a
human-edited config file holding key-and-value pairs. so just add to
your package a file called config.py, and import it.

for example, if that's our package structure:
PyApache/
__init__.py
config.py
server.py

then server.py would do:
....
import config
listener_sock.bind((config.host, config.port))
....

and config.py would look like:
# the port to bind to
port = 80
host = "localhost"
timeout = 300
enable_keep_alives = False
options = [1, 2, 3]
....

isn't python suitable enough to hold your configuration?

the second case, dynamic configuration, is when you need to alter your
configuration at runtime or programatically, so the configuration
doesnt need to be human-readable. for that case -- use pickle. and
Bunch (as shown on the aspn python cookbook)

class Bunch(object):
def __init__(self, **kw):
self.__dict__.update(kw)

create the initial config file:
config = Bunch(port = 80, host = "localhost", timeout = 300, ...)
pickle.dump(open("config.pkl", "wb"), config)

of course you can nest Bunch'es inside one another, i.e.,
config = Bunch(
# global config
port = 80,
host = "localhost",

# this is per-user configuration
users = {
"malcom_x" : Bunch(
http_path = "/home/joe/httpdocs",
cgi_path = "/home/joe/cgi-bin",
options = ["i love lucy", "bush is gay"]
),
...
},
...
)

and now you use it:
# global configuration
config = pickle.load(open("config.pkl"))
listener_sock.bind((config.host, config.port))
# and per-user configuration
from getpass import getuser
print config.users[getuser()].http_path
....

that way, if you need to programatically change your configuration,
just change and pickle.dump() it.

hope it helps,
-tomer

Mar 10 '06 #1
20 2574
and just as i was writing, this was added to lang.python.announce:

http://groups.google.com/group/comp....a7b35599f65794

-tomer

Mar 10 '06 #2
to*********@gmail.com enlightened us with:
i came to this conclusion a long time ago: YOU DON'T NEED CONFIG
FILES FOR PYTHON. why re-invent stuff and parse text by yourself,
why the interpreter can do it for you?
Because you generally don't want to give the configuration file writer
full control over the Python virtual machine.
for the most common case, static configuration, you just have a
human-edited config file holding key-and-value pairs. so just add to
your package a file called config.py, and import it.


Which only works if there is only one configuration file per
installation of your package, and is writable by the users that need
to configure it. For example, per-user database connection parameters
should be in $HOME/.programrc on UNIX systems. A program's preference
settings should be stored in a user-writable file to, preferably in
the user's homedir.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
Mar 10 '06 #3
if you are really so scared of letting others exploit your config
scripts, then use the second, pickled fashion. that way you can store
the file at $HOME/blah-config.pkl, and everybody's happy.

still, my point is we dont need special config mechanisms, since the
builtin ones, like object persistency (sp) or python scripts are good
enough, less buggy, and dont require you to learn thousands of config
formats.

and you can even edit pickled files by hand (protocol 0 i believe).
it's not that complicated.
-tomer

Mar 10 '06 #4
to*********@gmail.com wrote:
if you are really so scared of letting others exploit your config
scripts, then use the second, pickled fashion. that way you can store
the file at $HOME/blah-config.pkl, and everybody's happy.
Except the user who wants to edit the config file.
still, my point is we dont need special config mechanisms, since the
builtin ones, like object persistency (sp) or python scripts are good
enough, less buggy, and dont require you to learn thousands of config
formats.

and you can even edit pickled files by hand (protocol 0 i believe).
it's not that complicated.

Fine. Kindly write the "How to Edit Your Configuration" instructions for
naive users. I think you might find they object to such an obscure format.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd www.holdenweb.com
Love me, love my blog holdenweb.blogspot.com

Mar 10 '06 #5
I agree with Steve and I agree Sybren.

Also:
This is a Bad Idea, since you should never add more complexity than needed. Imports, computation, IO and so on are generally not needed for program configuration, so standard configfile syntax should therefore not allow it. Otherwise you may easily end up with hard-to-debug errors, or even worse - weird program behavior.

/Joel

Mar 10 '06 #6
to*********@gmail.com enlightened us with:
if you are really so scared of letting others exploit your config
scripts, then use the second, pickled fashion. that way you can
store the file at $HOME/blah-config.pkl, and everybody's happy.
Ehm... and how is a user supposed to edit that? I wouldn't be happy...
still, my point is we dont need special config mechanisms, since the
builtin ones, like object persistency (sp) or python scripts are
good enough, less buggy, and dont require you to learn thousands of
config formats.


Oh, and the ConfigParser module requires you to learn *thousands* of
config formats. Right.

I think you need to get real.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
Mar 10 '06 #7
<to*********@gmail.com> wrote:
isn't python suitable enough to hold your configuration?


that depends on the target application, and, more importantly, the
target audience and what kind of configuration they're expected to
do.

there's no "one rule to rule them all" for configuration issues.

(except, possibly, that zero configuration is often easier to use than
any configuration file format...)

</F>

Mar 10 '06 #8
i dont know about your experience with config files, but there
thousands of formats. on the python side -- just in this conversation,
we mentioned ConfigObj, ConfigParser and the Config module i linked to.
when everybody writes his own config, you get loads of unique formats.

anyway, for all the cry-babies here that can't edit pickle files. okay
-- just load() them, change what you want, and dump() them. don't cry.

and if you insist, i'm sure there's a python serializer to
XML/SOAP/whatever other readble format. persistency is far better for
configuration than config files. they are limited, have weird syntaxes,
hard to extend, and are never generic enough. with my approach --
anything you can do in python, or anything you can pickle -- is
possible.

and for security issues -- usually config files are edited by admins,
so that's not a problem. and per-user config files (at $HOME), can
easily be achieved with execfile(). the point is NOT TO WRITE A PARSER
for every config file.

you can easily crash your web server (or make it non functional) if you
pass an invalid port or host, or make it act weird by changing the
timeouts or paths... so yeah, if the admin writes a config script that
does os.system("rm -rf /"), well, too bad. but then the admin can do
stupid things at the shell level as well.

again -- the points are:
* python is readable and easy to write config files with
* usually admins change the configuration, and they have too much power
anyway
* if you worry about security/too much power, pickle your config
* if you need to edit your pickled config on a regular basis, serialize
it with some other textual serializer (xml, etc).

but inventing proprietary formats with unique syntaxes, and having to
write and debug parsers for them -- that's stupid. a configuration is
just a persistent state of your program. it shouldnt be any more
complex than that.

-tomer

Mar 10 '06 #9
to*********@gmail.com enlightened us with:
i dont know about your experience with config files, but there
thousands of formats.
All the config files I needed were either very easy to learn, or well
documented in comments.
on the python side -- just in this conversation, we mentioned
ConfigObj, ConfigParser and the Config module i linked to. when
everybody writes his own config, you get loads of unique formats.
Hence the Python modules.
anyway, for all the cry-babies here that can't edit pickle files.
okay -- just load() them, change what you want, and dump() them.
don't cry.
You really need to get real here. Configuration files are for *users*,
not programmers. You can't expect a user to learn about Python in
general and about pickle in specific.
and if you insist, i'm sure there's a python serializer to
XML/SOAP/whatever other readble format.
Which then gives you another configuration format to learn...
and for security issues -- usually config files are edited by
admins, so that's not a problem.
You go explain that to someone who just wants to edit his mail
client's config file.
and per-user config files (at $HOME), can easily be achieved with
execfile().
Which is then totally insecure. An exploit can easily be made then -
just inject a rootkit downloading & starting script into someone's
email client configuration file and boom, computer is hacked.
the point is NOT TO WRITE A PARSER for every config file.
Hence standard config file formats and parser modules.
* usually admins change the configuration, and they have too much
power anyway
Admins have too much power? Go get an education.
* if you worry about security/too much power, pickle your config
Sure, and where would you keep your comments explaining the
configuration fields?
but inventing proprietary formats with unique syntaxes, and having
to write and debug parsers for them -- that's stupid.


Which is why there are standard modules for them.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
Mar 10 '06 #10

to*********@gmail.com wrote:
if you are really so scared of letting others exploit your config
scripts, then use the second, pickled fashion. that way you can store
the file at $HOME/blah-config.pkl, and everybody's happy.

still, my point is we dont need special config mechanisms, since the
builtin ones, like object persistency (sp) or python scripts are good
enough, less buggy, and dont require you to learn thousands of config
formats.

Well... ConfigObj uses the same format as ConfigParser, which the basic
ini style.

The message is that config files are for users, and so should be in a
format convenient for them - not for the machine.

Call your users cry-babies if yu want, you won't have many...
and you can even edit pickled files by hand (protocol 0 i believe).
it's not that complicated.

If you're happy with a hardwired config file, you don't need a config
file at all.

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

-tomer


Mar 11 '06 #11
On Fri, 10 Mar 2006 06:48:03 -0800, tomerfiliba wrote:
hey

i've been seeing lots of config-file-readers for python. be it
ConfigObj (http://www.voidspace.org.uk/python/configobj.html) or the
like. seems like a trend to me.
i came to this conclusion a long time ago: YOU DON'T NEED CONFIG FILES
FOR PYTHON.
Of course you do.

Sometimes you have to be able to parse and understand existing config
files that have come from somewhere else. If your task is "read and parse
a .ini file", insisting the user re-writes their ini file as Python code
isn't helpful.

Separating code from data is always a good idea. I hope I don't need to
explain why. So you want config files, the only question is, what format
should they be in?

Sometimes it can be useful, especially for quick and dirty apps, to use a
Python module as a config file. But that's not a good idea for production
level code where end users are expected to edit the data:

# config.py
value = 2.5
colour = "blue"

The user edits value to 3.1, but accidentally puts in a typo "3,1".
Now when your application imports the config.py module, it silently
assigns the tuple (3, 1) to value, and your app dies an unpleasant death
somewhere a long way away. You have no idea why.

So you end up coding defensively to protect against user typos or
stupidity (and believe me, even if your users are technically minded IT
professionals, they will screw up your config files):

# config.py
import logger, sys
value = 2.5 # warning: value must be a float
colour = "blue" # warning: colour must be one of "red", "blue", "green"
# warning: quotes are compulsory
try:
colour = colour.strip()
except AttributeError:
pass
if type(value) != float or value < 0.0:
logger.log("Value is %s" % value)
print >>sys.stderr("Bad value, using default")
value = 2.5
if colour not in ("blue", "red", "green"):
logger.log("Colour is %s" % value)
print >>sys.stderr("Bad colour, using default")
colour = "bleu" # oops, a bug

and now your config file is code instead of data, and you expect your
users to hack code to change a default value. B--A--D idea.

Using a data config file means you can separate the user-editable data
from the code that verifies that it has sensible values. Your config file
becomes simple again:

# config.py
value = 2.5
colour = "blue"

and your users no longer have to wade through complex code to change a few
defaults, but you still have full flexibility to vet their data.
why re-invent stuff and parse text by yourself, why the
interpreter can do it for you? and anyway, i find this a very ugly
format:
http://www.voidspace.org.uk/python/c...ig-file-format


You are joking right? Pulling our legs?

Here is the config file format, according to the link you supply:

# comment line
# comment line
keyword = value # inline comment

Here is the equivalent written in pure Python:

# comment line
# comment line
keyword = value # inline comment
Why is the first uglier than the second?

--
Steven.

Mar 11 '06 #12
On Fri, 10 Mar 2006 09:08:36 -0800, tomerfiliba wrote:
you can easily crash your web server (or make it non functional) if you
pass an invalid port or host, or make it act weird by changing the
timeouts or paths... so yeah, if the admin writes a config script that
does os.system("rm -rf /"), well, too bad.
Not if the code is being run on YOUR webserver and the config file is
being edited on some compromised PC in Romania.

again -- the points are:
* python is readable and easy to write config files with
* usually admins change the configuration, and they have too much power
anyway
So why do you want to give them MORE power?
* if you worry about security/too much power, pickle your config
Huh? You think a competent sys admin can't learn enough Python to hack
your pickled file?

Binary configs only keep out legitimate users who don't have the time or
ability to learn how to hack the binary format. Black hats and power users
will break your binary format and hack them anyway.
* if you need to edit your pickled config on a regular basis, serialize
it with some other textual serializer (xml, etc).
But you forget the most important point of all:

* keep your data separate from your code.

but inventing proprietary formats with unique syntaxes, and having to
write and debug parsers for them -- that's stupid. a configuration is
just a persistent state of your program. it shouldnt be any more
complex than that.


Exactly. And that's why we have two or three common config file formats,
such as xml, ini files, etc. Pick one of them and stick to it.
--
Steven.

Mar 11 '06 #13
> Huh? You think a competent sys admin can't learn enough Python to hack
your pickled file?

Binary configs only keep out legitimate users who don't have the time or
ability to learn how to hack the binary format. Black hats and power users
will break your binary format and hack them anyway.


then you dont know what pickle is. pickle code is NOT python bytecode.
it's a bytecode they made in order to represent objects. you cannot
"exploit" in in the essence of running arbitrary code, unless you find
a bug in the pickle module. and that's less likely than you find a bug
in the parser of the silly config file formats you use.

i'm not hiding the configuration in "binary files", that's not the
point. pickle is just more secure by definition.

aah. you all are too stupid.
-tomer

Mar 11 '06 #14
>> Why is the first uglier than the second?
YES THATS THE POINT. PYTHON CAN BE USED JUST LIKE A CONFIG FILE.

and if your users did
timeout = "300"
instead of
timeout = 300

then either your config parser must be uber-smart and all-knowing, and
check the types of key-value pairs, or your server would crash. either
way is bad, and i prefer crash-on-use then
know-everything-and-check-at-the-parser-level.

good night,
-tomer

Mar 11 '06 #15
"gangesmaster" wrote:
Binary configs only keep out legitimate users who don't have the time or
ability to learn how to hack the binary format. Black hats and power users
will break your binary format and hack them anyway.


then you dont know what pickle is. pickle code is NOT python bytecode.
it's a bytecode they made in order to represent objects. you cannot
"exploit" in in the essence of running arbitrary code


import pickle
print pickle.loads("cos\nsystem\np0\n(S'echo really?'\np1\ntp2\nRp3\n.")

</F>

Mar 11 '06 #16
gangesmaster enlightened us with:
YES THATS THE POINT. PYTHON CAN BE USED JUST LIKE A CONFIG FILE.


AND CAN ALSO BE MISUSED AND HARDER TO USE THAN A SIMPLE CONFIG FILE.
Get it into your thick head that you're plain wrong here.

Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
Mar 11 '06 #17
gangesmaster wrote:
Huh? You think a competent sys admin can't learn enough Python to hack
your pickled file?

Binary configs only keep out legitimate users who don't have the time or
ability to learn how to hack the binary format. Black hats and power users
will break your binary format and hack them anyway.

then you dont know what pickle is. pickle code is NOT python bytecode.
it's a bytecode they made in order to represent objects. you cannot
"exploit" in in the essence of running arbitrary code, unless you find
a bug in the pickle module. and that's less likely than you find a bug
in the parser of the silly config file formats you use.

i'm not hiding the configuration in "binary files", that's not the
point. pickle is just more secure by definition.

aah. you all are too stupid.

Great way to win an argument. Pity we aren't as intelligent as you ...

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd www.holdenweb.com
Love me, love my blog holdenweb.blogspot.com

Mar 11 '06 #18
"gangesmaster" <to*********@gmail.com> wrote in
news:11**********************@j52g2000cwj.googlegr oups.com:
aah. you all are too stupid.


-1 QOTW.

--
rzed

Mar 11 '06 #19
On Sat, 11 Mar 2006 05:49:38 -0800, gangesmaster wrote:
Why is the first uglier than the second?

YES THATS THE POINT. PYTHON CAN BE USED JUST LIKE A CONFIG FILE.

and if your users did
timeout = "300"
instead of
timeout = 300

then either your config parser must be uber-smart and all-knowing, and
check the types of key-value pairs, or your server would crash. either
way is bad, and i prefer crash-on-use then
know-everything-and-check-at-the-parser-level.


Well, I think this puts a new light on the argument from Tomer: he'd
prefer his server to crash than to spend some time validating his data.

Would you mind telling us what software you've been involved in writing,
so we know what software to avoid?
--
Steven.

Mar 11 '06 #20
> > YES THATS THE POINT. PYTHON CAN BE USED JUST LIKE A CONFIG FILE.

AND CAN ALSO BE MISUSED AND HARDER TO USE THAN A SIMPLE CONFIG FILE.
Get it into your thick head that you're plain wrong here.


comp.lang.python sure isn't what it used to be :-(

</F>

Mar 11 '06 #21

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Tom Willis | last post by:
How are the expert pythoneers dealing with config files? Is there anything similair to .net's config files or java's .properties? A quick search on google didn't return anything that looked...
125
by: Sarah Tanembaum | last post by:
Beside its an opensource and supported by community, what's the fundamental differences between PostgreSQL and those high-price commercial database (and some are bloated such as Oracle) from...
22
by: Daniel Billingsley | last post by:
Ok, I wanted to ask this separate from nospam's ridiculous thread in hopes it could get some honest attention. VB6 had a some simple and fast mechanisms for retrieving values from basic text...
1
by: Lialie | last post by:
Hello,all I found it easy to read configures from a config file. But how can I set a special value to an item or write it into the original file? I have tried this: import ConfigParser config...
1
by: Nergock | last post by:
Hi, Does anyone know how to handle special characters like "<" and ">" in the web.config file? Something like this would cause my app to not startup. <add key="SenderEmail" value="John Doe...
12
by: wheels619 | last post by:
How can I get access for another user's special folder locations? A configuration file is stored in the users' appData folder and the program altering it will be ran under the admin.
3
by: DGleeson3 | last post by:
Hello All Hope Im posting in the right place. If not please point me elsewhere. We are developing a reasonably standard Web application. VS2005 ASP.NET SQL server 2005. The unusual...
21
by: Nick Craig-Wood | last post by:
Lance Gamet <lance@gamet.comwrote: I've found http://docs.python.org/lib/module-ConfigParser.html To be easy to use and built in. It makes human readable / editable ..ini - like files. ...
1
pradeepjain
by: pradeepjain | last post by:
hii. i have a db tables which i am exporting to csv or other formats . i will be making it format like name and value . so i need 3 special chars which will separate the 2 . like $name*pradeep%...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.