473,325 Members | 2,608 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,325 software developers and data experts.

Object Persistence Using a File System

Before I get too carried away with something that's probably
unnecessary, please allow me to throw around some ideas. I've been
looking for a method of transparent, scalable, and human-readable object
persistence, and I've tried the standard lib's Shelve, Zope's ZODB,
Divmod's Axiom, and others. However, while they're all useful, none
satisfies all my criteria. So I started writing some toy code of my own:
http://paste.plone.org/5227

All my code currently does is transparently keep track of object changes
without requiring any special coding on part of the user, and a function
to convert an object to a file system hierarchy of folders and files.
Please, let me know what you think.

Thanks,
Chris
Jul 12 '06 #1
5 1482
Chris Spencer wrote:
Before I get too carried away with something that's probably
unnecessary, please allow me to throw around some ideas. I've been
looking for a method of transparent, scalable, and human-readable object
persistence, and I've tried the standard lib's Shelve, Zope's ZODB,
Divmod's Axiom, and others. However, while they're all useful, none
satisfies all my criteria. So I started writing some toy code of my own:
http://paste.plone.org/5227

All my code currently does is transparently keep track of object changes
without requiring any special coding on part of the user, and a function
to convert an object to a file system hierarchy of folders and files.
Please, let me know what you think.
As you say, using filesystem for fine-grained persistance may not be the
most efficient solution. I also wonder how (if...) you intend to address
concurrent R/W access and transactions...

A few observations and questions :
- you should avoid tests on concrete types as much as possible - at
least use isinstance
- tuples are immutable containers. What about them ?
- what about multiple references to a same object ?

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
Jul 12 '06 #2
Chris,

Interesting concept. But why is there a need for a human readable
object persistence that is x10 slower than pickle? In other words
present a good use case with a rationale (i.e. your "criteria" that you
mentioned). The only one I can think of so far is debugging.

Also some objects are inherently not human readable (they are large, or
just binary/array data for example), or you could simply end up having
so many of them (10GB worth of disk space) that just due to the volume
they will become not very readable and you would need some kind of a
custom query mechanism (or find+grep) to search through your objects if
you wanted to read/edit values in them.

In your code comments I saw that another reason is resistance to
corruption. I think that a database that is backed up regularly is
probably a better solution. Besides, sometimes you want your failure to
be dramatic (go down with a bang!) so it gets your attention and you
can restore everything with a backup. Otherwise, with a tens of
thousands of files, some of them might end up being corrupted before
your next filesystem check gets to them, by then, the corruption could
spread (your program would load it, perform computations, persist the
wrong results and so on), and you wouldn't even notice it.

Hope these comments help,
Nick V.

Chris Spencer wrote:
Before I get too carried away with something that's probably
unnecessary, please allow me to throw around some ideas. I've been
looking for a method of transparent, scalable, and human-readable object
persistence, and I've tried the standard lib's Shelve, Zope's ZODB,
Divmod's Axiom, and others. However, while they're all useful, none
satisfies all my criteria. So I started writing some toy code of my own:
http://paste.plone.org/5227

All my code currently does is transparently keep track of object changes
without requiring any special coding on part of the user, and a function
to convert an object to a file system hierarchy of folders and files.
Please, let me know what you think.

Thanks,
Chris
Jul 12 '06 #3
Good point about isinstance. Here is a good explanation why:
http://www.canonical.org/~kragen/isinstance/
Also the frozenset should be added the list of immutable types.

Nick Vatamaniuc
Bruno Desthuilliers wrote:
Chris Spencer wrote:
Before I get too carried away with something that's probably
unnecessary, please allow me to throw around some ideas. I've been
looking for a method of transparent, scalable, and human-readable object
persistence, and I've tried the standard lib's Shelve, Zope's ZODB,
Divmod's Axiom, and others. However, while they're all useful, none
satisfies all my criteria. So I started writing some toy code of my own:
http://paste.plone.org/5227

All my code currently does is transparently keep track of object changes
without requiring any special coding on part of the user, and a function
to convert an object to a file system hierarchy of folders and files.
Please, let me know what you think.

As you say, using filesystem for fine-grained persistance may not be the
most efficient solution. I also wonder how (if...) you intend to address
concurrent R/W access and transactions...

A few observations and questions :
- you should avoid tests on concrete types as much as possible - at
least use isinstance
- tuples are immutable containers. What about them ?
- what about multiple references to a same object ?

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'o****@xiludom.gro'.split('@')])"
Jul 12 '06 #4
Nick Vatamaniuc a écrit :
(please don't top-post - corrected)
>
Bruno Desthuilliers wrote:
(snip)
>>A few observations and questions :
- you should avoid tests on concrete types as much as possible - at
least use isinstance

Good point about isinstance. Here is a good explanation why:
http://www.canonical.org/~kragen/isinstance/
Err... I'm sorry but justifying the use of explicit tests on concrete
type not even taking inheritance into consideration with a paper
explaining why type tests taking inheritance into consideration may be
bad seems rather strange to me... Or did I missed your point here ?
Jul 18 '06 #5
In message <1O%sg.3749$wZ.3322@trndny01>, Chris Spencer wrote:
I've been looking for a method of transparent, scalable, and
human-readable object persistence...
Don't do object persistence. What is an object? It's a combination of code
and data. Code structure is internal to your program--it has no business
being reflected in external data that may be saved to a persistent medium,
transmitted over an external channel or whatever. Otherwise when you
refactor your code, that external data no longer becomes readable without
major backward-compatibility hacks.

Use data abstraction instead: define a high-level data structure that is
independent of implementation details of your code. When you look at the
major data-interchange formats in use today--such as XML, ODF, MPEG,
WAV--there's a reason why none of them are built on object persistence.
Jul 19 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Paolo Losi | last post by:
Hi all, I'm pretty new to the python language so please excuse me if this is FAQ... I'm very glad to be part of the list! :-) I'm looking into a way to implement a generic workflow framework...
15
by: Ville Vainio | last post by:
Pythonic Nirvana - towards a true Object Oriented Environment ============================================================= IPython (by Francois Pinard) recently (next release - changes are...
2
by: Chris Murphy via DotNetMonster.com | last post by:
Hey guys, I've been hitting a brick wall with a problem I've come accross in developing an application. Background: The application uses one primary class that I'm trying to implement with the...
5
by: Michael Moreno | last post by:
Hello, In a class I have this code: public object Obj; If Obj is a COM object I would like to call in the Dispose() method the following code: ...
2
by: damian.jolly | last post by:
Is persisting objects to the SESSION the best way to prevent an object from being created every time a page is refreshed? It's a bit complicated but I'll try to be brief... I am using a guage...
6
by: Peter Richardson | last post by:
Hi, I'm wondering if someone can help me with some design questions I have. I'm trying to create a class in C# to represent my customers. I know how to create teh Customer class and all, but my...
1
by: Don | last post by:
I'm getting the following exception displayed in the task list at design time for my project: "Code generation for property 'Controls' failed. Error was: 'Object reference not set to an...
1
by: =?ISO-8859-1?Q?Lasse_V=E5gs=E6ther_Karlsen?= | last post by:
I get the above error in some of the ASP.NET web applications on a server, and I need some help figuring out how to deal with it. This is a rather long post, and I hope I have enough details that...
0
by: Bill McCormick | last post by:
I'm looking for the best (easiest, cheapest, fastest, most flexible) way to implement object persistence in .NET 3.5. The object data is native XML, so serialization is a good option as well as...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.