473,698 Members | 2,097 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

disk based dictionaries


Hi,

I want to store dictionaries on disk. I had a look at a few modules
like bsddb, shelve etc. However would it be possible for me to do the
following

hash[1] = [1, 2, 3] where the key is an int and not a string

bsddb requires that both the key,value are string.
shelve does support values being object but not the keys. Is there any
module which support keys which are not strings

Also how do i use disk based hashes for multidimensiona l hashes such as
below

#!/usr/bin/python

dict={}
dict['key1'] = {}
dict[('key1')][('key2')] = 'value'

key1=dict['key1']
print key1['key2']

I have read of mxBeeDict but was unable to get it work properly. I am
not sure if it supports what i need as i was unable to get any
documentation about it. Is the module used widely ?

Below is how i am using the module

bdict = BeeDict('/tmp/beedict')

bdict[1] = 1
print bdict.keys()

bdict.commit()
bdict.close()

bdict1 = BeeDict('/tmp/beedict')
print bdict1.keys()
print bdict1.values()
Would it be that using disk based dictionaries once opened are as fast
as in memory dictionaries ?

Thanks in advance,

Best Regards,
Shivram U


Confidentiality Notice

The information contained in this electronic message and any attachments to this message are intended
for the exclusive use of the addressee(s) and may contain confidential or privileged information. If
you are not the intended recipient, please notify the sender at Wipro or Ma*******@wipro .com immediately
and destroy all copies of this message and any attachments.
Jul 18 '05 #1
3 2369
You may also want to take a look at ZODB (Zope database).
It handles the pickling, storage and retrieval of all
Python objects (including dictionaries) very well. And yes
you can use ZODB without using Zope proper.

http://www.zope.org/Products/StandaloneZODB

http://zope.org/Members/adytumsoluti...LoveZODB_PartI

http://www.h7.dion.ne.jp/~harm/ZODB-Tutorial.py
Larry Bates
Shivram U wrote:
Hi,

I want to store dictionaries on disk. I had a look at a few modules
like bsddb, shelve etc. However would it be possible for me to do the
following

hash[1] = [1, 2, 3] where the key is an int and not a string

bsddb requires that both the key,value are string.
shelve does support values being object but not the keys. Is there any
module which support keys which are not strings

Also how do i use disk based hashes for multidimensiona l hashes such as
below

#!/usr/bin/python

dict={}
dict['key1'] = {}
dict[('key1')][('key2')] = 'value'

key1=dict['key1']
print key1['key2']

I have read of mxBeeDict but was unable to get it work properly. I am
not sure if it supports what i need as i was unable to get any
documentation about it. Is the module used widely ?

Below is how i am using the module

bdict = BeeDict('/tmp/beedict')

bdict[1] = 1
print bdict.keys()

bdict.commit()
bdict.close()

bdict1 = BeeDict('/tmp/beedict')
print bdict1.keys()
print bdict1.values()
Would it be that using disk based dictionaries once opened are as fast
as in memory dictionaries ?

Thanks in advance,

Best Regards,
Shivram U


Confidentiality Notice

The information contained in this electronic message and any attachments to this message are intended
for the exclusive use of the addressee(s) and may contain confidential or privileged information. If
you are not the intended recipient, please notify the sender at Wipro or Ma*******@wipro .com immediately
and destroy all copies of this message and any attachments.

Jul 18 '05 #2
You may also want to take a look at ZODB (Zope database).
It handles the pickling, storage and retrieval of all
Python objects (including dictionaries) very well. And yes
you can use ZODB without using Zope proper.

http://www.zope.org/Products/StandaloneZODB

http://zope.org/Members/adytumsoluti...LoveZODB_PartI

http://www.h7.dion.ne.jp/~harm/ZODB-Tutorial.py
Larry Bates
Shivram U wrote:
Hi,

I want to store dictionaries on disk. I had a look at a few modules
like bsddb, shelve etc. However would it be possible for me to do the
following

hash[1] = [1, 2, 3] where the key is an int and not a string

bsddb requires that both the key,value are string.
shelve does support values being object but not the keys. Is there any
module which support keys which are not strings

Also how do i use disk based hashes for multidimensiona l hashes such as
below

#!/usr/bin/python

dict={}
dict['key1'] = {}
dict[('key1')][('key2')] = 'value'

key1=dict['key1']
print key1['key2']

I have read of mxBeeDict but was unable to get it work properly. I am
not sure if it supports what i need as i was unable to get any
documentation about it. Is the module used widely ?

Below is how i am using the module

bdict = BeeDict('/tmp/beedict')

bdict[1] = 1
print bdict.keys()

bdict.commit()
bdict.close()

bdict1 = BeeDict('/tmp/beedict')
print bdict1.keys()
print bdict1.values()
Would it be that using disk based dictionaries once opened are as fast
as in memory dictionaries ?

Thanks in advance,

Best Regards,
Shivram U


Confidentiality Notice

The information contained in this electronic message and any attachments to this message are intended
for the exclusive use of the addressee(s) and may contain confidential or privileged information. If
you are not the intended recipient, please notify the sender at Wipro or Ma*******@wipro .com immediately
and destroy all copies of this message and any attachments.

Jul 18 '05 #3
I'd like to second this suggestion. While there are a few things you
need to be aware of when writing your code (mostly taken care of in
the latest release) it's a mostly trivial code change. (For me it was
replacing a few dictionaries with PersistentMap objects and changing
the base class of a few objects to Persistant from object.

FWIW, I'm using ZODB to help track EDI transactions for a help desk
application. Right now my database hovers in the 100MB range with
several ten of thousands of objects. I also use it for single object
temp storage, so I feel it works well from both the small and mid-size
scale. (It probably works fine for large projects as well, I just
don't have one right now...)

Chris
On Thu, 02 Dec 2004 18:53:44 -0600, Larry Bates <lb****@syscono nline.com> wrote:
You may also want to take a look at ZODB (Zope database).
It handles the pickling, storage and retrieval of all
Python objects (including dictionaries) very well. And yes
you can use ZODB without using Zope proper.

http://www.zope.org/Products/StandaloneZODB

http://zope.org/Members/adytumsoluti...LoveZODB_PartI

http://www.h7.dion.ne.jp/~harm/ZODB-Tutorial.py
Larry Bates


Shivram U wrote:
Hi,

I want to store dictionaries on disk. I had a look at a few modules
like bsddb, shelve etc. However would it be possible for me to do the
following

hash[1] = [1, 2, 3] where the key is an int and not a string

bsddb requires that both the key,value are string.
shelve does support values being object but not the keys. Is there any
module which support keys which are not strings

Also how do i use disk based hashes for multidimensiona l hashes such as
below

#!/usr/bin/python

dict={}
dict['key1'] = {}
dict[('key1')][('key2')] = 'value'

key1=dict['key1']
print key1['key2']

I have read of mxBeeDict but was unable to get it work properly. I am
not sure if it supports what i need as i was unable to get any
documentation about it. Is the module used widely ?

Below is how i am using the module

bdict = BeeDict('/tmp/beedict')

bdict[1] = 1
print bdict.keys()

bdict.commit()
bdict.close()

bdict1 = BeeDict('/tmp/beedict')
print bdict1.keys()
print bdict1.values()
Would it be that using disk based dictionaries once opened are as fast
as in memory dictionaries ?

Thanks in advance,

Best Regards,
Shivram U


Confidentiality Notice

The information contained in this electronic message and any attachments to this message are intended
for the exclusive use of the addressee(s) and may contain confidential or privileged information. If
you are not the intended recipient, please notify the sender at Wipro or Ma*******@wipro .com immediately
and destroy all copies of this message and any attachments.

--
http://mail.python.org/mailman/listinfo/python-list

--
"It is our responsibilitie s, not ourselves, that we should take
seriously." -- Peter Ustinov
Jul 18 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
2427
by: Chris | last post by:
I have a set of routines, the first of which reads lots and lots of data from disparate regions of disk. This read routine takes 40 minutes on a P3-866 (with IDE drives). This routine populates an array with a number of dictionaries, e.g., (not actually the data i'm reading) This information is acted upon by subsequent routines. These routines change very often, but the data changes very infrequently (the
8
3848
by: robin | last post by:
I need to do a search through about 50 million records, each of which are less than 100 bytes wide. A database is actually too slow for this, so I thought of optimising the data and putting it all in memory. There is a single key field, so a dictionary is an obvious choice for a structure, since Python optimises these nicely. But is there a better choice? Is it worth building some sort of tree?
8
4229
by: Foodbank | last post by:
Hi, Has anyone ever hashed a text file to disk before? I'm trying to convert an in-memory hash to disk hash and I can't find any relevant information to help me. Also, I'd like to use lseek, read, write, and fd if anyone is familiar with them. Any info would be greatly helpful. Thanks, James
3
3024
by: Bruce | last post by:
I am building a WinForms app that uses Web Services access to a server for most of its data input/output, but I also need to persist some of its data to the local disk (basically as a cache of some of the Web Services data) in XML format. Since the size of the XML local store could be rather large, I'd prefer to have a random access mechanism for reading and writing to it. It seems that XMLReader /XMLWriter are sequentially fast,...
57
10830
by: Chris Foote | last post by:
Hi all. I have the need to store a large (10M) number of keys in a hash table, based on a tuple of (long_integer, integer). The standard python dictionary works well for small numbers of keys, but starts to perform badly for me inserting roughly 5M keys: # keys dictionary metakit (both using psyco) ------ ---------- ------- 1M 8.8s 22.2s
1
1472
by: dopey483 | last post by:
I am manipulating lots of log files (about 500,000 files and about 30Gb in total) to get them into a little SQL db. Part of this process is "normalisation" and creating tables of common data. I am creating dictionaries for these in a simple {value,key} form. In terms of memory and performance what are the reasonable limits for a dictionary with a key and a 16 character string? eg; if I read in one of my tables from disk into a...
1
1854
by: qouify | last post by:
Hello, here is my problem. I have a dictionnary of words (e.g., type char) stored sequentially (but not sorted) in a file F and I want to insert in it some words stored in memory in a balanced tree T. some elements of T may already be in F and I do not want to have them twice in F after insertion. i want to do something like this: for each word in F - if word is in T
11
3060
by: GG | last post by:
Anybody knows of any collection where is not stored in memory but using hard disk instead? Thanks *** Sent via Developersdex http://www.developersdex.com ***
0
216
by: M.-A. Lemburg | last post by:
On 2008-07-31 02:29, python@bdurham.com wrote: If you don't have a problem with taking a small performance hit, then I'd suggest to have a look at mxBeeBase, which is an on-disk dictionary implementation: http://www.egenix.com/products/python/mxBase/mxBeeBase/ Of course, you could also use a database table for this. Together with a proper index that should work as well (but it's likely slower
0
8603
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9157
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9027
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8861
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7725
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6518
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5860
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4619
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3046
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.