473,666 Members | 2,565 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Best dbm to use?

I'm creating an persistant index of a large 63GB file
containing millions of peices of data. For this I would
naturally use one of python's dbm modules. But which is the
best to use?

The index would be created with something like this:
fh=open('file_t o_index')
db=dbhash.open( 'file_to_index. idx')
for obj in fh:
db[obj.name]=fh.tell()

The index should serve two purposes. Random access and
sequential stepped access. Random access could be dealt with
by the hash table ability for example:
fh.seek(db[name])
obj=fh.GetObj()

However, I may want to access the i'th element in the file.
Something like this:
fh.seek(db.GetE lement(i))
obj=fh.GetObj()

This is where the hash table breaks down and a b-tree would
serve my purpose better. Is there a unified data structure
that I could use or am I doomed to maintaining two seperate
index's?

Thanks in advance for any help.

-Brian
Sep 7 '05 #1
1 1458
br****@temple.e du wrote:
I'm creating an persistant index of a large 63GB file
containing millions of peices of data. For this I would
naturally use one of python's dbm modules. But which is the
best to use?


BDB4, but consider using sqlite - it's really simple, holds all data in
a single file and it's more supported (in the sense: there are bindings
for sqlite for almost any language/environment out there, and the file
format is stable). It's also very fast and you can later add more
information you want to store (by adding more fields to table).

Sep 7 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

15
2226
by: John J | last post by:
I've written the following code into a class to search for and display the results of all races entered (The complete code is in a previous thread). I wish to amend the code so as to display the best result only. Can anyone suggest a simple amendment to the following that will result in only the best result being displayed?
136
9307
by: Matt Kruse | last post by:
http://www.JavascriptToolbox.com/bestpractices/ I started writing this up as a guide for some people who were looking for general tips on how to do things the 'right way' with Javascript. Their code was littered with document.all and eval, for example, and I wanted to create a practical list of best practices that they could easily put to use. The above URL is version 1.0 (draft) that resulted. IMO, it is not a replacement for the FAQ,...
20
11305
by: hagai26 | last post by:
I am looking for the best and efficient way to replace the first word in a str, like this: "aa to become" -> "/aa/ to become" I know I can use spilt and than join them but I can also use regular expressions and I sure there is a lot ways, but I need realy efficient one
3
3994
by: Irene | last post by:
Hi all, I have set up a simple VB program (and later on an ASP interface) to manage an Athletics database. I'm using Access 2000. To simplify, I have the Athlets, the Competitions and the Scores tables. When I want to list of the best scores/ranking, I just do:
5
1806
by: l.woods | last post by:
I want your recommendation on which ASP.NET Shopping Cart software I should buy? Best code Best documentation Best support (if needed. I will buying source code, if possible) TIA, Larry Woods
10
3437
by: jojobar | last post by:
Hello, I am trying to use vs.net 2005 to migrate a project originally in vs.net 2003. I started with creation of a "web site", and then created folders for each component of the site. I read somewhere that each folder under the "web site" is compiled in separate assembly. I however, did not find that the "web site" creation in vs.net 2005 created any AssemblyInfo.cs file.
10
2982
by: Ren | last post by:
Hi All, I'm still rather new at vb.net and would like to know the proper way to access private varibables in a class. Do I access the variable directly or do I use the public property? public class MyClass private _variableName as integer public property VariableName as integer
4
1834
by: Ned Balzer | last post by:
Hi all, I am pretty new to asp.net; I've done lots of classic asp, but am just beginning to get my mind wrapped around .net. What I'd like to do is include some code that tests if a user is logged in, on each and every page, and redirects the user to a login page if s/he's not logged in. The login page will also take care of some standard setup, such as choosing/populating a user profile. I used to use <!-- #include ... --for this,...
24
2186
by: Earl | last post by:
I have all of my data operations in a separate library, so I'm looking for what might be termed "best practices" on a return type from those classes. For example, let's say I send an update from the UI layer to a method in a library class that calls the stored procedure. Best to return a boolean indicating success/failure, return a string with the exception message, or just return the entire exception?
11
4682
by: ankitmathur | last post by:
Hi, I'm trying to overcome a situation whereby I have to search through 4-5 columns and produce the results with an order by according to the values matched in these columns. Example: My Table Structure: Create Table TestPfx
0
8440
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8352
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
8863
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8636
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7378
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4192
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4358
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2765
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2005
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.