473,387 Members | 1,574 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,387 software developers and data experts.

Re: Optimizing size of very large dictionaries

On Wed, Jul 30, 2008 at 8:29 PM, <py****@bdurham.comwrote:
Background: I'm trying to identify duplicate records in very large text
based transaction logs. I'm detecting duplicate records by creating a SHA1
checksum of each record and using this checksum as a dictionary key. This
works great except for several files whose size is such that their
associated checksum dictionaries are too big for my workstation's 2G of RAM.
What are the values of this dictionary?

You can save memory by representing the checksums as long integers, if
you're currently using strings.

-Miles
Jul 31 '08 #1
0 693

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: robin | last post by:
I need to do a search through about 50 million records, each of which are less than 100 bytes wide. A database is actually too slow for this, so I thought of optimising the data and putting it all...
12
by: Anon | last post by:
Hello all, I'm hoping for some guidance here... I am a c/c++ "expert", but a complete python virgin. I'm trying to create a program that loads in the entire FreeDB database (excluding the...
1
by: DJTB | last post by:
zodb-dev@zope.org] Hi, I'm having problems storing large amounts of objects in a ZODB. After committing changes to the database, elements are not cleared from memory. Since the number of...
0
by: Aidan | last post by:
The setup: a Sax parser in a servlet and a Java client (same machine) which uploads an XML document containing CDATA elements which hold base 64 encoded binary files. The servlet then SAX parses...
21
by: James Moe | last post by:
Hello, I just joined this group and saw the discussion "What do you think of resizing 1em to 10px?" I am somewhat confused by what y'all think should be used as a reference size. I was surprised...
4
by: Flashman | last post by:
A little confusing with setting up optimizing options with 2003 .NET. Under the Optimization Tab. if you set to /O1 or /O2 is the program ignoring the settings for Inline Function expansion,...
5
by: Claudio Grondi | last post by:
I have just started to play around with the bsddb3 module interfacing the Berkeley Database. Beside the intended database file databaseFile.bdb I see in same directory also the __db.001...
8
by: placid | last post by:
Hi all, Just wondering if anyone knows how to pop up the dialog that windows pops up when copying/moving/deleting files from one directory to another, in python ? Cheers
2
by: Gabriel Genellina | last post by:
En Wed, 30 Jul 2008 21:29:39 -0300, <python@bdurham.comescribi�: You could use a different hash algorithm yielding a smaller value (crc32, by example, fits on an integer). At the expense of...
10
by: orsula | last post by:
Hi Guys, I have a class A composed of several string and int members. I would like to manage a huge amount (several thousands) of A objects in a dictionary where each object has its unique key....
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.