473,231 Members | 2,041 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,231 software developers and data experts.

hash_map in stl c++ memory issues

2
Hi!

I have written code using STL hash_map in c++ and am running into memory issues. I have two kinds of hash_maps:
A <int,int> which stores ~5million key-value pairs, and needs to be accessed often to store in memory, its not updated during the program.
B <int,hash_map<int,float>> which is also ~5m key-value pairs, but I can read this in parts from the disk. The inner hash_map has about 20 elements.

In my program, I read in A and B1(first 100K key-values), I perform the required computation, and say B1.clear(), and read the next set of 100k elements.

1) The program exits with an std::bad_alloc error. How can I fix this?

2) What hash_fns would be good for this large data?

3) It seems that clear does not free the memory. What can I do? When I say B1.clear(), will it free the inner hash_map also?

4) How can I find out what is using up the memory?

I am a novice to stl programing, any help would be appreciated.

Thanks!
Mar 18 '08 #1
1 4425
weaknessforcats
9,208 Expert Mod 8TB
1) The program exits with an std::bad_alloc error. How can I fix this?

2) What hash_fns would be good for this large data?

3) It seems that clear does not free the memory. What can I do? When I say B1.clear(), will it free the inner hash_map also?

4) How can I find out what is using up the memory?
1) You have run out of heap memory. Check your system and see what the heap limits are. That is, your program is in a process address space and you have to stay within that range of addresses. Becuae you have a 2GB machine does not mean you cna have a 2GB heap.

2) I have no idea. You need to research here. Check out The Art of Computer Programming by Donald Knuth Volume 2 - Sorting and Searching.

3) None of the STL containers gives up memory once it it acquired. This is by design. Allocating memory is slow operation and the STL is optimized for speed so once your container gloms onto memory, it keeps it. The B1.clear() just erases all of the elements in the container but does not reduce the memory size of the container. You can force the container to give up this memory by swapping it with a container that was created empty.

4) See (3) above.
Mar 18 '08 #2

Sign in to post your reply or Sign up for a free account.

Similar topics

5
by: Sabrina | last post by:
Can someone help? I have been trying to get the hash_map in C++ for .NET to work with strings and const char*. I am using the const char* as the key and a pointer to another class as the data...
5
by: Charles Herman | last post by:
I have the folowing program: #include <iostream> #include <ext/hash_map> using namespace std; using namespace __gnu_cxx; typedef hash_map<char*, long, hash<char*> > char_hash_t;
2
by: gibffe | last post by:
Hello I have a big problem with hash_map in stl. I'm trying to create hash_map<const char *, char *> map to keep key => val pairs as 4 byte C "strings". Everything is ok when i do let's say...
4
by: Dymus | last post by:
problem with deleting big sized hash_map. have defined a comporator, where defined min_bucket_size (for speed improvement, since i know that i need to store big ammount of data (~1Mln)),...
3
by: kony | last post by:
Hi there, I would much appreciate your help with the following problem. Below is the code that uses a hash_map. I want to release all the memory occupied by the hash_map for other use. Apparently...
4
by: lokki | last post by:
Hello, can anybody tell me what's wrong with following example code? char *k, *v; k = new char; strcpy(k, "a2"); v = new char;
4
by: newbie | last post by:
For STL containers like hash_map, need I explicitly call clear() before delete()ing it? e.g. hash_map<string, MyClass**my_map; my_map = new (hash_map<string, MyClass*>); .... ....
6
by: SimpleCode | last post by:
class COsgCar; class moving_vechicle; hash_map<moving_vechicle*, COsgCar*m_hash; hash_map<moving_vechicle*, COsgCar*>::iterator iter; iter = m_hash.find( pMovingVechicle ); if ( iter !=...
4
by: James Kanze | last post by:
On Jul 16, 10:53 pm, Mirco Wahab <wa...@chemie.uni-halle.dewrote: It depends. You might like to have a look at my "Hashing.hh" header (in the code at kanze.james.neuf.fr/code-en.html---the...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.