473,699 Members | 2,008 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

C# DIctionary size limit

Hi Guys,

I have a class A composed of several string and int members.
I would like to manage a huge amount (several thousands) of A objects
in a dictionary where each object has its unique key.

Is there any known limitation or best practice that you think I should
consider?

Thanks,
Orsula.z
Aug 12 '08 #1
10 23769
On Aug 12, 4:53*pm, orsula <orsul...@gmail .comwrote:
I have a class A composed of several string and int members.
I would like to manage a huge amount (several thousands) of A objects
in a dictionary where each object has its unique key.

Is there any known limitation or best practice that you think I should
consider?
"Several thousand" is actually reasonably small. Basically if you've
got enough room in memory for the objects and a bit of overhead, the
dictionary should be fine.

Jon
Aug 12 '08 #2
Orsula,

Be aware that as with every system collection, you are only adding the
references of the objects, not the values itself, therefore a dictionary can
be in fact very small.

Cor

"orsula" <or******@gmail .comschreef in bericht
news:59******** *************** ***********@k13 g2000hse.google groups.com...
Hi Guys,

I have a class A composed of several string and int members.
I would like to manage a huge amount (several thousands) of A objects
in a dictionary where each object has its unique key.

Is there any known limitation or best practice that you think I should
consider?

Thanks,
Orsula.z
Aug 12 '08 #3
On Aug 12, 7:11*pm, "Jon Skeet [C# MVP]" <sk...@pobox.co mwrote:
On Aug 12, 4:53*pm, orsula <orsul...@gmail .comwrote:
I have a class A composed of several string and int members.
I would like to manage a huge amount (several thousands) of A objects
in a dictionary where each object has its unique key.
Is there any known limitation or best practice that you think I should
consider?

"Several thousand" is actually reasonably small. Basically if you've
got enough room in memory for the objects and a bit of overhead, the
dictionary should be fine.

Jon
Thanks (again :) ) Jon
Aug 12 '08 #4
orsula wrote:
I have a class A composed of several string and int members.
I would like to manage a huge amount (several thousands) of A objects
in a dictionary where each object has its unique key.

Is there any known limitation or best practice that you think I should
consider?
No.

My understanding is that would be able to stuff 2GB/16B = 125 million
of entries in a Dictionary<stri ng,Aon 32 bit Windows.

Arne
Aug 13 '08 #5
On Aug 13, 1:21*am, Arne Vajhøj <a...@vajhoej.d kwrote:
No.

My understanding is that would be able to stuff 2GB/16B = 125 million
of entries in a Dictionary<stri ng,Aon 32 bit Windows.
Not quite. For one thing I think there's an upper limit of about 1GB
per object (including the entries array) in the CLR. I've also found
that for some reason (using a Dictionary<int, intto avoid creating
any extra objects) the memory per entry averages at around 40 bytes
rather than 16. No idea why at the moment - 16 would make sense to me
as well, although there's bound to be *some* extra overhead for the
buckets etc. Possibly alignment issues? Hmm...

Jon
Aug 13 '08 #6
A little bit difficult, because then all pairs should be referencing to null
and that is not possible.

Cor

"Arne Vajhøj" <ar**@vajhoej.d kschreef in bericht
news:48******** *************** @news.sunsite.d k...
orsula wrote:
>I have a class A composed of several string and int members.
I would like to manage a huge amount (several thousands) of A objects
in a dictionary where each object has its unique key.

Is there any known limitation or best practice that you think I should
consider?

No.

My understanding is that would be able to stuff 2GB/16B = 125 million
of entries in a Dictionary<stri ng,Aon 32 bit Windows.

Arne

Aug 13 '08 #7
On Aug 13, 1:18*am, "Jon Skeet [C# MVP]" <sk...@pobox.co mwrote:
On Aug 13, 1:21*am, Arne Vajhøj <a...@vajhoej.d kwrote:
No.
My understanding is that would be able to stuff 2GB/16B = 125 million
of entries in a Dictionary<stri ng,Aon 32 bit Windows.

Not quite. For one thing I think there's an upper limit of about 1GB
per object (including the entries array) in the CLR. I've also found
that for some reason (using a Dictionary<int, intto avoid creating
any extra objects) the memory per entry averages at around 40 bytes
rather than 16. No idea why at the moment - 16 would make sense to me
as well, although there's bound to be *some* extra overhead for the
buckets etc. Possibly alignment issues? Hmm...

Jon
I just Reflector'd it. In addition to the key and the value the hash
code (int) and a next pointer (int) are also stored on each entry.
That would account for 16B. There is also an int[] array that is used
for indexing the entries array. That gets us to 20B. The resizing
algorithm uses the first prime number greater than twice the
hashtable's current size and uses that for the new size. That would
easily get you to 40.

Since we're discussing very large dictionaries I noticed that the
prime table is preloaded up to 7,199,369 (which is not to say that all
primes up to that value are included). Once it reaches that size the
next resize operation will start using the naive algorithm for
primality testing, but will *probably* yield 14,398,753, 28,797,523,
and 57,595,063. That gives us jumps to 288, 576, and 1152 MB
respectively. If the 1GB limit is indeed in play for arrays then we
might expect ~28 million to be the upper limit. Pure speculation.
Aug 13 '08 #8
On Aug 13, 4:25*pm, Brian Gideon <briangid...@ya hoo.comwrote:
I just Reflector'd it. *In addition to the key and the value the hash
code (int) and a next pointer (int) are also stored on each entry.
That would account for 16B. *There is also an int[] array that is used
for indexing the entries array. *That gets us to 20B.
For some reason I didn't think that the int[] was the same length as
the main array - but I see that it is.
*The resizing
algorithm uses the first prime number greater than twice the
hashtable's current size and uses that for the new size. *That would
easily get you to 40.
Ah, I've just worked out a flaw in my test - I was only outputing the
result when the amount of memory changed, which would always give the
worst case. Printing out the best case as well shows around 20 bytes,
which now makes sense.
Since we're discussing very large dictionaries I noticed that the
prime table is preloaded up to 7,199,369 (which is not to say that all
primes up to that value are included). *Once it reaches that size the
next resize operation will start using the naive algorithm for
primality testing, but will *probably* yield 14,398,753, 28,797,523,
and 57,595,063. That gives us jumps to 288, 576, and 1152 MB
respectively. *If the 1GB limit is indeed in play for arrays then we
might expect ~28 million to be the upper limit. *Pure speculation.
Don't forget that the 1GB limit applies to individual objects AFAIK -
so only the 16 bytes per entry of the "main" array. The int[] is
separate. 57,595,063*16 = 921 521 008, which should be okay. However,
you've also got to have enough room for the old version as well, so
your whole process needs 1.5x as much memory, unless you preallocate
that capacity.

Unfortunately I don't know the exact details of all the limits, so I
won't analyse this any further, for fear of giving the wrong
information.

Jon
Aug 13 '08 #9
On Aug 13, 11:22*am, "Jon Skeet [C# MVP]" <sk...@pobox.co mwrote:
Don't forget that the 1GB limit applies to individual objects AFAIK -
so only the 16 bytes per entry of the "main" array. The int[] is
separate. 57,595,063*16 = 921 521 008, which should be okay. However,
you've also got to have enough room for the old version as well, so
your whole process needs 1.5x as much memory, unless you preallocate
that capacity.
Ah yes. Good points. Also in the sake of full disclosure, that
57595063 value was based on a quick statistical test for primality so
it may not even be prime for all I know. I'm sure there is a prime
somewhere in that ballpark so it's probably not going to skew the
numbers too much.
>
Unfortunately I don't know the exact details of all the limits, so I
won't analyse this any further, for fear of giving the wrong
information.
Understood.

Aug 13 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
2482
by: Benjamin Scott | last post by:
Hello. I attempted to build a compound dictionary: len(Lst)=1000 len(nuerLst)=250 len(nuestLst)=500 Dict={}
8
5585
by: Rodd Snook | last post by:
I have an application which makes extensive use of the Scripting.Dictionary object. I'm not doing anything silly like putting them outside the page scope -- just creating quite a few of them and stuffing quite a lot of data (from and MS SQL database) into them. On Windows 2000 server, everything is fine. If the data structures get really big it slows down, but for normal operation it's no problem. Recently our hosting provider moved to...
8
1702
by: Samuel | last post by:
I have a user complaining about a random error (sporadic, cannot be reliably reproduced): ================================= Microsoft VBScript runtime error '800a01fb' An exception occurred: 'objDict.add' =================================
5
2745
by: Fox | last post by:
Hi, I am working on a project which used dictionaries. I am having to remake part of this and have no experience with the scripting dictionary. I need to see how to create multiple dictionaries within one dictionary.
2
1670
by: John | last post by:
I am trying to design a dictionary in such a way that I can keep the interface and implementation separate. Here is my initial design. Anyone has any comments on it or suggestions on how I could make it better. Does anyone have a good code/pointer that discusses dictionary design. Thanks,
2
28590
by: Kums | last post by:
What is the maximum permissible size of a database? Is there any limitation. What is the maximum # of tablespace's allowed in a database? Thanks for your response.
5
47221
by: Andrew Robinson | last post by:
I need to serialize a dictionary so I can encode the contents. I have the following working but the size seems large. I am guessing that I am serializing the entire object not just the data. Is there a better way? MemoryStream stream = new MemoryStream(); BinaryFormatter formatter = new BinaryFormatter(); Dictionary<string, string> dictionary = new Dictionary<string, string>(); dictionary.Add("name", "andrew"); dictionary.Add("home",...
8
13335
by: Brian L. Troutwine | last post by:
I've got a problem that I can't seem to get my head around and hoped somebody might help me out a bit: I've got a dictionary, A, that is arbitarily large and may contains ints, None and more dictionaries which themselves may contain ints, None and more dictionaries. Each of the sub-dictionaries is also arbitrarily large. When pretty printing A, in the context I'm using A for, it's rather helpful to remove all key:value pairs where value...
11
11405
by: John | last post by:
I am coding a radix sort in python and I think that Python's dictionary may be a choice for bucket. The only problem is that dictionary is a mapping without order. But I just found that if the keys are numeric, the keys themselves are ordered in the dictionary. part of my code is like this: radix={} for i in range(256):
0
9184
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
1
8929
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8891
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7759
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
5878
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4380
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
3061
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2357
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2013
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.