By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
448,712 Members | 1,540 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 448,712 IT Pros & Developers. It's quick & easy.

Memory Mapped IO and Sample code for win2000

P: n/a
Hi All,
I have some software that currently loads database tables into sorted arrays
in memory to be binary searched by processes. (Long story as to why it does
this..)

Each process must either read the data from the in memory array or from the
database itself if the table is too large to be loaded into memory. (2GB is
usually the limit because the array is indexed by a 32 bit signed integer.
I have never had one of these tables go over 2GB, even when the client had
20M customers.)

This current set up is fine for companies with a few million customers and a
few million transactions per day. The software works just fine on win2000
and solaris/aix. None of my existing customers has large enough lookup
tables or so many transactions that it is a problem to load them into
memory.

However, one of the major 'upgrades' I want to make to this code is to
enable it to share these in-memory arrays between many processes. This
sharing of memory will make the product pretty much 'endlessly scalable',
and this is where I want to get it to.

The idea is that if the customer is a large telephone company then I can
split the Call Detail Records into say 12 input files and have 12 processes
run at the same time. By sharing the tables to be searched in memory I can
cut the memory requirement by a factor of 12. By putting all the data to be
looked up into memory the speed of processing the CDRs can be maximised
because a lookup to a database is at least 20 times slower than the lookup
to the array even if the data is in memory.

I have the code I need to do this under solaris.....but from the examples I
can see in VS.NET it looks like windows 2000 acts a little differently.

Questions....

Q1. Does anyone have a fragement of code for a producer/consumer for memory
mapped IO under win2000 that they are willing to share? Or is there a good
public example anywhere that someone here can point to? The examples I can
find with vs.net are in java or very old.

Q2. Is there a 2GB limit on creating a memory mapped file under win2000?

Q3. Will a memory mapped file like this run at the same speed as an in
memory array that is created by calloc like the current one? (Once the
producer has loaded the table into memory all other processes are
'read-only' and will be able to read the memory at the same time, no locks
required.)

Q4. Are there any other pointers the experts on this group might like to
pass along to someone going to do their very first Memory Mapped IO piece of
work? Are there any 'gotchas' for beginners in this area?

All assistance is much appreciated!!!

Best Regards

Peter Nolan
www.peternolan.com
Jul 22 '05 #1
Share this Question
Share on Google+
1 Reply


P: n/a
Peter Nolan wrote:
Hi All,
I have some software that currently loads database tables into sorted
arrays in memory to be binary searched by processes. (Long story as to why
it does this..) ... deleted All assistance is much appreciated!!!

Best Regards

Peter Nolan
www.peternolan.com


Hi, the ACE framework (http://www.cs.wustl.edu/~schmidt/ACE.html) provides
memory mapped files on all it's supported platforms (man unices, embedded
systems and win32)

Lothar
Jul 22 '05 #2

This discussion thread is closed

Replies have been disabled for this discussion.