472,780 Members | 1,288 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,780 software developers and data experts.

Memory Mapped IO and Sample code for win2000

Hi All,
I have some software that currently loads database tables into sorted arrays
in memory to be binary searched by processes. (Long story as to why it does

Each process must either read the data from the in memory array or from the
database itself if the table is too large to be loaded into memory. (2GB is
usually the limit because the array is indexed by a 32 bit signed integer.
I have never had one of these tables go over 2GB, even when the client had
20M customers.)

This current set up is fine for companies with a few million customers and a
few million transactions per day. The software works just fine on win2000
and solaris/aix. None of my existing customers has large enough lookup
tables or so many transactions that it is a problem to load them into

However, one of the major 'upgrades' I want to make to this code is to
enable it to share these in-memory arrays between many processes. This
sharing of memory will make the product pretty much 'endlessly scalable',
and this is where I want to get it to.

The idea is that if the customer is a large telephone company then I can
split the Call Detail Records into say 12 input files and have 12 processes
run at the same time. By sharing the tables to be searched in memory I can
cut the memory requirement by a factor of 12. By putting all the data to be
looked up into memory the speed of processing the CDRs can be maximised
because a lookup to a database is at least 20 times slower than the lookup
to the array even if the data is in memory.

I have the code I need to do this under solaris.....but from the examples I
can see in VS.NET it looks like windows 2000 acts a little differently.


Q1. Does anyone have a fragement of code for a producer/consumer for memory
mapped IO under win2000 that they are willing to share? Or is there a good
public example anywhere that someone here can point to? The examples I can
find with vs.net are in java or very old.

Q2. Is there a 2GB limit on creating a memory mapped file under win2000?

Q3. Will a memory mapped file like this run at the same speed as an in
memory array that is created by calloc like the current one? (Once the
producer has loaded the table into memory all other processes are
'read-only' and will be able to read the memory at the same time, no locks

Q4. Are there any other pointers the experts on this group might like to
pass along to someone going to do their very first Memory Mapped IO piece of
work? Are there any 'gotchas' for beginners in this area?

All assistance is much appreciated!!!

Best Regards

Peter Nolan
Jul 22 '05 #1
1 2375
Peter Nolan wrote:
Hi All,
I have some software that currently loads database tables into sorted
arrays in memory to be binary searched by processes. (Long story as to why
it does this..) ... deleted All assistance is much appreciated!!!

Best Regards

Peter Nolan

Hi, the ACE framework (http://www.cs.wustl.edu/~schmidt/ACE.html) provides
memory mapped files on all it's supported platforms (man unices, embedded
systems and win32)

Jul 22 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: Bryan Parkoff | last post by:
I work emulator project using 32 Bit emulated processor. The problem is that it attempts to allocate 4GB memory because some people want to use emulated instruction to read data from 4GB memory. ...
by: Justin Lazanowski | last post by:
Cross posting this question on the recommendation of an I have a .NET application that I am developing in C# I am loading information in from a dataset, and then pushing the dataset to a grid,...
by: Holger Hasselbach | last post by:
- The value of the object allocated by the malloc function is used ( - The value of any bytes in a new object allocated by the realloc function beyond the size of the old object are used...
by: vikas | last post by:
I have following structure in c++. typedef struct MMF_result_struct { int action; char text; int cols,rows; int month,day,year; } MMF_result; Now this structure is shared between C++ and C#...
by: Nirosh | last post by:
Hi All, Can any one suggest me a best way to do this .. I have a thrid party tool "EXE" that we need to use with our web service to manipulate some complex XML files, which reside in a...
by: Tomas | last post by:
I have two questions: (1) How (if possible) can you, with ASP.NET (and with the IIS 5 included with win2000) specify an maximum limit of the memory that a web application may consume, as an...
by: Maxwell2006 | last post by:
Hi, Is there a concept of shared named memory in .NET to allow multiple program use the name and then set/retrieve values? Thank you, Alan
by: Samshayam | last post by:
I have come across the application of placement new in memory mapped i/o in a number of books.I am not able to understand it completely, may be becaues of my lack of knowledge with memory mapped...
by: simonl | last post by:
Hi, I've been given the job of sorting out a crash in our product for which we have the crash information and an avi of the event (which can't possibly match but more of that later...) (btw this...
by: Rina0 | last post by:
Cybersecurity engineering is a specialized field that focuses on the design, development, and implementation of systems, processes, and technologies that protect against cyber threats and...
by: erikbower65 | last post by:
Using CodiumAI's pr-agent is simple and powerful. Follow these steps: 1. Install CodiumAI CLI: Ensure Node.js is installed, then run 'npm install -g codiumai' in the terminal. 2. Connect to...
by: erikbower65 | last post by:
Here's a concise step-by-step guide for manually installing IntelliJ IDEA: 1. Download: Visit the official JetBrains website and download the IntelliJ IDEA Community or Ultimate edition based on...
by: Taofi | last post by:
I try to insert a new record but the error message says the number of query names and destination fields are not the same This are my field names ID, Budgeted, Actual, Status and Differences ...
by: DJRhino1175 | last post by:
When I run this code I get an error, its Run-time error# 424 Object required...This is my first attempt at doing something like this. I test the entire code and it worked until I added this - If...
by: DJRhino | last post by:
Private Sub CboDrawingID_BeforeUpdate(Cancel As Integer) If = 310029923 Or 310030138 Or 310030152 Or 310030346 Or 310030348 Or _ 310030356 Or 310030359 Or 310030362 Or...
by: lllomh | last post by:
How does React native implement an English player?
by: Mushico | last post by:
How to calculate date of retirement from date of birth
by: DJRhino | last post by:
Was curious if anyone else was having this same issue or not.... I was just Up/Down graded to windows 11 and now my access combo boxes are not acting right. With win 10 I could start typing...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.