473,320 Members | 1,867 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

best memory allocation function for dealing with large (possiblyin the 100s of MBs) byte arrays?

I am working on some software which has to deal with data that could be as
large as 500mb or so. Currently I am using new[] and delete[] to manage
this memory but I find it is not ideal and sometimes gives out of memory
errors if I open one large data item then free that data item then another
large data item even though I have enough memory (including 2GB of physical
RAM and 100GB of free disk space for swap file). Are there any functions
(either in the CRT or in the win32 API) that would be better for this job
(specifically functions that are guaranteed to return the memory to the OS
and make task manager show that the memory is no longer in use as well as
having as little overhead as possible when using it)

The app in question runs only on windows XP and only has one thread
touching this memory at all.
Aug 18 '07 #1
1 1610
Hi Jonathan,

assuming that you are talking about a 32 bit process here, you only have
(on XP) 2 GB of virtual memory accessible in user mode. Even if you have
more physical RAM and plenty of paging file space, your process will not
be able to use more than 2GB of VM.

Your modules, stacks and heaps are sprinkled throughout these 2 GB of
VM, so you have to assume that the 2 GB are already quite a bit
fragmented. Given these constraints, it is pretty obvious that
allocating 500 MB of contiguous VM is very likely to fail.

So first you should ask yourself if you really need such large blocks of
contiguous virtual memory or if you can split them into chains of
multiple, much smaller sized blocks (of which maybe you do not even need
all in memory at once).

Finally, if you cannot avoid using large contiguous blocks of VM,
consider reserving a fair amount of VM early in the lifecycle of your
process and commit parts of it as you need them.

--Johannes

Jonathan Wilson wrote:
I am working on some software which has to deal with data that could be
as large as 500mb or so. Currently I am using new[] and delete[] to
manage this memory but I find it is not ideal and sometimes gives out of
memory errors if I open one large data item then free that data item
then another large data item even though I have enough memory (including
2GB of physical RAM and 100GB of free disk space for swap file). Are
there any functions (either in the CRT or in the win32 API) that would
be better for this job (specifically functions that are guaranteed to
return the memory to the OS and make task manager show that the memory
is no longer in use as well as having as little overhead as possible
when using it)

The app in question runs only on windows XP and only has one thread
touching this memory at all.

--
Johannes Passing - http://int3.de/
Aug 18 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Sourin | last post by:
Hi all, I am trying to write code for my experiments. My work involves huge datasets, and as such my code needs to be memory efficient. I did some hand calculations regarding the amount of...
5
by: Crimzon | last post by:
I am using MSVC++ 6.0 compiler. I am declaring an array char ch. Program works fine for these arbitrary sizes. But if I make the size of the array bigger like ch, the program gives me an error...
50
by: Joerg Schwerdtfeger | last post by:
Hi folks, how can I determine the total main-memory size and the size of free memory available in bytes? I tried to use mallinfo() from malloc.h - resulting some strange values in Windows...
5
by: swarsa | last post by:
Hi All, I realize this is not a Palm OS development forum, however, even though my question is about a Palm C program I'm writing, I believe the topics are relevant here. This is because I...
6
by: James | last post by:
I am using vb.net and need to keep in memory a large data structure, so I am looking for the best option. And after several test I am pretty confused. So I will be grateful if anyone can help me. ...
1
by: kiplring | last post by:
List<string> effectList = new List<string>(); effectList.Clear(); effectList = null; using (List<string> effectList = new List<string>()) { } If there are so many calls, I should save as...
1
by: trevor.farchild | last post by:
Hi, long time reader, first time poster I have an application that will be doing this 5 times a second: Get a bunch of data from a NetworkStream and convert it into a Bitmap Therefore, the...
53
by: fdmfdmfdm | last post by:
This is an interview question and I gave out my answer here, could you please check for me? Q. What are the memory allocation for static variable in a function, an automatic variable and global...
14
by: vivek | last post by:
i have some doubts on dynamic memory allocation and stacks and heaps where is the dynamic memory allocation used? in function calls there are some counters like "i" in the below function. Is...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.