473,288 Members | 1,705 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,288 software developers and data experts.

How lock memory so not paged? vs Unix mlock.

Ive got a time-critical application that runs on Windows XP and Unix. I've
got some timing issues that Ive traced to virtual memory getting paged in and
out. I think I can solve the problem if I can "lock" a certain huge memory
buffer into physical memory, so it is not paged out and in. The rest of my
program (code and other data) can get paged in and out: it is just this one
buffer that is critical.

Unix (some Unixs, at least) have an mlock() function that is working well
for me. Now Id like to do the same thing on Windows.

Question: Is there a technique in Windows XP (Im using Visual C++) to lock
a memory buffer (that my program allocated with malloc() ) into physical
memory so it is never paged out?

Thanks in advance for any replies,
neal

Jan 28 '06 #1
11 9294
I've done some hunting around and found the function
VirtualLock()
that may meet my requriements in Windows.

If anyone has any advice or tips about VirtualLock(), Id appreciate it.

Thanks,
neal

"noleander" wrote:
Ive got a time-critical application that runs on Windows XP and Unix. I've
got some timing issues that Ive traced to virtual memory getting paged in and
out. I think I can solve the problem if I can "lock" a certain huge memory
buffer into physical memory, so it is not paged out and in. The rest of my
program (code and other data) can get paged in and out: it is just this one
buffer that is critical.

Unix (some Unixs, at least) have an mlock() function that is working well
for me. Now Id like to do the same thing on Windows.

Question: Is there a technique in Windows XP (Im using Visual C++) to lock
a memory buffer (that my program allocated with malloc() ) into physical
memory so it is never paged out?

Thanks in advance for any replies,
neal

Jan 28 '06 #2
Hi noleander!
Question: Is there a technique in Windows XP (Im using Visual C++) to lock
a memory buffer (that my program allocated with malloc() ) into physical
memory so it is never paged out?


See: VirtualLock
http://msdn.microsoft.com/library/en...irtuallock.asp

It is not recommended to do this, but for special applications / PC it
might make sence...

--
Greetings
Jochen

My blog about Win32 and .NET
http://blog.kalmbachnet.de/
Jan 28 '06 #3
"noleander" <no*******@discussions.microsoft.com> wrote in message
news:72**********************************@microsof t.com...
I've done some hunting around and found the function
VirtualLock()
that may meet my requriements in Windows.

If anyone has any advice or tips about VirtualLock(), Id appreciate it.


I wouldn't know how mlock works so I don't know if it will do what you want
exactly.

But, to be clear the function locks some number of pages into the "working
set" of the calling process. (The working set is the collection of pages
that are backed by physical memory). Under memory pressure Windows is free
to swap out your process to make room for something more pressing, locked
pages included.

Regards,
Will
Jan 28 '06 #4
Hi William!

VirtualLock()


But, to be clear the function locks some number of pages into the "working
set" of the calling process.


AFAIK: Virtual Lock really locks the pages into *physical* memory!
The following will fail on the second call to "SetProcessWorkingSet",
(trimming to 0 is not allowed, because some pages are locked into
virtual memory).

char *data = new char[1024*1024]; // 1 MB
BOOL bRet = SetProcessWorkingSetSize(GetCurrentProcess(),
1024*1024*2, 1024*1024*10);
bRet = VirtualLock(data, 1024*1024); // LOCK into memory!
bRet = SetProcessWorkingSetSize(GetCurrentProcess(), 0, 0); // WILL
FAIL!
while(1) Sleep(1000);

You also can see that the working-set is never under 1 MB...

Of course, this will require "PROCESS_SET_QUOTA" permissions...

--
Greetings
Jochen

My blog about Win32 and .NET
http://blog.kalmbachnet.de/
Jan 29 '06 #5
Jochen,
Locking memory pages doesn't prevent paging-out, it only asks the memory
manager to keep these pages in memory as long as possible. Run following
code, and watch the WS getting trimmed to ~40-200Kb (your milleage may vary
depending on OS version and type of application) despite the allocation of
2MB of locked memory.
Another point is that SetProcessWorkingSetSize (user API) and WS trimming by
the memory balance manager are two different things, the balance set
manager can remove a complete process from memory irrespective the number of
locked pages.
int BytesToLock = 2*1024*1024;
char *data = new char[BytesToLock]; // 2 MB
bool bRet = SetProcessWorkingSetSize(GetCurrentProcess(), 1024*1024*2,
1024*1024*10);
bRet = VirtualLock(data,BytesToLock); // LOCK into memory!
bRet = SetProcessWorkingSetSize(GetCurrentProcess(), -1, -1); //
Succeeds
while(1) Sleep(1000);
Willy.

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:ul*************@TK2MSFTNGP12.phx.gbl...
| Hi William!
|
|
| >> VirtualLock()
| >
| > But, to be clear the function locks some number of pages into the
"working
| > set" of the calling process.
|
| AFAIK: Virtual Lock really locks the pages into *physical* memory!
| The following will fail on the second call to "SetProcessWorkingSet",
| (trimming to 0 is not allowed, because some pages are locked into
| virtual memory).
|
| char *data = new char[1024*1024]; // 1 MB
| BOOL bRet = SetProcessWorkingSetSize(GetCurrentProcess(),
| 1024*1024*2, 1024*1024*10);
| bRet = VirtualLock(data, 1024*1024); // LOCK into memory!
| bRet = SetProcessWorkingSetSize(GetCurrentProcess(), 0, 0); // WILL
| FAIL!
| while(1) Sleep(1000);
|
| You also can see that the working-set is never under 1 MB...
|
| Of course, this will require "PROCESS_SET_QUOTA" permissions...
|
| --
| Greetings
| Jochen
|
| My blog about Win32 and .NET
| http://blog.kalmbachnet.de/
Jan 29 '06 #6
Hi Willy!
Locking memory pages doesn't prevent paging-out,

I can´t verify your statement on my machine...
Even with the following code, the working set is always > 2MB!

#include <windows.h>
#include <tchar.h>

int _tmain()
{
char *data = new char[1024*1024*2]; // 2 MB
BOOL bRet = SetProcessWorkingSetSize(GetCurrentProcess(),
1024*1024*4, 1024*1024*10);
bRet = VirtualLock(data, 1024*1024*2);
while(1)
{
// swap-out the whole process:
bRet = SetProcessWorkingSetSize(GetCurrentProcess(), -1, -1);
Sleep(1000);
}
}
So I can´t verify that the working set gets trimmed < 2MB!
And that is also what the documentation of "VirtualLock" is saying...
--
Greetings
Jochen

My blog about Win32 and .NET
http://blog.kalmbachnet.de/
Jan 29 '06 #7
Jochen,

Are you sure you aren't looking at the "private bytes" instead of the
"working set"?
Also, you should not include the call to SetProcessWorkingSetSize in the
loop, this will increase the WS because you keep the process in the "active"
list, so the balance set manager keeps more pages in the WS (at least the
pages for Sleep, SetProcessWorkingSetSize and their callee's).

If I run this from the commandline (XP SP2), the WS drops to 132Kb and when
I minimize the cmd window it drops further to 48Kb. (On W2K3 SP1 the values
are resp. 115Kb and 32Kb).

int BytesToLock = 2*1024*1024;
char *data = new char[BytesToLock]; // 1 MB
bool bRet = SetProcessWorkingSetSize(GetCurrentProcess(), 1024*1024*2,
1024*1024*10);

bRet = VirtualLock(data,BytesToLock); // LOCK into memory!
bRet = SetProcessWorkingSetSize(GetCurrentProcess(),-1, -1); // WILL
FAIL!
Sleep(60000); // sleep 1'
return 0;

As far as I know, it's impossible to lock more pages than the "minimum
working set", that means that when you call SetProcessWorkingSetSize (after
VirtualLock) to reduce the WS to a value lower than the locked page value,
the WS will drop and a number of pages will unlock. This can be watched by
using the vadump utility.

Willy.
"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:e0**************@TK2MSFTNGP14.phx.gbl...
| Hi Willy!
|
| > Locking memory pages doesn't prevent paging-out,
|
|
| I can´t verify your statement on my machine...
| Even with the following code, the working set is always > 2MB!
|
| #include <windows.h>
| #include <tchar.h>
|
| int _tmain()
| {
| char *data = new char[1024*1024*2]; // 2 MB
| BOOL bRet = SetProcessWorkingSetSize(GetCurrentProcess(),
| 1024*1024*4, 1024*1024*10);
| bRet = VirtualLock(data, 1024*1024*2);
| while(1)
| {
| // swap-out the whole process:
| bRet = SetProcessWorkingSetSize(GetCurrentProcess(), -1, -1);
| Sleep(1000);
| }
| }
|
|
| So I can´t verify that the working set gets trimmed < 2MB!
|
|
| And that is also what the documentation of "VirtualLock" is saying...
|
|
| --
| Greetings
| Jochen
|
| My blog about Win32 and .NET
| http://blog.kalmbachnet.de/
Jan 29 '06 #8
Hi Willy!
Are you sure you aren't looking at the "private bytes" instead of the
"working set"?
Yes....
Also, you should not include the call to SetProcessWorkingSetSize in the
loop,
It just wanted to be sure that the WS will be trimmed to a very small
amount...
But I removed it from the loop and put the sleep to 100000000... still
same effect (XP-SP2). The working-Set is still min. 2MB...

If I run this from the commandline (XP SP2), the WS drops to 132Kb and when
I minimize the cmd window it drops further to 48Kb.
After starting the WS has 2160 KB and the Private bytes has 2276 KB.
After minimizing it hase 2084 KB and 2276 KB...

The observed behaviour is exactly the documented behaviour...


bool bRet = SetProcessWorkingSetSize(GetCurrentProcess(), 1024*1024*2,
1024*1024*10);
bRet = VirtualLock(data,BytesToLock); // LOCK into memory!
Are you sure that the adjustment of the min. working set and the
virtual-lock succeeds?

As far as I know, it's impossible to lock more pages than the "minimum
working set",
Yes. Thats the reason why we adjust the min. working set before we call
VirtualLock...
that means that when you call SetProcessWorkingSetSize (after
VirtualLock) to reduce the WS to a value lower than the locked page value,
the WS will drop and a number of pages will unlock. This can be watched by
using the vadump utility.


Here is (a small excerpt) of the output from vadump -o

Catagory Total Private Shareable
Pages KBytes KBytes KBytes
Page Table Pages 4 16 16 0
Other System 4 16 16 0
Code/StaticData 0 0 0 0
Heap 0 0 0 0
Stack 0 0 0 0
Teb 0 0 0 0
Mapped Data 0 0 0 0
Other Data 513 2052 2052 0

Total Modules 0 0 0 0
Total Dynamic Data 513 2052 2052 0
Total System 8 32 32 0
Grand Total Working Set 521 2084 2084 0
--
Greetings
Jochen

My blog about Win32 and .NET
http://blog.kalmbachnet.de/
Jan 29 '06 #9
Jochen,

Correct, my bad, the VirtualLock failed, the minimum WS must be at least the
locked page size + the minimum WS (I had to set this value to 140Kb + size
of locked memory).
In the posted sample both were equal so the call failed with error 1453. I
had the error check in place when I first tried your code snip but
uncommented it it when I changed the code, something you should never do,
lesson learned ;-)

Willy.

"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:%2****************@tk2msftngp13.phx.gbl...
| Hi Willy!
| > Are you sure you aren't looking at the "private bytes" instead of the
| > "working set"?
|
| Yes....
|
| > Also, you should not include the call to SetProcessWorkingSetSize in the
| > loop,
|
| It just wanted to be sure that the WS will be trimmed to a very small
| amount...
| But I removed it from the loop and put the sleep to 100000000... still
| same effect (XP-SP2). The working-Set is still min. 2MB...
|
|
| > If I run this from the commandline (XP SP2), the WS drops to 132Kb and
when
| > I minimize the cmd window it drops further to 48Kb.
|
| After starting the WS has 2160 KB and the Private bytes has 2276 KB.
| After minimizing it hase 2084 KB and 2276 KB...
|
| The observed behaviour is exactly the documented behaviour...
|
|
|
|
|
| > bool bRet = SetProcessWorkingSetSize(GetCurrentProcess(),
1024*1024*2,
| > 1024*1024*10);
| > bRet = VirtualLock(data,BytesToLock); // LOCK into memory!
|
| Are you sure that the adjustment of the min. working set and the
| virtual-lock succeeds?
|
|
| > As far as I know, it's impossible to lock more pages than the "minimum
| > working set",
|
| Yes. Thats the reason why we adjust the min. working set before we call
| VirtualLock...
|
|
|
| > that means that when you call SetProcessWorkingSetSize (after
| > VirtualLock) to reduce the WS to a value lower than the locked page
value,
| > the WS will drop and a number of pages will unlock. This can be watched
by
| > using the vadump utility.
|
| Here is (a small excerpt) of the output from vadump -o
|
| Catagory Total Private Shareable
| Pages KBytes KBytes KBytes
| Page Table Pages 4 16 16 0
| Other System 4 16 16 0
| Code/StaticData 0 0 0 0
| Heap 0 0 0 0
| Stack 0 0 0 0
| Teb 0 0 0 0
| Mapped Data 0 0 0 0
| Other Data 513 2052 2052 0
|
| Total Modules 0 0 0 0
| Total Dynamic Data 513 2052 2052 0
| Total System 8 32 32 0
| Grand Total Working Set 521 2084 2084 0
|
|
| --
| Greetings
| Jochen
|
| My blog about Win32 and .NET
| http://blog.kalmbachnet.de/
Jan 29 '06 #10
"Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in message
news:ul*************@TK2MSFTNGP12.phx.gbl...
But, to be clear the function locks some number of pages into the
"working set" of the calling process.


AFAIK: Virtual Lock really locks the pages into *physical* memory!
The following will fail on the second call to "SetProcessWorkingSet",
(trimming to 0 is not allowed, because some pages are locked into virtual
memory).


It could be, but the docs have always said otherwise.

At this link

http://msdn.microsoft.com/library/de...sdn_virtmm.asp

you'll find this

<quote>
Processes in Windows NT are granted subtle influence into this system
behavior with the VirtualLock and VirtualUnlock functions. Essentially, a
process can establish specific pages to lock into its working set. However,
this does not give the process free reign over its working set. It cannot
affect the number of pages that make up its working set (the system adjusts
the working set for each process routinely), and it cannot control when the
working set is in memory and when it is not.
</quote>

What you demonstrate is that the working set of a process can not be trimmed
down past the limit of what it has locked. I think that docs are pointing
out that the operating system may choose to page out an _entire_
application, locked pages and all, under severe memory pressure.

Now that article goes back to the early days of NT, and the real behavior
may differ. I don't know but it goes against the prevailing wisdom in the
kernel group.

Further, I'd never suggest something at odds with the docs unless I
scrutinized the source and confirmed the anomalous behavior or heard from
someone whom I trust who did.

Regards,
Will

Jan 30 '06 #11

"William DePalo [MVP VC++]" <wi***********@mvps.org> wrote in message
news:u8*************@TK2MSFTNGP12.phx.gbl...
| "Jochen Kalmbach [MVP]" <no********************@holzma.de> wrote in
message
| news:ul*************@TK2MSFTNGP12.phx.gbl...
| >> But, to be clear the function locks some number of pages into the
| >> "working set" of the calling process.
| >
| > AFAIK: Virtual Lock really locks the pages into *physical* memory!
| > The following will fail on the second call to "SetProcessWorkingSet",
| > (trimming to 0 is not allowed, because some pages are locked into
virtual
| > memory).
|
| It could be, but the docs have always said otherwise.
|
| At this link
|
|
http://msdn.microsoft.com/library/de...sdn_virtmm.asp
|
| you'll find this
|
| <quote>
| Processes in Windows NT are granted subtle influence into this system
| behavior with the VirtualLock and VirtualUnlock functions. Essentially, a
| process can establish specific pages to lock into its working set.
However,
| this does not give the process free reign over its working set. It cannot
| affect the number of pages that make up its working set (the system
adjusts
| the working set for each process routinely), and it cannot control when
the
| working set is in memory and when it is not.
| </quote>
|
| What you demonstrate is that the working set of a process can not be
trimmed
| down past the limit of what it has locked. I think that docs are pointing
| out that the operating system may choose to page out an _entire_
| application, locked pages and all, under severe memory pressure.
|
| Now that article goes back to the early days of NT, and the real behavior
| may differ. I don't know but it goes against the prevailing wisdom in the
| kernel group.
|
| Further, I'd never suggest something at odds with the docs unless I
| scrutinized the source and confirmed the anomalous behavior or heard from
| someone whom I trust who did.
|
| Regards,
| Will
|

This docs are still valid, as I said in another reply, there is a
significant difference between a user process initiated WS trimming action
(through SetProcessWorkingSet) and the OS "balance set manager" initiated WS
trimming action. The BSM is the master of the game here and he's allowed to
trim all pages from a process's WS, even the pages that are locked by
VirtualLock. The only pages that never gets removed from memory are those
allocated the non-paged pool, but these do not apply to user processes at
all.

Willy.

Jan 30 '06 #12

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Riadh Elloumi | last post by:
Hi, I have some problems when debugging memory allocation (for both malloc and new) in a program that uses standard lists and vectors. When I add an element to the vector, the overloaded...
5
by: countzero | last post by:
Why not have the "new" operator return a typed handle when you get memory on the heap (if you want it to)? And then use a "lock" operator to get a pointer to the memory, and the pointer is only...
6
by: Nick | last post by:
excuse me!! may i ask a simple problem here? if i dynamically allocate a memory(ex. new in C++ or malloc in C) in a sub-function and forget free the space end of the sub-function. does it...
8
by: vikram | last post by:
i have series of questions 1.How a c program is loaded in memory i mean the whats is the structure that the code segment?? data segment?? 2.When you say const int *p; where is p...
7
by: trialproduct2004 | last post by:
Hi all I am having application in c# where i am loading one table of database into dataset. My table is of large size. so whenever i am loading that into dataset my memory size is getting...
5
by: cikkamikka | last post by:
Hi friends, Sorry for such basic question. but I wanted to know where does new operator or malloc operator allocate memory? in actualy physical Main memory or virtual memory?
2
by: OgaW | last post by:
Hi, What is the difference bet. Physical memory usage and Kernal memory usage? Physical memory: available vs system cache? kernal memory: paged vs nonpaged? Anyone can explain in details...
1
by: chinmaya | last post by:
Please help me,is there any command in unix for crating a file with specified amount of memory like 10 kb etc thanx
5
by: pp | last post by:
hi, on window we can GlobalMemoryStatusEx() to get the memory status on the system do we have any equivalent function on Unix/linux? can we use BOOST library to memory information on the...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.