Unable to VirtualAllocEx more than 32664 items.

[Bob Karaban]

We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|P ROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT ,PAGE_READWRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory[i],0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[red floyd]

Bob Karaban wrote:

We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h> nonstandard header #include <iostream.h> nonstandard header, use <iostream>

My compiler has no clue what VirtualAllocEx is. You might try asking in
a VC newsgroup.

[Dave Townsend]

Just a guess, but VirtualAllocEx allocates in complete pages, so even though
you ask for 1k, you are getting a whole page worth, 8k ? That adds up to
256Meg,
do you have that much memory on your system available ?

dave
"Bob Karaban" <bN************@newNOSPAMworldsystems.com> wrote in message
news:10***************@nnrp1.phx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody has a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|P ROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT ,PAGE_READWRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory[i],0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Bob Karaban]

Actually, we do have the memory available - originally, we were thinking the
same thing. What we tried was changing how much memory we allocated - it
doesn't matter if we allocate in 1k blocks or 100k blocks, we can only do it
32,664 times! So, the amount of total memory we use may vary wildly (we
never actually run out) depending on the block size, but we never complete
request #32,665! Very strange! Any other guesses, please keep them coming!

Bob

"Dave Townsend" <da********@comcast.net> wrote in message
news:Ac********************@comcast.com...

Just a guess, but VirtualAllocEx allocates in complete pages, so even though you ask for 1k, you are getting a whole page worth, 8k ? That adds up to
256Meg,
do you have that much memory on your system available ?

dave
"Bob Karaban" <bN************@newNOSPAMworldsystems.com> wrote in message
news:10***************@nnrp1.phx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody

has

a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item.

Below

is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|P ROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT ,PAGE_READWRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory[i],0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)? Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Jochen Kalmbach]

Bob Karaban wrote:

We ran into a problem using VirtualAllocEx and were wondering if
anybody has a way around this. We have an executable that stores a
hash table in a remote process. The VirtualAllocEx function fails on
the 32665 item. Below is a sample project to show it.

Just a guess: Maybe your LDT is full !?

--
Greetings
Jochen

Do you need a memory-leak finder ?
http://www.codeproject.com/tools/leakfinder.asp

Do you need daily reports from your server?
http://sourceforge.net/projects/srvreport/

[Ivan Brugiolo [MSFT]]

VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x10000
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************@newNOSPAMworldsystems.com> wrote in message
news:10***************@nnrp1.phx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody has a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|P ROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT ,PAGE_READWRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory[i],0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Bob Karaban]

O.K., we think we understand - how would we reduce this granularity? If we
reduce the granularity, we should then be able to get proportionally more
VirtualAllocEx calls, correct?

Thanks for the help!
Bob

"Ivan Brugiolo [MSFT]" <iv******@online.microsoft.com> wrote in message
news:OF**************@TK2MSFTNGP11.phx.gbl...

VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x10000
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************@newNOSPAMworldsystems.com> wrote in message
news:10***************@nnrp1.phx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody

has

a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item.

Below

is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|P ROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT ,PAGE_READWRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory[i],0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)? Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Dave Townsend]

Yeah, I this makes sense, I just looked at the .NET documentation on
virtualallocex, it says the desired
address is rounded to the nearest 64k boundary, so it sounds like you've
blown
the addressability range of windows32.

Since you only need 1k for each entry, can you do some clever stuff to split
up
the 64k page amongst 64 separate entries perhaps ?

dave
"Ivan Brugiolo [MSFT]" <iv******@online.microsoft.com> wrote in message
news:OF**************@TK2MSFTNGP11.phx.gbl...

VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x10000
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************@newNOSPAMworldsystems.com> wrote in message
news:10***************@nnrp1.phx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody

has

a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item.

Below

is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|P ROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT ,PAGE_READWRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory[i],0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)? Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Ivan Brugiolo [MSFT]]

The granularity is pretty much hard-coded in the memory manager
per each OS and platform and architecture.
You should consider virtualallocating a 64K chunks in the reserved state,
and then commit the individual pages on demand.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************@newNOSPAMworldsystems.com> wrote in message
news:10***************@nnrp1.phx1.gblx.net...

O.K., we think we understand - how would we reduce this granularity? If we reduce the granularity, we should then be able to get proportionally more
VirtualAllocEx calls, correct?

Thanks for the help!
Bob

"Ivan Brugiolo [MSFT]" <iv******@online.microsoft.com> wrote in message
news:OF**************@TK2MSFTNGP11.phx.gbl...

VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x10000
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of

your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no

rights.

Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************@newNOSPAMworldsystems.com> wrote in message news:10***************@nnrp1.phx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody

has

a way around this. We have an executable that stores a hash table in

a remote process. The VirtualAllocEx function fails on the 32665 item.

Below

is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of

the above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|P ROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT ,PAGE_READWRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory[i],0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)? Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or directly via email - thanks in advance!

Bob

[Severian]

On Fri, 4 Jun 2004 15:39:43 -0400, "Bob Karaban"
<bN************@newNOSPAMworldsystems.com> wrote:

We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ| PROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMI T,PAGE_READWRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory[i],0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

If you're writing using C, use malloc() & free. If you insist on Win32
API, try HeapAlloc & friends.

IIRC, both only call VirtualAlloc(Ex) when necessary and subdivide the
large blocks.

--
Sev

[Alf P. Steinbach]

Why are you crossposting this drivel to comp.lang.c++?

* Bob Karaban:

We ran into a problem using VirtualAllocEx and were wondering if anybody has

--
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

[Severian]

On Sat, 05 Jun 2004 04:46:26 GMT, al***@start.no (Alf P. Steinbach)
wrote:

Why are you crossposting this drivel to comp.lang.c++?
I at least apologize abjectly. With only a cursory glance prior to my
earlier response, I thought they were all MS shite. I'm sorry for my
contribution to the pollution in comp.lang.c++!
* Bob Karaban:

We ran into a problem using VirtualAllocEx and were wondering if anybody has

--
Sev