Unable to VirtualAllocEx more than 32664 items.

[Bob Karaban]

We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProce ssId() << endl;
HANDLE object = CreateSemaphore (NULL,0,1,NULL) ;
WaitForSingleOb ject(object,INF INITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemo ry[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PRO CESS_VM_OPERATI ON|PROCESS_VM_R EAD|PROCESS_VM_ WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAl locEx(hProcess, NULL,1024,MEM_C OMMIT,PAGE_READ WRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUG H_MEMORY
}

////////////
// Free
for (i=0;i<100000;i ++)

VirtualFreeEx(h Process,(void*) remoteMemory[i],0,MEM_RELEASE) ;

CloseHandle(hPr ocess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[red floyd]

Bob Karaban wrote:

We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h> nonstandard header #include <iostream.h> nonstandard header, use <iostream>

My compiler has no clue what VirtualAllocEx is. You might try asking in
a VC newsgroup.

[Dave Townsend]

Just a guess, but VirtualAllocEx allocates in complete pages, so even though
you ask for 1k, you are getting a whole page worth, 8k ? That adds up to
256Meg,
do you have that much memory on your system available ?

dave
"Bob Karaban" <bN************ @newNOSPAMworld systems.com> wrote in message
news:10******** *******@nnrp1.p hx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody has a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProce ssId() << endl;
HANDLE object = CreateSemaphore (NULL,0,1,NULL) ;
WaitForSingleOb ject(object,INF INITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemo ry[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PRO CESS_VM_OPERATI ON|PROCESS_VM_R EAD|PROCESS_VM_ WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAl locEx(hProcess, NULL,1024,MEM_C OMMIT,PAGE_READ WRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUG H_MEMORY
}

////////////
// Free
for (i=0;i<100000;i ++)

VirtualFreeEx(h Process,(void*) remoteMemory[i],0,MEM_RELEASE) ;

CloseHandle(hPr ocess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Bob Karaban]

Actually, we do have the memory available - originally, we were thinking the
same thing. What we tried was changing how much memory we allocated - it
doesn't matter if we allocate in 1k blocks or 100k blocks, we can only do it
32,664 times! So, the amount of total memory we use may vary wildly (we
never actually run out) depending on the block size, but we never complete
request #32,665! Very strange! Any other guesses, please keep them coming!

Bob

"Dave Townsend" <da********@com cast.net> wrote in message
news:Ac******** ************@co mcast.com...

Just a guess, but VirtualAllocEx allocates in complete pages, so even though you ask for 1k, you are getting a whole page worth, 8k ? That adds up to
256Meg,
do you have that much memory on your system available ?

dave
"Bob Karaban" <bN************ @newNOSPAMworld systems.com> wrote in message
news:10******** *******@nnrp1.p hx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody

has

a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item.

Below

is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProce ssId() << endl;
HANDLE object = CreateSemaphore (NULL,0,1,NULL) ;
WaitForSingleOb ject(object,INF INITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemo ry[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PRO CESS_VM_OPERATI ON|PROCESS_VM_R EAD|PROCESS_VM_ WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAl locEx(hProcess, NULL,1024,MEM_C OMMIT,PAGE_READ WRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create break point here
// you should get err = 8 -->
ERROR_NOT_ENOUG H_MEMORY
}

////////////
// Free
for (i=0;i<100000;i ++)

VirtualFreeEx(h Process,(void*) remoteMemory[i],0,MEM_RELEASE) ;

CloseHandle(hPr ocess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)? Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Jochen Kalmbach]

Bob Karaban wrote:

We ran into a problem using VirtualAllocEx and were wondering if
anybody has a way around this. We have an executable that stores a
hash table in a remote process. The VirtualAllocEx function fails on
the 32665 item. Below is a sample project to show it.

Just a guess: Maybe your LDT is full !?

--
Greetings
Jochen

Do you need a memory-leak finder ?
http://www.codeproject.com/tools/leakfinder.asp

Do you need daily reports from your server?
http://sourceforge.net/projects/srvreport/

[Ivan Brugiolo [MSFT]]

VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x1000 0
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************ @newNOSPAMworld systems.com> wrote in message
news:10******** *******@nnrp1.p hx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody has a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProce ssId() << endl;
HANDLE object = CreateSemaphore (NULL,0,1,NULL) ;
WaitForSingleOb ject(object,INF INITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemo ry[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PRO CESS_VM_OPERATI ON|PROCESS_VM_R EAD|PROCESS_VM_ WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAl locEx(hProcess, NULL,1024,MEM_C OMMIT,PAGE_READ WRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUG H_MEMORY
}

////////////
// Free
for (i=0;i<100000;i ++)

VirtualFreeEx(h Process,(void*) remoteMemory[i],0,MEM_RELEASE) ;

CloseHandle(hPr ocess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Bob Karaban]

O.K., we think we understand - how would we reduce this granularity? If we
reduce the granularity, we should then be able to get proportionally more
VirtualAllocEx calls, correct?

Thanks for the help!
Bob

"Ivan Brugiolo [MSFT]" <iv******@onlin e.microsoft.com > wrote in message
news:OF******** ******@TK2MSFTN GP11.phx.gbl...

VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x1000 0
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************ @newNOSPAMworld systems.com> wrote in message
news:10******** *******@nnrp1.p hx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody

has

a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item.

Below

is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProce ssId() << endl;
HANDLE object = CreateSemaphore (NULL,0,1,NULL) ;
WaitForSingleOb ject(object,INF INITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemo ry[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PRO CESS_VM_OPERATI ON|PROCESS_VM_R EAD|PROCESS_VM_ WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAl locEx(hProcess, NULL,1024,MEM_C OMMIT,PAGE_READ WRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create break point here
// you should get err = 8 -->
ERROR_NOT_ENOUG H_MEMORY
}

////////////
// Free
for (i=0;i<100000;i ++)

VirtualFreeEx(h Process,(void*) remoteMemory[i],0,MEM_RELEASE) ;

CloseHandle(hPr ocess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)? Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Dave Townsend]

Yeah, I this makes sense, I just looked at the .NET documentation on
virtualallocex, it says the desired
address is rounded to the nearest 64k boundary, so it sounds like you've
blown
the addressability range of windows32.

Since you only need 1k for each entry, can you do some clever stuff to split
up
the 64k page amongst 64 separate entries perhaps ?

dave
"Ivan Brugiolo [MSFT]" <iv******@onlin e.microsoft.com > wrote in message
news:OF******** ******@TK2MSFTN GP11.phx.gbl...

VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x1000 0
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************ @newNOSPAMworld systems.com> wrote in message
news:10******** *******@nnrp1.p hx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody

has

a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item.

Below

is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProce ssId() << endl;
HANDLE object = CreateSemaphore (NULL,0,1,NULL) ;
WaitForSingleOb ject(object,INF INITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemo ry[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PRO CESS_VM_OPERATI ON|PROCESS_VM_R EAD|PROCESS_VM_ WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAl locEx(hProcess, NULL,1024,MEM_C OMMIT,PAGE_READ WRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create break point here
// you should get err = 8 -->
ERROR_NOT_ENOUG H_MEMORY
}

////////////
// Free
for (i=0;i<100000;i ++)

VirtualFreeEx(h Process,(void*) remoteMemory[i],0,MEM_RELEASE) ;

CloseHandle(hPr ocess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)? Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

Bob

[Ivan Brugiolo [MSFT]]

The granularity is pretty much hard-coded in the memory manager
per each OS and platform and architecture.
You should consider virtualallocati ng a 64K chunks in the reserved state,
and then commit the individual pages on demand.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************ @newNOSPAMworld systems.com> wrote in message
news:10******** *******@nnrp1.p hx1.gblx.net...

O.K., we think we understand - how would we reduce this granularity? If we reduce the granularity, we should then be able to get proportionally more
VirtualAllocEx calls, correct?

Thanks for the help!
Bob

"Ivan Brugiolo [MSFT]" <iv******@onlin e.microsoft.com > wrote in message
news:OF******** ******@TK2MSFTN GP11.phx.gbl...

VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x1000 0
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of

your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no

rights.

Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
"Bob Karaban" <bN************ @newNOSPAMworld systems.com> wrote in message news:10******** *******@nnrp1.p hx1.gblx.net...

We ran into a problem using VirtualAllocEx and were wondering if anybody

has

a way around this. We have an executable that stores a hash table in

a remote process. The VirtualAllocEx function fails on the 32665 item.

Below

is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProce ssId() << endl;
HANDLE object = CreateSemaphore (NULL,0,1,NULL) ;
WaitForSingleOb ject(object,INF INITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of

the above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemo ry[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PRO CESS_VM_OPERATI ON|PROCESS_VM_R EAD|PROCESS_VM_ WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualAl locEx(hProcess, NULL,1024,MEM_C OMMIT,PAGE_READ WRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create break point here
// you should get err = 8 -->
ERROR_NOT_ENOUG H_MEMORY
}

////////////
// Free
for (i=0;i<100000;i ++)

VirtualFreeEx(h Process,(void*) remoteMemory[i],0,MEM_RELEASE) ;

CloseHandle(hPr ocess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)? Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or directly via email - thanks in advance!

Bob

[Severian]

On Fri, 4 Jun 2004 15:39:43 -0400, "Bob Karaban"
<bN************ @newNOSPAMworld systems.com> wrote:

We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.
Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>
int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProce ssId() << endl;
HANDLE object = CreateSemaphore (NULL,0,1,NULL) ;
WaitForSingleOb ject(object,INF INITE);
return 0;
}
Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemo ry[0],75,1024);
////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PR OCESS_VM_OPERAT ION|PROCESS_VM_ READ|PROCESS_VM _WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory[i] =
(long)VirtualA llocEx(hProcess ,NULL,1024,MEM_ COMMIT,PAGE_REA DWRITE);
if (remoteMemory[i] == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOU GH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i ++)

VirtualFreeEx( hProcess,(void* )remoteMemory[i],0,MEM_RELEASE) ;

CloseHandle(hPr ocess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!

If you're writing using C, use malloc() & free. If you insist on Win32
API, try HeapAlloc & friends.

IIRC, both only call VirtualAlloc(Ex ) when necessary and subdivide the
large blocks.

--
Sev