malloc beyond 20MB fails. Why so limited?

[itsolution]

Hi Guru,

I need some data cache (which I want is able to keep tens of thousands
entries) containing DB data in my Unix running process. Each cache
entry is around 0.5k bytes. So, I want to create the Cache area in my
Unix process when the process starts up.
To the contrary to my expectation, once malloc(0.5k) is called
beyond 4000 times, then, trailing malloc() continues to fail.

Isn't it the heap space should be much bigger than 20MB in a Unix
process ?
Why malloc is failing at not big size allocation ?
Any workaround or advice?
thanks

[Malcolm McLean]

<it********@gmail.comwrote in message

I need some data cache (which I want is able to keep tens of thousands
entries) containing DB data in my Unix running process. Each cache
entry is around 0.5k bytes. So, I want to create the Cache area in my
Unix process when the process starts up.
To the contrary to my expectation, once malloc(0.5k) is called
beyond 4000 times, then, trailing malloc() continues to fail.

Isn't it the heap space should be much bigger than 20MB in a Unix
process ?
Why malloc is failing at not big size allocation ?
Any workaround or advice?

Firstly check everything else using a program monitor to make sure you don't
have a hog running simultaneously.
You should have much more than 20MB. Either you are passing the wrong size,
or there is a leak somewhere. Blocks of 500 bytes or so are rather
inefficient, but not that inefficient.

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

[vippstar]

On Dec 30, 12:30 pm, [e-mail] wrote:

Hi Guru,

I need some data cache (which I want is able to keep tens of thousands
entries) containing DB data in my Unix running process. Each cache
entry is around 0.5k bytes. So, I want to create the Cache area in my
Unix process when the process starts up.
To the contrary to my expectation, once malloc(0.5k) is called
beyond 4000 times, then, trailing malloc() continues to fail.

Isn't it the heap space should be much bigger than 20MB in a Unix
process ?
Why malloc is failing at not big size allocation ?
Any workaround or advice?

try <news:comp.unix.programmer>
As far as ISO C is concerned, malloc can allocate from 1 to SIZE_MAX.

[jacob navia]

it********@gmail.com wrote:

Hi Guru,

I need some data cache (which I want is able to keep tens of thousands
entries) containing DB data in my Unix running process. Each cache
entry is around 0.5k bytes. So, I want to create the Cache area in my
Unix process when the process starts up.
To the contrary to my expectation, once malloc(0.5k) is called
beyond 4000 times, then, trailing malloc() continues to fail.

Isn't it the heap space should be much bigger than 20MB in a Unix
process ?
Why malloc is failing at not big size allocation ?
Any workaround or advice?
thanks

Besides checking what Mr McLean told you, check the permissions.
(ulimit). Sometimes administrators limit what a user process can
allocate.

--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32

[Wolfgang Draxinger]

jacob navia wrote:

All that is true but the user is running in a Unix system, that
is unlikely to have such a bad malloc implementation...

I wouldn't be so sure about this. Some Unix libc have been
developed for speed and simplicity, at the sacrifice of memory
efficiency.

Wolfgang Draxinger
--
E-Mail address works, Jabber: he******@jabber.org, ICQ: 134682867

[itsolution]

The issue I have encountered occurs in a network device running BSD.
malloc(1024 * 20000) is ok but malloc(1024 * 30000) always fails.
Looks like due to ulimit issue or similar limit ... right?
Follow-up question (to overcome that limitation)

When implementing a memory pool of having 30MB or so,
using global static memory instead of using heap is the better
way ?
thanks

On Dec 30, 2:30 am, [e-mail] wrote:

Hi Guru,

I need some data cache (which I want is able to keep tens of thousands
entries) containing DB data in my Unix running process. Each cache
entry is around 0.5k bytes. So, I want to create the Cache area in my
Unix process when the process starts up.
To the contrary to my expectation, once malloc(0.5k) is called
beyond 4000 times, then, trailing malloc() continues to fail.

Isn't it the heap space should be much bigger than 20MB in a Unix
process ?
Why malloc is failing at not big size allocation ?
Any workaround or advice?

thanks

[Randy Howard]

On Sun, 30 Dec 2007 05:35:36 -0600, jacob navia wrote
(in article <fl**********@aioe.org>):

it********@gmail.com wrote:

>Hi Guru,

I need some data cache (which I want is able to keep tens of thousands
entries) containing DB data in my Unix running process. Each cache
entry is around 0.5k bytes. So, I want to create the Cache area in my
Unix process when the process starts up.
To the contrary to my expectation, once malloc(0.5k) is called
beyond 4000 times, then, trailing malloc() continues to fail.

Isn't it the heap space should be much bigger than 20MB in a Unix
process ?
Why malloc is failing at not big size allocation ?
Any workaround or advice?
thanks

Besides checking what Mr McLean told you, check the permissions.
(ulimit). Sometimes administrators limit what a user process can
allocate.

Yes, and there are also per-process limits, for example 2GB for a
32-bit Linux system is common, which you might be bumping up against
due to the size of other code and data in the process, or by an
unintentional memory leak, the inclusion of a bloatware library, there
could be a lot of potential reasons. You can /usually/ grab via malloc
something quite close to 2GB with a single malloc() call in small apps,
somewhere in the 1700MB range wouldn't surprise me at all.

However, none of this is really a standard C issue, and more an issue
with a particular platform/system setup, or an undetected error in the
code. Taking it up in a forum dedicated to your platform would make a
lot more sense.

--
Randy Howard (2reply remove FOOBAR)
"The power of accurate observation is called cynicism by those
who have not got it." - George Bernard Shaw

[Serve Lau]

<it********@gmail.comschreef in bericht
news:12**********************************@s19g2000 prg.googlegroups.com...

Hi Guru,

I need some data cache (which I want is able to keep tens of thousands
entries) containing DB data in my Unix running process. Each cache
entry is around 0.5k bytes. So, I want to create the Cache area in my
Unix process when the process starts up.
To the contrary to my expectation, once malloc(0.5k) is called
beyond 4000 times, then, trailing malloc() continues to fail.

Isn't it the heap space should be much bigger than 20MB in a Unix
process ?
Why malloc is failing at not big size allocation ?
Any workaround or advice?

If you know you will call malloc 4k times with 512 bytes (that is 2MB by the
way) why not call malloc once with that size and create a function to return
a pointer to the next 512 bytes at every call. This way all that memory will
be contiguous in memory

>

thanks

[SM Ryan]

# Isn't it the heap space should be much bigger than 20MB in a Unix
# process ?

No.

Yes.

No.

Depends. Are there resource limits? Do you have manually add
sectors to VM on your unix? Is your boot drive track limitting?

# Why malloc is failing at not big size allocation ?

Is that what the errno is saying?

# Any workaround or advice?

Many unices have performance tools that help you figure out
if you're exhausting VM, but it depends on which unix you've
got.

--
SM Ryan http://www.rawbw.com/~wyrmwif/
Wow. A sailboat.