473,659 Members | 2,666 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Dealing with naive malloc() implementations

Some recent posts got me thinking about how one might have dealt with
simplistic malloc() implementations which might return NULL for a 64K
request but might accept two 32K requests or four 16K requests. (I'm
assuming, perhaps incorrectly, that quality modern implementations
will coalesce free space as necessary and if possible to satisfy
requests.) I came up with the following (compilable but untested)
first cut at code to try a combination of smaller allocations:

#include <stdlib.h>
#define MAX_SUBCHUNKS 4

/* Functions for accessing individual chunk members omitted */

struct chunk {
void *data[MAX_SUBCHUNKS];
int num_subchunks;
/* Needed by chunk accessors to correctly calculate which subchunk a
* requested member will be in, given that the division of members
* in subchunks is done in a known way, as below */
size_t nmemb;
size_t size;
};

struct chunk *allocate_chunk ( struct chunk *chunk, size_t nmemb, size_t size ) {
size_t idx, jdx;
int succeeded;
chunk->nmemb=nmemb;
chunk->size=size;
for( idx=0; idx < MAX_SUBCHUNKS; idx++ ) {
chunk->num_subchunks= idx+1;
succeeded=1;
for( jdx=0; jdx < idx; jdx++ ) {
size_t num_members=nme mb/(idx+1)+( nmemb%(idx+1)>j dx ? 1 : 0 );
if( (chunk->data[jdx]=malloc(num_mem bers)) == NULL ) {
succeeded=0;
break;
}
}
if( !succeeded ) { /* One of the suballocations failed - free all
existing allocated space */
for( jdx=0; jdx < idx; jdx++ ) {
if( chunk->data[jdx] != NULL ) {
free( chunk->data[jdx] );
}
else {
break;
}
}
continue; /* Try again with more subchunks */
}
return chunk;
}
return NULL; /* Couldn't allocate memory in any combination of
subchunks */
}

int main(void) {
return 0;
}

Were (are?) there implementations where this strategy (or something
along the same lines but undoubtedly more sophisticated) might have
been required to work around a malloc() that was too silly to work out
the details for itself?

This question gives rise to another point - a function that allows the
user to query the malloc() implementation and determine the maximum
number of objects with a given size that it will accept an allocation
request for would really be handy. I wouldn't think that such a
function would be difficult for the malloc() implementor to provide,
assuming that the caller accepts the obvious performance penalty.
This would allow code that needs a substantial but arbitrary amount of
space to easily ask malloc(), "How much space can I have? Ok, that's
enough, give me all of it." Was such a function considered? (I can
take this followup question to comp.std.c if need be.)

--
C. Benson Manica | I *should* know what I'm talking about - if I
cbmanica(at)gma il.com | don't, I need to know. Flames welcome.
May 9 '07 #1
17 1707
Christopher Benson-Manica <at***@faeroes. freeshell.orgwr ote:
(a rather lengthy post)
struct chunk *allocate_chunk ( struct chunk *chunk, size_t nmemb, size_t size ) {
(snip)
for( jdx=0; jdx < idx; jdx++ ) {
size_t num_members=nme mb/(idx+1)+( nmemb%(idx+1)>j dx ? 1 : 0 );
if( (chunk->data[jdx]=malloc(num_mem bers)) == NULL ) {
if( (chunk->data[jdx]=malloc(num_mem bers*size)) == NULL ) {

Well, darn. I was really hoping I had managed to make mistakes that
were at least subtle enough so that *I* couldn't see them.

--
C. Benson Manica | I *should* know what I'm talking about - if I
cbmanica(at)gma il.com | don't, I need to know. Flames welcome.
May 9 '07 #2
Christopher Benson-Manica wrote On 05/09/07 10:15,:
[... simulate large allocation with multiple small allocations
and "accessor" functions ... code snipped (and examined only
briefly ...]

Were (are?) there implementations where this strategy (or something
along the same lines but undoubtedly more sophisticated) might have
been required to work around a malloc() that was too silly to work out
the details for itself?
A "buddy system" malloc (binary, Fibonacci, whatever)
will not coalesce adjacent free areas if they are not buddies.

Implementations that strive for other kinds of efficiencies
(e.g., minimal lock contention in multi-threaded environments)
might postpone coalescing adjacent free areas until the stars
are right.

Even if free areas are coalesced aggressively, there can
still be fragmentation problems. J.M. Robson showed that
fragmentation can cause allocation failure when memory is
only about two-thirds occupied, even if the allocations come
in only two sizes. (See Knuth TAOCP section 2.5 exercises
36 through 38.)

Conclusion: One need not presume a "silly" malloc to
make subdividing large allocations worth while.
This question gives rise to another point - a function that allows the
user to query the malloc() implementation and determine the maximum
number of objects with a given size that it will accept an allocation
request for would really be handy. I wouldn't think that such a
function would be difficult for the malloc() implementor to provide,
assuming that the caller accepts the obvious performance penalty.
This would allow code that needs a substantial but arbitrary amount of
space to easily ask malloc(), "How much space can I have? Ok, that's
enough, give me all of it." Was such a function considered? (I can
take this followup question to comp.std.c if need be.)
I have no idea whether such a facility was considered
in the standardization debates.

... but at a PPOE I did a fairly sizeable amount of work
on a rather more elaborate memory manager that provided a
related capability (for some of the memory pools it managed).
It answered the analogue of "How much larger can realloc()
make such-and-such an area without needing to relocate its
contents?" In the course of porting our product to its first
64-bit platform, I found that this function was expending
enormous amounts of time and started looking for ways to
speed it up.

The problem occurred when the area in question was at
the "end" of allocated memory: on a 64-bit system there was
an unthinkably large amount of unused virtual memory right
after that area, and the whole system was churning around
trying to figure out how much swap it would be able to find.
There didn't seem to be an obvious way to answer the question
with acceptable efficiency. So I turned my attention away
from the slow function and toward its callers; what use were
they making of the answer? Lo! there were only about half
a dozen callers, and they all did something like

if (fragSize(ptr) >= what_i_need)
do_this();
else
do_that();

So I wound up jettisoning the troublesome function and
replacing it with the analogue of "Would realloc() need to
relocate the area if asked to expand it to such-and-such
a size?", which was far more tractable. The callers all
became

if (canReallocInPl ace(ptr,what_i_ need))
do_this();
else
do_that();

.... and it was a beautiful day in the neighborhood.

The point of this long-winded anecdote is that I have
some reason to doubt the usefulness of "What's the most
space I can have?" Even if you could ask it, following
with "Give me all of it" seems a dangerous next step, one
that is likely to cause almost immediate memory exhaustion
leading to failure or bad performance (think of subsequent
malloc() calls all returning NULL, of fopen() failing or
producing only unbuffered I/O streams, and so on). It's a
question that may seem interesting at first glance, but
I'm not so sure it would turn out to be all that useful.

The related question "Can I have N bytes?" is another
matter altogether -- and the C library already provides
the means to answer it.

canReallocInPla ce() is not part of the library, and
something along those lines might make a useful addition.
Might not play well with multi-threaded programs, though
(by the time you get the answer, it could be outdated).

--
Er*********@sun .com
May 9 '07 #3
"Christophe r Benson-Manica" <at***@faeroes. freeshell.orgwr ote in message
news:f1******** **@chessie.cirr .com...
Some recent posts got me thinking about how one might have dealt with
simplistic malloc() implementations which might return NULL for a 64K
request but might accept two 32K requests or four 16K requests. (I'm
assuming, perhaps incorrectly, that quality modern implementations
will coalesce free space as necessary and if possible to satisfy
requests.) I came up with the following (compilable but untested)
first cut at code to try a combination of smaller allocations:
(...)
Were (are?) there implementations where this strategy (or something
along the same lines but undoubtedly more sophisticated) might have
been required to work around a malloc() that was too silly to work out
the details for itself?
In my allocator the free status of the block with a given granularity is
kept in a separate array of bits. Free just sets the appropriate number of
bits and returns, while malloc scans through the array looking for the
required number of adjacent bits set.
Several arenas with different granularities are used so that no allocation
requires more than 32 bits. You can check out the code at my website:
www.geocities.com/malbrain.
This question gives rise to another point - a function that allows the
user to query the malloc() implementation and determine the maximum
number of objects with a given size that it will accept an allocation
request for would really be handy.
You're welcome to add an appropriate bit scanner to determine these
quantities from the arenas.

karl m
May 9 '07 #4
Eric Sosman <Er*********@su n.comwrote:
"How much larger can realloc()make such-and-such an area
without needing to relocate its contents?"
The problem occurred when the area in question was at
the "end" of allocated memory: on a 64-bit system there was
an unthinkably large amount of unused virtual memory right
after that area, and the whole system was churning around
trying to figure out how much swap it would be able to find.
Aha, yes, I can certainly see that being a problem. I guess I'm
comforted by the fact that some folks got paid to make the same naive
assumption I made for free ;-)
The point of this long-winded anecdote is that I have
some reason to doubt the usefulness of "What's the most
space I can have?" Even if you could ask it, following
with "Give me all of it" seems a dangerous next step...
Yes, your (much appreciated) anecdote definitely demonstrated that my
initial question and response were poorly conceived.
The related question "Can I have N bytes?" is another
matter altogether -- and the C library already provides
the means to answer it.
Well, yes, although I do think perhaps it might be beneficial to be
able to ask it in a slightly different way as well as the current way.
For example, code that attempted to obtain a certain amount of memory
in any combination of chunks might (if I were writing it) want to ask,
"Can I have space for M obects of size N?" and get a response "No, but
you can have space for M-1 of those, and maybe if you ask me nicely I
might give you some more space for that last object". This question
can be answered with malloc() only by calling malloc( M*N ) up to M-1
times to see whether the request will succeed or not, which isn't very
efficient.

--
C. Benson Manica | I *should* know what I'm talking about - if I
cbmanica(at)gma il.com | don't, I need to know. Flames welcome.
May 9 '07 #5
Christopher Benson-Manica wrote:
Some recent posts got me thinking about how one might have dealt with
simplistic malloc() implementations which might return NULL for a 64K
request but might accept two 32K requests or four 16K requests. (I'm
assuming, perhaps incorrectly, that quality modern implementations
will coalesce free space as necessary and if possible to satisfy
requests.)
I think all implementations will coalesce free space, but only if it has
two (or more) *adjacent* free memory blocks.
The big problem is that most C implementations use raw memory addresses
for pointers, and are therefor not able to move pieces of allocated
memory around in order to create a larger block of free memory.

Therefor, a request for one large block may fail, while several requests
for smaller blocks (amounting to the same total amount of memory being
allocated) succeed.
I came up with the following (compilable but untested)
first cut at code to try a combination of smaller allocations:

#include <stdlib.h>
#define MAX_SUBCHUNKS 4

/* Functions for accessing individual chunk members omitted */

struct chunk {
void *data[MAX_SUBCHUNKS];
int num_subchunks;
/* Needed by chunk accessors to correctly calculate which subchunk a
* requested member will be in, given that the division of members
* in subchunks is done in a known way, as below */
size_t nmemb;
size_t size;
};

struct chunk *allocate_chunk ( struct chunk *chunk, size_t nmemb,
size_t size ) {
size_t idx, jdx;
int succeeded;
chunk->nmemb=nmemb;
chunk->size=size;
for( idx=0; idx < MAX_SUBCHUNKS; idx++ ) {
chunk->num_subchunks= idx+1;
succeeded=1;
for( jdx=0; jdx < idx; jdx++ ) {
size_t num_members=nme mb/(idx+1)+( nmemb%(idx+1)>j dx ? 1 : 0 );
if( (chunk->data[jdx]=malloc(num_mem bers)) == NULL ) {
succeeded=0;
break;
}
}
if( !succeeded ) { /* One of the suballocations failed - free all
existing allocated space */
for( jdx=0; jdx < idx; jdx++ ) {
if( chunk->data[jdx] != NULL ) {
free( chunk->data[jdx] );
}
else {
break;
}
}
continue; /* Try again with more subchunks */
}
return chunk;
}
return NULL; /* Couldn't allocate memory in any combination of
subchunks */
}

int main(void) {
return 0;
}

Were (are?) there implementations where this strategy (or something
along the same lines but undoubtedly more sophisticated) might have
been required to work around a malloc() that was too silly to work out
the details for itself?

This question gives rise to another point - a function that allows the
user to query the malloc() implementation and determine the maximum
number of objects with a given size that it will accept an allocation
request for would really be handy. I wouldn't think that such a
function would be difficult for the malloc() implementor to provide,
assuming that the caller accepts the obvious performance penalty.
This would allow code that needs a substantial but arbitrary amount of
space to easily ask malloc(), "How much space can I have? Ok, that's
enough, give me all of it." Was such a function considered? (I can
take this followup question to comp.std.c if need be.)
Such a function would only be useful if your program has exclusive
access to the memory and does not use <OTmultiple threads </OT>.
Otherwise, the value you got from that function could already be invalid
by the time you actually try to allocate the memory.

Bart v Ingen Schenau
--
a.c.l.l.c-c++ FAQ: http://www.comeaucomputing.com/learn/faq
c.l.c FAQ: http://www.eskimo.com/~scs/C-faq/top.html
c.l.c++ FAQ: http://www.parashift.com/c++-faq-lite/
May 9 '07 #6
Eric Sosman wrote:
canReallocInPla ce() is not part of the library, and
something along those lines might make a useful addition.
Might not play well with multi-threaded programs, though
(by the time you get the answer, it could be outdated).
Could that not be answered with a `reallocInPlace ` which
reuses the space if it can and returns null if it can't?
(The malloc family would have to be robust against multi-
threading anyway.)

--
Fragmented Hedgehog
"Who do you serve, and who do you trust?" /Crusade/

May 10 '07 #7
Chris Dollin wrote:
Eric Sosman wrote:
> canReallocInPla ce() is not part of the library, and
something along those lines might make a useful addition.
Might not play well with multi-threaded programs, though
(by the time you get the answer, it could be outdated).

Could that not be answered with a `reallocInPlace ` which
reuses the space if it can and returns null if it can't?
(The malloc family would have to be robust against multi-
threading anyway.)
That can be handled by the existing system:

if (tmp = realloc(whateve r, newsize)) {
if (tmp == whatever) tmp = OK; /* realloced in place */;
else realloc = tmp;
}
else {
/* realloc failed */
}

tmp == NULL signals failure. tmp == OK signals in place (OK).
Other values signal realloc success.

--
<http://www.cs.auckland .ac.nz/~pgut001/pubs/vista_cost.txt>
<http://www.securityfoc us.com/columnists/423>
<http://www.aaxnet.com/editor/edit043.html>
<http://kadaitcha.cx/vista/dogsbreakfast/index.html>
cbfalconer at maineline dot net
--
Posted via a free Usenet account from http://www.teranews.com

May 10 '07 #8
Chris Dollin wrote:
Eric Sosman wrote:
> canReallocInPla ce() is not part of the library, and
something along those lines might make a useful addition.
Might not play well with multi-threaded programs, though
(by the time you get the answer, it could be outdated).

Could that not be answered with a `reallocInPlace ` which
reuses the space if it can and returns null if it can't?
(The malloc family would have to be robust against multi-
threading anyway.)
That'd be one way to package it. Another might be a
realloc-like call that distinguished three outcomes: failure,
success (unmoved), and success (relocated).

Hmmm... reallocInPlace( ) would most likely be more usable.
The "in place" property is probably only important if you have
pointers aiming into the reallocated data (perhaps contained
within it), and if the allocation moves the data the pointers
need to be reconstituted. However, once the data moves it's
too late for `new_ptr = old_ptr - old_base + new_base'; using
the now-invalid old_ptr and old_base invokes undefined behavior.
After a reallocInPlace( ) failure, one could prepare a table of
offsets, then do an ordinary realloc(), and then repair the
pointers.

I'm not so sure there's a really compelling need for
the facility, though. It seems more useful than "Give me all
the memory I can get," but still may not be useful enough
to fret about.

--
Eric Sosman
es*****@acm-dot-org.invalid
May 10 '07 #9
CBFalconer wrote:
Chris Dollin wrote:
>Eric Sosman wrote:
>> canReallocInPla ce() is not part of the library, and
something along those lines might make a useful addition.
Might not play well with multi-threaded programs, though
(by the time you get the answer, it could be outdated).

Could that not be answered with a `reallocInPlace ` which
reuses the space if it can and returns null if it can't?
(The malloc family would have to be robust against multi-
threading anyway.)

That can be handled by the existing system:

if (tmp = realloc(whateve r, newsize)) {
That doesn't work, because if realloc needed to move it, /it has
already been moved/. The intent of the hedgehog's `reallocInPlace `
is clearly that if the existing space isn't big enough, /nothing
happens/, so you can take alternative action.

--
"Who do you serve, and who do you trust?" /Crusade/

Hewlett-Packard Limited Cain Road, Bracknell, registered no:
registered office: Berks RG12 1HN 690597 England

May 10 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

13
2543
by: Ardhendu Nandan | last post by:
Hi, Will someone please tell me, How malloc allocating memory from heap & what data structure it is using to do so. regards Ardhendu
12
818
by: Yevgen Muntyan | last post by:
Hey, Consider the following code: #include <stdlib.h> #define MAGIC_NUMBER 64 void *my_malloc (size_t n) { char *result = malloc (n + MAGIC_NUMBER);
3
1240
by: ais523 | last post by:
Consider the following program: #include <stdlib.h> #include <limits.h> int main(void) { size_t s=SIZE_MAX; size_t i; char* p;
82
31076
by: quiberon2 | last post by:
Hi, Sorry if it might be a stupid question but what should returns malloc(0) ? void *ptr = malloc(0); I am running gcc 3.3.5 and a non-null address is returned. ( in the compiler that I am currently implementing, NULL is returned. Is it wrong ?)
318
12926
by: jacob navia | last post by:
Rcently I posted code in this group, to help a user that asked to know how he could find out the size of a block allocated with malloc. As always when I post something, the same group of people started to try to find possible errors, a harmless passtime they seem to enjoy. One of their remarks was that I used "int" instead of "size_t" for the input of my allocator function.
6
1803
by: Dave Stallard | last post by:
So, I'm looking at some code that does 770K malloc calls in a row, of varying size, paired with corresponding freads from a binary file to initialize. In total, about 58 MB of data is allocated an initialized. Unsurprisingly, this takes a good bit of time. Question: Wouldn't it be a lot more efficient to allocate this 58 MB in one big gulp, then fread in the 58 MB of data from the file to initialize it, rather than a zillion small...
71
19093
by: desktop | last post by:
I have read in Bjarne Stroustrup that using malloc and free should be avoided in C++ because they deal with uninitialized memory and one should instead use new and delete. But why is that a problem? I cannot see why using malloc instead of new does not give the same result.
34
13371
by: niranjan.singh | last post by:
This is regarding to test an SDK memory stuff. In what situation malloc gets fail. any comment/reply pls.... regards
35
2070
by: =?utf-8?b?QXNiasO4cm4gU8OmYsO4?= | last post by:
This topic is a FAQ. But I have read the faq and spent a couple of hours browsing the group archives, and still have a few questions that I hope you can answer. My understanding is that recommended practice is to not cast the return value from malloc(). The rationale for this is that 1) the cast is not needed and 2) the cast may mask errors. I assume that the reason the cast is not needed has to do with the fact that the the pointer...
0
8427
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8850
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8746
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8626
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7355
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4175
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4334
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2749
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
1975
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.