Looking for an open source memory pool

[graham.keellings]

hi,

I'm looking for an open source memory pool. It's for use on an
embedded system, if that makes any difference. Something with garbage
collection/defragmentation would be nice. It should have the ability
to allocate different size chunks of memory not just a single size. It
should error check for double free, etc. And it should be usable by a
mixture of C and C++ subsystems.

If I get that, I'm happy. Thank you very much.

As a bonus, I'd like to detect memory leaks, although I'm willing to
code that bit myself to get what I want (I just don't want to reinvent
the wheel of a standard memory pool).

I see two approaches to detecting leaks. Both involve the memory pool
actually allocating more memory than was requested and keeping a
secret header for itself where it stores details of the caller (file/
line number or unique ID, plus timestamp).

After each unit test I would generally expect any allocated memory to
be freed, so comapring memory pool free size at start and end of test
should suffice. If I know that the test will cause my software to
allocate but not free some memory, I can adjust for that (for
instance, Software Under Test allocates a buffer to send a message
which will be freed by recipient, or a timer data block, or similar).

So much for unit test, which ought to be straightforward. Integration
& system verification test are more complex, with lots of background
tasks running. However, in an embedded system it is uncommon for
memory to remain allocated for long. So I can create a low priority
task which runs when the system is otherwise idle and checks
timestamps, looking for memory allocated for "a suspiciously long
time", which can then be investigated to see if someone forgot to free
it. Not perfect, but more than good enough.

Any comments? And any basic, Open Source, memory pool which I can use
as a base?

Thanks in advance...

[graham.keellings]

On Jul 25, 9:34*am, [e-mail] wrote:

hi,

* I'm looking for an open source memory pool. It's for use on an
embedded system, if that makes any difference. Something with garbage
collection/defragmentation would be nice. It should have the ability
to allocate different size chunks of memory not just a single size. It
should error check for double free, etc. And it should be usable by a
mixture of C and C++ subsystems.

If I get that, I'm happy. Thank you very much.

As a bonus, I'd like to detect memory leaks, although I'm willing to
code that bit myself to get what I want (I just don't want to reinvent
the wheel of a standard memory pool).

I see two approaches to detecting leaks. Both involve the memory pool
actually allocating more memory than was requested and keeping a
secret header for itself where it stores details of the caller (file/
line number or unique ID, plus timestamp).

After each unit test I would generally expect any allocated memory to
be freed, so comapring memory pool free size at start and end of test
should suffice. If I know that the test will cause my software to
allocate but not free some memory, I can adjust for that (for
instance, Software Under Test allocates a buffer to send a message
which will be freed by recipient, or a timer data block, or similar).

So much for unit test, which ought to be straightforward. Integration
& system verification test are more complex, with lots of background
tasks running. However, in an embedded system it is uncommon for
memory to remain allocated for long. So I can create a low priority
task which runs when the system is otherwise idle and checks
timestamps, looking for memory allocated for "a suspiciously long
time", which can then be investigated to see if someone forgot to free
it. Not perfect, but more than good enough.

Any comments? And any basic, Open Source, memory pool which I can use
as a base?

Thanks in advance...

Bosst++ is looking good. See
http://www.boost.org/doc/libs/1_35_0...d_storage.html

[Juha Nieminen]

gr**************@gmail.com wrote:

Something with garbage collection/defragmentation would be nice.

As a bonus, I'd like to detect memory leaks

Aren't those two things a bit mutually exclusive?

[Pete Becker]

On 2008-07-25 10:02:43 -0400, Juha Nieminen <no****@thanks.invalidsaid:

gr**************@gmail.com wrote:

>Something with garbage collection/defragmentation would be nice.

>As a bonus, I'd like to detect memory leaks

Aren't those two things a bit mutually exclusive?

No. When the application terminates, all reachable memory has been leaked.

--
Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com) Author of "The
Standard C++ Library Extensions: a Tutorial and Reference
(www.petebecker.com/tr1book)

[James Kanze]

On Jul 25, 5:03 pm, Pete Becker <[e-mail]:

On 2008-07-25 10:02:43 -0400, Juha Nieminen <[e-mail]:

[e-mail] wrote:

Something with garbage collection/defragmentation would be nice.

As a bonus, I'd like to detect memory leaks

Aren't those two things a bit mutually exclusive?

No. When the application terminates, all reachable memory has
been leaked.

We must have a different definition of memory leaks. When I
terminate an application program, the system recovers all of the
memory, so no memory leaks. The problem with memory leaks is
when the program doesn't terminate, but just keeps on using more
and more memory. (And such leaks are perfectly possible with
garbage collection, just not as likely.)

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[Pete Becker]

On 2008-07-25 17:10:38 -0400, James Kanze <ja*********@gmail.comsaid:

On Jul 25, 5:03 pm, Pete Becker <[e-mail]:

>On 2008-07-25 10:02:43 -0400, Juha Nieminen <[e-mail]:

>>[e-mail] wrote:
Something with garbage collection/defragmentation would be nice.

>>>As a bonus, I'd like to detect memory leaks

>>Aren't those two things a bit mutually exclusive?

>No. When the application terminates, all reachable memory has
been leaked.

We must have a different definition of memory leaks. When I
terminate an application program, the system recovers all of the
memory, so no memory leaks.

Nevertheless, the program leaked memory, leaving it for the OS to clean
up. Yes, from a system perspective, the memory probably wasn't lost,
but that's a different issue.

--
Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com) Author of "The
Standard C++ Library Extensions: a Tutorial and Reference
(www.petebecker.com/tr1book)

[James Kanze]

On Jul 26, 12:14 am, Pete Becker <[e-mail]:

On 2008-07-25 17:10:38 -0400, James Kanze <[e-mail]:

On Jul 25, 5:03 pm, Pete Becker <[e-mail]:

On 2008-07-25 10:02:43 -0400, Juha Nieminen <[e-mail]:

>[e-mail] wrote:
Something with garbage collection/defragmentation would be nice.

>Aren't those two things a bit mutually exclusive?

No. When the application terminates, all reachable memory has
been leaked.

We must have a different definition of memory leaks. When I
terminate an application program, the system recovers all of the
memory, so no memory leaks.

Nevertheless, the program leaked memory, leaving it for the OS
to clean up.

Again, it depends on your definition of leaked. As a pragmatic
developer, the only definition which interests me is the one
that concerns me: the program hasn't "freed" memory that it
doesn't need anymore. If a garbage collector is present, and
could collect the memory, it isn't "leaked" (since the
application could reuse it). If its a one time operation, for
example constructing a singleton which is never deleted, it
isn't leaked (any more than a static variable is a "leak"). If
the application keeps memory that it doesn't need any more, even
if it does have a pointer to it, and could reach it (e.g. it's
in a map, indexed by a key that has been retired, and won't be
used any more), it's been leaked.

I rather insist on this definition, because people use all sort
of useless definitions, to prove that you can't have a memory
leak in Java (although Sun has had a couple in their bug list),
or some other claim that is only true because the definition is
useless.

(Of course, even this definition leaves some questions open. A
simple compiler will parse all of the source first, then
generate code. Once the source is parsed, no more error
messages will appear. Has it leaked memory because the buffer
of std::cerr hasn't been freed? It's not going to use it any
more.)

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[Pete Becker]

On 2008-07-26 04:57:07 -0400, James Kanze <ja*********@gmail.comsaid:

On Jul 26, 12:14 am, Pete Becker <[e-mail]:

>On 2008-07-25 17:10:38 -0400, James Kanze <[e-mail]:

>>On Jul 25, 5:03 pm, Pete Becker <[e-mail]:
On 2008-07-25 10:02:43 -0400, Juha Nieminen <[e-mail]

id:

>>>>[e-mail] wrote:
>Something with garbage collection/defragmentation would be nice.

>>>>Aren't those two things a bit mutually exclusive?

>>>No. When the application terminates, all reachable memory has
been leaked.

>>We must have a different definition of memory leaks. When I
terminate an application program, the system recovers all of the
memory, so no memory leaks.

>Nevertheless, the program leaked memory, leaving it for the OS
to clean up.

Again, it depends on your definition of leaked.

Yes, that was my point.

As a pragmatic
developer, the only definition which interests me is the one
that concerns me: the program hasn't "freed" memory that it
doesn't need anymore.

And one way to see that is to terminate the program and look at what's
still in use.

--
Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com) Author of "The
Standard C++ Library Extensions: a Tutorial and Reference
(www.petebecker.com/tr1book)

[James Kanze]

On Jul 26, 12:55 pm, Pete Becker <[e-mail]:

On 2008-07-26 04:57:07 -0400, James Kanze <[e-mail]:

[...]

As a pragmatic
developer, the only definition which interests me is the one
that concerns me: the program hasn't "freed" memory that it
doesn't need anymore.

And one way to see that is to terminate the program and look
at what's still in use.

Which begs the point: what does it mean to be "still in use"?
If there is an active pointer to it, is it "still in use" (even
if the application would never have used that pointer)?

Purify (and I suppose most other similar tools) distinguishes
between a "memory leak" (no pointer to the memory, but it hasn't
been freed) and a "possible memory leak" (unfreed memory at the
end of the program, but still pointers to it in static memory).

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[Pete Becker]

On 2008-07-26 08:02:33 -0400, James Kanze <ja*********@gmail.comsaid:

On Jul 26, 12:55 pm, Pete Becker <[e-mail]:

>On 2008-07-26 04:57:07 -0400, James Kanze <[e-mail]:

[...]

>>As a pragmatic
developer, the only definition which interests me is the one
that concerns me: the program hasn't "freed" memory that it
doesn't need anymore.

>And one way to see that is to terminate the program and look
at what's still in use.

Which begs the point: what does it mean to be "still in use"?

Sigh. As I said before, still reachable.

If there is an active pointer to it, is it "still in use" (even
if the application would never have used that pointer)?

Purify (and I suppose most other similar tools) distinguishes
between a "memory leak" (no pointer to the memory, but it hasn't
been freed) and a "possible memory leak" (unfreed memory at the
end of the program, but still pointers to it in static memory).

Shrug. The original question was, in essence, how can garbage
collection help detect leaks, not what's the best possible approach to
leak detection. I answered the first question, and you're talking about
the second one.

--
Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com) Author of "The
Standard C++ Library Extensions: a Tutorial and Reference
(www.petebecker.com/tr1book)

[Greg Comeau]

In article <4f**********************************@i76g2000hsf. googlegroups.com>,
James Kanze <ja*********@gmail.comwrote:

>On Jul 26, 12:14 am, Pete Becker <[e-mail]:

>On 2008-07-25 17:10:38 -0400, James Kanze <[e-mail]:

On Jul 25, 5:03 pm, Pete Becker <[e-mail]:
On 2008-07-25 10:02:43 -0400, Juha Nieminen <[e-mail]=

id:

>>[e-mail] wrote:
Something with garbage collection/defragmentation would be nice.

>>Aren't those two things a bit mutually exclusive?

>No. When the application terminates, all reachable memory has
been leaked.

We must have a different definition of memory leaks. When I
terminate an application program, the system recovers all of the
memory, so no memory leaks.

>Nevertheless, the program leaked memory, leaving it for the OS
to clean up.

Again, it depends on your definition of leaked. As a pragmatic
developer, the only definition which interests me is the one
that concerns me: the program hasn't "freed" memory that it
doesn't need anymore. If a garbage collector is present, and
could collect the memory, it isn't "leaked" (since the
application could reuse it). If its a one time operation, for
example constructing a singleton which is never deleted, it
isn't leaked (any more than a static variable is a "leak"). If
the application keeps memory that it doesn't need any more, even
if it does have a pointer to it, and could reach it (e.g. it's
in a map, indexed by a key that has been retired, and won't be
used any more), it's been leaked.

I rather insist on this definition, because people use all sort
of useless definitions, to prove that you can't have a memory
leak in Java (although Sun has had a couple in their bug list),
or some other claim that is only true because the definition is
useless.

(Of course, even this definition leaves some questions open. A
simple compiler will parse all of the source first, then
generate code. Once the source is parsed, no more error
messages will appear. Has it leaked memory because the buffer
of std::cerr hasn't been freed? It's not going to use it any
more.)

You raise some excellent points, and ones that should definitely
be considered. For another $0.02 I usually try to remove the
aspect of the environment from the application to determine
if it _could_ have a leak. That is to say, if I moved this application
to another OS, would it then have a leak. If so, then is it worth
fixing. I rather think this does negate anything being said above,
it may not even address what you're saying, and just another data point,
even if deemed inconsistent. :)
--
Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
Comeau C/C++ ONLINE == http://www.comeaucomputing.com/tryitout
World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

[coal]

On Jul 26, 3:57*am, James Kanze <[e-mail]:

>
Again, it depends on your definition of leaked. *As a pragmatic
developer, the only definition which interests me is the one
that concerns me: the program hasn't "freed" memory that it
doesn't need anymore. *If a garbage collector is present, and
could collect the memory, it isn't "leaked" (since the
application could reuse it). *If its a one time operation, for
example constructing a singleton which is never deleted, it
isn't leaked (any more than a static variable is a "leak"). *If
the application keeps memory that it doesn't need any more, even
if it does have a pointer to it, and could reach it (e.g. it's
in a map, indexed by a key that has been retired, and won't be
used any more), it's been leaked.

I rather insist on this definition, because people use all sort
of useless definitions, to prove that you can't have a memory
leak in Java (although Sun has had a couple in their bug list),
or some other claim that is only true because the definition is
useless.

(Of course, even this definition leaves some questions open. *A
simple compiler will parse all of the source first, then
generate code. *Once the source is parsed, no more error
messages will appear. *Has it leaked memory because the buffer
of std::cerr hasn't been freed? *It's not going to use it any
more.)

I'm surprised you say that. I've read other posts where you mention
disks filling up and the errors that stem from that. I guess 99.9%
of
the total errors would occur while parsing, but a few could come up
later. But isn't it more interesting to consider a compiler that
doesn't exit after one request? In that case I wouldn't free the
resource because there would be a need for it again on the next
iteration. If you faced tough resource constraints that might be
an area where you could save some at the expense of losing the .1% of
error messages. If you stop getting output you'll probably
eventually figure out to check if the disk is full.

Brian Wood
http://webEbenezer.net

[James Kanze]

On Jul 26, 11:56 pm, [e-mail] wrote:

On Jul 26, 3:57 am, James Kanze <[e-mail]:

(Of course, even this definition leaves some questions open. A
simple compiler will parse all of the source first, then
generate code. Once the source is parsed, no more error
messages will appear. Has it leaked memory because the buffer
of std::cerr hasn't been freed? It's not going to use it any
more.)

I'm surprised you say that. I've read other posts where you
mention disks filling up and the errors that stem from that.
I guess 99.9% of the total errors would occur while parsing,
but a few could come up later.

Yes. It was meant as a simplistic example, but it is probably
too simple: std::cerr (or the underlying physical file) might be
used in case of assertion failure, or if the compiler detects
some internal error. Shouldn't happen, but defensive
programming will allow for it anyway.

A more realistic example might be if the compiler outputs its
usual messages to std::cout, rather than std::cerr, and reserves
std::cerr for more critical things, such as write errors or
internal errors.

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[graham.keellings]

On Jul 26, 5:10*am, James Kanze <[e-mail]:

On Jul 25, 5:03 pm, Pete Becker <[e-mail]:

On 2008-07-25 10:02:43 -0400, Juha Nieminen <[e-mail]:

[e-mail] wrote:
>Something with garbage collection/defragmentation would be nice.
>As a bonus, I'd like to detect memory leaks
* Aren't those two things a bit mutually exclusive?

No. When the application terminates, all reachable memory has
been leaked.

We must have a different definition of memory leaks. *When I
terminate an application program, the system recovers all of the
memory, so no memory leaks. *The problem with memory leaks is
when the program doesn't terminate, but just keeps on using more
and more memory. *(And such leaks are perfectly possible with
garbage collection, just not as likely.)

--
James Kanze (GABI Software) * * * * * * email:[e-mail]
Conseils en informatique orientée objet/
* * * * * * * * * *Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

Sigh! I was hoping for a recommendation of a memory pool, not a flame
war^h^h^h^h^h^h^h^h^h learned discourse :-)

Of course, that won't stop me from jumping in :-)

Since it's an embedded application it is never meant to terminate; it
is supposed to run "forever".

My definition of leak (ymmv) is when a process allocates memory which
no process ever frees.

I hope to test for this at unit test, s/w integration and system
verification test - so, yes, my memory pool should help me test for
leaks (even if I have to add a helper process to scan the pool when
the system is otherwise idle).

When testing has found all(!) memory leaks, then garbage collection is
obviously useful. So, they are not mutually elusive, although they are
likely to be used at different times.

Now, back to the practical... Does anyone know of a decent memory pool
in C/C++ ? The best which I have found so far is Boost++ which I
propose to extend. I just wondered if there is another which is more
suited to my porpoises.

Btw, I am also on the look out for an efficient debug trace system for
embedded software...

Thanks for the replies; I did enjoy the discussion.

[James Kanze]

On Jul 28, 3:51 am, [e-mail] wrote:

On Jul 26, 5:10 am, James Kanze <[e-mail]:

[...]

Since it's an embedded application it is never meant to
terminate; it is supposed to run "forever".

My definition of leak (ymmv) is when a process allocates
memory which no process ever frees.

Even if it only does it once, on start-up? By this definition,
I've never seen a C++ program which didn't leak.

The buffers used by cin, cout and cerr are never freed. Many
implementations of the STL also leak by this definition. With
older versions of Sun CC (I've not checked recently), array new
leaked.

I hope to test for this at unit test, s/w integration and
system verification test - so, yes, my memory pool should help
me test for leaks (even if I have to add a helper process to
scan the pool when the system is otherwise idle).

I don't use a memory pool for this, but a replacement operator
new/operator delete (which of course supposes that all
allocations are done with new and delete, and not malloc and
free). From experience with using it, with a number of
different implementations of C++ and its libraries, I've found
that 1) you have to wait until after start up to turn it on,
2) you may need to artificially exercise some components before
turning it on (e.g. create a couple of vectors of string, with
strings of various lengths in them), and 3) have some way of
temporarily turning it off.

When testing has found all(!) memory leaks, then garbage
collection is obviously useful. So, they are not mutually
elusive, although they are likely to be used at different
times.

Now, back to the practical... Does anyone know of a decent
memory pool in C/C++ ? The best which I have found so far is
Boost++ which I propose to extend. I just wondered if there is
another which is more suited to my porpoises.

It depends on what level you're looking for. I use the Boehm
collector a lot, both for garbage collection and leak detection.
I use my own debugging operator new/operator delete, not just
for leak detection, but for simulating out of memory situations,
ensuring that dangling pointers don't accidentally work, etc.
(My debugging operators are available at my site:
http://kanze.james.neuf.fr/code-en.html; they're part of the
Test component, but you'll need the Port component as well if
you want to compile them---unlike some other components, the
memory checking component depends very heavily on some of the
code in Port. For the Boehm collector, see
http://www.hpl.hp.com/personal/Hans_Boehm/gc/.)

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

[graham.keellings]

On Jul 28, 3:46*pm, James Kanze <[e-mail]:

On Jul 28, 3:51 am, [e-mail] wrote:

On Jul 26, 5:10 am, James Kanze <[e-mail]:

* * [...]

Since it's an embedded application it is never meant to
terminate; it is supposed to run "forever".
My definition of leak (ymmv) is when a process allocates
memory which no process ever frees.

Even if it only does it once, on start-up? *By this definition,
I've never seen a C++ program which didn't leak.

Yes, of course, applications dynamically allocate their "Static"
memory at startup, before entering their main loop. And, yes, if you
want to split hairs, that memory will not be freed, so you can call it
a "leak". But we both know that it's not a bug ...

The buffers used by cin, cout and cerr are never freed. *Many
implementations of the STL also leak by this definition. *With
older versions of Sun CC (I've not checked recently), array new
leaked.

I hope to test for this at unit test, s/w integration and
system verification test - so, yes, my memory pool should help
me test for leaks (even if I have to add a helper process to
scan the pool when the system is otherwise idle).

I don't use a memory pool for this, but a replacement operator
new/operator delete (which of course supposes that all
allocations are done with new and delete, and not malloc and
free). *

Unfortunately, I am stuck with a mix of C and C++.

From experience with using it, with a number of
different implementations of C++ and its libraries, I've found
that 1) you have to wait until after start up to turn it on,

Yes, that makes sense from the discussion above.

2) you may need to artificially exercise some components before
turning it on (e.g. create a couple of vectors of string, with
strings of various lengths in them), and 3) have some way of
temporarily turning it off.

When testing has found all(!) memory leaks, then garbage
collection is obviously useful. So, they are not mutually
elusive, although they are likely to be used at different
times.
Now, back to the practical... Does anyone know of a decent
memory pool in C/C++ ? * The best which I have found so far is
Boost++ which I propose to extend. I just wondered if there is
another which is more suited to my porpoises.

It depends on what level you're looking for. *I use the Boehm
collector a lot, both for garbage collection and leak detection.

Looks like http://www.hpl.hp.com/personal/Hans_Boehm/gc/ I will
check it out, thanks.

I use my own debugging operator new/operator delete, not just
for leak detection, but for simulating out of memory situations,
ensuring that dangling pointers don't accidentally work, etc.
(My debugging operators are available at my site:http://kanze.james.neuf.fr/code-en.html;

Thanks. I will check this out.

they're part of the
Test component, but you'll need the Port component as well if
you want to compile them---unlike some other components, the
memory checking component depends very heavily on some of the
code in Port. *For the Boehm collector, see http://www.hpl.hp.com/personal/Hans_Boehm/gc/.)

--
James Kanze (GABI Software) * * * * * * email:[e-mail]
Conseils en informatique orientée objet/
* * * * * * * * * *Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]

Thanks, James

[James Kanze]

On Jul 29, 4:01 am, [e-mail] wrote:

On Jul 28, 3:46 pm, James Kanze <[e-mail]:

[...]

I don't use a memory pool for this, but a replacement operator
new/operator delete (which of course supposes that all
allocations are done with new and delete, and not malloc and
free).

Unfortunately, I am stuck with a mix of C and C++.

In practice, you can usually replace malloc/free (although
unlike the case of new/delete, this isn't guaranteed by the
standard). I chose to replace new/delete because I don't have
any C (of my own), it is guaranteed by the standard, and above
all, because it's much easier to do when you can use malloc/free
for the actual allocation and deallocation. The same mechanisms
can easily be applied to malloc/free, but you'll have to
implement your own allocator as well. (With the GNU linker, I
think that there's an option to rename symbols in a library, so
you could rename the standard malloc/free to something else, and
use them. Or just take the sources, and modify them.)

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]