By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
458,068 Members | 867 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 458,068 IT Pros & Developers. It's quick & easy.

Software memory management reengineering

P: n/a
Suppose that you have a module that always allocates
memory without ever releasing it because the guy that
wrote it was lazy, as lazy as me.

Now, you want to reuse it in a loop. What do you do?

Contrary to some people that will start crying to that
&@@""#~ programmer that wrote this sh!!!! you keep
your cool and you do the following:

1) Since the program always call malloc and never free()
you allocate a big chunk of memory and each time that the
program calls malloc, you replace those calls with another
that allocates from that memory heap.

2) When the program finishes you free all memory witha single
free() call.

Caveats. The program should never call directly
realloc. Watch for calloc too.

I have done this many times. Actually programs that use the
lazy memory allocation strategy are very easy to convert to a heap
based strategy if the need ever arises.

Programs that leak memory but contain a lots of calls to free()
are much harder to modify like this, since you would have to
repeat the functionality of free().

Of course, a lazy strategy is much easier to upgrade to a GC
based one. In that case absolutely no code needs to be
modified. malloc is replaced with GC_malloc or similar,
and there is nothing else to do.

--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32
Oct 13 '07 #1
Share this Question
Share on Google+
34 Replies


P: n/a
jacob navia wrote:
Suppose that you have a module that always allocates
memory without ever releasing it because the guy that
wrote it was lazy, as lazy as me.

Now, you want to reuse it in a loop. What do you do?

Contrary to some people that will start crying to that
&@@""#~ programmer that wrote this sh!!!! you keep
your cool and you do the following:

1) Since the program always call malloc and never free()
you allocate a big chunk of memory and each time that the
program calls malloc, you replace those calls with another
that allocates from that memory heap.
How does that help if the new application calls leaky parts of the old
one enough times to exhaust the heap?

--
Ian Collins.
Oct 13 '07 #2

P: n/a
Ian Collins said:
jacob navia wrote:
>Suppose that you have a module that always allocates
memory without ever releasing it because the guy that
wrote it was lazy, as lazy as me.

Now, you want to reuse it in a loop. What do you do?

Contrary to some people that will start crying to that
&@@""#~ programmer that wrote this sh!!!! you keep
your cool and you do the following:

1) Since the program always call malloc and never free()
you allocate a big chunk of memory and each time that the
program calls malloc, you replace those calls with another
that allocates from that memory heap.
How does that help if the new application calls leaky parts of the old
one enough times to exhaust the heap?
It doesn't, obviously. The correct fix is to plough through the code,
working out the right places to free up the memory - which would have been
a lot easier (and therefore cheaper) if it had been done at the time the
original code was written. Writing your own memory manager is not a fix -
it's an error-prone bodge that doesn't address the real issue, as you
rightly point out.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 13 '07 #3

P: n/a
jacob wrote:
) 1) Since the program always call malloc and never free()
) you allocate a big chunk of memory and each time that the
) program calls malloc, you replace those calls with another
) that allocates from that memory heap.
) <snip>
) Programs that leak memory but contain a lots of calls to free()
) are much harder to modify like this, since you would have to
) repeat the functionality of free().

No, you just make free() a no-op. That will make it leak a lot more
memory, but you've already fixed that, right ?
SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
Oct 13 '07 #4

P: n/a
Richard Heathfield <rj*@see.sig.invalidwrites:
Writing your own memory manager is not a fix - it's an
error-prone bodge that doesn't address the real issue, as you
rightly point out.
I'm rather a fan of pool-based memory allocation, as done in
e.g. Apache: add memory blocks (and other resources) to a list,
for convenient freeing in a group.

Would you contend that this is also an "error-prone bodge"? If
not, at what point does pool-based allocation descend to that
level?
--
A competent C programmer knows how to write C programs correctly,
a C expert knows enough to argue with Dan Pop, and a C expert
expert knows not to bother.
Oct 13 '07 #5

P: n/a
Ben Pfaff said:
Richard Heathfield <rj*@see.sig.invalidwrites:
>Writing your own memory manager is not a fix - it's an
error-prone bodge that doesn't address the real issue, as you
rightly point out.

I'm rather a fan of pool-based memory allocation, as done in
e.g. Apache: add memory blocks (and other resources) to a list,
for convenient freeing in a group.

Would you contend that this is also an "error-prone bodge"? If
not, at what point does pool-based allocation descend to that
level?
At the point where you try to retrofit it into a program (such as I
described upthread) that was written without pool-based allocation in
mind, in the hope of magically fixing the failure to deal with memory
housekeeping properly. If pool-based allocation is part of a sensible,
coherent design strategy that is part of the original program design,
that's obviously a very different matter.

The point is that you have to think about this stuff up front. To malloc
all over the place and never bother to free is sloppy programming, and
overlaying a customised memory manager on top of that mess is what I
referred to as "an error-prone bodge".

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 13 '07 #6

P: n/a
Richard Heathfield wrote:
Ben Pfaff said:
[snip]
>... at what point does pool-based allocation descend to that
level?

At the point where you try to retrofit it into a program (such as I
described upthread) that was written without pool-based allocation in
mind, in the hope of magically fixing the failure to deal with memory
housekeeping properly. If pool-based allocation is part of a sensible,
coherent design strategy that is part of the original program design,
that's obviously a very different matter.
This is wrong.

In a pool managed memory allocation strategy you make a lot of
allocations that go into the same pool, then release the pool.

This is exactly the same as a program that is "retrofitted" to
do this because it never frees any memory. The only part that is
changed is that at the end of the program, instead of leaving the
freeing of the pool to the operating system, it is done
explicitely.

Note that lazy allocation is always actually a pool based allocation
since the operating system maintains a pool for each process.
The point is that you have to think about this stuff up front.
This is wrong.

There is absolutely no need to over engineer a program, making it manage
memory up front. You allocate memory without freeing it. If this ever
becomes a problem you add a pool based strategy or a garbage collector.

The lazy allocation strategy is adapted to transient programs, where the
memory pool is released by the operating system.

Utilities that produce directory listings, linkers, compilers, filters,
text searching tools, there are many examples of transient programs.
In many of them, ignoring memory allocation problems makes them faster,
leaner, and less error prone, since all the bugs associated with
"free()" calls can be avoided.


--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32
Oct 13 '07 #7

P: n/a

jacob navia wrote:
Richard Heathfield wrote:
The point is that you have to think about this stuff up front.

This is wrong.

There is absolutely no need to over engineer a program, making it manage
memory up front. You allocate memory without freeing it. If this ever
becomes a problem you add a pool based strategy or a garbage collector.
Navia really would make a fascinating subject for a psychologist. He
makes himself look like a complete fool by taking some utterly
ludicrous position, then when he gets called on it he just digs
himself further and further into the hole.

The killer is that he comes out with this incompetence and dribbling
nonsense and then has the chutzpah to write a tutorial trying to teach
other people how to write C!

Oct 13 '07 #8

P: n/a
jacob navia wrote:
Richard Heathfield wrote:
>The point is that you have to think about this stuff up front.

This is wrong.

There is absolutely no need to over engineer a program, making it manage
memory up front. You allocate memory without freeing it. If this ever
becomes a problem you add a pool based strategy or a garbage collector.
I can see you have never worked on embedded systems.
Utilities that produce directory listings, linkers, compilers, filters,
text searching tools, there are many examples of transient programs.
In many of them, ignoring memory allocation problems makes them faster,
leaner, and less error prone, since all the bugs associated with
"free()" calls can be avoided.
You fail to realise that your leaky directory listing routine may get
invoked recursively in something like a backup utility, or get called
hundreds or even thousands of times in a test suite. Just clean up your
mess, it isn't that hard.

--
Ian Collins.
Oct 13 '07 #9

P: n/a
jacob navia said:
Richard Heathfield wrote:
>Ben Pfaff said:
[snip]
>>... at what point does pool-based allocation descend to that
level?

At the point where you try to retrofit it into a program (such as I
described upthread) that was written without pool-based allocation in
mind, in the hope of magically fixing the failure to deal with memory
housekeeping properly. If pool-based allocation is part of a sensible,
coherent design strategy that is part of the original program design,
that's obviously a very different matter.
This is wrong.
No, it isn't. You need to learn to tell right from wrong. For the past few
years, you appear to have got them mixed up with monotonous regularity.
>
In a pool managed memory allocation strategy you make a lot of
allocations that go into the same pool, then release the pool.
Fine, provided there's enough memory, which - in the example I gave - there
wasn't.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 13 '07 #10

P: n/a
Ian Collins wrote:
jacob navia wrote:
>Richard Heathfield wrote:
>>The point is that you have to think about this stuff up front.
This is wrong.

There is absolutely no need to over engineer a program, making it manage
memory up front. You allocate memory without freeing it. If this ever
becomes a problem you add a pool based strategy or a garbage collector.
I can see you have never worked on embedded systems.
I am speaking of transient programs, i.e. programs that do
something, then exit. Embedded systems do have transient programs
if they have an OS. Otherwise they don't.
>Utilities that produce directory listings, linkers, compilers, filters,
text searching tools, there are many examples of transient programs.
In many of them, ignoring memory allocation problems makes them faster,
leaner, and less error prone, since all the bugs associated with
"free()" calls can be avoided.
You fail to realise that your leaky directory listing routine may get
invoked recursively in something like a backup utility, or get called
hundreds or even thousands of times in a test suite.
A transient program, as I have explained several times already, leaves
its cleanup to the operating system. Thus, it can be called thousands
of times, it will always cleanup implicitly.

I can repeat this again and again and apparently there will be always
the same "argument"...

Even in a recursive call if you call my program with some way like
system("dodir"); or whatever, it will always cleanup! And this will not
change even if it is called hundred billion times!
--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32
Oct 13 '07 #11

P: n/a
jacob navia said:

<snip>
A transient program, as I have explained several times already, leaves
its cleanup to the operating system. Thus, it can be called thousands
of times, it will always cleanup implicitly.
I have already given an example, upthread, of what you call a "transient"
program which became a "non-transient" program five years later. Skimp on
the housekeeping now, and pay for it many times over in the future.
I can repeat this again and again and apparently there will be always
the same "argument"...
No matter how many times you repeat an incorrect argument, it remains
incorrect.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 13 '07 #12

P: n/a
jacob navia wrote:
Ian Collins wrote:
>jacob navia wrote:
>>Richard Heathfield wrote:

The point is that you have to think about this stuff up front.
This is wrong.

There is absolutely no need to over engineer a program, making it manage
memory up front. You allocate memory without freeing it. If this ever
becomes a problem you add a pool based strategy or a garbage collector.
I can see you have never worked on embedded systems.

I am speaking of transient programs, i.e. programs that do
something, then exit. Embedded systems do have transient programs
if they have an OS. Otherwise they don't.
"Transient programs" contain components. Those components, if they are
any use, will get reused. Unless you stick a great big fluorescent
label on them "Danger, the authour was too slack to free memory" the
component may be reused in something other than a "transient programs".
>You fail to realise that your leaky directory listing routine may get
invoked recursively in something like a backup utility, or get called
hundreds or even thousands of times in a test suite.

A transient program, as I have explained several times already, leaves
its cleanup to the operating system. Thus, it can be called thousands
of times, it will always cleanup implicitly.
I wasn't talking about the program, I was talking about the leaky
routine that reads a directory. Or do you write all of you code in main?

--
Ian Collins.
Oct 13 '07 #13

P: n/a
Ian Collins wrote:
I wasn't talking about the program, I was talking about the leaky
routine that reads a directory. Or do you write all of you code in main?
This is the problem. I am always talking about transient
*program* and you are talking about some routine!

I repeat:

Lazy allocation can be done only at the program level. I have
always maintained this.

--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32
Oct 13 '07 #14

P: n/a
jacob navia wrote:
Ian Collins wrote:
>I wasn't talking about the program, I was talking about the leaky
routine that reads a directory. Or do you write all of you code in main?

This is the problem. I am always talking about transient
*program* and you are talking about some routine!

I repeat:

Lazy allocation can be done only at the program level. I have
always maintained this.
But what is is program if it isn't a collection of routines?

--
Ian Collins.
Oct 13 '07 #15

P: n/a
Ian Collins wrote:
jacob navia wrote:
>I repeat:

Lazy allocation can be done only at the program level. I have
always maintained this.
But what is is program if it isn't a collection of routines?
Maybe you haven't noticed it yet, but after a program finishes the
operating system cleans up. Not so for a routine.

And you know what?

"There is no blinder person as the one that doesn't want to see".
--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32
Oct 13 '07 #16

P: n/a
jacob navia wrote:
Ian Collins wrote:
>jacob navia wrote:
>>I repeat:

Lazy allocation can be done only at the program level. I have
always maintained this.
But what is is program if it isn't a collection of routines?

Maybe you haven't noticed it yet, but after a program finishes the
operating system cleans up. Not so for a routine.
So? Enough of this, I'm off to find an open pub and celebrate England's
victory over France in the wold cup....

--
Ian Collins.
Oct 13 '07 #17

P: n/a
jacob navia wrote:
And you know what?

"There is no blinder person as the one that doesn't want to see".
Said without the slightest trace of irony...

Oct 13 '07 #18

P: n/a
jacob wrote:
) This is the problem. I am always talking about transient
) *program* and you are talking about some routine!
)
) I repeat:
)
) Lazy allocation can be done only at the program level. I have
) always maintained this.

Maybe I should butt in and also repeat:

Programs often get reused and turned (partly) into routines.
SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
Oct 13 '07 #19

P: n/a
Richard Heathfield wrote:
jacob navia said:

<snip>
>A transient program, as I have explained several times already, leaves
its cleanup to the operating system. Thus, it can be called thousands
of times, it will always cleanup implicitly.

I have already given an example, upthread, of what you call a "transient"
program which became a "non-transient" program five years later. Skimp on
the housekeeping now, and pay for it many times over in the future.
Engineering is about balancing considerations that are
sometimes in conflict. No one consideration or goal is always
paramount. Besides, even when "The program must be easy to
modify" is important, the designer faces the challenge of
trying to anticipate the likely modifications. Sometimes
the crystal ball works, sometimes it's cloudy, and hindsight
is always twenty-twenty.
>I can repeat this again and again and apparently there will be always
the same "argument"...

No matter how many times you repeat an incorrect argument, it remains
incorrect.
"Incorrect" is too absolute a term to apply here. Can
you think of a good reason for the Unix `shutdown' program
to free() its allocated memory before returning? Yes, a
leaky `shutdown' would be hard to use in a loop, but ...

For what it's worth, many of my own programs follow the
general line Jacob describes, at least in part. "Library"
or "utility" functions are scrupulous about their memory
bookkeeping because they don't know the context they run in;
they're careful about memory for the same reasons they don't
call abort(). But if there's a bunch of long-lived data
that represents the program's "global state," I see nothing
wrong with letting exit() clean it up.

--
Eric Sosman
es*****@ieee-dot-org.invalid
Oct 13 '07 #20

P: n/a
Eric Sosman said:
Richard Heathfield wrote:
>jacob navia said:

<snip>
>>A transient program, as I have explained several times already, leaves
its cleanup to the operating system. Thus, it can be called thousands
of times, it will always cleanup implicitly.

I have already given an example, upthread, of what you call a
"transient" program which became a "non-transient" program five years
later. Skimp on the housekeeping now, and pay for it many times over in
the future.

Engineering is about balancing considerations that are
sometimes in conflict. No one consideration or goal is always
paramount.
Certainly true, but I would not classify "can't be bothered to manage
resources properly" as an engineering consideration, any more than I would
classify "can't be bothered to wash the dishes" as being a household
management consideration.

<snip>
>No matter how many times you repeat an incorrect argument, it remains
incorrect.

"Incorrect" is too absolute a term to apply here.
Sorry, Eric, but I can't agree. I consider the argument he is making to be
incorrect.
Can
you think of a good reason for the Unix `shutdown' program
to free() its allocated memory before returning?
Actually, I can't even think of a good reason for 'shutdown' to allocate
memory dynamically.

<snip>

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 13 '07 #21

P: n/a
On Sun, 14 Oct 2007 07:56:40 +1300, Ian Collins
<ia******@hotmail.comwrote:
>jacob navia wrote:
>Richard Heathfield wrote:
>>The point is that you have to think about this stuff up front.

This is wrong.

There is absolutely no need to over engineer a program, making it manage
memory up front. You allocate memory without freeing it. If this ever
becomes a problem you add a pool based strategy or a garbage collector.
I can see you have never worked on embedded systems.
>Utilities that produce directory listings, linkers, compilers, filters,
text searching tools, there are many examples of transient programs.
In many of them, ignoring memory allocation problems makes them faster,
leaner, and less error prone, since all the bugs associated with
"free()" calls can be avoided.
You fail to realise that your leaky directory listing routine may get
invoked recursively in something like a backup utility, or get called
hundreds or even thousands of times in a test suite. Just clean up your
mess, it isn't that hard.
While what you say is quite true, you are talking about something
different than what Jacob is talking about. Jacob is talking
about transient *programs*, which you changed into *routines*.
If I write a program that lists a single directory specified in
its calling sequence then it can't be "invoked recursively". The
backup utility executes the program multiple times, each
execution being a separate process (or equivalent concept in the
OS of your choice).

Jacob's notion is plausible on the surface - if you have a
collection of utilities that are bounded in their own right and
you connecting them in scripts, then one could save execution
time by not bothering to free storage. That said, there are
several objections that occur to me.

The first is that people do take standalone programs and convert
them to internal functions. What was acceptable in the original
is not in the conversion. Secondly, it is not necessarily the
case that a simple standalone program will have bounded memory
usage. For example, a program that reads lines from stdin and
prints will exhaust memory if a new buffer is allocated for each
line. Thirdly the savings in execution time are strictly
nominal; the costs of starting up and terminating these little
transient programs dominates the cost of freeing storage.

Richard Harter, cr*@tiac.net
http://home.tiac.net/~cri, http://www.varinoma.com
But the rhetoric of holistic harmony can generate into a kind of
dotty, Prince Charles-style mysticism. -- Richard Dawkins
Oct 13 '07 #22

P: n/a
Richard Heathfield wrote:
Eric Sosman said:
>[...]
Can
you think of a good reason for the Unix `shutdown' program
to free() its allocated memory before returning?

Actually, I can't even think of a good reason for 'shutdown' to allocate
memory dynamically.
Whether the reason is "good" or not I can't say, but
at least one version certainly does allocate dynamic memory
and does not free it. See line 711 of

http://cvs.opensolaris.org/source/xr...own/shutdown.c

From the copyright notices it appears that this module
is at least twenty-three years old; there are also stylistic
suggestions that its origins antedate the C Standard. Yet in
all that time, it seems no one has felt impelled to embed
`shutdown' as a routine in a larger program that could call
it in a loop! Maybe they would have, were it not for the
memory leak? What an awful thought: a `shutdown' that gobbled
up all the swap space and crashed the system ...

--
Eric Sosman
es*****@ieee-dot-org.invalid
Oct 13 '07 #23

P: n/a
Eric Sosman said:
Richard Heathfield wrote:
>Eric Sosman said:
>>[...]
Can
you think of a good reason for the Unix `shutdown' program
to free() its allocated memory before returning?

Actually, I can't even think of a good reason for 'shutdown' to allocate
memory dynamically.

Whether the reason is "good" or not I can't say, but
at least one version certainly does allocate dynamic memory
and does not free it. See line 711 of

http://cvs.opensolaris.org/source/xr...own/shutdown.c

<cough:-)
From the copyright notices it appears that this module
is at least twenty-three years old; there are also stylistic
suggestions that its origins antedate the C Standard. Yet in
all that time, it seems no one has felt impelled to embed
`shutdown' as a routine in a larger program that could call
it in a loop!
I can't imagine why...
Maybe they would have, were it not for the
memory leak? What an awful thought: a `shutdown' that gobbled
up all the swap space and crashed the system ...
Yes, it's very bad practice - if you're unlucky, it might even shut down
the machine. Tut tut.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 13 '07 #24

P: n/a
Richard Heathfield <rj*@see.sig.invalidwrites:
[...]
Certainly true, but I would not classify "can't be bothered to manage
resources properly" as an engineering consideration, any more than I would
classify "can't be bothered to wash the dishes" as being a household
management consideration.
[...]

I don't wash paper plates before I throw them away.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
Oct 13 '07 #25

P: n/a
Keith Thompson said:
Richard Heathfield <rj*@see.sig.invalidwrites:
[...]
>Certainly true, but I would not classify "can't be bothered to manage
resources properly" as an engineering consideration, any more than I
would classify "can't be bothered to wash the dishes" as being a
household management consideration.
[...]

I don't wash paper plates before I throw them away.
That's the thing about analogies - they are illustrations, not proofs.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 14 '07 #26

P: n/a

"jacob navia" <ja***@nospam.orgwrote in message
news:47**********************@news.orange.fr...
Suppose that you have a module that always allocates
memory without ever releasing it because the guy that
wrote it was lazy, as lazy as me.

Now, you want to reuse it in a loop. What do you do?

Contrary to some people that will start crying to that
&@@""#~ programmer that wrote this sh!!!! you keep
your cool and you do the following:

1) Since the program always call malloc and never free()
you allocate a big chunk of memory and each time that the
program calls malloc, you replace those calls with another
that allocates from that memory heap.

2) When the program finishes you free all memory witha single
free() call.
actually, this is more or less what I do in my "lower compiler" (converts my
RPN-IL or RIL language to assembler).
it is convinient since the memory in question is not shared between runs.
so, I run the basic parts of the compiler, and simply "rewind" the heap when
I am done.

note that my "upper compiler", which works with ASTs, is garbage collected,
and my assembler/linker core, does its own (manual) memory management.
for anyone that knows:
this was also a major memory-management technique in Quake 1 and 2 (maybe Q3
and later, but I have not really investigated this). these engines actually
had 2 memory managers: the hunk (as above), and Z_Malloc (from what I
remember, a circular linked-list, rover, and memory-block based system,
basically, we scan along until we find a free block, and maybe split and
merge as needed, and a rover keeps track of where in the heap we are). the
hunk was used for most game data (maps, models, textures, wordstate, ...),
wheras the Z_Malloc system was used for longer-lived data (such as the
'cvars', or engine variables).

major cost:
one up-front commits a decent-sized chunk of memory to be used in this
manner, and "overflowing" is a potentially fatal condition (in my case, for
example if someone tries to compile some obscenely large source file...).

Caveats. The program should never call directly
realloc. Watch for calloc too.
the way to deal with realloc, is simply to relocate the object to the end of
the heap...
this can be done much the same as above, a wrapper to the custom allocator
and a memcpy (downsizing though can be made into a no-op).

I have done this many times. Actually programs that use the
lazy memory allocation strategy are very easy to convert to a heap
based strategy if the need ever arises.
maybe, maybe not.

Programs that leak memory but contain a lots of calls to free()
are much harder to modify like this, since you would have to
repeat the functionality of free().
or, very more the case: code that actually needs to keep track of the memory
after it finishes...
if we need to create, pass around, and return objects, then this approach is
totally ill-suited.

Of course, a lazy strategy is much easier to upgrade to a GC
based one. In that case absolutely no code needs to be
modified. malloc is replaced with GC_malloc or similar,
and there is nothing else to do.
ok.
--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32
Oct 14 '07 #27

P: n/a

"Eric Sosman" <es*****@ieee-dot-org.invalidwrote in message
>
But if there's a bunch of long-lived data
that represents the program's "global state," I see nothing
wrong with letting exit() clean it up.
Writing a clean-up function forces you to keep track of that global data's
pointer structure. If you can't do this easily it is a useful warning that
the program might be getting out of control.

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

Oct 14 '07 #28

P: n/a
Joachim Schmitz said:
"Richard Heathfield" <rj*@see.sig.invalidschrieb im Newsbeitrag
news:CZ******************************@bt.com...
>Keith Thompson said:
>>Richard Heathfield <rj*@see.sig.invalidwrites:
[...]
Certainly true, but I would not classify "can't be bothered to manage
resources properly" as an engineering consideration, any more than I
would classify "can't be bothered to wash the dishes" as being a
household management consideration.
[...]

I don't wash paper plates before I throw them away.

That's the thing about analogies - they are illustrations, not proofs.
It's not a bad analogy though: if you want to reuse the dish (or code),
do the washing (or free()ing), if you don't, just dump it.

It's quite easy to tell the difference between paper plates and china,
it's not equally easy to tell reusable code from non-reusable though.
Right. The Principle of Conservatism applies.

<good example snipped>

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 14 '07 #29

P: n/a
Richard Heathfield <rj*@see.sig.invalidwrote:
jacob navia said:

<snip>
>A transient program, as I have explained several times already, leaves
its cleanup to the operating system. Thus, it can be called thousands
of times, it will always cleanup implicitly.

I have already given an example, upthread, of what you call a "transient"
program which became a "non-transient" program five years later. Skimp on
the housekeeping now, and pay for it many times over in the future.
And besides, there is no guarantee that the operating system will clean up
after the program. Depending on the OS doing that is a bad idea if you want
your programs to be portable.

>
>I can repeat this again and again and apparently there will be always
the same "argument"...

No matter how many times you repeat an incorrect argument, it remains
incorrect.
--
<Insert your favourite quote here.>
Erik Trulsson
er******@student.uu.se
Oct 14 '07 #30

P: n/a
jacob navia <ja***@nospam.orgwrote:
Ian Collins wrote:
>jacob navia wrote:
>>I repeat:

Lazy allocation can be done only at the program level. I have
always maintained this.
But what is is program if it isn't a collection of routines?

Maybe you haven't noticed it yet, but after a program finishes the
operating system cleans up.
Not all operating systems will do that.
Not so for a routine.

And you know what?

"There is no blinder person as the one that doesn't want to see".

--
<Insert your favourite quote here.>
Erik Trulsson
er******@student.uu.se
Oct 14 '07 #31

P: n/a
On Sun, 14 Oct 2007 18:21:04 +0000, Erik Trulsson wrote:
jacob navia <ja***@nospam.orgwrote:
>Ian Collins wrote:
>>jacob navia wrote:
I repeat:

Lazy allocation can be done only at the program level. I have always
maintained this.

But what is is program if it isn't a collection of routines?

Maybe you haven't noticed it yet, but after a program finishes the
operating system cleans up.

Not all operating systems will do that.
True, but the operating systems that don't do that aren't necessarily
operating systems the software might ever be ported to. Another reason to
clean up everything, one that is more interesting to me personally, is
because if I ignore memory leaks that don't cause problems in practice,
tools such as valgrind have no way of warning me of memory leaks that do
cause problems in practice.
Oct 14 '07 #32

P: n/a
jacob navia wrote:
1) Since the program always call malloc and never free()
you allocate a big chunk of memory and each time that the
program calls malloc, you replace those calls with another
that allocates from that memory heap.

2) When the program finishes you free all memory witha single
free() call.
If you're already diverting malloc to your own replacement, then why not
making it do something useful? Such as trace where they are called from,
how much they allocate etc - and your final free equivalent can then
dump the statistics to aid you in fixing the source. There are
ready-made tools for that, but so are malloc and free :-)
Oct 14 '07 #33

P: n/a

"Peter Pichler" <us****@pichler.co.ukwrote in message
news:47**********@mk-nntp-2.news.uk.tiscali.com...
jacob navia wrote:
>1) Since the program always call malloc and never free()
you allocate a big chunk of memory and each time that the
program calls malloc, you replace those calls with another
that allocates from that memory heap.

2) When the program finishes you free all memory witha single
free() call.

If you're already diverting malloc to your own replacement, then why not
making it do something useful? Such as trace where they are called from,
how much they allocate etc - and your final free equivalent can then dump
the statistics to aid you in fixing the source. There are ready-made tools
for that, but so are malloc and free :-)
though this is true, this does not rule out the original idea as a
possibility.
there are some cases where variations on the original idea are sensible.

however, usually one does not free and re-allocated the memory, rather it is
kept around and re-used each time.
the main advantages: it can offer better performance, and can help avoid
fragmenting the main heap.
basically, we create garbage yes, but one can simply forget about it and it
is gone.

this is useful for temporary-use situations (such as in a dynamic compiler,
where once the output is produced, most of the intermediate state is,
simply, no longer relevant, even to the level of being properly freed).

so, we produce a mass of small objects defining the various declared
functions and variables (consider all the crap contained in a typical set of
system headers...), small chunks of intermediate bytecoded blocks,
register-allocator info, various stacks and contexts, ... easily tearing
through a good number of MB in a matter of milliseconds.

and on to the next source file, where we forget it all and start all over
again...
a tradeoff (more memory conservative at least), is to keep track of all the
allocations (say, in an array), or maybe to implement a segmented pool, and
then to free everything in the array/pool.

this is not to say all this is all that good of an idea (or even at all
relevant for most 'general' situations), but in certain cases, it works
pretty well...

or such...

Oct 14 '07 #34

P: n/a
cr88192 wrote:
"jacob navia" <ja***@nospam.orgwrote in message
>Suppose that you have a module that always allocates
memory without ever releasing it because the guy that
wrote it was lazy, as lazy as me.

Now, you want to reuse it in a loop. What do you do?

Contrary to some people that will start crying to that
&@@""#~ programmer that wrote this sh!!!! you keep
your cool and you do the following:

1) Since the program always call malloc and never free()
you allocate a big chunk of memory and each time that the
program calls malloc, you replace those calls with another
that allocates from that memory heap.

2) When the program finishes you free all memory witha single
free() call.
This is reminiscent of the mark/release used on SOME versions of
p-code interpreters for Pascal. It was sufficient to handle the
symbol tables of the compiler, but was sadly wanting for general
use. One of the unique features of PascalP was that it allowed
mark/release to be mixed with new/dispose, and all functioned
correctly.

I.e. experience shows that this is not a satisfactory technique.

--
Chuck F (cbfalconer at maineline dot net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home.att.net>

--
Posted via a free Usenet account from http://www.teranews.com

Oct 19 '07 #35

This discussion thread is closed

Replies have been disabled for this discussion.