hard memory limits

[Maurice LING]

Hi,

I think I've hit a system limit in python when I try to construct a list
of 200,000 elements. My error is

malloc: vm_allocate (size = 2400256) failed......

Just wondering is this specific to my system or what? Will adding more
RAM helps in this case?

Thanks and cheers
Maurice

[Fredrik Lundh]

Maurice LING wrote:

I think I've hit a system limit in python when I try to construct a list
of 200,000 elements.
there's no such limit in Python.
My error is

malloc: vm_allocate (size = 2400256) failed......

Just wondering is this specific to my system or what?
that doesn't look like a Python error (Python usually raises
MemoryError exceptions when it runs out of memory), and
there's no sign of any vm_allocate function in the Python
sources, so yes, it's a system-specific problem.
Will adding more RAM helps in this case?

probably. more swap space might also help. or you could use a
smarter malloc package. posting more details on your platform,
toolchain, python version, and list building approach might also
help.

(are you perhaps building multiple lists piece by piece, interleaved
with other object allocations? if so, it's probably a fragmentation
problem. to check this, watch the process size. if if grows at a
regular rate, and then explodes just before you get the above error,
you may need to reconsider the design a bit).

</F>

[Mike Meyer]

"Fredrik Lundh" <fr*****@python ware.com> writes:

Maurice LING wrote:

Will adding more RAM helps in this case?

probably. more swap space might also help. or you could use a
smarter malloc package. posting more details on your platform,
toolchain, python version, and list building approach might also
help.

Without platform information, it's hard to say. On a modern Unix
system, you only run into system resource limits when the system is
heavily loaded. Otherwise, you're going to hit per-process limits. In
the latter case, adding RAM or swap won't help at all. Raising the
per-process limits is the solution.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Bill Mill]

On 5/6/05, Mike Meyer <mw*@mired.or g> wrote:

"Fredrik Lundh" <fr*****@python ware.com> writes:

Maurice LING wrote:

Will adding more RAM helps in this case?
probably. more swap space might also help. or you could use a
smarter malloc package. posting more details on your platform,
toolchain, python version, and list building approach might also
help.

Without platform information, it's hard to say. On a modern Unix
system, you only run into system resource limits when the system is
heavily loaded. Otherwise, you're going to hit per-process limits. In
the latter case, adding RAM or swap won't help at all. Raising the
per-process limits is the solution.

A quick google shows it to be mac os X, and a pretty frequent error message..

http://www.google.com/search?hl=en&q...=Google+Search

Peace
Bill Mill
bill.mill at gmail.com
<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.
--
http://mail.python.org/mailman/listinfo/python-list

[Fredrik Lundh]

Mike Meyer wrote:

Without platform information, it's hard to say. On a modern Unix
system, you only run into system resource limits when the system is
heavily loaded. Otherwise, you're going to hit per-process limits. In
the latter case, adding RAM or swap won't help at all. Raising the
per-process limits is the solution.

does Mac OS X ship with memory limits set by default? isn't that
a single-user system?

</F>

[James Stroud]

On Friday 06 May 2005 10:29 am, Fredrik Lundh wrote:

Mike Meyer wrote:

Without platform information, it's hard to say. On a modern Unix
system, you only run into system resource limits when the system is
heavily loaded. Otherwise, you're going to hit per-process limits. In
the latter case, adding RAM or swap won't help at all. Raising the
per-process limits is the solution.

does Mac OS X ship with memory limits set by default? isn't that
a single-user system?

</F>

Dear original poster or whoever is interested in OS X:

OS X is not a single user system. It is BSD based unix. And its fv@king
sweeeeeeeeeeeee et! (Though I'm using only Linux right now :o/

If configurable memory limits are a problem and if running python from the
shell, do:

% unlimit

You can also change this in your .cshrc, .tcshrc, .bashrc, .k[whatever for
korn], etc. if you run a custom shell.

Your shell settings for each user are in NetInfo Manager.

If you are completely clueless as to what the hell I'm talking about, then
stop and try this:

1. start a Terminal
2. type this at the prompt:

% echo unlimit >> .bashrc

3. Quit that terminal.
4. Start a new terminal.
5. Start python and make your list.

Hope it works.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

[Fredrik Lundh]

James Stroud wrote:

does Mac OS X ship with memory limits set by default? isn't that
a single-user system?

Dear original poster or whoever is interested in OS X:

OS X is not a single user system. It is BSD based unix. And its fv@king
sweeeeeeeeeeeee et! (Though I'm using only Linux right now :o/

Well, Apple's marketing materials contain no signs whatsoever that the
systems Apple sells are designed for massive numbers of users, compared
to systems from RedHat, Sun, HP, etc. (if you look at apple.com in this
very moment, it talks a lot about "your mac" and "your desktop" and "your
computer", not "the mac/desktop/computer you share with hundreds of
other users").

So why would Apple insist on setting unusably low process limits, when
the others don't?

</F>

[James Stroud]

On Friday 06 May 2005 11:27 am, Fredrik Lundh wrote:

James Stroud wrote:

does Mac OS X ship with memory limits set by default? isn't that
a single-user system?

Dear original poster or whoever is interested in OS X:

OS X is not a single user system. It is BSD based unix. And its fv@king
sweeeeeeeeeeeee et! (Though I'm using only Linux right now :o/

Well, Apple's marketing materials contain no signs whatsoever that the
systems Apple sells are designed for massive numbers of users, compared
to systems from RedHat, Sun, HP, etc. (if you look at apple.com in this
very moment, it talks a lot about "your mac" and "your desktop" and "your
computer", not "the mac/desktop/computer you share with hundreds of
other users").

So why would Apple insist on setting unusably low process limits, when
the others don't?

</F>

I think that two different markets exist for this computer:

1. Joe user who has never seen a command line interface. These people need a
nice, cozy little user environment that takes as little understanding as
possible to use. They also buy the most computers and are probably most
responsive to fluffy advertising campaigns. Hence the targeted advertising on
apple.com In this case, my guess is that memory allocation, etc, is left to
the application. For cocoa apps it is the objective c runtime handling this
kind of thing and for carbon apps, it is probobably tacked on during the
process of carbonizing. But I should say that I really don't know much about
the low level workings of either.

2. Scientists/Engineers/Programmer types. These people configure their own
limits instinctively and probably forgot that they ever put that unlimit in
their rc files (like I did) and the dozens of other customizations they did
to get their OS X boxes just so for unix use. When Joe User makes the
crossover, such customizations don't seem very intuitive. Plus, I remember
having to ulimit my IRIX account on SGIs I used back in the day--so other
*nixes seem to have similar requirements.

To answer your question, my guess is that no one has complained yet.

James

--
James Stroud
UCLA-DOE Institute for Genomics and Proteomics
Box 951570
Los Angeles, CA 90095

http://www.jamesstroud.com/

[Mike Meyer]

"Fredrik Lundh" <fr*****@python ware.com> writes:

James Stroud wrote:

> does Mac OS X ship with memory limits set by default? isn't that
> a single-user system?

Dear original poster or whoever is interested in OS X:

OS X is not a single user system. It is BSD based unix. And its fv@king
sweeeeeeeeeeeee et! (Though I'm using only Linux right now :o/

So why would Apple insist on setting unusably low process limits, when
the others don't?

You're making an unwarranted assumption here - that the OP wasn't
creating a large process of some kind. IIRC, all we ever saw was the
size of the request that triggered the error, with no indication of
the total process size.

FWIW, OS X has a Mach kernel. The failing vm_malloc call listed in the
OP is a Mach call, not a Unix call. These days, the userland code is
largely FreeBSD. It used to include "the best of" OpenBSD, NetBSD and
FreeBSD at a relatively small level, but that headache was dropped in
favor of tracking one external system. The legacy of the mixed
heritage is utilities from NetBSD and OpenBSD that aren't in
FreeBSD. shlock comes to mind as an obvious example.

<mike
--
Mike Meyer <mw*@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[John Machin]

On Fri, 06 May 2005 18:24:21 +1000, Maurice LING <ma*********@ac m.org>
wrote:

Hi,

I think I've hit a system limit in python when I try to construct a list
of 200,000 elements. My error is

malloc: vm_allocate (size = 2400256) failed......

Just wondering is this specific to my system or what? Will adding more
RAM helps in this case?

Not if it's an OS limit (see other posts). Not if you are doing
something so weird or drastic that you will use up the extra RAM and
still get the same message.

If you were to reply to Fredrik's question (HOW are you creating your
list), and this one: WHAT is an "element", we might be able to help
you avoid a trip to the Apple dealer.

As a bit of a reality check for you:
[numbers based on a 32-bit machine, CPython]

An extra list of 200000 ints will take up 800000 bytes (plus small
change) if the ints are all in range(-1, 101) and thus cached --
that's a 4-byte pointer (i.e. PyObject *) each.

If all the ints are outside that range, and are distinct, they'll take
(worst case) 16 bytes each (assuming you aren't using a debug build of
Python). The additional 12 bytes are for the int object: a reference
counter, a pointer to the type object, and the actual value. So you're
looking at approx 3.2MB.

In general, reckon on each element taking up 12+sizeof(eleme nt_value).

I'd suspect that your "elements" are not quite as elementary as ints,
and/or you are doing a whole lot of *other* memory allocation. Confess
all ...

Cheers,
John