How many items can be stored in a Python list? I have close to 70,000
items... is this within a lists limits? 5 2307
It is possible to store 70,000 items in a list (try "l =
range(70000)"), but the best way to check if you can store all the
items you need to store is to try it. After all if they are all very
large you might potentially run out of memory.
Chris
I'm referring to Python 2.2's C headers as I answer this question. I
believe some of may have changed by 2.4.
The number of elements in a "variable-sized object" (those with
Py_VAR_HEAD; I believe this includes lists, tuples, and strings) is
stored in a platform 'int'.
On most (desktop) systems, this means the limit of any sized object is
no more than 2**31-1, or about 2 billion.
On those same systems, a list object of length 128 million, give or
take, approximately 512 megabytes of memory will be allocated for the
list object itself (4 bytes for each pointer-to-element). If each
element of the list is distinct, those objects will each require
additional memory---The smallest useful Python object takes around 16
bytes, IIRC, which would bring the total memory required to around 2560
megabytes if I didn't screw up my arithmetic. This is close to (or
over, depending on the system) the maximum amount of physical RAM these
machines can accomodate, and the maximum amount of address space
available to a single program.
While performing list-resizing operations, there may be a temporary need
for two copies of the list object itself, bumping the memory used up to 3
gigs.
Finally, the speed of some operations (l.index(item), l.pop(0),
l.insert(0, item)) are related linearly to the size of the list, so your
program may slow down as the lists it manipulates grow. Others, such as
l[i], l.pop(), and l.append(item), are constant-time or amortized-constant-
time.
Jeff
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)
iD8DBQFBxw6pJd01MZaTXX0RAljUAJoD8/bR3hBgQOb3I40h+yKm5r9PqQCdEiBo
s1rX4NrllE9a+vWoQQpqFe4=
=F8Db
-----END PGP SIGNATURE-----
> How many items can be stored in a Python list? I have close to 70,000 items... is this within a lists limits?
Lists store pointers to objects. Unless you have a lot of duplicates, it is the
objects themselves that will consume most of your memory. The list itself will
likely be small in comparison.
Raymond Hettinger
Raymond Hettinger wrote: How many items can be stored in a Python list? I have close to 70,000 items... is this within a lists limits?
Lists store pointers to objects. Unless you have a lot of duplicates, it is the objects themselves that will consume most of your memory. The list itself will likely be small in comparison.
Given the size of the counter, is it actually physically possible for a list to
run out of room before the application runs out memory?
Even list(None for x in xrange(sys.maxint)) wouldn't do the trick, since each of
those pointers to None is taking 4 bytes of memory, and Python's internal
structures are already chewing up some of the address space.
Cheers,
Nick.
--
Nick Coghlan | nc******@email.com | Brisbane, Australia
--------------------------------------------------------------- http://boredomandlaziness.skystorm.net
Nick Coghlan wrote: Given the size of the counter, is it actually physically possible for a list to run out of room before the application runs out memory?
depends on the system architecture, of course: consider an 64-bit computer
with 32-bit integers and 256 GB of memory...
</F> This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Jason |
last post by:
Hi,
I have a few million data items being received by my program and I wish to
store them in a list, with each item being inserted in any position in the
list. Any performance tips so that my...
|
by: Ross A. Finlayson |
last post by:
Hi,
I hope you can help me understand the varargs facility.
Say I am programming in ISO C including stdarg.h and I declare a
function as so:
void log_printf(const char* logfilename, const...
|
by: zacks |
last post by:
I have written a serialized class that has several properties that are
typed as a list of type class. When I deserialize an XML file, the list
is populated just fine. But I am having trouble...
|
by: Spiro Trikaliotis |
last post by:
Hello,
does anyone know of any list which presents the headers that differ
between C90 and c99, and where they differ? Searching on the web, I
found many web sites about the differences between...
|
by: dmp |
last post by:
What are Linked list? Please somebody show some ready made programs of
linked list
|
by: santosh |
last post by:
Hello all,
In K&R2 one exercise asks the reader to compute and print the limits for
the basic integer types. This is trivial for unsigned types. But is it
possible for signed types without...
|
by: Josip |
last post by:
I'm trying to limit a value stored by object (either int or float):
class Limited(object):
def __init__(self, value, min, max):
self.min, self.max = min, max
self.n = value
def...
|
by: pereges |
last post by:
Hi, I am wondering which of the two data structures (link list or
array) would be better in my situation. I have to create a list of
rays for my ray tracing program.
the data structure of ray...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new...
| |