Hi!
I would like to create a pretty big list of lists; a list 3,000,000
long, each entry containing 5 empty lists. My application will append
data each of the 5 sublists, so they will be of varying lengths (so no
arrays!).
Does anyone know the most efficient way to do this? I have tried:
list = [[[],[],[],[],[]] for _ in xrange(3000000)]
but its not soooo fast. Is there a way to do this without looping?
David. 19 8595
Dr Mephesto <dn****@googlem ail.comwrites:
Hi!
I would like to create a pretty big list of lists; a list 3,000,000
long, each entry containing 5 empty lists. My application will append
data each of the 5 sublists, so they will be of varying lengths (so no
arrays!).
Does anyone know the most efficient way to do this? I have tried:
list = [[[],[],[],[],[]] for _ in xrange(3000000)]
but its not soooo fast. Is there a way to do this without looping?
You can do:
[[[],[],[],[],[]]] * 3000000
although I don't know if it performs any better than what you already
have.
Paul Rudin wrote:
Dr Mephesto <dn****@googlem ail.comwrites:
>Hi!
I would like to create a pretty big list of lists; a list 3,000,000 long, each entry containing 5 empty lists. My application will append data each of the 5 sublists, so they will be of varying lengths (so no arrays!).
Does anyone know the most efficient way to do this? I have tried:
list = [[[],[],[],[],[]] for _ in xrange(3000000)]
but its not soooo fast. Is there a way to do this without looping?
You can do:
[[[],[],[],[],[]]] * 3000000
although I don't know if it performs any better than what you already
have.
You are aware that this is hugely different, because the nested lists are
references, not new instances? Thus the outcome is most probably (given the
gazillion of times people stumbled over this) not the desired one...
Diez
Paul Rudin wrote:
Dr writes:
>I would like to create a pretty big list of lists; a list 3,000,000 long, each entry containing 5 empty lists. My application will append data each of the 5 sublists, so they will be of varying lengths (so no arrays!).
Does anyone know the most efficient way to do this? I have tried:
list = [[[],[],[],[],[]] for _ in xrange(3000000)]
but its not soooo fast. Is there a way to do this without looping?
You can do:
[[[],[],[],[],[]]] * 3000000
although I don't know if it performs any better than what you already
have.
Actually, that produces list of 3000000 references to the same
5-element list. A reduced example:
>>lst = [[[],[],[],[],[]]] * 3 lst[1][1].append(42) print lst
[[[], [42], [], [], []], [[], [42], [], [], []], [[], [42], [], [], []]]
--
--Bryan
"Diez B. Roggisch" <de***@nospam.w eb.dewrites:
Paul Rudin wrote:
>Dr Mephesto <dn****@googlem ail.comwrites:
>>Hi!
I would like to create a pretty big list of lists; a list 3,000,000 long, each entry containing 5 empty lists. My application will append data each of the 5 sublists, so they will be of varying lengths (so no arrays!).
Does anyone know the most efficient way to do this? I have tried:
list = [[[],[],[],[],[]] for _ in xrange(3000000)]
but its not soooo fast. Is there a way to do this without looping?
You can do:
[[[],[],[],[],[]]] * 3000000
although I don't know if it performs any better than what you already have.
You are aware that this is hugely different, because the nested lists are
references, not new instances? Thus the outcome is most probably (given the
gazillion of times people stumbled over this) not the desired one...
Err, yes sorry. I should try to avoid posting before having coffee in
the mornings.
yep, thats why I'm asking :)
On Sep 5, 12:22 pm, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
Paul Rudin wrote:
Dr Mephesto <dnh...@googlem ail.comwrites:
Hi!
I would like to create a pretty big list of lists; a list 3,000,000
long, each entry containing 5 empty lists. My application will append
data each of the 5 sublists, so they will be of varying lengths (so no
arrays!).
Does anyone know the most efficient way to do this? I have tried:
list = [[[],[],[],[],[]] for _ in xrange(3000000)]
but its not soooo fast. Is there a way to do this without looping?
You can do:
[[[],[],[],[],[]]] * 3000000
although I don't know if it performs any better than what you already
have.
You are aware that this is hugely different, because the nested lists are
references, not new instances? Thus the outcome is most probably (given the
gazillion of times people stumbled over this) not the desired one...
Diez
In article <11************ *********@k79g2 000hse.googlegr oups.com>,
Dr Mephesto <dn****@googlem ail.comwrote:
> I would like to create a pretty big list of lists; a list 3,000,000 long, each entry containing 5 empty lists. My application will append data each of the 5 sublists, so they will be of varying lengths (so no arrays!).
Why do you want to pre-create this? Why not just create the big list and
sublists as you append data to the sublists?
--
Aahz (aa**@pythoncra ft.com) <* http://www.pythoncraft.com/
"Many customs in this life persist because they ease friction and promote
productivity as a result of universal agreement, and whether they are
precisely the optimal choices is much less important." --Henry Spencer http://www.lysator.liu.se/c/ten-commandments.html
On Sep 5, 7:50 pm, Dr Mephesto <dnh...@googlem ail.comwrote:
Hi!
I would like to create a pretty big list of lists; a list 3,000,000
long, each entry containing 5 empty lists. My application will append
data each of the 5 sublists, so they will be of varying lengths (so no
arrays!).
Will each and every of the 3,000,000 slots be used? If not, you may be
much better off storagewise if you used a dictionary instead of a
list, at the cost of slower access.
Cheers,
John
Dr Mephesto <dn****@googlem ail.comwrites:
I would like to create a pretty big list of lists; a list 3,000,000
long, each entry containing 5 empty lists. My application will
append data each of the 5 sublists, so they will be of varying
lengths (so no arrays!).
Does anyone know the most efficient way to do this? I have tried:
list = [[[],[],[],[],[]] for _ in xrange(3000000)]
You might want to use a tuple as the container for the lower-level
lists -- it's more compact and costs less allocation-wise.
But the real problem is not list allocation vs tuple allocation, nor
is it looping in Python; surprisingly, it's the GC. Notice this:
$ python
Python 2.5.1 (r251:54863, May 2 2007, 16:56:35)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.
>>import time t0=time.time( ); l=[([],[],[],[],[]) for _ in xrange(3000000)]; t1=time.time( ) t1-t0
143.89971613883 972
Now, with the GC disabled:
$ python
Python 2.5.1 (r251:54863, May 2 2007, 16:56:35)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.
>>import gc gc.disable( ) import time t0=time.time( ); l=[([],[],[],[],[]) for _ in xrange(3000000)]; t1=time.time( ) t1-t0
2.9048631191253 662
The speed difference is staggering, almost 50-fold. I suspect GC
degrades the (amortized) linear-time list building into quadratic
time. Since you allocate all the small lists, the GC gets invoked
every 700 or so allocations, and has to visit more and more objects in
each pass. I'm not sure if this can be fixed (shouldn't the
generational GC only have to visit the freshly created objects rather
than all of them?), but it has been noticed on this group before.
If you're building large data structures and don't need to reclaim
cyclical references, I suggest turning GC off, at least during
construction.
On 6 Sep., 01:34, "Delaney, Timothy (Tim)" <tdela...@avaya .comwrote:
Hrvoje Niksic wrote:
Dr Mephesto <dnh...@googlem ail.comwrites:
I would like to create a pretty big list of lists; a list 3,000,000
long, each entry containing 5 empty lists. My application will
append data each of the 5 sublists, so they will be of varying
lengths (so no arrays!).
Does anyone know the most efficient way to do this? I have tried:
list = [[[],[],[],[],[]] for _ in xrange(3000000)]
If you're building large data structures and don't need to reclaim
cyclical references, I suggest turning GC off, at least during
construction.
This is good advice, but another question is whether you really want
such a list. You may well be better off with a database of some kind -
they're designed for manipulating large amounts of data.
Tim Delaney
I need some real speed! a database is waaay to slow for the algorithm
im using. and because the sublists are of varying size, i dont think I
can use an array... This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Todd MacCulloch |
last post by:
Suppose I have a master list and I need to create two derived lists,
something like:
s0 =
s1 =
s2 =
But suppose generating s0 is expensive and/or s0 is big. In otherwords I'd
like to go over only once and I don't want to keep it around any longer than
I have to.
|
by: Jeff Wagner |
last post by:
I've spent most of the day playing around with lists and tuples to get a really good grasp on what
you can do with them. I am still left with a question and that is, when should you choose a list or
a tuple? I understand that a tuple is immutable and a list is mutable but there has to be more to it
than just that. Everything I tried with a...
|
by: Tech |
last post by:
I have a table tblEmails which contains a list of email ids.
ID,address_id,list_name is the structure of it. I need to create a sub
list for a particular list_name say "Google Customers" , then say my
sublist "Random Internal Google customers" should contain 1000
address_ids from "Google Customes" when I create a new sublist say
"Random...
|
by: Odd-R. |
last post by:
I have to lists, A and B, that may, or may not be equal. If they are not
identical, I want the output to be three new lists, X,Y and Z where X has
all the elements that are in A, but not in B, and Y contains all the
elements that are B but not in A. Z will then have the elements that are
in both A and B.
One way of doing this is of course...
|
by: Ryan Ternier |
last post by:
I'm currently run into a snag on one of my projects. We need to create an
ordered list (Mutli levels).
Ie:
1. Some Title
....A....Something here
....B....Something Else
2. Another Title
| |
by: Richard Thornley |
last post by:
Hello,
I need some clarification in creating objects. Consider the following
code... (note: The function InitializeListCombo initializes the combobox)
Private daLists As New OleDbDataAdapter
|
by: m.k.ball |
last post by:
What is the best way to achieve alphabetically ordered lists that will
index phrases starting with 'The' as if the word was at the end rather
than the start of the phrase?
Michael
|
by: Scott |
last post by:
I'm sorry if most of my question's seem "petty", but as I've said before, I
need to know the petty just because I need to know.
This question is more along the lines of just having you guys either agree
or disgree with me, and if disagreeing to give the reasoning behind it, to
further my understanding of lists.
Am I safe in assuming that...
|
by: Prateek |
last post by:
I have 3 variable length lists of sets. I need to find the common
elements in each list (across sets) really really quickly.
Here is some sample code:
# Doesn't make sense to union the sets - we're going to do
intersections later anyway
l1 = reduce(operator.add, list(x) for x in l1)
l2 = reduce(operator.add, list(x) for x in l2)
l3 =...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
| |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in...
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...
| |