trouble writing results to files

[lisa.engblom]

I have two semi related questions...

First, I am trying to output a list of strings to a csv file using the
csv module. The output file separates each letter of the string with a
comma and then puts each string on a separate line. So the code is:

import csv
output = csv.writer(open ('/Python25/working/output.csv', 'a'))
a = ["apple", "cranberry" , "tart"]
for elem in range(len(a)):
output.writerow (a[elem])
.... and it would write to the file:
a,p,p,l,e
c,r,a,n,b,e,r,r ,y
t,a,r,t

How do I get it to write "apple", "cranberry" , "tart" ?

Second, there is a significant delay (5-10 minutes) between when the
program finishes running and when the text actually appears in the
file. Any ideas for why this happens? It is the same for writing with
the csv module or the standard way.

thanks!
Lisa

[Peter Otten]

Steven D'Aprano wrote:

And remember that if alist is truly huge, you may take a performance hit
due to duplicating all those megabytes of data when you slice it.

Having the same object in two lists simultaneously does not double the total
amount of memory; you just need space for an extra pointer.

If you
are modifying the original, better to skip making a slice.

I wrote a piece of code the other day that had to walk along a list,
swapping adjacent elements like this:

for i in xrange(0, len(alist)-1, 2):
alist[i],*alist[i+1]*=*alist[i+1],*alist[i]
The version using enumerate is no improvement:

for i, x in enumerate(alist[0:len(alist)-1:2]):
alist[i*2],*alist[i*2+1]*=*alist[i*2+1],*x
In my opinion, it actually is harder to understand what it is doing.
Swapping two items using "a,b = b,a" is a well known and easily recognised
idiom. Swapping two items using "a,b = b,c" is not.

That example was chosen to prove your point. The real contender for the
"swap items" problem are slices.

def swap_slice(item s):
left = items[::2]
items[::2] = items[1::2]
items[1::2] = left
return items

def swap_loop(items ):
for i in xrange(0, len(items)-1, 2):
k = i+1
items[i], items[k] = items[k], items[i]
return items

$ python2.5 -m timeit -s 'from swap import swap_loop as swap; r =
range(10**6)' 'swap(r)'
10 loops, best of 3: 326 msec per loop
$ python2.5 -m timeit -s 'from swap import swap_slice as swap; r =
range(10**6)' 'swap(r)'
10 loops, best of 3: 186 msec per loop

$ python2.5 -m timeit -s 'from swap import swap_loop as swap; r =
range(10**7)' 'swap(r)'
10 loops, best of 3: 3.27 sec per loop
$ python2.5 -m timeit -s 'from swap import swap_slice as swap; r =
range(10**7)' 'swap(r)'
10 loops, best of 3: 2.29 sec per loop

With 10**7 items in the list I hear disc access on my system, so the
theoretical sweet spot where swap_slice() is already swapping while
swap_loop() is not may be nonexistent...

Peter

[Duncan Booth]

Peter Otten <__*******@web. dewrote:

That example was chosen to prove your point. The real contender for the
"swap items" problem are slices.

def swap_slice(item s):
left = items[::2]
items[::2] = items[1::2]
items[1::2] = left
return items

It makes no difference to the time or memory use, but you can of course
also write swap_slice using the aforementioned 'well known idiom':

items[::2], items[1::2] = items[1::2], items[::2]

[Peter Otten]

Duncan Booth wrote:

items[::2], items[1::2] = items[1::2], items[::2]

Cool. I really should have found that myself.

Peter

[Steven D'Aprano]

On Thu, 30 Nov 2006 11:17:17 +0100, Peter Otten wrote:

Steven D'Aprano wrote:

>And remember that if alist is truly huge, you may take a performance hit
due to duplicating all those megabytes of data when you slice it.

Having the same object in two lists simultaneously does not double the total
amount of memory; you just need space for an extra pointer.

Sure. But if you have millions of items in a list, the pointers themselves
take millions of bytes -- otherwise known as megabytes.
[snip]

>In my opinion, it actually is harder to understand what it is doing.
Swapping two items using "a,b = b,a" is a well known and easily
recognised idiom. Swapping two items using "a,b = b,c" is not.

That example was chosen to prove your point.

Well, I thought about choosing an example that disproved my point, but I
couldn't think of one :-)

The real contender for the "swap items" problem are slices.

def swap_slice(item s):
left = items[::2]
items[::2] = items[1::2]
items[1::2] = left
return items

I always forget that extended slices can be assigned to as well as
assigned from! Nice piece of code... if only it worked.

>>def swap_slice(item s):

.... left = items[::2]
.... items[::2] = items[1::2]
.... items[1::2] = left
.... return items
....

>>alist

[0, 1, 2, 3, 4]

>>swap_slice(al ist)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in swap_slice
ValueError: attempt to assign sequence of size 2 to extended slice of size 3

Everybody always forgets corner cases... like lists with odd number of
items... *wink*

Here is a version that handles both odd and even length lists:

def swap_slice(item s):
left = items[:len(items)-1:2]
items[:len(items)-1:2] = items[1::2]
items[1::2] = left
return items
Replacing the explicit Python for loop with an implicit loop in C makes it
very much faster.
--
Steven.

[Duncan Booth]

Steven D'Aprano <st***@REMOVE.T HIS.cybersource .com.auwrote:

Everybody always forgets corner cases... like lists with odd number of
items... *wink*

I didn't forget. I just assumed that raising an exception was a more useful
response.

Here is a version that handles both odd and even length lists:

def swap_slice(item s):
left = items[:len(items)-1:2]
items[:len(items)-1:2] = items[1::2]
items[1::2] = left
return items

I guess a viable alternative to raising an exception would be to pad the
list to even length:

[1, 2, 3] -[2, 1, Padding, 3]

for some value of Padding. I don't think I would expect swap_slice to
silently fail to swap the last element.

[Peter Otten]

Steven D'Aprano wrote:

On Thu, 30 Nov 2006 11:17:17 +0100, Peter Otten wrote:

>Steven D'Aprano wrote:

>>And remember that if alist is truly huge, you may take a performance hit
due to duplicating all those megabytes of data when you slice it.

Having the same object in two lists simultaneously does not double the
total amount of memory; you just need space for an extra pointer.

Sure. But if you have millions of items in a list, the pointers themselves
take millions of bytes -- otherwise known as megabytes.

I don't know the exact layout of an int, but let's assume 4 bytes for the
class, the value, the refcount and the initial list entry -- which gives
you 16 bytes per entry for what is probably the class with the smallest
footprint in the python universe. For the range(N) example the slicing
approach then needs an extra 4 bytes or 25 percent. On the other hand, if
you are not dealing with unique objects (say range(100) * (N//100)) the
amount of memory for the actual objects is negligable and consequently the
total amount doubles.
You should at least take that difference into account when you choose the
swapping algorithm.

>That example was chosen to prove your point.

Well, I thought about choosing an example that disproved my point, but I
couldn't think of one :-)

Lack of fantasy :-)

>The real contender for the "swap items" problem are slices.

def swap_slice(item s):
left = items[::2]
items[::2] = items[1::2]
items[1::2] = left
return items

I always forget that extended slices can be assigned to as well as
assigned from! Nice piece of code... if only it worked.

>>>def swap_slice(item s):

... left = items[::2]
... items[::2] = items[1::2]
... items[1::2] = left
... return items
...

>>>alist

[0, 1, 2, 3, 4]

>>>swap_slice(a list)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in swap_slice
ValueError: attempt to assign sequence of size 2 to extended slice of size
3

Everybody always forgets corner cases... like lists with odd number of
items... *wink*

True in general, but on that one I'm with Duncan.

Here is another implementation that cuts maximum memory down from 100 to
50%.

from itertools import islice
def swap(items):
items[::2], items[1::2] = islice(items, 1, None, 2), items[::2]
return items

Peter

[Peter Otten]

Peter Otten wrote:

Here is another implementation that cuts maximum memory down from 100 to
50%.

from itertools import islice
def swap(items):
****items[::2],*items[1::2]*=*islice(items ,*1,*None,*2),* items[::2]
****return*item s

Unfortunately, the following

>>a = [1, 2, 3]
a[::2] = iter([10, 20, 30])

Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: attempt to assign sequence of size 3 to extended slice of size 2

>>a

[1, 2, 3]

does not support my bold claim :-( Since the list is not changed there must
be an intermediate copy.

Peter