iterator wrapper

[alf]

Hi,

I have a following task: let's say I do have an iterator returning the
some objects:

>>i=<iterator>
i.next()

1

>>i.next()

'abgfdgdfg'

>>i.next()

<some object>
For some reason I need to wrap thos objects with a list. I thought I
could have a smart lamda or simple function class yielding following result:

>>i=<iterator>
i=list_wrapper(i)
i.next()

[1]

>>i.next()

['abgfdgdfg']

>>i.next()

[<some object>]
What would thesolution?
Thx,

A.

[John Machin]

alf wrote:

Hi,

I have a following task: let's say I do have an iterator returning the
some objects:

>>i=<iterator>
>>i.next()

1

>>i.next()

'abgfdgdfg'

>>i.next()

<some object>
For some reason I need to wrap thos objects with a list. I thought I
could have a smart lamda or simple function class yielding following result:

>>i=<iterator>
>>i=list_wrapper(i)
>>i.next()

[1]

>>i.next()

['abgfdgdfg']

>>i.next()

[<some object>]
What would thesolution?

If I understand you correctly, something like this:

>>stuff = ['a', 'b', 'c', 'd']
i1 = iter(stuff)
i1.next()

'a'

>>i1.next()

'b'

>>class LW(object): # ListWrapper

.... def __init__(self, i):
.... self.theiter = i
.... def next(self):
.... return [self.theiter.next()]
....

>>i2 = iter(stuff)
x = LW(i2)
x.next()

['a']

>>x.next()

['b']

>>x.next()

['c']

>>x.next()

['d']

>>x.next()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 5, in next
StopIteration

Cheers,
John

[Simon Forman]

John Machin wrote:

alf wrote:

Hi,

I have a following task: let's say I do have an iterator returning the
some objects:

>>i=<iterator>
>>i.next()

1

>>i.next()

'abgfdgdfg'

>>i.next()

<some object>
For some reason I need to wrap thos objects with a list. I thought I
could have a smart lamda or simple function class yielding following result:

>>i=<iterator>
>>i=list_wrapper(i)
>>i.next()

[1]

>>i.next()

['abgfdgdfg']

>>i.next()

[<some object>]
What would thesolution?

If I understand you correctly, something like this:

>stuff = ['a', 'b', 'c', 'd']
i1 = iter(stuff)
i1.next()

'a'

>i1.next()

'b'

>class LW(object): # ListWrapper

... def __init__(self, i):
... self.theiter = i
... def next(self):
... return [self.theiter.next()]
...

>i2 = iter(stuff)
x = LW(i2)
x.next()

['a']

>x.next()

['b']

>x.next()

['c']

>x.next()

['d']

>x.next()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 5, in next
StopIteration

Cheers,
John

You could also use a generator comprehension:

([n] for n in i)

|>i = iter(xrange(3)) # iter() just for kicks..
|>I = ([n] for n in i)
|>I.next()
[0]
|>I.next()
[1]
|>I.next()
[2]
|>I.next()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

Peace,
~Simon

[alf]

Simon Forman wrote:

>>>>>class LW(object): # ListWrapper

... def __init__(self, i):
... self.theiter = i
... def next(self):
... return [self.theiter.next()]

I hoped one lamda would take care of it but looks like it is a simplest
choice.

|>I = ([n] for n in i)

This is nice but I am iterating thru hude objects (like MBs) so you know ...
Thx for responding ...
A.

[Paul Rubin]

alf <ask@mewrites:

|>I = ([n] for n in i)

This is nice but I am iterating thru hude objects (like MBs) so you know ...

I don't understand the objection-- the above is entirely correct and
produces the same iterator you'd get from

def wrap(i):
for x in i:
yield [x]
I = wrap(i)

You could also use

import itertools
I = itertools.imap(lambda x: [x], i)

which again does the same thing.

[Simon Forman]

alf wrote:

Simon Forman wrote:

>>>>class LW(object): # ListWrapper
... def __init__(self, i):
... self.theiter = i
... def next(self):
... return [self.theiter.next()]

I hoped one lamda would take care of it but looks like it is a simplest
choice.

|>I = ([n] for n in i)

This is nice but I am iterating thru hude objects (like MBs) so you know ...
Thx for responding ...
A.

No, I don't know... :-)

My friend, I think you've misunderstood. Observe:

|>L = [n for n in range(3)]
|>G = (n for n in range(3))
|>L
[0, 1, 2]
|>G
<generator object at 0xb7d982ec>
|>L.next()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'list' object has no attribute 'next'
|>G.next()
0

List comprehensions [] create lists, generator comprehensions () create
generators.

Generator comprehensions work "just-in-time", pulling items from
whatever they're iterating over as they themselves are iterated over,
as I hope this example makes clear:

|>i = iter(xrange(3))
|>G = ([n] for n in i)
|>G.next()
[0]
|>i.next()
1
|>G.next()
[2]
|>G.next()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

So, ([n] for n in i) is indeed sufficient to your needs, as I
understand them.

BTW, the ()'s of a function call serve to create generator
comprehensions:

|>sum(n * 2 for n in range(3))
6

Neat, eh?

HTH,
~Simon

"Some say it is better to give than to receive. We say it is better to
take than to receive. Note the subtle difference."

[alf]

Simon Forman wrote:

>>

>>>|>I = ([n] for n in i)

This is nice but I am iterating thru hude objects (like MBs) so you know ...

No, I don't know... :-)

potentially my source lists are huge - so wanted to avoid unnecessary
memory allocation

My friend, I think you've misunderstood. Observe:

|>L = [n for n in range(3)]
|>G = (n for n in range(3))
|>L
[0, 1, 2]
|>G
<generator object at 0xb7d982ec>

well, I am in the python 2.3 word where generator comprehensions seem
not to work. So I just took it a preallocated list comprehention. still
wonder if there would be a difference between:

G = (n for n in range(300000000)) - this creates the huge list there
G = (n for n in xrange(300000000)) - this (at least to my understanding)
does not

>
List comprehensions [] create lists, generator comprehensions () create
generators.

Generator comprehensions work "just-in-time", pulling items from
whatever they're iterating over as they themselves are iterated over,
as I hope this example makes clear:

[...]

got it now ... thx or the lesson....
A.

[alf]

Paul Rubin wrote:

alf <ask@mewrites:

>>>|>I = ([n] for n in i)

This is nice but I am iterating thru hude objects (like MBs) so you know ...

I don't understand the objection-- the above is entirely correct and
produces the same iterator you'd get from

def wrap(i):
for x in i:
yield [x]
I = wrap(i)

You could also use

import itertools
I = itertools.imap(lambda x: [x], i)

which again does the same thing.

I did see [] instead of () :-). this was a source of confusion.
okay, we have 3 functionally equal solution - which is most pythonic :-)
..... seroiusly ...

--
alf

[Simon Forman]

alf wrote:

Simon Forman wrote:

>
|>I = ([n] for n in i)

This is nice but I am iterating thru hude objects (like MBs) so you know ...

No, I don't know... :-)

potentially my source lists are huge - so wanted to avoid unnecessary
memory allocation

My friend, I think you've misunderstood. Observe:

|>L = [n for n in range(3)]
|>G = (n for n in range(3))
|>L
[0, 1, 2]
|>G
<generator object at 0xb7d982ec>

well, I am in the python 2.3 word where generator comprehensions seem
not to work. So I just took it a preallocated list comprehention. still
wonder if there would be a difference between:

G = (n for n in range(300000000)) - this creates the huge list there
G = (n for n in xrange(300000000)) - this (at least to my understanding)
does not

Yes, you've got it, the xrange() version will not allocate a huge list.

It's not part of your main question, and I understand that there may be
reasons why you can't, but consider upgrading to 2.4 (or very soon now
2.5...)

Meanwhile, in 2.3 both John Machin's and Paul Rubin's solutions will do
the trick (as you already know ;-) )

List comprehensions [] create lists, generator comprehensions () create
generators.

Generator comprehensions work "just-in-time", pulling items from
whatever they're iterating over as they themselves are iterated over,
as I hope this example makes clear:

>[...]

got it now ... thx or the lesson....
A.

No problem. These are things I learned over time, it's fun to share
the knowledge. :)

Peace,
~Simon

[alf]

Simon Forman wrote:

Yes, you've got it, the xrange() version will not allocate a huge list.

It's not part of your main question, and I understand that there may be
reasons why you can't, but consider upgrading to 2.4 (or very soon now
2.5...)

upgrade to 2.4 is on the roadmap and will take place soon .... the app
is just too big and must be stable so since all works in 2.3 well we
stick to it.

there are some parts (related to wx where I noted that 2.4 behaves
differently so it needs some attention) ...
other reason is there is a number of users using customized python (on
top of standard install there is wx, twistedmatrix, matplot and a lot of
small libs/modules == 200MB) so it is kind of pin to migrate that from
the deployment point of view. Tried to have Python on network drive but
it is painfully slow. wonder what strategy should I choose while the
priority is the stable app.

--
alf