473,287 Members | 1,501 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

Using iterators to write in the structure being iterated through?

Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

template <class H>
class MyClass
{
public:

MyClass(H& o1, H& o2) : object1(o1), object2(o2) {}

void compute();

private:
H& object1;
H& object2;

};

template <class H>
void MyClass::compute()
{
typedef typename H::iterator I;

I o1_begin = object1.begin();
I o2_begin = object2.begin();
I o1_end = object1.end();

for(I io1 = o1_begin, io2 = o2_begin; io1 != o1_end; ++io1, ++io2)
{
// Do something with *io1 and *io2, for instance:
// *io1 += *io2;
}
}

This is all nice: any object having a forward iterator works in there.

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

Thanks!

Pierre

Jul 26 '06 #1
10 1590
I think you are going to need to provide some python minimal code as an
example of what is not working for you.

-Chris
On Wed, Jul 26, 2006 at 11:39:36AM -0400, Pierre Thibault wrote:
Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

template <class H>
class MyClass
{
public:

MyClass(H& o1, H& o2) : object1(o1), object2(o2) {}

void compute();

private:
H& object1;
H& object2;

};

template <class H>
void MyClass::compute()
{
typedef typename H::iterator I;

I o1_begin = object1.begin();
I o2_begin = object2.begin();
I o1_end = object1.end();

for(I io1 = o1_begin, io2 = o2_begin; io1 != o1_end; ++io1, ++io2)
{
// Do something with *io1 and *io2, for instance:
// *io1 += *io2;
}
}

This is all nice: any object having a forward iterator works in there.

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

Thanks!

Pierre

--
http://mail.python.org/mailman/listinfo/python-list
Jul 26 '06 #2
Pierre Thibault wrote:
Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

template <class H>
class MyClass
{
public:

MyClass(H& o1, H& o2) : object1(o1), object2(o2) {}

void compute();

private:
H& object1;
H& object2;

};

template <class H>
void MyClass::compute()
{
typedef typename H::iterator I;

I o1_begin = object1.begin();
I o2_begin = object2.begin();
I o1_end = object1.end();

for(I io1 = o1_begin, io2 = o2_begin; io1 != o1_end; ++io1, ++io2)
{
// Do something with *io1 and *io2, for instance:
// *io1 += *io2;
}
}

This is all nice: any object having a forward iterator works in there.

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?
You need a temporary variable (out in the example below):
>>accus = [[] for i in range(3)]
ins = ("abc" for i in range(3))
outs = iter(accus)
while 1:
.... out = outs.next()
.... out += ins.next()
....
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
StopIteration

In idiomatic Python that becomes
>>accus = [[] for i in range(3)]
ins = ("abc" for i in range(3))
outs = iter(accus)
from itertools import izip
for out, in_ in izip(outs, ins):
.... out += in_
....
>>accus
[['a', 'b', 'c'], ['a', 'b', 'c'], ['a', 'b', 'c']]
Peter

Jul 26 '06 #3
On Wed, 26 Jul 2006 18:54:39 +0200, Peter Otten wrote:
Pierre Thibault wrote:
>Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

template <class H>
class MyClass
{
public:

MyClass(H& o1, H& o2) : object1(o1), object2(o2) {}

void compute();

private:
H& object1;
H& object2;

};

template <class H>
void MyClass::compute()
{
typedef typename H::iterator I;

I o1_begin = object1.begin();
I o2_begin = object2.begin();
I o1_end = object1.end();

for(I io1 = o1_begin, io2 = o2_begin; io1 != o1_end; ++io1, ++io2)
{
// Do something with *io1 and *io2, for instance:
// *io1 += *io2;
}
}

This is all nice: any object having a forward iterator works in there.

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

You need a temporary variable (out in the example below):
>>>accus = [[] for i in range(3)]
ins = ("abc" for i in range(3))
outs = iter(accus)
while 1:
... out = outs.next()
... out += ins.next()
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
StopIteration

In idiomatic Python that becomes
>>>accus = [[] for i in range(3)]
ins = ("abc" for i in range(3))
outs = iter(accus)
from itertools import izip
for out, in_ in izip(outs, ins):
... out += in_
...
>>>accus
[['a', 'b', 'c'], ['a', 'b', 'c'], ['a', 'b', 'c']]
Peter
Hum, this example seems like a special case not really appropriate for my
needs. Let me make my problem a little more precise. The objects I'll want
to iterate through will always contain some floats. Very often, I guess
they will be numpy.ndarray instances, but they can be something else as
well. For instance, I will have to deal with arrays having internal
symmetries (crystallographic datasets). The most efficient thing to do
with those is save only the unique values and keep track of the symmetry
operations needed to extract other values.

A 1d examples: I have a class which represents a unit cell
which has the additional mirror symmetry in the middle of the cell. Say
C is an instance of this class representing data on a 50-long array. Only
25 datum will be stored in C, and the class takes care of giving the value
of C[0] if C[49] is asked, C[1] for C[48], and so on. Since crystals are
periodic, the translational symmetry can also be managed (C[-1] == C[49]
== C[0], C[-2] == C[48] == C[1]...). In any case, the user of this object
need not know how the data is stored: for him, C is just a funny
array-like object defined from -infinity to +infinity on a 1-D lattice.

Now, I want to do simple math operations on the data in C. Doing a loop
from 0 to 49 would loop twice through the actual data. In this
context, an iterator is perfect since it can take care internally of going
through the data only once, so it would be great to access _AND MODIFY_
data over which I iterate.

I now realize I don't know how to do that with numpy.ndarray either. It
looks like the only way I can modify the content of an array is by using
the [] notation (that is, in random access). For instance, the
following will not change the state of a:

import numpy
a = numpy.array([[1.,2.,3.],[4.,5.,6.],[7.,8.,9.]])

for c in a.flat
c += 2.

Of course, I know that a += .2 would be alright, but this is not general
enough for more complicated classes.

So I guess my problem is worst than I thought.

Thanks anyway for your help!

Pierre
Jul 26 '06 #4
Pierre Thibault wrote:
Hum, this example seems like a special case not really appropriate for my
needs. Let me make my problem a little more precise. The objects I'll want
to iterate through will always contain some floats. Very often, I guess
they will be numpy.ndarray instances, but they can be something else as
well. For instance, I will have to deal with arrays having internal
symmetries (crystallographic datasets). The most efficient thing to do
with those is save only the unique values and keep track of the symmetry
operations needed to extract other values.

A 1d examples: I have a class which represents a unit cell
which has the additional mirror symmetry in the middle of the cell. Say
C is an instance of this class representing data on a 50-long array. Only
25 datum will be stored in C, and the class takes care of giving the value
of C[0] if C[49] is asked, C[1] for C[48], and so on. Since crystals are
periodic, the translational symmetry can also be managed (C[-1] == C[49]
== C[0], C[-2] == C[48] == C[1]...). In any case, the user of this object
need not know how the data is stored: for him, C is just a funny
array-like object defined from -infinity to +infinity on a 1-D lattice.

Now, I want to do simple math operations on the data in C. Doing a loop
from 0 to 49 would loop twice through the actual data. In this
context, an iterator is perfect since it can take care internally of going
through the data only once, so it would be great to access _AND MODIFY_
data over which I iterate.
That would not only be nice but crucial to the integrity of your data. You'd
have to explicitly forbid operations on individual cells as applying an
operation to a single cell would change two or more cells, depending of the
kind of symmetry.
I now realize I don't know how to do that with numpy.ndarray either. It
looks like the only way I can modify the content of an array is by using
the [] notation (that is, in random access). For instance, the
following will not change the state of a:

import numpy
a = numpy.array([[1.,2.,3.],[4.,5.,6.],[7.,8.,9.]])

for c in a.flat
c += 2.

Of course, I know that a += .2 would be alright, but this is not general
enough for more complicated classes.

So I guess my problem is worst than I thought.
Indeed, but I don't think it would be easier in C++...

Peter

Jul 26 '06 #5
Pierre Thibault, some people here surely know enough Python (and C++)
to solve your problem, but often the problem is understanding the
problem. I have understood your problem just partially, so the
following are just ideas.

First of all I suggest you to use a normal Python list to keep the
data, and later modify the code to use an array if it is too much slow
or if you need arrays to do some special things, like plotting, etc.
Developing using normal Python lists is a bit simpler.

With python you can redefine the __getitem__ and __setitem__ of an
object according to its crystal symmetries. Then you can define some
method like iterkeys that yields (with yield) the only indexes that the
symmetry allows to be set.

Then you can iterate on those indexes and update the values in the list
attribute.

Bye,
bearophile

Jul 26 '06 #6

Pierre Thibault wrote:
Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:
<SNIP>
Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

Thanks!

Pierre
Pierre,
You should be able to write
io1.next().param += io2.next().param
If iter(object1) and iter(object2) both return classes or instances
with the appropriate parameter.
Here is what I was thinking of:
class ParamHolder(object):
def __init__(self, n):
self.param = n

class C1(object):
def __init__(self,m):
self.value = [ParamHolder(n) for n in range(m)]
def __getitem__(self, p):
return self.value[p]

obj1 = C1(5)
obj2 = C1(5)

io1 = iter(obj1)
io2 = iter(obj2)

print "obj1 pre loop",[r.param for r in obj1.value]

try:
while 1:
io1.next().param += io2.next().param
except StopIteration:
pass

print "obj1 post loop",[r.param for r in obj1.value]

- Paddy.

Jul 26 '06 #7

Paddy wrote:
Pierre Thibault wrote:
Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:
<SNIP>
Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

Thanks!

Pierre

Pierre,
You should be able to write
io1.next().param += io2.next().param
If iter(object1) and iter(object2) both return classes or instances
with the appropriate parameter.
Here is what I was thinking of:
class ParamHolder(object):
def __init__(self, n):
self.param = n

class C1(object):
def __init__(self,m):
self.value = [ParamHolder(n) for n in range(m)]
def __getitem__(self, p):
return self.value[p]

obj1 = C1(5)
obj2 = C1(5)

io1 = iter(obj1)
io2 = iter(obj2)

print "obj1 pre loop",[r.param for r in obj1.value]

try:
while 1:
io1.next().param += io2.next().param
except StopIteration:
pass

print "obj1 post loop",[r.param for r in obj1.value]

- Paddy.
I don't like the try/except code and would write something like the
following:
>>obj1 = C1(5)
obj2 = C1(5)
from itertools import izip
for x,y in izip(obj1, obj2):
.... x.param += y.param
....
>>print "obj1 post for loop",[r.param for r in obj1.value]
obj1 post for loop [0, 2, 4, 6, 8]
>>>
- Paddy.

Jul 26 '06 #8
On Wed, 26 Jul 2006 16:11:48 -0700, Paddy wrote:
>
Paddy wrote:
>Pierre Thibault wrote:
Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:
<SNIP>
Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

Thanks!

Pierre

Pierre,
You should be able to write
io1.next().param += io2.next().param
If iter(object1) and iter(object2) both return classes or instances
with the appropriate parameter.
Here is what I was thinking of:
class ParamHolder(object):
def __init__(self, n):
self.param = n

class C1(object):
def __init__(self,m):
self.value = [ParamHolder(n) for n in range(m)]
def __getitem__(self, p):
return self.value[p]

obj1 = C1(5)
obj2 = C1(5)

io1 = iter(obj1)
io2 = iter(obj2)

print "obj1 pre loop",[r.param for r in obj1.value]

try:
while 1:
io1.next().param += io2.next().param
except StopIteration:
pass

print "obj1 post loop",[r.param for r in obj1.value]

- Paddy.

I don't like the try/except code and would write something like the
following:
>>>obj1 = C1(5)
obj2 = C1(5)
from itertools import izip
for x,y in izip(obj1, obj2):
... x.param += y.param
...
>>>print "obj1 post for loop",[r.param for r in obj1.value]
obj1 post for loop [0, 2, 4, 6, 8]
>>>>

- Paddy.
Thanks Paddy,

This looks like the closest thing to what I wanted, though the need for
this "param" makes it not very general to my taste. Besides, this method
would not work with ndarrays anyways.

Thanks again for taking the time to answer,

Pierre

Jul 27 '06 #9
On Wed, 26 Jul 2006 20:59:12 +0200, Peter Otten wrote:
Pierre Thibault wrote:
>Hum, this example seems like a special case not really appropriate for my
needs. Let me make my problem a little more precise. The objects I'll want
to iterate through will always contain some floats. Very often, I guess
they will be numpy.ndarray instances, but they can be something else as
well. For instance, I will have to deal with arrays having internal
symmetries (crystallographic datasets). The most efficient thing to do
with those is save only the unique values and keep track of the symmetry
operations needed to extract other values.

A 1d examples: I have a class which represents a unit cell
which has the additional mirror symmetry in the middle of the cell. Say
C is an instance of this class representing data on a 50-long array. Only
25 datum will be stored in C, and the class takes care of giving the value
of C[0] if C[49] is asked, C[1] for C[48], and so on. Since crystals are
periodic, the translational symmetry can also be managed (C[-1] == C[49]
== C[0], C[-2] == C[48] == C[1]...). In any case, the user of this object
need not know how the data is stored: for him, C is just a funny
array-like object defined from -infinity to +infinity on a 1-D lattice.

Now, I want to do simple math operations on the data in C. Doing a loop
from 0 to 49 would loop twice through the actual data. In this
context, an iterator is perfect since it can take care internally of going
through the data only once, so it would be great to access _AND MODIFY_
data over which I iterate.

That would not only be nice but crucial to the integrity of your data. You'd
have to explicitly forbid operations on individual cells as applying an
operation to a single cell would change two or more cells, depending of the
kind of symmetry.
>I now realize I don't know how to do that with numpy.ndarray either. It
looks like the only way I can modify the content of an array is by using
the [] notation (that is, in random access). For instance, the
following will not change the state of a:

import numpy
a = numpy.array([[1.,2.,3.],[4.,5.,6.],[7.,8.,9.]])

for c in a.flat
c += 2.

Of course, I know that a += .2 would be alright, but this is not general
enough for more complicated classes.

So I guess my problem is worst than I thought.

Indeed, but I don't think it would be easier in C++...

Peter
Well, no, C++ is better in this case because the thing you iterate over is
a generalized pointer, which can be dereferenced to get access to the
memory. So in other word, iterator.next() (in python) is a copy, while
*iterator (in C++) is a reference.

Thanks anyway for your comments!

Pierre

Jul 27 '06 #10
Pierre Thibault wrote:
On Wed, 26 Jul 2006 18:54:39 +0200, Peter Otten wrote:
>Pierre Thibault wrote:
[...]
>
Now, I want to do simple math operations on the data in C. Doing a loop
from 0 to 49 would loop twice through the actual data. In this
context, an iterator is perfect since it can take care internally of going
through the data only once, so it would be great to access _AND MODIFY_
data over which I iterate.
The biggest problem I think is that numbers are immutable in Python.
Thus you cannot modify a number if you have a variable on it! All you
can do is modify the content of the variable.

IMHO, the way to go is to iterate on the indices and not on the values
(not much slower in Python). For example you can do something like:

lst1 = [1,2,3,4]
lst2 = [4,3,4,5]

for i in xrange(len(lst1)):
lst1[i] += lst2[i]

If you *really* want to iterate (for example to support something else
that lists), then you will need to create a new structure like that:

from itertool import izip

lst1 = MyList1()
lst2 = MyList2()

result = [0]*len(lst1) # If possible

for i,(e1,e2) in enumerate(izip(lst1, lst2)):
result[i] = e1+e2

lst1[:] = result # If you really need it

An intermediate possibility would be (if lst1 is indexable):

for i,e2 in enumerate(lst2):
lst1[i] += e2

But it all depend on the kind of structure you get.

Pierre
Jul 28 '06 #11

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

9
by: Jess Austin | last post by:
hi, I like the way that Python does lists, and I love the way it does iterators. But I've decided I don't like what it does with iterators of lists. Lists are supposed to be mutable sequences,...
7
by: marduk | last post by:
I have a weird request. I want to be able to say def myvalues(): while True: # stuff that determines a new somevalue yield somevalue x = "Hello, %s, this is a %s with %s and %s on top of...
5
by: Carlos Ribeiro | last post by:
Hello all, I'm posting this to the list with the intention to form a group of people interested in this type of solution. I'm not going to spam the list with it, unless for occasional and...
2
by: dgront | last post by:
Greetings! I believe someone can spare me a minute and give me some ideas. Once upon a time I had a C program: double propert; char symb; char* descript; But now I want to make it in C++....
11
by: Grasshopper | last post by:
Hi, I am automating Access reports to PDF using PDF Writer 6.0. I've created a DTS package to run the reports and schedule a job to run this DTS package. If I PC Anywhere into the server on...
2
by: Wavemaker | last post by:
I've been playing with C# v2.0 today, and having quite a bit of fun. The new version has added iterators. The iterators are coded directly into the class to be iterated. For example: public...
2
by: Thormod Johansen | last post by:
Hi, I have just read the article about iterators at http://www.oreillynet.com/pub/a/network/2005/10/18/what-is-iterator-in-c-plus-plus.html?page=last and I am wondering about what is ment by the...
3
by: n.torrey.pines | last post by:
Hi I work with a nested tree-like data structure template (that happens to be a tuple of vectors of tuples of trees of something, with different tuple elements having different types) I need...
3
by: Jess | last post by:
Hello, Iterators are typically put into five different categories, namely input iterator, output iterator, forward iterator, bidirectional iterator and random iterator. The differences come...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.