Forevery() writable iterator mechanism in C# 3

[shawnk]

I searched the net to see if other developers have been looking for a
writable iterator in C#. I found much discussion and thus this post.

Currently (C# 2) you can not pass ref and out arguments to an iterator
method (one returning IEnumerable). I WOULD like to do this for
transformative operations on a collection.

I realize the need to lock target, etc.

Does anyone know how to handle 'writable iterators' in C# 2?

Is anyone aware of feature requests for C# to provide a forevery() with a
matching interface like IEnumerable?

Thanks ahead of time. I want to put in a product feature request but want to
do some research before hand.

[Jon Skeet [C# MVP]]

shawnk <Sh****@discuss ions.microsoft. com> wrote:

I searched the net to see if other developers have been looking for a
writable iterator in C#. I found much discussion and thus this post.

Currently (C# 2) you can not pass ref and out arguments to an iterator
method (one returning IEnumerable). I WOULD like to do this for
transformative operations on a collection.

I realize the need to lock target, etc.

Does anyone know how to handle 'writable iterators' in C# 2?

Is anyone aware of feature requests for C# to provide a forevery() with a
matching interface like IEnumerable?

Thanks ahead of time. I want to put in a product feature request but want to
do some research before hand.

I'm not entirely clear about what you want. Could you give a sample
program that you'd *like* to work?

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Mike]

"shawnk" <Sh****@discuss ions.microsoft. com> wrote in message
news:B5******** *************** ***********@mic rosoft.com...

I searched the net to see if other developers have been looking for a
writable iterator in C#. I found much discussion and thus this post.

Currently (C# 2) you can not pass ref and out arguments to an iterator
method (one returning IEnumerable). I WOULD like to do this for
transformative operations on a collection.

I realize the need to lock target, etc.

Does anyone know how to handle 'writable iterators' in C# 2?
Is anyone aware of feature requests for C# to provide a forevery() with a
matching interface like IEnumerable?

Thanks ahead of time. I want to put in a product feature request but want
to
do some research before hand.

Why would the interface need ref/out itself?
There's nothing stopping the "yield"ing function from destructively
modifying it's collection regardless of what it returns, is there? (Although
it's certainly not common or expected.)
A destructive operation will by definition be refelected in all references
to the collection, so "out" is not needed.
And if it's not destructive, you can just write a little "collect" wrapper
that'll go though the enum and return a new ICollection.

IMHO, if you are:
- making a copied list (non-destructive) or returning each element, then
use a return value
- if you are performing a destructive operation, make the fn "void" and
do not return a value - this make it clear what's going on

If all your types are reference types (class instances), then each returned
element from the enumerator is a shared, mutable, object - so you can always
change it through normal means. (This isn't C++, class instances aren't
passed by value on the stack) If you want to change the "stored" element
itself of the current iteration in the underlying collection from *outside*
of the enumertor function, that's another story -- but that's really bad
practice -- especially considering that there may be no actual underlying
collection at all - that's really a major point of "yield", to provide
referential transparency in such matters. Consider:
yield return 1;
yield return 2;
yield return 3;

Your not going to be able to "write" to this interator no matter what you
do.

One way to do this would be to expose an iterator that returned proxies for
the storage of each element, if in fact there is a backing-store for the
enumeration. For example:

// Of course, generics too...
interface IEnumStoragePro xy { object Value { get; set; }};
interface IEnumHasStorage { GetStorageProxy Enum(); }

foreach ( object elem in coll)
elem.Change(bla h); // changing object inside collection as usual

foreach ( IEnumStoragePro xy cell in coll.GetStorage ProxyEnum())
{
object oldVal = c.Value;
c.Value = new MyObject(); // replace the object in the collection
}

This requires no new language features, though the interfaces would need to
be exposed from the framework collections.

m

[Mike]

Typos...

interface IEnumHasStorage { GetStorageProxy Enum(); } Should be: interface IEnumHasStorage { IEnumeration GetStorageProxy Enum(); } object oldVal = c.Value;
c.Value = new MyObject(); // replace the object in the collection Should be: object oldVal = cell.Value;
cell.Value = new MyObject(); // replace the object in the
collection

but I'm sure you got the idea...
I'm not saying the names of the interfaces/methods are so nice either :)

[Shawnk]

Please forgive the slow response. I'm swamped:-)

I will use the following category to articulate different iterator types;

Iterators
Indexors [] : This[] (_idr)
Sequencer r/w iterator : ISequencer - forevery (_sqr)
Enumerator ro iterator : IEnumerable - foreach (_emr)

Example Context of use; Migrating items in two nullable collections.

100 items that move between nullable enumerable collections with
constructors A_col(100) and B_col(100).
So there are 100 'slots' for the items to 'live in'.

The identity of each item is its index - for example item '6'.
Item '6' is always in A_col[6] or B_col[6].
When not 'in residence' in A_col[] the A_col[6] = null.
Same for B_col[].

Problem assignment:

I want to be able to move any items in B_col[] to A_col[]
and then print only the items in residence in A_col[].

Sequencer solution:

forevery ( My_item_cls Itm_A_sqr
in A_col_ins,
My_item_cls Itm_B_sqr
in B_col_ins
){

if ( Itm_B_sqr.In_re sidence() )
{
Itm_A_sqr = Itm_B_sqr.Move_ out();
}

if ( Itm_A_sqr.In_re sidence() )
{
Console.WriteLi ne(" Item[{0}] - {1}",
Itm_A_sqr.ID
Itm_A_sqr.Value
);
}
}

The 'forevery' is 'stackable' to handle multiple collections.
The 'sequencer' (_sqr) is 'writable' so I can assign in an elegant type safe
manner.

For simplicity ignore all threading and synchronization issues and
conventions.
These would be done under the covers by the collection classes.

I hope this helps.
Thank you for your response.
I will review Mike's excellent response and reply asap.
This (next) code example will show why ref and out parameters are useful in
iterator methods
(those returning IEnumerable). Please be patient with my response time.

Shawnk

PS. The smallest 'example program' is big.

"Jon Skeet [C# MVP]" wrote:

shawnk <Sh****@discuss ions.microsoft. com> wrote:

I searched the net to see if other developers have been looking for a
writable iterator in C#. I found much discussion and thus this post.

Currently (C# 2) you can not pass ref and out arguments to an iterator
method (one returning IEnumerable). I WOULD like to do this for
transformative operations on a collection.

I realize the need to lock target, etc.

Does anyone know how to handle 'writable iterators' in C# 2?

Is anyone aware of feature requests for C# to provide a forevery() with a
matching interface like IEnumerable?

Thanks ahead of time. I want to put in a product feature request but want to
do some research before hand.

I'm not entirely clear about what you want. Could you give a sample
program that you'd *like* to work?

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

Shawnk wrote:

<snip>

PS. The smallest 'example program' is big.

That in itself counts against the proposal, IMO. However, having seen
your sample, is this the kind of thing you mean?

using System;
using System.Collecti ons.Generic;

class Test
{
static void Main()
{
List<int> ints = new List<int>();
// Set up the initial list
for (int i=0; i < 5; i++)
{
ints.Add(i);
}

// Square each element
forevery (int x in ints)
{
x = x*x;
}

// Print them out
foreach (int x in ints)
{
Console.WriteLi ne (x);
}
}
}

would result in:
0
1
4
9
16

If that's the case, I don't think there's any need to use ref/out.
Instead, the interface would need to be just like IEnumerator, but with
Current having a setter as well as a getter. Any assignment to the
variable would end up as a call to the setter.

I'm not sure it's particularly necessary, but I'm not entirely averse
to it...

Jon

[Shawnk]

Mike,

Imagine the same situation as in my reply to Jon Skeet with A_col[] and
B_col[].
The collection could be an int?[] array of nullable ints.

The '_itr_mth' LLP tag means 'Iterator Method'.

Problem assignment: Print out list of resident items with their identity

Code solution : Using 'out parameter' in an iterator method (one returning
Enumerable).

int l_actual_itm_id x = 0; // The actual int?[l_idx] non-null item index in
a sparse array of int?[100] items.

l_temp_itm_col_ ins = m_source_itm_co l_ins.
Actual_items_wi th_index_itr_mt h( l_actual_itm_id x );

foreach( int l_int_emr in m_actual_item_i nt_based_gnc_co l_ins )
{
System.Console. WriteLine("Item #{0} = {1}", l_actual_itm_id x,
l_int_emr );
}

If you work it out (and I didn't screw it up :-0) the output skips over
nullable 'slots' ia the terator method internals and only returns existing
items.

In addition it will set the l_actual_itm_id x (item index) to the index in
the int?[100]array which, in this contrived example, is the identity of the
item. (see response to Jon Skeet).

The temporary collection we generate (l_temp_itm_col _ins) acts as an
'iterative delegate' that can return the value of the item identity
(l_actual_itm_i dx) along with the item.

Note that identity here, and this is key, is determined by the collection.

Although the 'identity' is contrived the key point is the collective
responsibility for determining datum l_actual_itm_id x. For collectively
determined pheonomea that exists 'in aggregate' the collection needs a 'data
channel' back to the iterative body (WriteLine).

Note, again key point, the item has no knowledge of its 'context state'
which is only internal to the collection and may, in fact, be dependent upon,
the sequence of iteration itself.

The traveling salesman problem and other 'path oriented solutions' can have
'weights' as sortable item properties and, therefore, expressed in a
mechanism similar to our contrived, but relevant, example.

I hope this answers your intitial question 'why would the inteface need
ref/out itself'

My next post will address transformative operations using sequencers.
Context is changeble, growing objects persisted in XML over a long period.

Key point. The data structure itself changes from one Item XML schema to
another.

This is schema composition (basically)
"Mike" wrote:

"shawnk" <Sh****@discuss ions.microsoft. com> wrote in message
news:B5******** *************** ***********@mic rosoft.com...

I searched the net to see if other developers have been looking for a
writable iterator in C#. I found much discussion and thus this post.

Currently (C# 2) you can not pass ref and out arguments to an iterator
method (one returning IEnumerable). I WOULD like to do this for
transformative operations on a collection.

I realize the need to lock target, etc.

Does anyone know how to handle 'writable iterators' in C# 2?
Is anyone aware of feature requests for C# to provide a forevery() with a
matching interface like IEnumerable?

Thanks ahead of time. I want to put in a product feature request but want
to
do some research before hand.

Why would the interface need ref/out itself?
There's nothing stopping the "yield"ing function from destructively
modifying it's collection regardless of what it returns, is there? (Although
it's certainly not common or expected.)
A destructive operation will by definition be refelected in all references
to the collection, so "out" is not needed.
And if it's not destructive, you can just write a little "collect" wrapper
that'll go though the enum and return a new ICollection.

IMHO, if you are:
- making a copied list (non-destructive) or returning each element, then
use a return value
- if you are performing a destructive operation, make the fn "void" and
do not return a value - this make it clear what's going on

If all your types are reference types (class instances), then each returned
element from the enumerator is a shared, mutable, object - so you can always
change it through normal means. (This isn't C++, class instances aren't
passed by value on the stack) If you want to change the "stored" element
itself of the current iteration in the underlying collection from *outside*
of the enumertor function, that's another story -- but that's really bad
practice -- especially considering that there may be no actual underlying
collection at all - that's really a major point of "yield", to provide
referential transparency in such matters. Consider:
yield return 1;
yield return 2;
yield return 3;

Your not going to be able to "write" to this interator no matter what you
do.

One way to do this would be to expose an iterator that returned proxies for
the storage of each element, if in fact there is a backing-store for the
enumeration. For example:

// Of course, generics too...
interface IEnumStoragePro xy { object Value { get; set; }};
interface IEnumHasStorage { GetStorageProxy Enum(); }

foreach ( object elem in coll)
elem.Change(bla h); // changing object inside collection as usual

foreach ( IEnumStoragePro xy cell in coll.GetStorage ProxyEnum())
{
object oldVal = c.Value;
c.Value = new MyObject(); // replace the object in the collection
}

This requires no new language features, though the interfaces would need to
be exposed from the framework collections.

m

[Mike]

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:11******** *************@e 56g2000cwe.goog legroups.com...

Shawnk wrote:

<snip>
I'm not sure it's particularly necessary, but I'm not entirely averse
to it...

<more snips>
That's how I feel about it. I think its noble to make something that works
no matter what the underlying collection is (again, assuming there is one)
but there are some problems:
(1) To me it's not necessarily easier to read -- it can actually be so
"generic looking" that it's hard to tell whats actually happening. I think
destructive operations should be explicit. (Maybe if I came from an STL
background I'd feel differently.)
(2) You're kind of assuming iterators that work by maintaining a "cursor"
though a collection. C# iterators are moving toward using "yield" which
saves activation state and acts as a limited sort of co-routine, which has
many advantages; however, making Current writable is not possible in general
using this method of writing iterators. (But I guess your distinction of
Sequencer vs. Enumerator is basically a cursor.)
(3) Different collections have different interfaces -- for example, if you
had a priority queue, and you do a "move out", where does the priority (or
other baggage for a particular container class) go which needs to stay
semantically "attached" to the item before it can be added to another queue?
(4) IMHO, the basic collection type is usually an obvious choice up-front
90% of the time, so spending too much time on anything past read-only
iterators isn't worth the complexity. Plus, if you have the collection
black-boxed behind an opaque type anyway and the interface to the outside
world is stable, you can change the complex data-munging code without
disturbing your clients - better to not let them get a hold of your
collections themselves anyway - publish your own GetEnumerator() calls that
can return the interior ones, or ones you code.
(5) I think writable iterators are weird -- they appear very "functional ",
but go against that paradigm as they mutate rather than copy.

But I guess if there is a solution that is natural in the c# world (whether
the "proxy" version or an out/ref version) then I could be convinced -
still, I wonder if the effort is worth the gain.

m
Jon

[Shawnk]

Mike,

I think your solution (an actually Jon Skeet's solution) is what I was
looking for in tems of a 'writable item' enumerator. Thanks to you both for
clarifying that issue for me (a writable sequencer vs a read only enumerator).

So (I think) we can put to rest the 'forevery()' request because the
'writable item' functionality is already there in 'foreach'.

I think, perhaps, some of my C# V 1 thinking is still in my mind :-)

I do want to clarify the issue of the ref/out with a better, really small
example, just like Jon Skeet did. I'm looking for a 'data channel' to flow
data from the iterative method back to the code block containing the
foreach() clause.

This is for 'collective phenomena' that exists above the 'item' level and
should really stay at that level of stratification (the collective level) to
partition the complexity of the problem space correctly (I hope thats not
obtuse).

In simple terms I would like to be able NOT to use the item to pass back or
process this kind of 'collective data'.

Thank you for your patience on my lack of better examples. I'll try to add a
beter example of the weekend to this post just to summarize both your
recommendations .

Thanks again.

Shawnk

"Mike" wrote:

"shawnk" <Sh****@discuss ions.microsoft. com> wrote in message
news:B5******** *************** ***********@mic rosoft.com...

I searched the net to see if other developers have been looking for a
writable iterator in C#. I found much discussion and thus this post.

Currently (C# 2) you can not pass ref and out arguments to an iterator
method (one returning IEnumerable). I WOULD like to do this for
transformative operations on a collection.

I realize the need to lock target, etc.

Does anyone know how to handle 'writable iterators' in C# 2?
Is anyone aware of feature requests for C# to provide a forevery() with a
matching interface like IEnumerable?

Thanks ahead of time. I want to put in a product feature request but want
to
do some research before hand.

Why would the interface need ref/out itself?
There's nothing stopping the "yield"ing function from destructively
modifying it's collection regardless of what it returns, is there? (Although
it's certainly not common or expected.)
A destructive operation will by definition be refelected in all references
to the collection, so "out" is not needed.
And if it's not destructive, you can just write a little "collect" wrapper
that'll go though the enum and return a new ICollection.

IMHO, if you are:
- making a copied list (non-destructive) or returning each element, then
use a return value
- if you are performing a destructive operation, make the fn "void" and
do not return a value - this make it clear what's going on

If all your types are reference types (class instances), then each returned
element from the enumerator is a shared, mutable, object - so you can always
change it through normal means. (This isn't C++, class instances aren't
passed by value on the stack) If you want to change the "stored" element
itself of the current iteration in the underlying collection from *outside*
of the enumertor function, that's another story -- but that's really bad
practice -- especially considering that there may be no actual underlying
collection at all - that's really a major point of "yield", to provide
referential transparency in such matters. Consider:
yield return 1;
yield return 2;
yield return 3;

Your not going to be able to "write" to this interator no matter what you
do.

One way to do this would be to expose an iterator that returned proxies for
the storage of each element, if in fact there is a backing-store for the
enumeration. For example:

// Of course, generics too...
interface IEnumStoragePro xy { object Value { get; set; }};
interface IEnumHasStorage { GetStorageProxy Enum(); }

foreach ( object elem in coll)
elem.Change(bla h); // changing object inside collection as usual

foreach ( IEnumStoragePro xy cell in coll.GetStorage ProxyEnum())
{
object oldVal = c.Value;
c.Value = new MyObject(); // replace the object in the collection
}

This requires no new language features, though the interfaces would need to
be exposed from the framework collections.

m

[Shawnk]

Mike and Jon,

Relative to the collection ontology issues of foreach()/forever() and the
read/write nature of iteration.

I did some research over the weekend. I covered the linguistics of
'iteration' in human language, the mathematical community and (of course)
programming. I ended up with about 60 PDF files of blogs, articles, and
specs.

My final focus was the evolution of the C# 'iteration' specifications in
Ander's (Inventor of C# - and his MS dev team) statements in the C# specs.

The intent of this investigation was to verify your statements on the read
only (or not) nature of the 'enumerator' in C#.

The final (and most important) reference point is Anders specifications
for C# V 2. From the C# V2 spec he writes. (Start Quoted section)

(Hejlsberg.book Page 470 Friday, October 10, 2003 7:35 PM)

The generic and nongeneric enumerable interfaces contain a single member,
a GetEnumerator method that takes no arguments and returns an enumerator
interface. An enumerable acts as an enumerator factory. Properly
implemented enumerables generate independent enumerators each time their
GetEnumerator method is called. Assuming the internal state of the
enumerable has not changed between two calls to GetEnumerator, the two
enumerators returned should produce the same set of values in the same
order. This should hold even if the lifetime of the enumerators overlap
as in the following code sample.

using System;
using System.Collecti ons.Generic;
class Test
{
static IEnumerable<int > FromTo(int from, int to) {
while (from <= to) yield return from++;
}
static void Main() {
IEnumerable<int > e = FromTo(1, 10);
foreach (int x in e) {
foreach (int y in e) {
Console.Write(" {0,3} ", x * y);
}
Console.WriteLi ne();
}
}
}

(End Quoted section)

Numerable articles and presentations mention the 'read only' nature of
'enumerators' to emphasize the 'repeatable results' aspect of 'iterators'
above.
I contrast the above reference point (Anders statement) with the
mathematical definition of 'Iterative Method' found at;

www.wikipedia.org

which states;

(Quoted section)

In computational mathematics, an iterative method attempts to solve a
problem (for example an equation or system of equations) by finding
successive approximations to the solution starting from an initial guess.
This approach is in contrast to direct methods, which attempt to solve the
problem in one-shot (like solving a linear system of equations Ax = b by
finding the inverse of the matrix A).

......

Probably the first iterative method appeared in a letter of Gauss to a
student of his. He proposed solving a 4-by-4 system of equations by
repeatedly solving the component in which the residual was the largest.

The theory of stationary iterative methods was solidly established with
the work of D.M. Young starting in the 1950s. The Conjugate Gradient
method was also invented in the 1950s, with independent developments by
Cornelius Lanczos, Magnus Hestenes and Eduard Stiefel, but its nature and
applicability were misunderstood at the time. Only in the 1970s was it
realized that conjugacy based methods work very well for partial
differential equations, especially the elliptic type.

(End quoted section)

The proposed C# V3 forevery() - ISequencer feature would be used in code
structures where collections are inherently writable, changable, and
iterations have changing results. Collections are seen more as 'living
places' for 'traveling items' that move about under their own volition or
under external impetus. The collections are also exist in a 'binding
matrix' that is used for navigation and migration decisions.

Thus, the 'stackable' forevery() computation features support an
indeterminate 'changing' nature of iteration results.

A set of 'dynamic conventions', such as new items (created during in
iteration pass)
go to end of list, provide for repeatablity of process (not result).

Real world examples would exist in the context of incoming data
composition of perception mechanisms for satellite telemetry, a machine
trying to understand what a person is saying and underwater robots.

In these applications data items go through stages of data processing
fostered by their location (which collection they belong to).

Writable iteration (sequencers) applies constantly changing 'aggregate
state space' phenomena to individual items in their individual journey
through the data processing system.

I am still in research mode and wish to review more material.

Later this week I will post a summary statement on
need for a authoritative collection ontology from the Microsoft folks that
can help to reduce some of the confusion (mine and others) related to
iterative terminology in C#. (I quickly checked Microsoft eBooks on
computing before any research).

I note that your combined understanding of iterators in C# is much better
than mine.

The code syntax issues of (1) ref/out [data channel], (2) terminology and
(3) foreach() vs forevery() are separate but inter-related. I will focus
the remaining posts (in this thread) to issues 2 and 3. These issues are
basically issues of semantics in the C# collection ontology.

I do have one short post on 'iterative data channels' (return, the enumerator
itself (in the foreach() statement), the ref/out). I will put that in a
separate
post.

I would like to defer to terms used in the C# C5 collection library out of
Copenhagen (IT University Technical Report Series TR-2006-76) but I need
to complete my linguistic groundwork first.

Mike, your excellent comment;

I think writable iterators are weird -- they appear very "functional ", but
go against that paradigm as they mutate rather than copy.

is very true. The application context (mentioned above) has a
fundamentally different view of mutability let alone the deterministic
implications because of the constant change nature of the collection
content. Thus your use of the word 'paradigm' is exactly the sematic issue
I am trying to understand.

This is the exact impetus; semantic clarity of the C# collection ontology
and its resulting syntactic expression that I wish to understand. With
this in mind I will be reviewing both of your statements on the mutable
nature of 'enumerators' this week.

Thanks, again, for your patience with my response time.

With all due respect I want to thank you ahead of time for any continued
input. Also thank you again for you input in helping me understand the 'C#
enumerator semantics'.

Shawnk

PS. The forevery() establishes an focused complementary collective processing
paradigm to complement foreach() via a read vs write functional basis.

"Mike" wrote:

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:11******** *************@e 56g2000cwe.goog legroups.com...

Shawnk wrote:

<snip>
I'm not sure it's particularly necessary, but I'm not entirely averse
to it...

<more snips>

That's how I feel about it. I think its noble to make something that works
no matter what the underlying collection is (again, assuming there is one)
but there are some problems:
(1) To me it's not necessarily easier to read -- it can actually be so
"generic looking" that it's hard to tell whats actually happening. I think
destructive operations should be explicit. (Maybe if I came from an STL
background I'd feel differently.)
(2) You're kind of assuming iterators that work by maintaining a "cursor"
though a collection. C# iterators are moving toward using "yield" which
saves activation state and acts as a limited sort of co-routine, which has
many advantages; however, making Current writable is not possible in general
using this method of writing iterators. (But I guess your distinction of
Sequencer vs. Enumerator is basically a cursor.)
(3) Different collections have different interfaces -- for example, if you
had a priority queue, and you do a "move out", where does the priority (or
other baggage for a particular container class) go which needs to stay
semantically "attached" to the item before it can be added to another queue?
(4) IMHO, the basic collection type is usually an obvious choice up-front
90% of the time, so spending too much time on anything past read-only
iterators isn't worth the complexity. Plus, if you have the collection
black-boxed behind an opaque type anyway and the interface to the outside
world is stable, you can change the complex data-munging code without
disturbing your clients - better to not let them get a hold of your
collections themselves anyway - publish your own GetEnumerator() calls that
can return the interior ones, or ones you code.
(5) I think writable iterators are weird -- they appear very "functional ",
but go against that paradigm as they mutate rather than copy.

But I guess if there is a solution that is natural in the c# world (whether
the "proxy" version or an out/ref version) then I could be convinced -
still, I wonder if the effort is worth the gain.

m

Jon