generator functions in another language

[castironpi]

I'm actually curious if there's a way to write a generator function
(not a generator expression) in C, or what the simplest way to do it
is... besides link the Python run-time.

[Marc 'BlackJack' Rintsch]

On Sat, 03 May 2008 16:39:43 -0700, castironpi wrote:

I'm actually curious if there's a way to write a generator function
(not a generator expression) in C, or what the simplest way to do it
is... besides link the Python run-time.

The reference implementation of Python is written in C, so obviously there
must be a way to write something like generators in C.

Ciao,
Marc 'BlackJack' Rintsch

[Gabriel Genellina]

En Sun, 04 May 2008 01:08:34 -0300, Marc 'BlackJack' Rintsch <bj****@gmx.netescribió:

On Sat, 03 May 2008 16:39:43 -0700, castironpi wrote:

>I'm actually curious if there's a way to write a generator function
(not a generator expression) in C, or what the simplest way to do it
is... besides link the Python run-time.

The reference implementation of Python is written in C, so obviously there
must be a way to write something like generators in C.

Yes and no. Generators are tied to frames, and frames execute Python code, not C. There is no simple way to write generators in C, but there are some generator-like examples in the itertools module.
See this thread http://groups.google.com/group/comp....2f72f2d0e88fc/

--
Gabriel Genellina

[David]

On Sun, May 4, 2008 at 1:39 AM, <ca********@gmail.comwrote:

I'm actually curious if there's a way to write a generator function
(not a generator expression) in C, or what the simplest way to do it
is... besides link the Python run-time.
--

Here's the itertools C code:

http://svn.python.org/view/python/tr...34&view=markup

David.

[castironpi]

On May 4, 12:21*am, "Gabriel Genellina" <[e-mail]>
wrote:

En Sun, 04 May 2008 01:08:34 -0300, Marc 'BlackJack' Rintsch <[e-mail]ó:

On Sat, 03 May 2008 16:39:43 -0700, castironpi wrote:

I'm actually curious if there's a way to write a generator function
(not a generator expression) in C, or what the simplest way to do it
is... besides link the Python run-time.

The reference implementation of Python is written in C, so obviously there
must be a way to write something like generators in C.

Yes and no. Generators are tied to frames, and frames execute Python code,not C. There is no simple way to write generators in C, but there are some generator-like examples in the itertools module.
See this threadhttp://groups.google.com/group/comp.lang.python/browse_thread/thread/...

--
Gabriel Genellina

Gabriel,
How did your attempt turn out from last May? At first look, it's
outside the scope of Python, but it is not the scope of C
necessarily. Generators offer a lot of simplicity (which I haven't
read about extensively, but am starting to see) that could gain some
reputation for Python. What is the midpoint at which C could meet
Python?

There is no such thing as a 'frame' per se in C; byte code is
integral. As there is no such thing as suspended state without
frames, and no such thing as generators without suspended state. It's
a hard thing to Google for without knowing the prior terminology for
the work that's already been done on them in C. What work is there?
Are any devs interested in pursuing it?

The frame implementation.

http://svn.python.org/projects/pytho.../frameobject.h
http://svn.python.org/projects/pytho.../frameobject.c

The generator code.

http://svn.python.org/projects/pytho...de/genobject.h
http://svn.python.org/projects/pytho...ts/genobject.c

I used Microsoft's search engine (python frame generator
site:svn.python.org , links 3 and 5) to find it.

[Gabriel Genellina]

En Sun, 04 May 2008 08:11:35 -0300, <ca********@gmail.comescribió:

On May 4, 12:21*am, "Gabriel Genellina" <[e-mail]>
wrote:

>En Sun, 04 May 2008 01:08:34 -0300, Marc 'BlackJack' Rintsch <[e-mail]ó:

On Sat, 03 May 2008 16:39:43 -0700, castironpi wrote:

>I'm actually curious if there's a way to write a generator function
(not a generator expression) in C, or what the simplest way to do it
is... besides link the Python run-time.

The reference implementation of Python is written in C, so obviously there
must be a way to write something like generators in C.

Yes and no. Generators are tied to frames, and frames execute Python code, not C. There is no simple way to write generators in C, but there are some generator-like examples in the itertools module.
See this threadhttp://groups.google.com/group/comp.lang.python/browse_thread/thread/...

Gabriel,
How did your attempt turn out from last May? At first look, it's
outside the scope of Python, but it is not the scope of C
necessarily. Generators offer a lot of simplicity (which I haven't
read about extensively, but am starting to see) that could gain some
reputation for Python. What is the midpoint at which C could meet
Python?

There is no such thing as a 'frame' per se in C; byte code is
integral. As there is no such thing as suspended state without
frames, and no such thing as generators without suspended state. It's
a hard thing to Google for without knowing the prior terminology for
the work that's already been done on them in C. What work is there?
Are any devs interested in pursuing it?

The frame and generator implementations are very tightly coupled to Python code, they aren't useful for implementing generators in C. Don't bother to read them.

Unfortunately, converting a C function into something like a generator isn't as easy as using the "yield" statement... Although the idea is simple, the implementation may be hard sometimes: You have to wrap the function within an object, maintain all state information into that object, and do all computation in the "next" method. Also, __iter__ should return itself so it can be called as an iterator.
(those objects are sometimes called "functors" <http://en.wikipedia.org/wiki/Function_objectnot the same meaning as functors in Mathematics)

All examples that I have at hand are propietary code so I can't post them. The itertools module may be used as reference - "cycle" and "chain" are the easiest I think, although they might be *too* easy to understand the structure. "groupby" is a more complex example but at the same time harder to understand. See http://svn.python.org/projects/pytho...rtoolsmodule.c

Ok, I'll try to use Python code as an example. A generator for Fibonacci numbers:

def fibo():
a = b = 1
while True:
a, b = b, a+b
yield b

We can convert that function into this object; it should be written in C, not Python, but the idea is the same:

class fibo:
def __init__(self):
self.a = 1
self.b = 1
def next(self):
temp = self.a + self.b
self.a = self.b
self.b = temp
return temp
def __iter__(self):
return self

It behaves exactly the same as the generator above; we can even use the same code to test it:

pyfor n in fibo():
.... if n>100: break
.... print n
....
2
3
5
8
13
21
34
55
89

Converting that class into C code should be straightforward. And then you have a generator-like function written in C.

--
Gabriel Genellina

[hdante]

On May 4, 8:11*am, [e-mail] wrote:

On May 4, 12:21*am, "Gabriel Genellina" <[e-mail]>
wrote:

En Sun, 04 May 2008 01:08:34 -0300, Marc 'BlackJack' Rintsch <bj_...@gmx..netescribió:

On Sat, 03 May 2008 16:39:43 -0700, castironpi wrote:

>I'm actually curious if there's a way to write a generator function
>(not a generator expression) in C, or what the simplest way to do it
>is... besides link the Python run-time.

The reference implementation of Python is written in C, so obviously there
must be a way to write something like generators in C.

Yes and no. Generators are tied to frames, and frames execute Python code, not C. There is no simple way to write generators in C, but there are some generator-like examples in the itertools module.
See this threadhttp://groups.google.com/group/comp.lang.python/browse_thread/thread/...

--
Gabriel Genellina

Gabriel,
How did your attempt turn out from last May? *At first look, it's
outside the scope of Python, but it is not the scope of C
necessarily. *Generators offer a lot of simplicity (which I haven't
read about extensively, but am starting to see) that could gain some
reputation for Python. *What is the midpoint at which C could meet
Python?

There is no such thing as a 'frame' per se in C; byte code is
integral. *As there is no such thing as suspended state without
frames, and no such thing as generators without suspended state. *It's
a hard thing to Google for without knowing the prior terminology for
the work that's already been done on them in C. *What work is there?
Are any devs interested in pursuing it?

The frame implementation.

http://svn.python.org/projects/pytho.../frameobject.c

The generator code.

http://svn.python.org/projects/pytho...ts/genobject.c

I used Microsoft's search engine (python frame generator
site:svn.python.org , links 3 and 5) to find it.

Isn't this guy a bot ? :-) It's learning fast. I believe there is a
"frame" in C, composed of its stack and globals. For generators in C,
you may look for "coroutines". For example, see:

http://www.chiark.greenend.org.uk/~s...oroutines.html

A sample code follows:

#define crBegin static int state=0; switch(state) { case 0:
#define crReturn(i,x) do { state=i; return x; case i:; } while (0)
#define crFinish }
int function(void) {
static int i;
crBegin;
for (i = 0; i < 10; i++)
crReturn(1, i);
crFinish;
}

The "suspended state" is saved in the static variable. It's not
necessary to save the complete "frame", only the suspended state.

[Terry Reedy]

A 'generator function' -- a function that when called returns a generator
object, a specific type of iterator, is a rather Python specific concept.
Better to think, I think, in terms of writing an iterator 'class' (in C,
struct with associated function). Looking at the implementation of two of
the itertools in the link below (previously posted), I could see the
'template' that each was based on and that would be the basis for writing
another. (And I have not programmed C for a decade. Raymond's code is
quite clear.)

=================================
[Outlook Express does not 'quote' this post properly]

"Gabriel Genellina" <ga*******@yahoo.com.arwrote in message
news:op***************@a98gizw.cpe.telecentro.net. ar...

The itertools module may be used as reference - "cycle" and "chain" are
the easiest I think, although they might be *too* easy to understand the
structure. "groupby" is a more complex example but at the same time harder
to understand. See
http://svn.python.org/projects/pytho...rtoolsmodule.c

Ok, I'll try to use Python code as an example. A generator for Fibonacci
numbers:

def fibo():
a = b = 1
while True:
a, b = b, a+b
yield b

We can convert that function into this object; it should be written in C,
not Python, but the idea is the same:

class fibo:
def __init__(self):
self.a = 1
self.b = 1
def next(self):
temp = self.a + self.b
self.a = self.b
self.b = temp
return temp
def __iter__(self):
return self

=============================
Terry again: one can think of a generator function as an abbreviation of an
iterator class. Calling the gen. func. produces an iterator instance just
like calling the class object.
=============================
[back to Gabriel]
It behaves exactly the same as the generator above; we can even use the
same code to test it:

pyfor n in fibo():
.... if n>100: break
.... print n
....
2
3
5
8
13
21
34
55
89

Converting that class into C code should be straightforward. And then you
have a generator-like function written in C.

==================================

[Gabriel Genellina]

En Mon, 05 May 2008 00:09:02 -0300, hdante <hd****@gmail.comescribió:

Isn't this guy a bot ? :-) It's learning fast. I believe there is a
"frame" in C, composed of its stack and globals. For generators in C,
you may look for "coroutines". For example, see:

http://www.chiark.greenend.org.uk/~s...oroutines.html

A sample code follows:

#define crBegin static int state=0; switch(state) { case 0:
#define crReturn(i,x) do { state=i; return x; case i:; } while (0)
#define crFinish }
int function(void) {
static int i;
crBegin;
for (i = 0; i < 10; i++)
crReturn(1, i);
crFinish;
}

The "suspended state" is saved in the static variable. It's not
necessary to save the complete "frame", only the suspended state.

Quoting the author himself, "this is the worst piece of C hackery ever seen"

--
Gabriel Genellina

[castironpi]

On May 5, 12:28*am, "Gabriel Genellina" <[e-mail]>
wrote:

En Mon, 05 May 2008 00:09:02 -0300, hdante <[e-mail]ó:

*Isn't this guy a bot ? :-) It's learning fast. I believe there is a
"frame" in C, composed of its stack and globals. For generators in C,
you may look for "coroutines". For example, see:

http://www.chiark.greenend.org.uk/~s...oroutines.html

*A sample code follows:

#define crBegin static int state=0; switch(state) { case 0:
#define crReturn(i,x) do { state=i; return x; case i:; } while (0)
#define crFinish }
int function(void) {
* * static int i;
* * crBegin;
* * for (i = 0; i < 10; i++)
* * * * crReturn(1, i);
* * crFinish;
}

*The "suspended state" is saved in the static variable. It's not
necessary to save the complete "frame", only the suspended state.

Quoting the author himself, "this is the worst piece of C hackery ever seen"

--
Gabriel Genellina- Hide quoted text -

- Show quoted text -

At some point, code goes "on" and "off" the processor, which knowledge
I do owe to spending money. Thus, if the group news is a localcy
(other dimension of currency), that's bounce check the house dollar.
What do [second person plural] spend on [vernacular]?

More strictly speaking, processor freely goes on and off binaries. Is
Goto a hardware instruction? Perhaps a Linux-Python-binding could
yield a frame, pun intended. Do [you] have access the file system?

Could yield merely be implemented as multiple strings of processses?
Get the variables on disk. Question is, is it worth optimizing away
from flash rom? There's a gig of chips in a bowling ball.

[J. Cliff Dyer]

On Mon, 2008-05-05 at 10:08 -0700, ca********@gmail.com wrote:

At some point, code goes "on" and "off" the processor, which knowledge
I do owe to spending money. Thus, if the group news is a localcy
(other dimension of currency), that's bounce check the house dollar.
What do [second person plural] spend on [vernacular]?

Sometimes I'm not sure whether castironpi is a bot or a really good
troll who wants us to think he or she is a bot.

At any rate, somebody's having a chuckle.

[metawilm]

On May 4, 1:11 pm, [e-mail] wrote:

There is no such thing as a 'frame' per se in C; byte code is
integral. As there is no such thing as suspended state without
frames, and no such thing as generators without suspended state.

Well, for implementing generators there are alternatives to using
frames + byte code. In CLPython generators are implemented with
closures. (Which does not help much when it comes to reimplementing
generators in C, alas.)

- Willem

[castironpi]

On May 5, 1:55*pm, "[e-mail]" <[e-mail]:

On May 4, 1:11 pm, [e-mail] wrote:

There is no such thing as a 'frame' per se in C; byte code is
integral. *As there is no such thing as suspended state without
frames, and no such thing as generators without suspended state.

Well, for implementing generators there are alternatives to using
frames + byte code. In CLPython generators are implemented with
closures. (Which does not help much when it comes to reimplementing
generators in C, alas.)

- Willem

There's a process decorator to functions in a module.

[supposes]

@process
def datafile( processdict ):
processdict.modify( )
op= yield
op.call( ) in processdict
# op.call( ) in namespace

More simply:

@process
def datafile( processdict ):
processdict.modify( )
interaction_object= yield
invoke( interaction_object, processdict )
#or interaction_object.invoke( **processdict )
processdict.commit( )

Making each execution of process a self-modifying disk file.
Interaction_object could be binary or text. If text, we'd have a
cross-language executable, but I'm suspicious it wouldn't be vacuous
as a Python of Python composition: would other languages pass other
than Python programs?

I think relational databases are the only in there. In other words, I
think Python would make a wholly better stored procedure and query
language than those of databases. Certainly I'm suspicious that's
naivete speaking up; I question, would it? Is there more risk of or
opportunity for malicious execution? Simply put, can we get
primitives in to databases? What's an example of a [optionally real]
operation on the primitives that doesn't map pretty well into SQL?
How do you want to write base-case scenarios if they're going to be
literals?

[hdante]

On May 6, 12:28*am, [e-mail] wrote:

>
There's a process decorator to functions in a module.

[supposes]

@process
def datafile( processdict ):
* *processdict.modify( )
* *op= yield
* *op.call( ) in processdict
* *# op.call( ) in namespace

More simply:

@process
def datafile( processdict ):
* *processdict.modify( )
* *interaction_object= yield
* *invoke( interaction_object, processdict )
* *#or interaction_object.invoke( **processdict )
* *processdict.commit( )

Making each execution of process a self-modifying disk file.
Interaction_object could be binary or text. *If text, we'd have a
cross-language executable, but I'm suspicious it wouldn't be vacuous
as a Python of Python composition: would other languages pass other
than Python programs?

I think relational databases are the only in there. *In other words, I
think Python would make a wholly better stored procedure and query
language than those of databases. *Certainly I'm suspicious that's
naivete speaking up; I question, would it? *Is there more risk of or
opportunity for malicious execution? *Simply put, can we get
primitives in to databases? *What's an example of a [optionally real]
operation on the primitives that doesn't map pretty well into SQL?
How do you want to write base-case scenarios if they're going to be
literals?

Datafiles could be implemented with SQL. Consider SQL:

SELECT * FROM process_dict where id = 1

[castironpi]

On May 6, 1:00*pm, hdante <[e-mail]:

On May 6, 12:28*am, [e-mail] wrote:

There's a process decorator to functions in a module.

[supposes]

@process
def datafile( processdict ):
* *processdict.modify( )
* *op= yield
* *op.call( ) in processdict
* *# op.call( ) in namespace

More simply:

@process
def datafile( processdict ):
* *processdict.modify( )
* *interaction_object= yield
* *invoke( interaction_object, processdict )
* *#or interaction_object.invoke( **processdict )
* *processdict.commit( )

Making each execution of process a self-modifying disk file.
Interaction_object could be binary or text. *If text, we'd have a
cross-language executable, but I'm suspicious it wouldn't be vacuous
as a Python of Python composition: would other languages pass other
than Python programs?

I think relational databases are the only in there. *In other words, I
think Python would make a wholly better stored procedure and query
language than those of databases. *Certainly I'm suspicious that's
naivete speaking up; I question, would it? *Is there more risk of or
opportunity for malicious execution? *Simply put, can we get
primitives in to databases? *What's an example of a [optionally real]
operation on the primitives that doesn't map pretty well into SQL?
How do you want to write base-case scenarios if they're going to be
literals?

*Datafiles could be implemented with SQL. Consider SQL:

*SELECT * FROM process_dict where id = 1
*.
*SELECT * FROM process_dict where id = 1

*Obviously, this is all a supposition, I'm not claiming the code is
perfect (or am I ?). If datafiles could be implemented with SQL and
storing text requires datafiles, then I can store text with datafiles
(haha !). Would you consider doing this ? One could even store python
decorated serialized code using SQL [supposition]. But would it be
useful [haha !] ?

*I don't know.- Hide quoted text -

- Show quoted text -

Serialized code has line numbers, which is fine in Python. (Tab-
indent even comes easier, though it's not clear what granularity
you're storing at, whether each "indention block" gets its own table
or not.)

You are asking for a per-object table.

a= { }
b= [ ]

a[ 'what' ]= object( )

id= new_id( )
process_vars[ id ]= object( )
insert into process_dict.variable_9782 values ( id )
insert into process_dict.variable_9371 values ( 'what', NONPRIMITIVE,
id )

b.append( None )
insert into process_dict.variable_8224 values ( NULL )

b.insert( 0, 0 )
update process_dict.variable_8224 set "column_1"= 0 where index= 0