473,287 Members | 1,880 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

how to iterate over sequence and non-sequence ?

hello,

I generate dynamically a sequence of values,
but this "sequence" could also have length 1 or even length 0.

So I get some line in the form of:
line = '(2,3,4)'
line = ''
line = '(2)'
(in fact these are not constant numbers, but all kind of integer
variables, coming from all over the program, selected from a tree, that
shows all "reachable" variables)

So in fact I get the value from an exec statement, like this
exec 'signals = ' + line

Now I want to iterate over "signals", which works perfect if there are 2
or more signals,
but it fails when I have none or just 1 signal.
for value in signals :
do something

As this meant for real-time signals, I want it fast, so (I think) I
can't afford extensive testing.

Any smart solution there ?

thanks,
Stef Mientki
Oct 19 '07 #1
13 2373
On Oct 19, 12:24 am, stef mientki <stef.mien...@gmail.comwrote:
I generate dynamically a sequence of values,
but this "sequence" could also have length 1 or even length 0.

So I get some line in the form of:
line = '(2,3,4)'
line = ''
line = '(2)'
(in fact these are not constant numbers, but all kind of integer
variables, coming from all over the program, selected from a tree, that
shows all "reachable" variables)

So in fact I get the value from an exec statement, like this
exec 'signals = ' + line

Now I want to iterate over "signals", which works perfect if there are 2
or more signals,
but it fails when I have none or just 1 signal.
for value in signals :
do something

As this meant for real-time signals, I want it fast, so (I think) I
can't afford extensive testing.

Any smart solution there ?
First: don't collect data into strings - python has many container
types which you can use.

Next, your strings look like they're supposed to contain tuples. In
fact, tuples are a bit awkward sometimes because you have to use
'(a,') for a tuple with one element - (2) isn't a tuple of length one,
it's the same as 2. Either cope with this special case, or use lists.
Either way, you'll have to use () or [] for an empty sequence.

--
Paul Hankin

Oct 19 '07 #2
On Fri, 19 Oct 2007 01:24:09 +0200, stef mientki wrote:
hello,

I generate dynamically a sequence of values, but this "sequence" could
also have length 1 or even length 0.

So I get some line in the form of:
line = '(2,3,4)'
line = ''
line = '(2)'
(in fact these are not constant numbers, but all kind of integer
variables, coming from all over the program, selected from a tree, that
shows all "reachable" variables)

So in fact I get the value from an exec statement, like this
exec 'signals = ' + line
And then, one day, somebody who doesn't like you will add the following
to your input data:

"0; import os; os.system('rm # -rf /')"

[ Kids: don't try this at home! Seriously, running that command will be
bad for your computer's health. Or at least it would, if I hadn't put a
spike in it. ]

Don't use exec in production code unless you know what you're doing. In
fact, don't use exec in production code.

Now I want to iterate over "signals", which works perfect if there are 2
or more signals,
but it fails when I have none or just 1 signal.
for value in signals :
do something

No, I would say it already failed before it even got there.
>>line = ''
exec 'signals = ' + line
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<string>", line 1
signals =
^
SyntaxError: unexpected EOF while parsing

This is the right way to deal with your data:

input_data = """ (2, 3 , 4)

(2)
(3,4,5)
( 1, 2,3)
"""

for line in input_data.split('\n'):
line = line.strip().strip('()')
values = line.split(',')
for value in values:
value = value.strip()
if value:
print(value)

As this meant for real-time signals, I want it fast, so (I think) I
can't afford extensive testing.
Don't guess, test it and see if it is fast enough. Some speed ups:

If you're reading from a file, you can just say: "for line in file:"
instead of slurping the whole lot into one enormous string, then
splitting over newlines.

If you can guarantee that there is no extra whitespace in the file, you
can change the line

line = line.strip().strip('()')

to the following:

line = line.strip('\n()')

and save a smidgen of time per loop. Likewise, drop the "value =
value.strip()" in the inner loop.
--
Steven.
Oct 19 '07 #3
On Oct 19, 10:58 am, Steven D'Aprano
<ste...@REMOVE.THIS.cybersource.com.auwrote:
On Fri, 19 Oct 2007 01:24:09 +0200, stef mientki wrote:
hello,
I generate dynamically a sequence of values, but this "sequence" could
also have length 1 or even length 0.
So I get some line in the form of:
line = '(2,3,4)'
line = ''
line = '(2)'
(in fact these are not constant numbers, but all kind of integer
variables, coming from all over the program, selected from a tree, that
shows all "reachable" variables)
So in fact I get the value from an exec statement, like this
exec 'signals = ' + line

And then, one day, somebody who doesn't like you will add the following
to your input data:

"0; import os; os.system('rm # -rf /')"

[ Kids: don't try this at home! Seriously, running that command will be
bad for your computer's health. Or at least it would, if I hadn't put a
spike in it. ]

Don't use exec in production code unless you know what you're doing. In
fact, don't use exec in production code.
Now I want to iterate over "signals", which works perfect if there are 2
or more signals,
but it fails when I have none or just 1 signal.
for value in signals :
do something

No, I would say it already failed before it even got there.
>line = ''
exec 'signals = ' + line

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<string>", line 1
signals =
^
SyntaxError: unexpected EOF while parsing

This is the right way to deal with your data:

input_data = """ (2, 3 , 4)

(2)
(3,4,5)
( 1, 2,3)
"""

for line in input_data.split('\n'):
line = line.strip().strip('()')
values = line.split(',')
for value in values:
value = value.strip()
if value:
print(value)
As this meant for real-time signals, I want it fast, so (I think) I
can't afford extensive testing.

Don't guess, test it and see if it is fast enough. Some speed ups:

If you're reading from a file, you can just say: "for line in file:"
instead of slurping the whole lot into one enormous string, then
splitting over newlines.

If you can guarantee that there is no extra whitespace in the file, you
can change the line

line = line.strip().strip('()')

to the following:

line = line.strip('\n()')

and save a smidgen of time per loop. Likewise, drop the "value =
value.strip()" in the inner loop.

--
Steven.
why not:
>>for i in eval('(1,2,3)'):
.... print i
1
2
3

Oct 19 '07 #4
Nils <ni**********@gmail.comwrote:
why not:
>>>for i in eval('(1,2,3)'):
... print i
1
2
3
For the exact same reason Steven already gave you: one day someone will
give you bad data.

For eval you need to use slightly more complicated expressions. e.g.
"__import__('os').system('rm # -rf /')"
will be sufficient to mess you up.

Oct 19 '07 #5
Paul Hankin wrote:
On Oct 19, 12:24 am, stef mientki <stef.mien...@gmail.comwrote:
>I generate dynamically a sequence of values,
but this "sequence" could also have length 1 or even length 0.

So I get some line in the form of:
line = '(2,3,4)'
line = ''
line = '(2)'
(in fact these are not constant numbers, but all kind of integer
variables, coming from all over the program, selected from a tree, that
shows all "reachable" variables)

So in fact I get the value from an exec statement, like this
exec 'signals = ' + line

Now I want to iterate over "signals", which works perfect if there are 2
or more signals,
but it fails when I have none or just 1 signal.
for value in signals :
do something

As this meant for real-time signals, I want it fast, so (I think) I
can't afford extensive testing.

Any smart solution there ?

First: don't collect data into strings - python has many container
types which you can use.
Well I'm not collecting data, I'm collecting pointers to data.
This program simulates a user written program in JAL.
As Python doesn't support pointers, instead I collect names.
The names are derived from an analysis of the user program under test,
so the danger some of you are referring to, is not there,
or at least is not that simple.
Besides it's a local application where the goal is to let a user test
his program (and hardware),
so if the user want to hack, he can better type directly "format c:\".
Next, your strings look like they're supposed to contain tuples. In
fact, tuples are a bit awkward sometimes because you have to use
'(a,') for a tuple with one element - (2) isn't a tuple of length one,
it's the same as 2. Either cope with this special case, or use lists.
Either way, you'll have to use () or [] for an empty sequence.
Of course, thanks Paul,
if I change tuple to list, everything works ok, even with empty lists.

cheers,
Stef Mientki
--
Paul Hankin

Oct 19 '07 #6
On Fri, 19 Oct 2007 16:19:32 +0200, stef wrote:
Well I'm not collecting data, I'm collecting pointers to data.
I beg to differ, you're collecting data. How that data is to be
interpreted (a string, a number, a pointer...) is a separate issue.

This
program simulates a user written program in JAL. As Python doesn't
support pointers, instead I collect names.
This doesn't make any sense to me. If your user-written program is
supplying pointers (that is, memory addresses like 0x15A8), how do you
get a name from the memory address?
If you are trying to emulate pointer-manipulation, then the usual way to
simulate a pointer is with an integer offset into an array:

# initialise your memory space to all zeroes:
memory = [chr(0)]*1024*64 # 64K of memory space, enough for anyone
NULL = 0
pointer = 45
memory[pointer:pointer + 5] = 'HELLO'
pointer += 6
memory[pointer:pointer + 5] = 'WORLD'

The names are derived from an
analysis of the user program under test, so the danger some of you are
referring to, is not there, or at least is not that simple.
What about accidental clashes between your program's names and the names
you are collecting? Are you sure there are no corner cases where
something you pass to exec can interact badly with your code?

The thing is, exec is stomping through your program's namespace with
great big steel-capped boots, crushing anything that gets in the way.
Even if it is safe in your specific example, it is still bad practice, or
at least risky practice. Code gets reused, copied, and one day a piece of
code you wrote for the JAL project ends up running on a webserver and now
you have a serious security hole.

(Every security hole ever started off with a programmer thinking "This is
perfectly safe to do".)

But more importantly, what makes you think that exec is going to be
faster and more efficient than the alternatives? By my simple test, I
find exec to be about a hundred times slower than directly executing the
same code:
>>timeit.Timer("a = 1").timeit()
0.26714611053466797
>>timeit.Timer("exec s", "s = 'a = 1'").timeit()
25.963317155838013
--
Steven
Oct 19 '07 #7
Steven D'Aprano wrote:
On Fri, 19 Oct 2007 16:19:32 +0200, stef wrote:

>Well I'm not collecting data, I'm collecting pointers to data.

I beg to differ, you're collecting data. How that data is to be
interpreted (a string, a number, a pointer...) is a separate issue.
>This
program simulates a user written program in JAL. As Python doesn't
support pointers, instead I collect names.

This doesn't make any sense to me. If your user-written program is
supplying pointers (that is, memory addresses like 0x15A8), how do you
get a name from the memory address?
If you are trying to emulate pointer-manipulation, then the usual way to
simulate a pointer is with an integer offset into an array:

# initialise your memory space to all zeroes:
memory = [chr(0)]*1024*64 # 64K of memory space, enough for anyone
NULL = 0
pointer = 45
memory[pointer:pointer + 5] = 'HELLO'
pointer += 6
memory[pointer:pointer + 5] = 'WORLD'
If there is a better way, I'ld like to hear it.
I understand that execute is dangerous.

I don't have pointers, I've just names (at least I think).
Let me explain a little bit more,
I want to simulate / debug a user program,
the user program might look like this:

x = 5
for i in xrange(10):
x = x + 1

So now I want to follow the changes in "x" and "i",
therefor in the background I change the user program a little bit, like
this

def user_program():
x = 5 ; _debug(2)
global x,i
_debug (3)
for i in xrange(10):
_debug (3)
x = x + 1 ; _debug (4)

And this modified user program is now called by the main program.
Now in the _debug procedure I can set breakpoints and watch x and i.
But as in this case both a and i are simple integers,
I can not reference them and I need to get their values through their
names,
and thus a execute statement.

I couldn't come up with a better solution ;-)
(There may be no restrictions laid upon the user program, and indeed
name clashing is an accepted risk).

cheers,
Stef


Oct 19 '07 #8
On Oct 19, 5:38 pm, stef mientki <stef.mien...@gmail.comwrote:
... snip hand-coded debugger
I couldn't come up with a better solution ;-)
Does pdb not suffice?

Even if it doesn't; you can look up variables without using exec,
using locals()['x'] or globals()['x']

--
Paul Hankin

Oct 19 '07 #9
stef mientki a écrit :
Steven D'Aprano wrote:
>On Fri, 19 Oct 2007 16:19:32 +0200, stef wrote:
>>Well I'm not collecting data, I'm collecting pointers to data.


I beg to differ, you're collecting data. How that data is to be
interpreted (a string, a number, a pointer...) is a separate issue.

>>This
program simulates a user written program in JAL. As Python doesn't
support pointers, instead I collect names.


This doesn't make any sense to me. If your user-written program is
supplying pointers (that is, memory addresses like 0x15A8), how do you
get a name from the memory address?
If you are trying to emulate pointer-manipulation, then the usual way
to simulate a pointer is with an integer offset into an array:

# initialise your memory space to all zeroes:
memory = [chr(0)]*1024*64 # 64K of memory space, enough for anyone
NULL = 0
pointer = 45
memory[pointer:pointer + 5] = 'HELLO'
pointer += 6
memory[pointer:pointer + 5] = 'WORLD'

If there is a better way, I'ld like to hear it.
I understand that execute is dangerous.

I don't have pointers, I've just names (at least I think).
Let me explain a little bit more,
I want to simulate / debug a user program,
the user program might look like this:

x = 5
for i in xrange(10):
x = x + 1

So now I want to follow the changes in "x" and "i",
therefor in the background I change the user program a little bit, like
this

def user_program():
x = 5 ; _debug(2)
global x,i
_debug (3)
for i in xrange(10):
_debug (3)
x = x + 1 ; _debug (4)
You do know that Python exposes all of it's compilation / AST / whatever
machinery, don't you ? IOW, you can take a textual program, compile it
to a code object, play with the AST, add debug hooks, etc... Perhaps you
should spend a little more time studying the modules index ?
Oct 19 '07 #10
On Fri, 19 Oct 2007 18:38:06 +0200, stef mientki wrote:
I don't have pointers, I've just names (at least I think). Let me
explain a little bit more,
I want to simulate / debug a user program, the user program might look
like this:

x = 5
for i in xrange(10):
x = x + 1

I thought you were writing a JAL interpreter:

http://en.wikipedia.org/wiki/JAL_(compiler)

but the code above is Python.
So now I want to follow the changes in "x" and "i", therefor in the
background I change the user program a little bit, like this

def user_program():
x = 5 ; _debug(2)
global x,i
_debug (3)
for i in xrange(10):
_debug (3)
x = x + 1 ; _debug (4)

And this modified user program is now called by the main program. Now in
the _debug procedure I can set breakpoints and watch x and i.
Python already has a debugger. Try this:

import pdb
help(pdb)

--
Steven.
Oct 20 '07 #11
Paul Hankin wrote:
On Oct 19, 5:38 pm, stef mientki <stef.mien...@gmail.comwrote:
>... snip hand-coded debugger
I couldn't come up with a better solution ;-)

Does pdb not suffice?
thanks very much Paul,
Never heard of that before,
I looked it up, just 1 page in my book of 500 pages ;-)
I'm certainly going to study that.
Even if it doesn't; you can look up variables without using exec,
using locals()['x'] or globals()['x']

Didn't know that either,
I'll try.

thanks,
Stef Mientki
--
Paul Hankin

Oct 20 '07 #12
<snip>
>def user_program():
x = 5 ; _debug(2)
global x,i
_debug (3)
for i in xrange(10):
_debug (3)
x = x + 1 ; _debug (4)

You do know that Python exposes all of it's compilation / AST / whatever
machinery, don't you ? IOW, you can take a textual program, compile it
to a code object, play with the AST, add debug hooks, etc... Perhaps you
should spend a little more time studying the modules index ?
thanks Bruno,
but you're talking about terminology I don't know:
compilation / AST / IOW / module index ???

But to be honest it's not my main goal.
My goal is to write a functional simulator.
You can compare it with a little travelling,
I want to go from A to B.
Normally I'ld travel by bus or train,
but in this case there isn't going a bus to B.
So in this case I take a car,
ask someone how to start it,
and drive to B.

But anyway thanks very much for this and other answers.
cheers,
Stef Mientki
Oct 20 '07 #13
Steven D'Aprano wrote:
On Fri, 19 Oct 2007 18:38:06 +0200, stef mientki wrote:

>I don't have pointers, I've just names (at least I think). Let me
explain a little bit more,
I want to simulate / debug a user program, the user program might look
like this:

x = 5
for i in xrange(10):
x = x + 1


I thought you were writing a JAL interpreter:

http://en.wikipedia.org/wiki/JAL_(compiler)

hi Steven, you're completely right,
in fact I want to some more: also taken hardware and physics into account.
You can see one of my first demo's here, to see what I mean:
http://stef.mientki.googlepages.com/...o_robot1a.html
and some more demos can be found over here:

http://oase.uci.kun.nl/~mientki/data...ted_demos.html
In fact we already used the simulator a number of times in real
applications with great success,
but didn't release it yet, because you still need Python knowledge to
run it reliable.
but the code above is Python.
Yes, right again ;-)
The simulator translates JAL into Python,
In the beginning I thought that was the easiest,
I'm not sure about that anymore at the moment,
but on the other hand that's just replacing one module.
Python already has a debugger. Try this:

import pdb
help(pdb)

Yes, Paul also pointed me into that direction,
and to be honest, I expected there would be such a module,
but I never searched for it, because ....
If I see that I can cook dinner,
when I (very seldom) test the program (I'm writing) in the debug mode,
(assuming the IDE uses the same pdb),
then this is far too slow :-(

I'll report back what my experiences are with pdb.

cheers,
Stef Mientki

Oct 20 '07 #14

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Randall Smith | last post by:
In the case I know how may times I want to iterate, one way to do it is like this: for i in range(100000): dothis() I like how clean this syntax is, but this method is very wasteful. The...
2
by: martin | last post by:
hi, I am using vb.net. I have wrote a data access class and one of my methods returns a dataset. I would like to iterate through this, although this is proving problematic for me. Can...
6
by: eBob.com | last post by:
How do you make a loop iterate early without using a GoTo? (I guess I've done too much structured programming and I really don't like using GoTos.) Here's my code ... For Each Thing As OFI...
2
by: nickheppleston | last post by:
I'm trying to iterate through repeating elements to extract data using libxml2 but I'm having zero luck - any help would be appreciated. My XML source is similar to the following - I'm trying to...
5
by: | last post by:
Hello, I have an array like this: Array ( =Array ( =Array ( =7A585DC0
5
by: ameshkin | last post by:
What I want to do is very simple, but I'm pretty new at PHP and its a little hard for me. I have one page, where there are a rows of checkboxes. A button selects all checkboxes, and then...
1
by: D2 | last post by:
Hi, I need to find all the non-ui controls like errorproviders that have been dropped in the form. Unlike form.controls property, I dont see any collection that maintains all the non-ui controls...
2
by: Efrat Regev | last post by:
Hello, Let's say a probability vector of dimension d is x_1, ..., x_d, where each one is a non-negative term, and they all sum up to 1. Now I'd like to iterate over all probability vectors, but...
15
RMWChaos
by: RMWChaos | last post by:
As usual, an overly-long, overly-explanatory post. Better too much info than too little, right? A couple weeks ago, I asked for some assistance iterating through a JSON property list so that my...
1
by: prathna | last post by:
Hi .. I have a logic:iterate tag which will display 5 rows each row with a drop downlist and 2 textfields.now by default all the rows will be shown.how do i hide all the rows except the first...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.