Possible improvement to slice opperations.

[Ron Adam]

After considering several alternatives and trying out a few ideas with a
modified list object Bengt Richter posted, (Thank You), I think I've
found a way to make slice operation (especially far end indexing)
symmetrical and more consistent.

So to find out if this is indeed a possibility, it would be nice to get
a few opinions at this point. So blast away... or hopefully tell me
what you like about it instead. ;-)

(Any suggestions or contributions to make it better would be appreciated.)

Cheers,
Ron Adam

"""
IMPROVED SLICING
=============== =

Slicing is one of the best features of Python in my opinion, but
when you try to use negative index's and or negative step increments
it can be tricky and lead to unexpected results.

This topic has come up fairly often on comp.lang.pytho n, and often
times, the responses include:

* Beginners should avoid negative extended slices.

* Slices with negative step increments are for advanced
python programmers.

* It's not broke if you look at it in a different way.

* You should do it a different way.

All of these and various responses similar to them are unsatisfactory in
my opinion and it's a good indicator it's not as good as it could be.
Indexing and slice operations are vary common in Python and I don't
think we should be discouraging people from learning and using them.
COMPATIBILITY
-------------
Because the following suggested changes will break current code,
it probably can not be implemented prior to Python 3000.

+ Direct indexing with both positive and negative values
returns the same items as they do now.

+ Extended slices with all positive and or empty default
values remain unchanged.

- Extended slices with negative values return values that
have less items than currently.

- Slices with negative step values return entirely different
results.
REVERSE ORDER STEPPING
----------------------
When negative steps are used, a slice operation
does the following. (or the equivalent)

1. reverse the list
2. cut the reversed sequence using start and stop
3. iterate forward using the absolute value of step.

* This order results in an inverse selection and I believe should be
considered a bug.

Changing the order in the following way results in a much
more predictable pattern which is both easier to understand and use.

1. cut sequence using start and stop.
2 reverse the order of the results.
3. iterate forward using the absolute value of step.
CURRENT INDEXING
----------------

Given a range [a,b,c]:

Positive indexing

| a | b | c |
+---+---+---+
0 1 2 3

Current negative indexing.

| a | b | c |
+---+---+---+
-3 -2 -1 -0
When a single index is used the item to the
right of the index for both positive and
negative index's is returned.

With slices, the items between start, and
stop index's are returned.

Accessing a range at the end of a list numerically
becomes inconvenient when negative index's are used
as the '-0'th position can not be specified numerically
with negative values.
ONES BASED NEGATIVE INDEXING
----------------------------
Making negative index's Ones based, and selecting
individual item to the left of negative index's would enable
addressing the end of the list numerically.

Ones based negative index's.

| a | b | c |
+---+---+---+
-4 -3 -2 -1

Then:

a[-1] -> c # item left of index, same result as now.

a[-3:-2] -> b # item between index's

a[-1:-1] = [d] # insert item at the end.

USE OF '~' IN PLACE OF NEGATIVE INDEX'S
---------------------------------------

The '~' is the binary not symbol which when used
with integers returns the two's compliment. This
works nice with indexing from the end of a list
because convieniently ~0 == -1.

This creates a numerical symmetry between positive
indexing and '~' "End of list" indexing.

a[0] -> first item in the list.
a[~0] -> last item in the list.

a[0:~0] -> whole list.

a[1:~1] -> center, one position from both ends.

* Note: using '~' works as described here in place of single negative
index's in current versions of Python. It does not work as described
here for extended slices.

"""

# TEST LIST CLASS.
"""
A list class to Test '~' end of list indexing.

* This class modifies the slice before returning
a value. The final implementation may do this by
modifying slice objects directly or the underlying
C code of sequences.

"""

class nxlist(object):

def __init__(self, value):
self.value = value

def normslc(self, slc):
start,stop,step = slc.start, slc.stop, slc.step
if type(start) == int and start<0:
start = len(self.value) +start+1
if type(stop) == int and stop<0:
stop = len(self.value) +stop+1
return slice(start,sto p,step)

def __getitem__(sel f, i):
tp = i.__class__
if tp == int:
if i>=0:
return self.value[i]
else:
return self.value[ len(self.value) +i ]
if tp == slice:
slc = self.normslc(i)
value = self.value[slc.start:slc.s top]
if type(i.step) == int and i.step<0:
value.reverse()
slc = slice(None,None ,-i.step)
else:
slc = slice(None,None ,i.step)
return value[slc]

#def __setitem__(sel f, i, v):
#Not emplimented yet.

def __repr__(self): return 'nxlist(%r)'%se lf.value
a = nxlist(range(10 ))
print a

testdata = [
('a[0]'),
('a[1]'),
('a[~0]'),
('a[~1]'),
('a[:]'),
('a[:~1]'),
('a[~1:]'),
('a[::]'),
('a[0:~0]'),
('a[1:~1]'),
('a[~1:1]'),
('a[::-2]'),
('a[:3]'),
('a[3:]'),
('a[~3:]'),
('a[:~3]'),
('a[:3:-1]'),
('a[3::-1]'),
('a[~3::-1]'),
('a[:~3:-1]'),
('a[:3:-2]'),
('a[3::-2]'),
('a[~3::-2]'),
('a[:~3:-2]'),
]

for n, s in enumerate(testd ata):
print '%r. %s = %r' % (n,s,eval(s))
"""
nxlist([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
0. a[0] = 0
1. a[1] = 1
2. a[~0] = 9
3. a[~1] = 8
4. a[:] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
5. a[:~1] = [0, 1, 2, 3, 4, 5, 6, 7, 8]
6. a[~1:] = [9]
7. a[::] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
8. a[0:~0] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
9. a[1:~1] = [1, 2, 3, 4, 5, 6, 7, 8]
10. a[~1:1] = []
11. a[::-2] = [9, 7, 5, 3, 1]
12. a[:3] = [0, 1, 2]
13. a[3:] = [3, 4, 5, 6, 7, 8, 9]
14. a[~3:] = [7, 8, 9]
15. a[:~3] = [0, 1, 2, 3, 4, 5, 6]
16. a[:3:-1] = [2, 1, 0]
17. a[3::-1] = [9, 8, 7, 6, 5, 4, 3]
18. a[~3::-1] = [9, 8, 7]
19. a[:~3:-1] = [6, 5, 4, 3, 2, 1, 0]
20. a[:3:-2] = [2, 0]
21. a[3::-2] = [9, 7, 5, 3]
22. a[~3::-2] = [9, 7]
23. a[:~3:-2] = [6, 4, 2, 0]
"""
r = range(10)
a = nxlist(r)

print r[~0],a[~0] # ok
print r[~3:],a[~3:] # one off
print r[~3::-1],a[~3::-1] # other side
print r[~3::-2],a[~3::-2] # other side
"""
Comparisons of negative indexing and '~'
indexing with same values.

current, proposed

9 9
[6, 7, 8, 9] [7, 8, 9]
[6, 5, 4, 3, 2, 1, 0] [9, 8, 7]
[6, 4, 2, 0] [9, 7]
"""

[Terry Reedy]

"Ron Adam" <rr*@ronadam.co m> wrote in message
news:FR******** **********@torn ado.tampabay.rr .com...

Slicing is one of the best features of Python in my opinion, but
when you try to use negative index's and or negative step increments
it can be tricky and lead to unexpected results.

This topic has come up fairly often on comp.lang.pytho n, and often times,
the responses include:
* Beginners should avoid negative extended slices.
* Slices with negative step increments are for advanced
python programmers.
* It's not broke if you look at it in a different way.
* You should do it a different way.

You omitted the slice-specific response I gave before and repeat here with
more detail by quoting from What's New in Python 2.3.
http://www.python.org/doc/2.3/whatsn...on-slices.html
"
15 Extended Slices
Ever since Python 1.4, the slicing syntax has supported an optional third
``step'' or ``stride'' argument. For example, these are all legal Python
syntax: L[1:10:2], L[:-1:1], L[::-1]. This was added to Python at the
request of the developers of Numerical Python, which uses the third
argument extensively. However, Python's built-in list, tuple, and string
sequence types have never supported this feature, raising a TypeError if
you tried it.
"
Again, extended slices were probably designed by and certainly designed for
Numerical Python and for 7 years were used at least mainly by Numerical
Python. They were not designed for other users like you. The responses
you summarized and distain pretty much all derive from this history.

So, I am pretty sure that changes to the core would have to be upwards
compatible with current usage. On the other hand, your nxlist subclass of
list seems to work pretty well now.The 2.2+ ability to do this sort of
thing is what type subclassing was made for.

Or one could just write an extended slice function similar to your .normslc
method. I might even use such a thing one day.

Terry J. Reedy

[Patrick Maupin]

> After considering several alternatives and trying out a few ideas with a

modified list object Bengt Richter posted, (Thank You), I think I've
found a way to make slice operation (especially far end indexing)
symmetrical and more consistent.

I don't know that it makes it more consistent. I could be persuaded,
but it would have to be by real-life examples with calculated slice
indices and stride. I do this thing all the time, and find the current
rules simple and very consistent. Occasionally, I might wish that
things were a little different, but there is always a workaround. I
would have to see some real code examples, of sufficient scope to see
that there are fewer workarounds with this proposal than with the
current implementation.

FWIW, there is a reasonable workaround for the case where the indices
might be negative and you would like zero or greater to mean 'end of
list'. If "x" is the index variable, you can use the expression (x<0
and x or None) for the index value in the slice. If you find yourself
doing this often, you can write a little function for it -- def
EndIndex(x): return x<0 and x or None.

But in real code, I fear you might need a similar helper function for
similar issues with your change. I just don't know what those are
without more thought.

Regards,
Pat

[Ron Adam]

Terry Reedy wrote:

"Ron Adam" <rr*@ronadam.co m> wrote in message
news:FR******** **********@torn ado.tampabay.rr .com...

Slicing is one of the best features of Python in my opinion, but
when you try to use negative index's and or negative step increments
it can be tricky and lead to unexpected results.

This topic has come up fairly often on comp.lang.pytho n, and often times,
the responses include:
* Beginners should avoid negative extended slices.
* Slices with negative step increments are for advanced
python programmers.
* It's not broke if you look at it in a different way.
* You should do it a different way.

You omitted the slice-specific response I gave before and repeat here with
more detail by quoting from What's New in Python 2.3.
http://www.python.org/doc/2.3/whatsn...on-slices.html
"
15 Extended Slices
Ever since Python 1.4, the slicing syntax has supported an optional third
``step'' or ``stride'' argument. For example, these are all legal Python
syntax: L[1:10:2], L[:-1:1], L[::-1]. This was added to Python at the
request of the developers of Numerical Python, which uses the third
argument extensively. However, Python's built-in list, tuple, and string
sequence types have never supported this feature, raising a TypeError if
you tried it.
"

I didn't omit anything, I previously gave examples of using various
forms of slices with the intent of showing how some of those forms are
not symmetrical and difficult to understand. And then you explained
more precisely how those slices were indexed.

It would be useful to do a side by side comparison of those cases and
similar ones to this new method. Then maybe the benefits of making the
changes I'm suggesting will be clearer.

I looked at the documents in Numerical Python. From what I could tell,
it uses the form L[::-1] quite a bit, which will still work *exactly* as
it does now. There were no examples of partial negative step
operations, so I couldn't determine if/how they are used. I'll install
it and experiment with it and see if/how it works with Numerical Python.

After Python 3000 is released Numerical Python will need to be updated
even without these changes. But it is a good idea to consult with the
Numerical Python developers on this.

Again, extended slices were probably designed by and certainly designed for
Numerical Python and for 7 years were used at least mainly by Numerical
Python. They were not designed for other users like you. The responses
you summarized and distain pretty much all derive from this history.
Other users like me? And what kind of user am I?

Is it really a good idea to maintain that some things shouldn't be done
by some, but is ok for others?

I think you mean to say that extended slices with negative strides,
should only (or primarily) be used with Numerical Python. I however
disagree with that view. I think they have a broader range of uses.

So, I am pretty sure that changes to the core would have to be upwards
compatible with current usage. ...
This is why I said these changes probably couldn't be made before Python
3000. It is upward (backward?) compatible for the majority of use
cases, all direct indexing operations, and all positive extended slices.
In Python's library, only negative extended slices will need to be
adjusted. As far as I could tell there are no partial negative strides
cases that will need to be adjusted. I wish there was a few, because
then I could show how this would improve those cases.

Now even though this suggestion may be sound, that alone isn't reason
enough to include it in Python 3000. It would need clear use case
benefits and support by users as well as support from Guido.

At this point, I'm only trying to clarify the issues and develop working
examples.
... On the other hand, your nxlist subclass of list seems to work
pretty well now. The 2.2+ ability to do this sort of thing is what
type subclassing was made for.
Yes, it does work well, (so far) :-) Which is the motivation for me
posting it. Lets make it better.

Or one could just write an extended slice function similar to your .normslc
method. I might even use such a thing one day.
Unfortunately one can not sub-class slice objects directly. And the ':'
in a slice index isn't a true operator with it's own methods. So
changing existing slice behavior will require changing the core. But
for now I will write a function and a complete list sub-class and post
it here.

Cheers,
Ron
Terry J. Reedy

[Ron Adam]

Patrick Maupin wrote:

After considering several alternatives and trying out a few ideas with a
modified list object Bengt Richter posted, (Thank You), I think I've
found a way to make slice operation (especially far end indexing)
symmetrical and more consistent.

I don't know that it makes it more consistent. I could be persuaded,
but it would have to be by real-life examples with calculated slice
indices and stride. I do this thing all the time, and find the current
rules simple and very consistent. Occasionally, I might wish that
things were a little different, but there is always a workaround. I
would have to see some real code examples, of sufficient scope to see
that there are fewer workarounds with this proposal than with the
current implementation.

I'll post a few examples once I'm sure the list object works correctly
and then you can play around with it and try it out as well.
FWIW, there is a reasonable workaround for the case where the indices
might be negative and you would like zero or greater to mean 'end of
list'. If "x" is the index variable, you can use the expression (x<0
and x or None) for the index value in the slice. If you find yourself
doing this often, you can write a little function for it -- def
EndIndex(x): return x<0 and x or None.
Thanks, this is a good example.

Yes it does remove the need for a work around in those cases.

You still need to check for cases where you may cross the -1,0 boundary
while incrementing or decrementing an index. But that is an expected
condition and easily handled with an 'if' or 'for' loop.

But in real code, I fear you might need a similar helper function for
similar issues with your change. I just don't know what those are
without more thought.

Regards,
Pat

Thanks for the feedback, it was helpful.

Cheers,
Ron

[Szabolcs Nagy]

with the current syntax L[i:i+1] returns [L[i]], with nxlist it returns
L[i+1] if i<0.

L=range(10)
L[1:2]==[L[1]]==[1]
L[-2:-1]==[L[-2]]==[8]

L=nxlist(range( 10))
L[1:2]==[L[1]]==[1]
L[-2:-1]==[L[-1]]==[9] # not [L[-2]]

IMHO in this case current list slicing is more consistent.

[Magnus Lycka]

Ron Adam wrote:

Slicing is one of the best features of Python in my opinion, but
when you try to use negative index's and or negative step increments
it can be tricky and lead to unexpected results.
Hm... Just as with positive indexes, you just need to understand
the concept properly.
This topic has come up fairly often on comp.lang.pytho n, and often
times, the responses include:

* Beginners should avoid negative extended slices.

* Slices with negative step increments are for advanced
python programmers.
I certainly wouldn't respond like that...
* It's not broke if you look at it in a different way.

* You should do it a different way.
Those are correct, but need to be complemented with helpful
explanations. It's often like this, that we think things seem
strange, because we don't see it right. We need a new perspective.
For instance, it took me a long time to understand OOP, but once
the concept was clear, things fell into place. Often, we fail to
see things due to preconceptions that mislead us. Like Yoda said:
"You need to unlearn what you have learnt."
- Extended slices with negative values return values that
have less items than currently.

- Slices with negative step values return entirely different
results.
Over my dead body! ;) Honestly, I can't imagine you'll get agreement
over such a change.
REVERSE ORDER STEPPING
----------------------
When negative steps are used, a slice operation
does the following. (or the equivalent)

1. reverse the list
2. cut the reversed sequence using start and stop
3. iterate forward using the absolute value of step.
I think you are looking at this from the wrong perspective.

Whatever sign c has:
For s[a:b:c], a is the index for the first item to include,
b is the item after the last to include (just like .end() in
C++ iterators for instance), and c describes the step size.

To get a non-empty result, you obviously must have a > b iff
c < 0.

a defaults to 0, b defaults to None (which represents the
item beyond the end of the sequence) and c defaults to 1.

This is basically all you need to understand before you use
a or b < 0. There are no special cases or exceptions.

The concept of negative indices are completely orthogonal
to the concept of slicing today. You can learn and
understand them independently, and will automatically
be able to understand how to use the concepts together,
iff you actually understood both concepts correctly.
1. cut sequence using start and stop.
2 reverse the order of the results.
3. iterate forward using the absolute value of step.
With this solution, you suddenly have two different cases
to consider. You're suggesting that with c < 0, a should be
the end, and b should be the start. Now, it's no longer
obvious whether a or b should be excluded from the result.
I'm pretty sure the number of bugs and questions regarding
negative slices would grow quite a lot.
CURRENT INDEXING
----------------

Given a range [a,b,c]:

Positive indexing

| a | b | c |
+---+---+---+
0 1 2 3

Current negative indexing.

| a | b | c |
+---+---+---+
-3 -2 -1 -0
Almost right, there's no such thing as -0 in Python.
It's more like this: Current negative indexing.

| a | b | c |
+---+---+---+
-3 -2 -1 None Of course, [a,b,c][None] won't work, since it would
access beyond the list. It actually returns a TypeError
rather than an IndexError, and you might argue whether this
is the right thing to do. No big deal in my opinion. For
slices, None works as intended, giving the default in all
three positions. Thid means that if you want a negative
index that might be "-0" for the slice end, you simply
write something like

seq[start:end or None]
Accessing a range at the end of a list numerically
becomes inconvenient when negative index's are used
as the '-0'th position can not be specified numerically
with negative values.
But None works as is...
ONES BASED NEGATIVE INDEXING
----------------------------
Making negative index's Ones based, and selecting
individual item to the left of negative index's would enable
addressing the end of the list numerically.

Ones based negative index's.

| a | b | c |
+---+---+---+
-4 -3 -2 -1
Right. 0-based positive indices, and 1-based negative indices, but
you don't actually get the indexed value, but the one before...
I fear that your cure is worse then the "disease".

Today, a[x:x+1 or None] is always the same as [a[n]] for cases
where the latter doesn't yield an exception. With your change,
it won't be. That's not an improvement in my mind.
a[-3:-2] -> b # item between index's But a[-3] -> a! Shrug!
The '~' is the binary not symbol which when used
with integers returns the two's compliment. This
works nice with indexing from the end of a list
because convieniently ~0 == -1.
But that's only helpful if you want to write literal sequence
indexes, and that's not how real programming works. Further,
it's only an advantage if we agree that your suggestion is
broken and we want some kind of half-baked fix, to make it
work as it should.
This creates a numerical symmetry between positive
indexing and '~' "End of list" indexing.

a[0] -> first item in the list.
a[~0] -> last item in the list.
You can use this silly trick today to, if you think
it looks prettier than a[-1], but for the newbie,
this either means that you introduce magic, or that
you must teach two concepts instead of one.
a[0:~0] -> whole list.
I don't see why that's better than a[:], or a[0:None].
After all, the visual aspect only appears when you type
literals, which isn't very interesting. For calculated
values on the slice borders, you still have -1 as end
value.
a[1:~1] -> center, one position from both ends.

This is just a convoluted way of writing a[1:-2], which
is exactly the same as you would write today.

[Scott David Daniels]

Magnus Lycka wrote:

Ron Adam wrote:

ONES BASED NEGATIVE INDEXING
I think Ron's idea is taking off from my observation that if one's
complement, rather than negation, was used to specify measure-from-
right, we would have a simple consistent system (although I also
observed it is far too late to do that to Python now). Using such
a system should not define things as below:
| a | b | c |
+---+---+---+
-4 -3 -2 -1 but rather use a form like: | a | b | c |
+---+---+---+
~3 ~2 ~1 ~0 The '~' is the binary not symbol which when used
with integers returns the two's compliment. Actually, the ~ operator is the one's complement operator.

For calculated values on the slice borders, you still
have -1 as end value.

But if you are defining the from-right as ones complement,
you use one's complement on the calculated values and
all proceeds happily. Since this could happen in Python,
perhaps we should call it Pythoñ.
a[1:~1] -> center, one position from both ends.

This is just a convoluted way of writing a[1:-2], which
is exactly the same as you would write today.

Actually, a[1 : -1] is how you get to drop the first and
last characters today. I suspect you knew this and were
just a bit in a hurry criticizing a lame-brained scheme.

-Scott David Daniels
Sc***********@A cm.Org

[Ron Adam]

Scott David Daniels wrote:

Magnus Lycka wrote:

Ron Adam wrote:

ONES BASED NEGATIVE INDEXING

I think Ron's idea is taking off from my observation that if one's
complement, rather than negation, was used to specify measure-from-
right, we would have a simple consistent system (although I also
observed it is far too late to do that to Python now). Using such
a system should not define things as below:

I didn't start with your observation in mind, but it ended up there
because the only way to extend the indexes past the last item is to not
use 0 along with negative index's. And once that's done the '~' forms
"just work" without any additional changes. ;-)

Yes, it may be too late for changing Python's built in indexing. And it
may also be that the majority may rather use it as is, rather than
change it to fix this edge case.

| a | b | c |
+---+---+---+
-4 -3 -2 -1
but rather use a form like: >> | a | b | c |
>> +---+---+---+
>> ~3 ~2 ~1 ~0 The '~' is the binary not symbol which when used
with integers returns the two's compliment.
Actually, the ~ operator is the one's complement operator.

Yes, thanks, my mistake.

> For calculated values on the slice borders, you still
> have -1 as end value.

But if you are defining the from-right as ones complement,
you use one's complement on the calculated values and
all proceeds happily. Since this could happen in Python,
perhaps we should call it Pythoñ.
a[1:~1] -> center, one position from both ends.

This is just a convoluted way of writing a[1:-2], which
is exactly the same as you would write today.

Actually, a[1 : -1] is how you get to drop the first and
last characters today. I suspect you knew this and were
just a bit in a hurry criticizing a lame-brained scheme.

-Scott David Daniels
Sc***********@A cm.Org

[Steve Holden]

Scott David Daniels wrote:

Magnus Lycka wrote: [...]

The '~' is the binary not symbol which when used
with integers returns the two's compliment.
Actually, the ~ operator is the one's complement operator.

Actually the two are exactly the same thing. Could we argue about
substantive matters, please? ;-)

> For calculated values on the slice borders, you still
> have -1 as end value.

But if you are defining the from-right as ones complement,
you use one's complement on the calculated values and
all proceeds happily. Since this could happen in Python,
perhaps we should call it Pythoñ.

:-) And how would that be pronounced? I understood that "ñ" would only
appear between two vowels. a[1:~1] -> center, one position from both ends.
This is just a convoluted way of writing a[1:-2], which
is exactly the same as you would write today.

It does have the merit (if you think of it as a merit) of allowing
someone to write

a[n, ~n]

to remove n characters from each end of the string. Frankly I'd rather
deal with the Python that exists now than wrap my head around this
particular suggestion.
Actually, a[1 : -1] is how you get to drop the first and
last characters today. I suspect you knew this and were
just a bit in a hurry criticizing a lame-brained scheme.

Yes, I've been surprised how this thread has gone on and on.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC http://www.holdenweb.com/