Identifying the start of good data in a list

[tkpmep]

I have a list that starts with zeros, has sporadic data, and then has
good data. I define the point at which the data turns good to be the
first index with a non-zero entry that is followed by at least 4
consecutive non-zero data items (i.e. a week's worth of non-zero
data). For example, if my list is [0, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8,
9], I would define the point at which data turns good to be 4 (1
followed by 2, 3, 4, 5).

I have a simple algorithm to identify this changepoint, but it looks
crude: is there a cleaner, more elegant way to do this?

flag = True
i=-1
j=0
while flag and i < len(retHist)-1:
i += 1
if retHist[i] == 0:
j = 0
else:
j += 1
if j == 5:
flag = False

del retHist[:i-4]

Thanks in advance for your help

Thomas Philips

[castironpi]

On Aug 27, 3:42*pm, George Sakkis <george.sak...@ gmail.comwrote:

Below are two more versions that pass all the doctests: the first
works only for lists and modifies them in place and the second works
for arbitrary iterables:

def clean_inplace(s eq, good_ones=4):
* * start = 0
* * n = len(seq)
* * while start < n:
* * * * try: end = seq.index(0, start)
* * * * except ValueError: end = n
* * * * if end-start >= good_ones:
* * * * * * break
* * * * start = end+1
* * del seq[:start]

def clean_iter(iter able, good_ones=4):
* * from itertools import chain, islice, takewhile, dropwhile
* * iterator = iter(iterable)
* * is_zero = float(0).__eq__
* * while True:
* * * * # consume all zeros up to the next non-zero
* * * * iterator = dropwhile(is_ze ro, iterator)
* * * * # take up to `good_ones` non-zeros
* * * * good = list(islice(tak ewhile(bool,ite rator), good_ones))
* * * * if not good: # iterator exhausted
* * * * * * return iterator
* * * * if len(good) == good_ones:
* * * * * * # found `good_ones` consecutive non-zeros;
* * * * * * # chain them to the rest items and return them
* * * * * * return chain(good, iterator)

HTH,
George

You gave me an idea-- maybe an arbitrary 'lookahead' iterable could be
useful. I haven't seen them that much on the newsgroup, but more than
once. IOW a buffered consumer. Something that you could check a
fixed number of next elements of. You might implement it as a
iterator with a __getitem__ method.

Example, unproduced:

>>import itertools
a= itertools.count ( )
a.next()

0

>>a.next()

1

>>a[ 3 ]

5

>>a.next()

2

>>a[ 3 ]

6

Does this make sense at all?

[tdmj]

On Aug 27, 11:50 am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com .auwrote:

On Tue, 26 Aug 2008 17:04:19 -0700, tdmj wrote:

On Aug 26, 5:49 pm, tkp...@hotmail. com wrote:

I have a list that starts with zeros, has sporadic data, and then has
good data. I define the point at which the data turns good to be the
first index with a non-zero entry that is followed by at least 4
consecutive non-zero data items (i.e. a week's worth of non-zero data).
For example, if my list is [0, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], I
would define the point at which data turns good to be 4 (1 followed by
2, 3, 4, 5).

...

With regular expressions:

Good grief. If you're suggesting that as a serious proposal, and not just
to prove it can be done, that's surely an example of "when all you have
is a hammer, everything looks like a nail" thinking.

In this particular case, your regex "solution" gives the wrong result,
indicating that you didn't test your code before posting. Hint:

re.search(r'[1-9]{5, }', "123456")

returns None.

The obvious fix for that specific bug is to use r'[1-9]{5,5}', but even
that will fail. Hint: what happens if an item has more than one digit?

Before posting another regex solution, make sure it does the right thing
with this:

[0, 0, 101, 0, 1002, 203, 3050, 4105, 5110, 623, 777]

--
Steven

Hey, it's clearer than a lot of the other proposals here. Too bad it
doesn't work. This is why you don't post after 8 p.m. after being at
work all day. I was seeing what I now recall as incorrect answers, but
at the time I was in the midst of a brainfart and for some reason took
them to be right. It can be made to work by removing the space in
"{5, }", inserting some kind of marker between the numbers, and using
the right regular expression to recognize nonzero numbers between the
markers, but I think I've already said too much in this thread.

Tommy McDaniel

[Jorgen Grahn]

On 27 Aug 2008 15:50:14 GMT, Steven D'Aprano <st***@REMOVE-THIS-cybersource.com .auwrote:

On Tue, 26 Aug 2008 17:04:19 -0700, tdmj wrote:

>On Aug 26, 5:49 pm, tkp...@hotmail. com wrote:

>>I have a list that starts with zeros, has sporadic data, and then has
good data. I define the point at which the data turns good to be the
first index with a non-zero entry that is followed by at least 4
consecutive non-zero data items (i.e. a week's worth of non-zero data).
For example, if my list is [0, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], I
would define the point at which data turns good to be 4 (1 followed by
2, 3, 4, 5).

...

>With regular expressions:

Good grief. If you're suggesting that as a serious proposal, and not just
to prove it can be done, that's surely an example of "when all you have
is a hammer, everything looks like a nail" thinking.

Maybe I'm stumbling into a "REs are evil" flamewar here. Anyway:

He has a point though: this *can* be seen as a regex problem. Only a
solution which builds a string first is only good for laughs or
(possibly) quick hacks. What's missing is an RE library for lists of
objects, rather than just strings and Unicode strings.

Not sure such a library would be worth implementing -- problems like
this one are rare, I think.

/Jorgen

--
// Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
\X/ snipabacken.se R'lyeh wgah'nagl fhtagn!

[castironpi]

On Aug 29, 9:43*am, Jorgen Grahn <grahn+n...@sni pabacken.sewrot e:

On 27 Aug 2008 15:50:14 GMT, Steven D'Aprano <[e-mail] .auwrote:

On Tue, 26 Aug 2008 17:04:19 -0700, tdmj wrote:

On Aug 26, 5:49 pm, tkp...@hotmail. com wrote:
I have a list that starts with zeros, has sporadic data, and then has
good data. I define the point at *which the data turns good to be the
first index with a non-zero entry that is followed by at least 4
consecutive non-zero data items (i.e. a week's worth of non-zero data).
For example, if my list is [0, 0, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], I
would define the point at which data turns good to be 4 (1 followed by
2, 3, 4, 5).

He has a point though: this *can* be seen as a regex problem. Only a
solution which builds a string first is only good for laughs or
(possibly) quick hacks. What's missing is an RE library for lists of
objects, rather than just strings and Unicode strings.

Not sure such a library would be worth implementing -- problems like
this one are rare, I think.

Every now and then, you see a proposal or a package for a finite state
machine--- how would you encode comparing of values into a string, if
you're not comparing a string?