Hi all,
I have a collection of ordered numerical data in a list. The numbers
when plotted on a line chart make a lowhighlowhighhighlow (random)
pattern. I need an algorithm to extract the "significan t" high and low
points from this data.
Here is some sample data:
data = [0.10, 0.50, 0.60, 0.40, 0.39, 0.50, 1.00, 0.80, 0.60, 1.20,
1.10, 1.30, 1.40, 1.50, 1.05, 1.20, 0.90, 0.70, 0.80, 0.40, 0.45, 0.35,
0.10]
In this data, some of the significant points include:
data[0]
data[2]
data[4]
data[6]
data[8]
data[9]
data[13]
data[14]
.....
How do I sort through this data and pull out these points of
significance?
Thanks for your help!
Erik 12 1742
erikcw wrote:
I have a collection of ordered numerical data in a list. The numbers
when plotted on a line chart make a lowhighlowhighhighlow (random)
pattern. I need an algorithm to extract the "significan t" high and low
points from this data.
....
>
How do I sort through this data and pull out these points of
significance?
Get a book on statistics. One idea is as follows. If you expect the points
to be centred around a single value, you can calculate the median or mean
of the points, calculate their standard deviation (aka spread), and remove
points which are more than Ntimes the standard deviation from the median.
Jeremy

Jeremy Sanders http://www.jeremysanders.net/
"erikcw" wrote:
I have a collection of ordered numerical data in a list. The numbers
when plotted on a line chart make a lowhighlowhighhighlow (random)
pattern. I need an algorithm to extract the "significan t" high and low
points from this data.
Here is some sample data:
data = [0.10, 0.50, 0.60, 0.40, 0.39, 0.50, 1.00, 0.80, 0.60, 1.20,
1.10, 1.30, 1.40, 1.50, 1.05, 1.20, 0.90, 0.70, 0.80, 0.40, 0.45, 0.35,
0.10]
silly solution:
for i in range(1, len(data)1):
if data[i1] < data[i] data[i+1] or data[i1] data[i] < data[i+1]:
print i
(the above doesn't handle the "edges", but that's easy to fix)
</F>
erikcw <er***********@ gmail.comwrote:
I have a collection of ordered numerical data in a list. The numbers
when plotted on a line chart make a lowhighlowhighhighlow (random)
pattern. I need an algorithm to extract the "significan t" high and low
points from this data.
I am not sure, what you mean by 'ordered' in this context. As
pointed out by Jeremy, you need to find an appropriate statistical test.
The appropriateness depend on how your data is (presumably) distributed
and what exactly you are trying to test. E.g. do te data pints come from
differetn groupos of some kind? Or are you just looking for extreme
values (outliers maybe?)?
So it's more of statistical question than a python one.
cu
Philipp

Dr. Philipp Pagel Tel. +49816171 2131
Dept. of Genome Oriented Bioinformatics Fax. +49816171 2186
Technical University of Munich http://mips.gsf.de/staff/pagel
erikcw wrote:
Hi all,
I have a collection of ordered numerical data in a list. The numbers
when plotted on a line chart make a lowhighlowhighhighlow (random)
pattern. I need an algorithm to extract the "significan t" high and low
points from this data.
Here is some sample data:
data = [0.10, 0.50, 0.60, 0.40, 0.39, 0.50, 1.00, 0.80, 0.60, 1.20,
1.10, 1.30, 1.40, 1.50, 1.05, 1.20, 0.90, 0.70, 0.80, 0.40, 0.45, 0.35,
0.10]
In this data, some of the significant points include:
data[0]
data[2]
data[4]
data[6]
data[8]
data[9]
data[13]
data[14]
....
How do I sort through this data and pull out these points of
significance?
I think you are looking for "extrema":
def w3(items):
items = iter(items)
view = None, items.next(), items.next()
for item in items:
view = view[1:] + (item,)
yield view
for i, (a, b, c) in enumerate(w3(da ta)):
if a b < c:
print i+1, "min", b
elif a < b c:
print i+1, "max", b
else:
print i+1, "", b
Peter
If the order doesn't matter, you can sort the data and remove x * 0.5 *
n where x is the proportion of numbers you want. If you have too many
similar values though, this falls down. I suggest you check out
quantiles in a good statistics book.
Alan.
Peter Otten wrote:
erikcw wrote:
Hi all,
I have a collection of ordered numerical data in a list. The numbers
when plotted on a line chart make a lowhighlowhighhighlow (random)
pattern. I need an algorithm to extract the "significan t" high and low
points from this data.
Here is some sample data:
data = [0.10, 0.50, 0.60, 0.40, 0.39, 0.50, 1.00, 0.80, 0.60, 1.20,
1.10, 1.30, 1.40, 1.50, 1.05, 1.20, 0.90, 0.70, 0.80, 0.40, 0.45, 0.35,
0.10]
In this data, some of the significant points include:
data[0]
data[2]
data[4]
data[6]
data[8]
data[9]
data[13]
data[14]
....
How do I sort through this data and pull out these points of
significance?
I think you are looking for "extrema":
def w3(items):
items = iter(items)
view = None, items.next(), items.next()
for item in items:
view = view[1:] + (item,)
yield view
for i, (a, b, c) in enumerate(w3(da ta)):
if a b < c:
print i+1, "min", b
elif a < b c:
print i+1, "max", b
else:
print i+1, "", b
Peter
erikcw wrote:
I have a collection of ordered numerical data in a list. The numbers
when plotted on a line chart make a lowhighlowhighhighlow (random)
pattern. I need an algorithm to extract the "significan t" high and low
points from this data.
In calculus, you identify high and low points by looking where the
derivative changes its sign. When working with discrete samples, you can
look at the sign changes in finite differences:
>>data = [...] diff = [data[i + 1]  data[i] for i in range(len(data) )] map(str, diff)
['0.4', '0.1', '0.2', '0.01', '0.11', '0.5', '0.2', '0.2', '0.6',
'0.1', '0.2', '0.1', '0.1', '0.45', '0.15', '0.3', '0.2', '0.1',
'0.4', '0.05', '0.1', '0.25']
The high points are those where diff changes from + to , and the low
points are those where diff changes from  to +.
HTH,

Roberto Bonvallet
>>>>Jeremy Sanders <je************ *******@jeremys anders.netwrite s:
>How do I sort through this data and pull out these points of significance ?
Get a book on statistics. One idea is as follows. If you expect the points
to be centred around a single value, you can calculate the median or mean
of the points, calculate their standard deviation (aka spread), and remove
points which are more than Ntimes the standard deviation from the median.
Standard deviation was the first thought that jumped to my mind
too. However, that's not what the OP is after. He's seems to be looking for
points when the direction changes.
Ganesan

Ganesan Rajagopal
"erikcw" <er***********@ gmail.comwrote:
I have a collection of ordered numerical data in a list. The numbers
when plotted on a line chart make a lowhighlowhighhighlow (random)
pattern. I need an algorithm to extract the "significan t" high and low
points from this data.
I think you want a control chart. A good place to start might be http://en.wikipedia.org/wiki/Control_chart. Even if you don't actually
graph the data, understanding the math behind control charts might help you
with your analysis.
Wow. I think this is the first time I'm actually used something I learned
by sitting though those stupid Six Sigma training classes :)
erikcw wrote:
Hi all,
I have a collection of ordered numerical data in a list.
Called a "time series" in statistics.
The numbers
when plotted on a line chart make a lowhighlowhighhighlow (random)
pattern. I need an algorithm to extract the "significan t" high and low
points from this data.
Here is some sample data:
data = [0.10, 0.50, 0.60, 0.40, 0.39, 0.50, 1.00, 0.80, 0.60, 1.20,
1.10, 1.30, 1.40, 1.50, 1.05, 1.20, 0.90, 0.70, 0.80, 0.40, 0.45, 0.35,
0.10]
In this data, some of the significant points include:
data[0]
data[2]
data[4]
data[6]
data[8]
data[9]
data[13]
data[14]
....
How do I sort through this data and pull out these points of
significance?
The best place to ask about an algorithm for this is not
comp.lang.pytho n  maybe sci.stat.math would be better. Once you have
an algorithm, coding it in Python should not be difficult. I'd suggest
using the NumPy array rather than the native Python list, which is not
designed for crunching numbers. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics 
by: Jim Garrison 
last post by:
I need to create a ResultSet in an Oracle Java Stored Procedure and
return it to a PL/SQL caller. I've done quite a bit of research in
Oracle's manuals and on the Web, and have found lots of references to
going the other direction (PL/SQL passing/returning REF CURSOR to
Java), and Java returning an existing ResultSet (obtained from the...

by: James A. Donald 
last post by:
I am contemplating getting into Python, which is used by engineers I
admire  google and Bram Cohen, but was horrified to read
"no variable or argument declarations are necessary."
Surely that means that if I misspell a variable name, my program will
mysteriously fail to work with no error message.
If you don't declare variables, you...

by: Matt 
last post by:
I have 2 questions:
1. strlen returns an unsigned (size_t) quantity. Why is an unsigned
value more approprate than a signed value? Why is unsighned value less
appropriate?
2. Would there be any advantage in having strcat and strcpy return a
pointer to the "end" of the destination string rather than returning a

by: Rico 
last post by:
Hello,
I have an application that I'm converting to Access 2003 and SQL Server 2005
Express. The application uses extensive use of DAO and the SEEK method on
indexes. I'm having an issue when the recordset opens a table. When I
write
Set rst = db.OpenRecordset("MyTable",dbOpenTable, dbReadOnly)
I get an error. I believe it's...

by: jht5945 
last post by:
For example I wrote a function:
function Func()
{
// do something
}
we can call it like:
var obj = new Func(); // call it as a constructor
or
var result = Func(); // call it as a function
 
by: =?Utf8?B?YjF1Y2VyZWU=?= 
last post by:
Hi, still very new to programming but am a little stumped by how to do this
idea i have.
I need to make a moving average which takes every nth value in a data series
to build a running total which can then be divided by the length of the
moving average. The data series will be gaining a new value every x amount of
time t so the average has...

by: none 
last post by:
Hello,
IIRC, I once saw an explanation how Python doesn't have "variables" in
the sense that, say, C does, and instead has bindings from names to
objects. Does anyone have a link?
Thanks,
Ami

by: Bob Altman 
last post by:
Hi all,
I want to write a generic class that does this:
Public Class X (Of T)
Public Sub Method(param As T)
dim x as T = param >3
End Sub
End Class

by: Astley Le Jasper 
last post by:
Sorry for the numpty question ...
How do you find the reference name of an object?
So if i have this
bob = modulename.objectname()
how do i find that the name is 'bob'

by: marktang 
last post by:
ONU (Optical Network Unit) is one of the key components for providing highspeed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...

by: Hystou 
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language...
 
by: Oralloy 
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bitfields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it. ...

by: Hystou 
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...

by: agi2029 
last post by:
Let's talk about the concept of autonomous AI software engineers and nocode agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...

by: isladogs 
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes...

by: conductexam 
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert...

by: 6302768590 
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
 
by: bsmnconsultancy 
last post by:
In today's digital era, a welldesigned website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...
 