Is there a way I can do time series calculation, such as a moving
average in list comprehension syntax? I'm new to python but it looks
like list comprehension's 'head' can only work at a value at a time. I
also tried using the reduce function and passed in my list and another
function which calculates a moving average outside the list comp. ...
but I'm not clear how to do it. Any ideas? Thanks. 9 3098
falcon wrote: Is there a way I can do time series calculation, such as a moving average in list comprehension syntax? I'm new to python but it looks like list comprehension's 'head' can only work at a value at a time. I also tried using the reduce function and passed in my list and another function which calculates a moving average outside the list comp. ... but I'm not clear how to do it. Any ideas? Thanks.
I suggest that statistical data, including time series, be stored and
processed in arrays, such as the one found in NumPy. You can compute
averages using the "sum" function and array slices.
"falcon" <sh******@gmail .com> wrote in message
news:11******** **************@ v46g2000cwv.goo glegroups.com.. . Is there a way I can do time series calculation, such as a moving average in list comprehension syntax? I'm new to python but it looks like list comprehension's 'head' can only work at a value at a time. I also tried using the reduce function and passed in my list and another function which calculates a moving average outside the list comp. ... but I'm not clear how to do it. Any ideas? Thanks.
Write explicit for loops, possibly with nested if conditionals, that do
exactly what you want. The functional expressions are abbreviations for
certain patterns of induction. Except as an educational exercise, I do not
think it worthwhile to go through contortions to force fit a problem to a
pattern it does not really fit.
Terry Jan Reedy
falcon wrote: Is there a way I can do time series calculation, such as a moving average in list comprehension syntax? I'm new to python but it looks like list comprehension's 'head' can only work at a value at a time. I also tried using the reduce function and passed in my list and another function which calculates a moving average outside the list comp. ... but I'm not clear how to do it. Any ideas? Thanks.
I agree with others that reduce is not the best way to do this. But,
to satisfy your curiosity, I offer this horribly inefficient way to use
"reduce" to calculate the average of a list:
from __future__ import division
def reduceaverage(a cc, x):
return [acc[0] + x, acc[1] + 1, (acc[0] + x) / (acc[1] + 1) ]
numbers = [4, 8, 15, 16, 23, 42]
print reduce(reduceav erage, numbers, [0,0,0])[2]
....basically, the idea is to write a function that takes as its first
argument the accumulated values, and as its second argument the next
value in the list. In Python, this is almost always the wrong way to
do something, but it is kind of geeky and LISP-ish.
"falcon" <sh******@gmail .com> writes: Is there a way I can do time series calculation, such as a moving average in list comprehension syntax? I'm new to python but it looks like list comprehension's 'head' can only work at a value at a time. I also tried using the reduce function and passed in my list and another function which calculates a moving average outside the list comp. ... but I'm not clear how to do it. Any ideas? Thanks.
Do you mean something like this?
for i in xrange(5, len(ts)):
# compute and print moving average from i-5 to i
print i, sum(ts[i-5:i]) / 5.
Well, you could iterate over an index into the list:
from __future__ import division
def moving_average( sequence, n):
return [sum(sequence[i:i+n])/n for i in
xrange(len(sequ ence)-n+1)]
Of course, that's hardly efficient. You really want to use the value
calculated for the i_th term in the (i+1)th term's evaluation. While
it's not easy (or pretty) to store state between iterations in a list
comprehension, this is the perfect use for a generator:
def generator_to_li st(f):
return lambda *args,**keyword s: list(f(*args,** keywords))
@generator_to_l ist
def moving_average( sequence, n):
assert len(sequence) >= n and n > 0
average = sum(sequence[:n]) / n
yield average
for i in xrange(1, len(sequence)-n+1):
average += (sequence[i+n-1] - sequence[i-1]) / n
yield average
Lonnie Princehouse wrote: You*really*want *to*use*the*val ue calculated for the i_th term in the (i+1)th term's evaluation.**
It may sometimes be necessary to recalculate the average for every iteration
to avoid error accumulation. Another tradeoff with your optimization is
that it becomes harder to switch the accumulation function from average to
max, say.
While it's not easy (or pretty) to store state between iterations in a list comprehension, this is the perfect use for a generator:
**def*generator _to_list(f): ****return*lamb da**args,**keyw ords:*list(f(*a rgs,**keywords) )
**@generator_to _list **def*moving_av erage(sequence, *n): ****assert*len( sequence)*>=*n* and*n*>*0 ****average*=*s um(sequence[:n])*/*n ****yield*avera ge ****for*i*in*xr ange(1,*len(seq uence)-n+1): ******average*+ =*(sequence[i+n-1]*-*sequence[i-1])*/*n ******yield*ave rage
Here are two more that work with arbitrary iterables:
from __future__ import division
from itertools import islice, tee, izip
from collections import deque
def window(items, n):
it = iter(items)
w = deque(islice(it , n-1))
for item in it:
w.append(item)
yield w # for a robust implementation:
# yield tuple(w)
w.popleft()
def moving_average1 (items, n):
return (sum(w)/n for w in window(items, n))
def moving_average2 (items, n):
first_items, last_items = tee(items)
accu = sum(islice(last _items, n-1))
for first, last in izip(first_item s, last_items):
accu += last
yield accu/n
accu -= first
While moving_average1 () is even slower than your inefficient variant,
moving_average2 () seems to be a tad faster than the efficient one.
Peter
In article <11************ **********@v46g 2000cwv.googleg roups.com>,
falcon <sh******@gmail .com> wrote: Is there a way I can do time series calculation, such as a moving average in list comprehension syntax? I'm new to python but it looks like list comprehension's 'head' can only work at a value at a time. I also tried using the reduce function and passed in my list and another function which calculates a moving average outside the list comp. ... but I'm not clear how to do it. Any ideas? Thanks.
I used the following to return an array of the average of the last n
values -it's not particularly pretty, but it works
# set number of values to average
weighting = 10
# an array of values we want to calculate a running average on
ratings = []
# an array of running averages
running_avg = []
# some routine to fill ratings with the values
r = random.Random()
for i in range(0, 20):
ratings.append( float(r.randint (0, 99)))
for i in range(1, 1 + len(ratings)):
if i < weighting:
running_avg.app end(ratings[i - 1])
else:
running_avg.app end(reduce(lamb da s, a: s+ a,
ratings[i - weighting : i]) /
len(ratings[i - weighting : i]))
for i in range(0, len(ratings)):
print "%3d: %3d %5.2f" % (i, ratings[i], running_avg[i])
sample output:
0: 34 34.00
1: 28 28.00
2: 58 58.00
3: 16 34.00
4: 74 44.00
5: 32 45.00
6: 74 49.00
7: 21 50.25
8: 78 51.25
9: 28 50.25
10: 32 39.75
11: 93 57.75
12: 2 38.75
13: 7 33.50
14: 8 27.50
15: 30 11.75
16: 1 11.50
17: 8 11.75
18: 40 19.75
19: 8 14.25
For all but the first 3 rows, the third column is the average of the
values in the 2nd column for this and the preceding 3 rows.
--
Jim Segrave (je*@jes-2.demon.nl)
[Peter Otten] from __future__ import division
from itertools import islice, tee, izip
. . . def moving_average2 (items, n): first_items, last_items = tee(items) accu = sum(islice(last _items, n-1)) for first, last in izip(first_item s, last_items): accu += last yield accu/n accu -= first
While moving_average1 () is even slower than your inefficient variant, moving_average2 () seems to be a tad faster than the efficient one.
This is nicely done and scales-up well. Given an n-average of m-items,
it has O(n) memory consumption and O(m) running time. In contrast, the
other variants do more work than necessary by pulling the whole
sequence into memory or by re-summing all n items at every step,
resulting in O(m) memory consumption and O(m*n) running time.
This recipe gets my vote for the best solution.
Raymond
Wow, thanks for all the reponses. Very helpful! This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: jsaul |
last post by:
Hi there,
wouldn't it be useful to have a 'while' conditional in addition to
'if' in list comprehensions?
foo =
for i in bar:
if len(i) == 0:
break
foo.append(i)
|
by: Chris P. |
last post by:
Hi. I've made a program that logs onto a telnet server, enters a
command, and then creates a list of useful information out of the
information that is dumped to the screen as a result of the command.
Here's a generic version of the code in question:
#####
# Prior code opens telnet connection "tn" and logs in.
tn.read_until('> ')
tn.write('THE COMMAND IS HERE\n')
|
by: jena |
last post by:
hello,
when i create list of lambdas:
l=]
then l() returns 'C', i think, it should be 'A'
my workaround is to define helper class with __call__ method:
class X:
def __init__(self,s): self.s=s
def __call__(self): return self.s.upper()
l=]
|
by: a |
last post by:
can someone tell me how to use them
thanks
|
by: Gregory Guthrie |
last post by:
Sorry for a simple question- but I don't understand how to parse this use of
a list comprehension.
The "or" clauses are odd to me.
It also seems like it is being overly clever (?) in using a lc expression as
a for loop to drive the recursion.
Thanks for any insight!
Gregory
| |
by: bullockbefriending bard |
last post by:
Given:
class Z(object):
various defs, etc.
class ZList(list):
various defs, etc.
i would like to be able to replace
|
by: Debajit Adhikary |
last post by:
I have two lists:
a =
b =
What I'd like to do is append all of the elements of b at the end of
a, so that a looks like:
a =
|
by: beginner |
last post by:
Hi All,
If I have a list comprehension:
ab=
c = "ABC"
print c
|
by: Ken Pu |
last post by:
Hi all,
I observed an interesting yet unpleasant variable scope behaviour with
list comprehension in the following code:
print
print x
It outputs:
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |