473,320 Members | 2,020 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

time series data and NumPy

Good morning,

I store time series data in a SQL database. The results of a typical
query using pyodbc look like this.

Date Close
"2007-01-17" 22.57

Where Date is a datetime.date object and Close is a float.

I'd like to put this data in a NumPy array for processing, but am
unsure as to how to handle the date. In the past I've used lists, but I
am looking to boost speed a bit as I wish to do a large number of
transformations and comparisons.

Can one index an array using datetime objects?

For example it would be nice to do a union of two arrays so that any
dates missing in either one were eliminated.

Thoughts on doing rolling operations, such as an n-period average or
variance?

Thoughts on working with time series data in arrays in general?

Thanks in advance,

jab--who is very happily returning to Python after a sojourn in
R-land

Jan 26 '07 #1
6 10225
BBands wrote:
Good morning,

I store time series data in a SQL database. The results of a typical
query using pyodbc look like this.

Date Close
"2007-01-17" 22.57

Where Date is a datetime.date object and Close is a float.

I'd like to put this data in a NumPy array for processing, but am
unsure as to how to handle the date. In the past I've used lists, but I
am looking to boost speed a bit as I wish to do a large number of
transformations and comparisons.

Can one index an array using datetime objects?

For example it would be nice to do a union of two arrays so that any
dates missing in either one were eliminated.

Thoughts on doing rolling operations, such as an n-period average or
variance?

Thoughts on working with time series data in arrays in general?
I'm pretty sure you're out of luck here - even _if_ NumPy would handle
arbitrary data-types (AFAIK it doesn't, but then I'm not a total expert
there), it certainly won't be able to make its hi-performance functions
work on them.

What you could do would be to convert the date-column into a timestamp,
which is a int/long, and use that. Would that help?

diez
Jan 26 '07 #2
On Jan 26, 9:29 am, "Diez B. Roggisch" <d...@nospam.web.dewrote:
What you could do would be to convert the date-column into a timestamp,
which is a int/long, and use that. Would that help?
Actually that might help, as all I need the date for is to index
values.

Thanks, I'll give it a spin.

jab

Jan 26 '07 #3
On 2007-01-26 10:54:02 -0700, "BBands" <bb****@gmail.comsaid:
On Jan 26, 9:29 am, "Diez B. Roggisch" <d...@nospam.web.dewrote:
>What you could do would be to convert the date-column into a timestamp,
which is a int/long, and use that. Would that help?

Actually that might help, as all I need the date for is to index
values.

Thanks, I'll give it a spin.

jab
You're using the Python-MySQL module mysqldb, right? You can select
the data from the database and have MySQL do the conversion with an SQL
command.

select to_days(<your date column name here>), <your close columnfrom
<your table>;

Bob
Jan 26 '07 #4
Diez B. Roggisch wrote:
I'm pretty sure you're out of luck here - even _if_ NumPy would handle
arbitrary data-types (AFAIK it doesn't, but then I'm not a total expert
there), it certainly won't be able to make its hi-performance functions
work on them.
Yes, one can make numpy arrays with "object" as its type. One can even extend
the C-level parts as well. For example, we have an experimental package in the
scipy sandbox for uniform time series that uses mx.DateTime.

http://www.scipy.org/TimeSeriesPackage
What you could do would be to convert the date-column into a timestamp,
which is a int/long, and use that. Would that help?
This is frequently what I do. For dates, I like Modified Julian Day Numbers
although I am sure that would horrify some people more knowledgeable than I.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

Jan 26 '07 #5
On Jan 26, 10:18 am, Bob Greschke wrote:
You're using the Python-MySQL module mysqldb, right?
Actually I using MySQL with pyodbc as the mysqldb Windows binaries for
Python 2.5 aren't out yet. :-(
You can select the data from the database and have
MySQL do the conversion with an SQL command.

select to_days(<your date column name here>),
<your close columnfrom <your table>;
That works very nicely indeed.

Thanks,

jab

Jan 26 '07 #6
On Jan 26, 10:46 am, Robert Kern <robert.k...@gmail.comwrote:
Yes, one can make numpy arrays with "object" as its type. One can even extend
the C-level parts as well. For example, we have an experimental package in the
scipy sandbox for uniform time series that uses mx.DateTime.

http://www.scipy.org/TimeSeriesPackage
I saw that, but I am working with irregular time series.
This is frequently what I do. For dates, I like Modified Julian Day Numbers
although I am sure that would horrify some people more knowledgeable than I.
Not horrified, just doing it. ;-)

jab

Jan 26 '07 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: sdhyok | last post by:
Hi, I am trying to build up a system handling time series data a lot. Do you know any well-designed python class specially for time series data? Thanks in advance. Shin, Daehyok
9
by: falcon | last post by:
Is there a way I can do time series calculation, such as a moving average in list comprehension syntax? I'm new to python but it looks like list comprehension's 'head' can only work at a value at...
6
by: Gaz | last post by:
Hi guys. I've been lookig for this in the numpy pdf manual, in this group and on google, but i could not get an answer... Is there a way to create a custom data type (eg: Name: string(30), Age:...
13
by: Gary Wessle | last post by:
Hi I am trying to install NumPy in my debian/testing linux 2.6.15-1-686. with no numpy for debian/testing, I am left alone, since the experimental version available by debian will result in a...
5
by: Ray Tomes | last post by:
Hi Folks I am an old codger who has much experience with computers in the distant past before all this object oriented stuff. Also I have loads of software in such languages as FORTRAN and...
11
by: Fie Pye | last post by:
Hallo I would like to have a high class open source tools for scientific computing and powerful 2D and 3D data visualisation. Therefore I chosepython, numpy and scipy as a base. Now I am in...
10
by: Tyler | last post by:
Hello All: After trying to find an open source alternative to Matlab (or IDL), I am currently getting acquainted with Python and, in particular SciPy, NumPy, and Matplotlib. While I await the...
33
by: desktop | last post by:
In the C++ standard sec 23.1.2 table 69 it says that erase(q) where q is a pointer to an element can be done in amortized constant time. I guess that is not worst case since std::set is...
5
by: Mars creature | last post by:
Hi guys, I am new to Python, and thinking about migrating to it from matlab as it is a really cool language. Right now, I am trying to figure out how to control read and write binary data, like...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.