Good morning,
I store time series data in a SQL database. The results of a typical
query using pyodbc look like this.
Date Close
"2007-01-17" 22.57
Where Date is a datetime.date object and Close is a float.
I'd like to put this data in a NumPy array for processing, but am
unsure as to how to handle the date. In the past I've used lists, but I
am looking to boost speed a bit as I wish to do a large number of
transformations and comparisons.
Can one index an array using datetime objects?
For example it would be nice to do a union of two arrays so that any
dates missing in either one were eliminated.
Thoughts on doing rolling operations, such as an n-period average or
variance?
Thoughts on working with time series data in arrays in general?
Thanks in advance,
jab--who is very happily returning to Python after a sojourn in
R-land 6 10225
BBands wrote:
Good morning,
I store time series data in a SQL database. The results of a typical
query using pyodbc look like this.
Date Close
"2007-01-17" 22.57
Where Date is a datetime.date object and Close is a float.
I'd like to put this data in a NumPy array for processing, but am
unsure as to how to handle the date. In the past I've used lists, but I
am looking to boost speed a bit as I wish to do a large number of
transformations and comparisons.
Can one index an array using datetime objects?
For example it would be nice to do a union of two arrays so that any
dates missing in either one were eliminated.
Thoughts on doing rolling operations, such as an n-period average or
variance?
Thoughts on working with time series data in arrays in general?
I'm pretty sure you're out of luck here - even _if_ NumPy would handle
arbitrary data-types (AFAIK it doesn't, but then I'm not a total expert
there), it certainly won't be able to make its hi-performance functions
work on them.
What you could do would be to convert the date-column into a timestamp,
which is a int/long, and use that. Would that help?
diez
On Jan 26, 9:29 am, "Diez B. Roggisch" <d...@nospam.web.dewrote:
What you could do would be to convert the date-column into a timestamp,
which is a int/long, and use that. Would that help?
Actually that might help, as all I need the date for is to index
values.
Thanks, I'll give it a spin.
jab
On 2007-01-26 10:54:02 -0700, "BBands" <bb****@gmail.comsaid:
On Jan 26, 9:29 am, "Diez B. Roggisch" <d...@nospam.web.dewrote:
>What you could do would be to convert the date-column into a timestamp, which is a int/long, and use that. Would that help?
Actually that might help, as all I need the date for is to index
values.
Thanks, I'll give it a spin.
jab
You're using the Python-MySQL module mysqldb, right? You can select
the data from the database and have MySQL do the conversion with an SQL
command.
select to_days(<your date column name here>), <your close columnfrom
<your table>;
Bob
Diez B. Roggisch wrote:
I'm pretty sure you're out of luck here - even _if_ NumPy would handle
arbitrary data-types (AFAIK it doesn't, but then I'm not a total expert
there), it certainly won't be able to make its hi-performance functions
work on them.
Yes, one can make numpy arrays with "object" as its type. One can even extend
the C-level parts as well. For example, we have an experimental package in the
scipy sandbox for uniform time series that uses mx.DateTime. http://www.scipy.org/TimeSeriesPackage
What you could do would be to convert the date-column into a timestamp,
which is a int/long, and use that. Would that help?
This is frequently what I do. For dates, I like Modified Julian Day Numbers
although I am sure that would horrify some people more knowledgeable than I.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
On Jan 26, 10:18 am, Bob Greschke wrote:
You're using the Python-MySQL module mysqldb, right?
Actually I using MySQL with pyodbc as the mysqldb Windows binaries for
Python 2.5 aren't out yet. :-(
You can select the data from the database and have
MySQL do the conversion with an SQL command.
select to_days(<your date column name here>),
<your close columnfrom <your table>;
That works very nicely indeed.
Thanks,
jab
On Jan 26, 10:46 am, Robert Kern <robert.k...@gmail.comwrote:
Yes, one can make numpy arrays with "object" as its type. One can even extend
the C-level parts as well. For example, we have an experimental package in the
scipy sandbox for uniform time series that uses mx.DateTime.
http://www.scipy.org/TimeSeriesPackage
I saw that, but I am working with irregular time series.
This is frequently what I do. For dates, I like Modified Julian Day Numbers
although I am sure that would horrify some people more knowledgeable than I.
Not horrified, just doing it. ;-)
jab This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: sdhyok |
last post by:
Hi, I am trying to build up a system handling time series data a lot.
Do you know any well-designed python class specially for time series data?
Thanks in advance.
Shin, Daehyok
|
by: falcon |
last post by:
Is there a way I can do time series calculation, such as a moving
average in list comprehension syntax? I'm new to python but it looks
like list comprehension's 'head' can only work at a value at...
|
by: Gaz |
last post by:
Hi guys. I've been lookig for this in the numpy pdf manual, in this
group and on google, but i could not get an answer...
Is there a way to create a custom data type (eg: Name: string(30), Age:...
|
by: Gary Wessle |
last post by:
Hi
I am trying to install NumPy in my debian/testing linux
2.6.15-1-686.
with no numpy for debian/testing, I am left alone, since the
experimental version available by debian will result in a...
|
by: Ray Tomes |
last post by:
Hi Folks
I am an old codger who has much experience with computers
in the distant past before all this object oriented stuff.
Also I have loads of software in such languages as FORTRAN
and...
|
by: Fie Pye |
last post by:
Hallo
I would like to have a high class open source tools for scientific computing and powerful 2D and 3D data visualisation. Therefore I chosepython, numpy and scipy as a base. Now I am in...
|
by: Tyler |
last post by:
Hello All:
After trying to find an open source alternative to Matlab (or IDL), I
am currently getting acquainted with Python and, in particular SciPy,
NumPy, and Matplotlib. While I await the...
|
by: desktop |
last post by:
In the C++ standard sec 23.1.2 table 69 it says that erase(q) where q is
a pointer to an element can be done in amortized constant time.
I guess that is not worst case since std::set is...
|
by: Mars creature |
last post by:
Hi guys,
I am new to Python, and thinking about migrating to it from matlab
as it is a really cool language. Right now, I am trying to figure out
how to control read and write binary data, like...
|
by: DolphinDB |
last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation.
Take...
|
by: DolphinDB |
last post by:
Tired of spending countless mintues downsampling your data? Look no further!
In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
|
by: ryjfgjl |
last post by:
ExcelToDatabase: batch import excel into database automatically...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: jfyes |
last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
|
by: PapaRatzi |
last post by:
Hello,
I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
|
by: Defcon1945 |
last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
|
by: af34tf |
last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
|
by: Faith0G |
last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
| |