473,706 Members | 2,014 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Re: Fastest way to store ints and floats on disk

On 2008-08-07 20:41, Laszlo Nagy wrote:
>
Hi,

I'm working on a pivot table. I would like to write it in Python. I
know, I should be doing that in C, but I would like to create a cross
platform version which can deal with smaller databases (not more than a
million facts).

The data is first imported from a csv file: the user selects which
columns contain dimension and measure data (and which columns to
ignore). In the next step I would like to build up a database that is
efficient enough to be used for making pivot tables. Here is my idea for
the database:

Original CSV file with column header and values:

"Color","Year", "Make","Price", "VMax"
Yellow,2000,Fer rari,100000,254
Blue,2003,Volvo ,50000,210

Using the GUI, it is converted to this:

dimensions = [
{ 'name':'Color', 'colindex:0, 'values':[ 'Red', 'Blue', 'Green',
'Yellow' ], },
{ 'name':'Year', colindex:1, 'values':[
1995,1999,2000, 2001,2002,2003, 2007 ], },
{ 'name':'Make', colindex:2, 'value':[ 'Ferrari', 'Volvo', 'Ford',
'Lamborgini' ], },
]
measures = [
{ 'name', 'Price', 'colindex':3 },
{ 'name', 'Vmax', 'colindex':4 },
]
facts = [
( (3,2,0),(100000 .0,254.0) ), # ( dimension_value _indexes,
measure_values )
( (1,5,1),(50000. 0,210.0) ),
.... # Some million rows or less
]
The core of the idea is that, when using a relatively small number of
possible values for each dimension, the facts table becomes
significantly smaller and easier to process. (Processing the facts would
be: iterate over facts, filter out some of them, create statistical
values of the measures, grouped by dimensions.)

The facts table cannot be kept in memory because it is too big. I need
to store it on disk, be able to read incrementally, and make statistics.
In most cases, the "statistic" will be simple sum of the measures, and
counting the number of facts affected. To be effective, reading the
facts from disk should not involve complex conversions. For this reason,
storing in CSV or XML or any textual format would be bad. I'm thinking
about a binary format, but how can I interface that with Python?

I already looked at:

- xdrlib, which throws me DeprecationWarn ing when I store some integers
- struct which uses format string for each read operation, I'm concerned
about its speed

What else can I use?
>>import marshal
marshal.dump( 1, open('test.db', 'wb'))
marshal.load( open('test.db', 'rb'))
1

It also very fast at dumping/loading lists, tuples, dictionaries,
floats, etc.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Aug 07 2008)
>>Python/Zope Consulting and Support ... http://www.egenix.com/
mxODBC.Zope.D atabase.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
_______________ _______________ _______________ _______________ ____________

:::: Try mxODBC.Zope.DA for Windows,Linux,S olaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
Aug 7 '08 #1
3 2342
On Aug 7, 2:27*pm, "M.-A. Lemburg" <m...@egenix.co mwrote:
On 2008-08-07 20:41, Laszlo Nagy wrote:


*Hi,
I'm working on a pivot table. I would like to write it in Python. I
know, I should be doing that in C, but I would like to create a cross
platform version which can deal with smaller databases (not more than a
million facts).
The data is first imported from a csv file: the user selects which
columns contain dimension and measure data (and which columns to
ignore). In the next step I would like to build up a database that is
efficient enough to be used for making pivot tables. Here is my idea for
the database:
Original CSV file with column header and values:
"Color","Year", "Make","Price", "VMax"
Yellow,2000,Fer rari,100000,254
Blue,2003,Volvo ,50000,210
Using the GUI, it is converted to this:
dimensions = [
* *{ 'name':'Color', 'colindex:0, 'values':[ 'Red', 'Blue', 'Green',
'Yellow' ], },
* *{ 'name':'Year', colindex:1, 'values':[
1995,1999,2000, 2001,2002,2003, 2007 ], },
* *{ 'name':'Make', colindex:2, 'value':[ 'Ferrari', 'Volvo', 'Ford',
'Lamborgini' ], },
]
measures = [
* *{ 'name', 'Price', 'colindex':3 },
* *{ 'name', 'Vmax', 'colindex':4 },
]
facts = [
* *( (3,2,0),(100000 .0,254.0) *), # ( dimension_value _indexes,
measure_values )
* *( (1,5,1),(50000. 0,210.0) ),
* .... # Some million rows or less
]
The core of the idea is that, when using a relatively small number of
possible values for each dimension, the facts table becomes
significantly smaller and easier to process. (Processing the facts would
be: iterate over facts, filter out some of them, create statistical
values of the measures, grouped by dimensions.)
The facts table cannot be kept in memory because it is too big. I need
to store it on disk, be able to read incrementally, and make statistics..
In most cases, the "statistic" will be simple sum of the measures, and
counting the number of facts affected. To be effective, reading the
facts from disk should not involve complex conversions. For this reason,
storing in CSV or XML or any textual format would be bad. I'm thinking
about a binary format, but how can I interface that with Python?
I already looked at:
- xdrlib, which throws me DeprecationWarn ing when I store some integers
- struct which uses format string for each read operation, I'm concerned
about its speed
What else can I use?

*>>import marshal
*>>marshal.dump (1, open('test.db', 'wb'))
*>>marshal.load (open('test.db' , 'rb'))
1

It also very fast at dumping/loading lists, tuples, dictionaries,
floats, etc.
Depending on how hard-core you want to be, store the int, float,
string, and long C structures directly to disk, at a given offset.
Either use fixed-length strings, or implement (or find) a memory
manager. Anyone have a good alloc-realloc-free library, C or Python?
Aug 9 '08 #2
On Aug 10, 4:58 am, castironpi <castiro...@gma il.comwrote:
On Aug 7, 2:27 pm, "M.-A. Lemburg" <m...@egenix.co mwrote:
On 2008-08-07 20:41, Laszlo Nagy wrote:
Hi,
I'm working on a pivot table. I would like to write it in Python. I
know, I should be doing that in C, but I would like to create a cross
platform version which can deal with smaller databases (not more than a
million facts).
The data is first imported from a csv file: the user selects which
columns contain dimension and measure data (and which columns to
ignore). In the next step I would like to build up a database that is
efficient enough to be used for making pivot tables. Here is my idea for
the database:
Original CSV file with column header and values:
"Color","Year", "Make","Price", "VMax"
Yellow,2000,Fer rari,100000,254
Blue,2003,Volvo ,50000,210
Using the GUI, it is converted to this:
dimensions = [
{ 'name':'Color', 'colindex:0, 'values':[ 'Red', 'Blue', 'Green',
'Yellow' ], },
{ 'name':'Year', colindex:1, 'values':[
1995,1999,2000, 2001,2002,2003, 2007 ], },
{ 'name':'Make', colindex:2, 'value':[ 'Ferrari', 'Volvo', 'Ford',
'Lamborgini' ], },
]
measures = [
{ 'name', 'Price', 'colindex':3 },
{ 'name', 'Vmax', 'colindex':4 },
]
facts = [
( (3,2,0),(100000 .0,254.0) ), # ( dimension_value _indexes,
measure_values )
( (1,5,1),(50000. 0,210.0) ),
.... # Some million rows or less
]
The core of the idea is that, when using a relatively small number of
possible values for each dimension, the facts table becomes
significantly smaller and easier to process. (Processing the facts would
be: iterate over facts, filter out some of them, create statistical
values of the measures, grouped by dimensions.)
The facts table cannot be kept in memory because it is too big. I need
to store it on disk, be able to read incrementally, and make statistics.
In most cases, the "statistic" will be simple sum of the measures, and
counting the number of facts affected. To be effective, reading the
facts from disk should not involve complex conversions. For this reason,
storing in CSV or XML or any textual format would be bad. I'm thinking
about a binary format, but how can I interface that with Python?
I already looked at:
- xdrlib, which throws me DeprecationWarn ing when I store some integers
- struct which uses format string for each read operation, I'm concerned
about its speed
What else can I use?
>>import marshal
>>marshal.dump( 1, open('test.db', 'wb'))
>>marshal.load( open('test.db', 'rb'))
1
It also very fast at dumping/loading lists, tuples, dictionaries,
floats, etc.

Depending on how hard-core you want to be, store the int, float,
string, and long C structures directly to disk, at a given offset.
Either use fixed-length strings, or implement (or find) a memory
manager. Anyone have a good alloc-realloc-free library, C or Python?
A long time ago, when I last needed to bother about such things (to
override the memory allocator in the DJGPP RTL), Doug Lea's malloc did
the trick.

A memory allocator written in Python? That's a novel concept.
Aug 9 '08 #3
On Aug 9, 4:43*pm, John Machin <sjmac...@lexic on.netwrote:
On Aug 10, 4:58 am, castironpi <castiro...@gma il.comwrote:
On Aug 7, 2:27 pm, "M.-A. Lemburg" <m...@egenix.co mwrote:
On 2008-08-07 20:41, Laszlo Nagy wrote:
*Hi,
I'm working on a pivot table. I would like to write it in Python. I
know, I should be doing that in C, but I would like to create a cross
platform version which can deal with smaller databases (not more than a
million facts).
The data is first imported from a csv file: the user selects which
columns contain dimension and measure data (and which columns to
ignore). In the next step I would like to build up a database that is
efficient enough to be used for making pivot tables. Here is my idea for
the database:
Original CSV file with column header and values:
"Color","Year", "Make","Price", "VMax"
Yellow,2000,Fer rari,100000,254
Blue,2003,Volvo ,50000,210
Using the GUI, it is converted to this:
dimensions = [
* *{ 'name':'Color', 'colindex:0, 'values':[ 'Red', 'Blue', 'Green',
'Yellow' ], },
* *{ 'name':'Year', colindex:1, 'values':[
1995,1999,2000, 2001,2002,2003, 2007 ], },
* *{ 'name':'Make', colindex:2, 'value':[ 'Ferrari', 'Volvo', 'Ford',
'Lamborgini' ], },
]
measures = [
* *{ 'name', 'Price', 'colindex':3 },
* *{ 'name', 'Vmax', 'colindex':4 },
]
facts = [
* *( (3,2,0),(100000 .0,254.0) *), # ( dimension_value _indexes,
measure_values )
* *( (1,5,1),(50000. 0,210.0) ),
* .... # Some million rows or less
]
The core of the idea is that, when using a relatively small number of
possible values for each dimension, the facts table becomes
significantly smaller and easier to process. (Processing the facts would
be: iterate over facts, filter out some of them, create statistical
values of the measures, grouped by dimensions.)
The facts table cannot be kept in memory because it is too big. I need
to store it on disk, be able to read incrementally, and make statistics.
In most cases, the "statistic" will be simple sum of the measures, and
counting the number of facts affected. To be effective, reading the
facts from disk should not involve complex conversions. For this reason,
storing in CSV or XML or any textual format would be bad. I'm thinking
about a binary format, but how can I interface that with Python?
I already looked at:
- xdrlib, which throws me DeprecationWarn ing when I store some integers
- struct which uses format string for each read operation, I'm concerned
about its speed
What else can I use?
*>>import marshal
*>>marshal.dump (1, open('test.db', 'wb'))
*>>marshal.load (open('test.db' , 'rb'))
1
It also very fast at dumping/loading lists, tuples, dictionaries,
floats, etc.
Depending on how hard-core you want to be, store the int, float,
string, and long C structures directly to disk, at a given offset.
Either use fixed-length strings, or implement (or find) a memory
manager. *Anyone have a good alloc-realloc-free library, C or Python?

A long time ago, when I last needed to bother about such things (to
override the memory allocator in the DJGPP RTL), Doug Lea's malloc did
the trick.

A memory allocator written in Python? That's a novel concept.
For strings and longs, you need variable-length records. Wrap Lea's
malloc in Python calls, and design a Python class where Year, Price,
and VMax are stored as ints at given offsets from the start of the
file, and Color and Make are stored as strings at given offsets.
Don't bother to cache, just seek and read. Part 1 of the file, or
File 1, looks like:

Car 1 color_offset year_offset make_offset price_offset vmax_offset
Car 2 color_offset year_offset make_offset price_offset vmax_offset

Store them directly as bytes, not string reps. of numbers.

1024 1050 1054 1084 1088
1092 1112 1116 1130 1134

Part 2 looks like

1024 Yell
1028 ow
1050 2000
1054 Ferr
1058 ari
1084 100000
1088 254
1092 Blue
1112 2003
1116 Volv

and so on.
Aug 10 '08 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
4091
by: Jenny | last post by:
Hi, I have a class foo which will construct some objects in my code. some of the objects store int values into the data deque, while others store float values to the deque. template <class TYPE> class foo { protected:
4
1448
by: mchoya | last post by:
Hello all, I want to display a list of ints for example 100, 1000, 10000 as prices $1.00, $10.00, $100.00. Can I do this? I guess I have to use the iomanip lib. But the only manipulators I know deal with floats.
2
2091
by: Daniel Mori | last post by:
Hi, Im hoping someone can give me some advice. Im doing some development using VS Whidbey 2005 beta 1. Im about to implement some highly time critical code related to a managed collection of floats. I basically require the fastest way to convert a managed array of floats into an unmanaged array of floats. As far as the managed collection is implemented, ive read that the
5
4498
by: Tales Normando | last post by:
The title says it all. Anyone?
15
12310
by: Buddy Home | last post by:
Hello, I'm trying to speed up a piece of code that is causing performance issues with our product. The problem is we are using serialization to convert the object to a string, this is costing us performance degrade. Does anyone know any better way to archive this which just not degrade performance. The code is written in C#. Thanks
21
1584
by: Pieter | last post by:
Hi, I need some type of array/list/... In which I can store objects together with a unique key. The most important thing is performance: I will need to do the whole time searches in the list of given keys. Which datastructure will be best suited for this? A Hashtable? The list may contain upto 10^12 items, bit more proably most of the time +- 10^9 of even 10^6... Thanks a lot in advance,
2
1143
by: Laszlo Nagy | last post by:
Hi, I'm working on a pivot table. I would like to write it in Python. I know, I should be doing that in C, but I would like to create a cross platform version which can deal with smaller databases (not more than a million facts). The data is first imported from a csv file: the user selects which columns contain dimension and measure data (and which columns to ignore). In the next step I would like to build up a database that is
0
188
by: Emile van Sebille | last post by:
Laszlo Nagy wrote: Hmm... I wrote an browser based analysis tool and used the working name pyvot... I found Numeric to provide the best balance of memory footprint and speed. I also segregated data prep into a separate process to avoid excessive memory use at run time. Turns out python For the site I'm at, I've got 10 years sales history recapped from
5
1528
by: castironpi | last post by:
Hi, I've got an "in-place" memory manager that uses a disk-backed memory- mapped buffer. Among its possibilities are: storing variable-length strings and structures for persistence and interprocess communication with mmap. It allocates segments of a generic buffer by length and returns an offset to the reserved block, which can then be used with struct to pack values to store. The data structure is adapted from the GNU PAVL
0
8783
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
8697
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
1
9056
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8999
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7912
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6615
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
4454
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4710
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
2503
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.