I'm not that experienced with database design and I cannot figure out the best way to design this. I have annual hourly data (8760 values) for a million sites. The total size of the data is not a problem, but how would one store the data such that I can manipulate annual sets of data?
I often read that designing tables with 100s of columns is a mistake. Clearly, I cannot store the data as I would in a program, which is in an array with 8760 columns. So, would monthly tables work (744 columns) or tables with one week of data (168 columns)? Or perhaps 365 tables with a single day of data per record?
Any advice is appreciated.
6  4562 
did you mean columns or rows? I see no reason not to store 1 table with millions of rows.
To clarify, I have a million sets of data, and each set contains 8760 values. This is hourly data for an entire year for 1 million sites.
The question is, how to efficiently store the data in a way that allows me to quickly "view" the data for particular set of sites? My application needs to load all the hourly data for a subset of rows. The simplest thing to do would be to store the data in a table with 8760 columns, one for every hour of the year. Apparently, this is not only bad practice, it's not even possible in most database architectures.
So, I could store the data in 365 tables, one for each day of the year and each with 24 columns (plus a column for the ID of course). There would still be 1 million rows in each table. To load the data for one site, however, I would have to query 365 tables. That seems painful. Is there a standard way of dealing with data of these dimensions?
I'm posting from my phone so ill help more later but ill just say there is no reason to have separate tables for each day just have a field in the row for the day and sort them out after.
Rabbit 12,516
Recognized Expert Moderator MVP
To elaborate on what NetDynamic said, you only need one table with 3 fields. Site, DateTimeField, and ValueField.
Ahhhh ... I hadn't thought of that solution. So, instead of a wide table with a million rows, I'd have a narrow table with 8.7 billion rows. I have to admit, the idea of "billions and billions" of rows (with a nod to Carl Sagan) sort of scares me, but that's probably my problem, not postgre's problem.
Since the current Site ID is a long character string, I imagine a Site ID table should be created with an auto-increment index. The index would be used in the large data table.
Hmmm, makes sense. Thanks for the clue.
If you have a ton of tiny data sets just use a delimiter and use only 1 column for each set, then in your script just explode it into an array or something and use as you wish.
Sign in to post your reply or Sign up for a free account.
Similar topics | |
by: Steve |
last post by:
How do I store files in a database? I have done something similar before in
a none microsoft language in a previous lifetime, byt breaking it down into
chunks and storing it in a BLOB field. Is this the same in SQL?
I have an app written for windows in C# with SQL 2000 on the back. I have a
print engine which users XML documents as the print document template, and
was hoping I could store these in the database, rather than shipping...
|
by: Jake_adl |
last post by:
Is there any way to create a
Microsoft.Practices.EnterpriseLibrary.Data.Database object without reading
from a configuration file?
I am writing a utility that manages databases in SQL Server. The utility
queries SQL Server for the database names. So I don't know the names of the
databases beforehand, but I will know the names and connection strings once
the server has been queried.
However, I have a class that I'd like to work with...
|
by: mukeshpatil |
last post by:
How to get data from .CSV file using php & store it into Mysql database
I want to read data from two different .CSV files using php & How to store it into Mysql Database tables
Please help me to do this.
|
by: dianaj86 |
last post by:
Hello All,
I have a SQL Server, My question is how can I store images in a database table??
Thanks,
|
by: hemashiki |
last post by:
hi
i need help..how can i store image in mysql database
and i want to retrive that image
can anyone suggest plz
| | |
by: shriku |
last post by:
i have problem that i can't store audio file in database. i m using .net & sqlserver .i need detail information. please help me about it . it's very urgent. My email id is shri_chiku24@yahoo.co.in u can send ur suggetions on it. please do urgent reply.
|
by: Abandoned |
last post by:
Hi..
I have some dictionary as {2:3, 4:6, 5:7.........}
I want to pickle to these dictionary and save to database but i
couldn't..
My code for adding..
a=eval(dict)
a=pickle.dumps(a, -1)
cursor2.execute("INSERT INTO cache VALUES ('%s', %s)", (x,a))
conn2.commit()
|
by: sganeshsvk |
last post by:
sir,
i want to store the same data values in two different databases at that same time in mysql using php programming.
suppose any one databases data will lose then we use the other databases.
suppose there is any query for store the same data values in two different databases at the same time.
for eg:
databases sample1
|
by: rahulephp |
last post by:
Hi there,
can you please help me?
i have data in array and i want to insert it in a single table.
structure of table and array to be store is shown below:
Please let me know how to store this array in database?
Database Structure:
|
by: Jyotiranjan Sahoo |
last post by:
How do to read a xml file and store to in a database using Hibernate?
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
| | |
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
| | |
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
| |
