473,769 Members | 7,388 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Should I split a large table into several smaller ones for query efficency?

bugboy
160 New Member
Does the total number of rows in a table determine the amount of resources required for a query?.. or is it primarily determined by the number of rows used by the query? ..Does an INDEX mean it doesn't have to read every row to find data?

Context:
I have a large dataset and am wondering if i should split it up into several smaller tables to reduce the resources required to search it... i would then have to predetermine which table to do the query on so i could avoid joins. The large table is normalized.. it's just that it's going to be really big.. over 2 billion rows...

Thanks!
Oct 5 '07 #1
2 2852
pbmods
5,821 Recognized Expert Expert
Heya, bugboy.

That could work. It'll make things a little tricky on the PHP side when you need to be able to calculate where your data is located, but if done properly, it will make things run a lot faster.

You may also want to see if you can create 'index' tables. That is, you would create a table that represents the result of a complicated -- but frequently-accessed -- query, then run a query to update it periodically.

An example of this would be the 'top 10 posters' box on the right side of each page on this site. The numbers in that box are only updated every few minutes to reduce the load on the database server.
Oct 6 '07 #2
bugboy
160 New Member
I'm going to forward this discussion to another as i feel the topics have merged:

http://www.thescripts. com/forum/thread719146.ht ml

Thanks!
Oct 8 '07 #3

Sign in to post your reply or Sign up for a free account.

Similar topics

4
2515
by: Dag Sunde | last post by:
Just wondering if anyone have looked into this? How to split up ones JavaScript library? A lot of very specific (and small) .js files, or a few larger files. I'm thinking about load-time here... I have a gut-feeling that it will be better to use (a lot of ) smaller (very specific) files, giving you
2
1353
by: Marc Thompson | last post by:
Hi, I have several large XML files (500-700 MB) that get updated once a day. I have an app that will need to Query (read only operations is all I'll ever need) these files. Obviously, querying a file this large will be somewhat of a challenge because it can't be loaded into memory all at once. Also, I am skeptical of the speed of Xpath/Xquery. I am wondering if I shouldn't just create a table in SQL server, then create a routine...
7
2177
by: mef526 | last post by:
I have had this problem for months now and it has been nagging me. I have a large project that has several C++ DLL's, one of them uses malloc / calloc / free for the other DLL's. I process very large datafiles (100MB to 300 MB) that have more than 524,000 lines with 5 columns of numbers in text. I allocate 8 arrays with 524,000 or more doubles , and 10 arrays of 32,768 doubles. Is there a min size for malloc / calloc required to...
3
1534
by: Scythen | last post by:
I thought that including template headers such as <vector> in the pch is not useful. Templates can’t be compiled without the template arguments so there is no way to pre compile them. Is this true? Also, it seems that this could actually be bad since it will significantly increase the size of the pch file which I would think could slow the compilation processes due to the additional memory overhead.
6
3292
by: Vlado Jasovic | last post by:
Hello, We're developing application in VS2005 using vb.net. Our application exe file is ~20mb when compiled. Recently, we have developed auto-update feature that goes on our web-site, authenticates through WebServices and if neccessary downloads updated app.exe file. We had to use custom solution vs. ClickOnce for several different reasons.
1
1976
by: datapro01 | last post by:
X-No-Archive: Yes I have a questionabout reorging very large tables. Running DB2 8.1 Fixpack 6 on AIX 5.2 supporting Siebel. I've read through the docs I could find and the postings in this newsgroup on the challenges in reorging very large tables, using partitioning, MDC, union all views of smaller tables etc. However in supporting Siebel we can't partition. The online reorg runs for a week,
10
3059
by: ARC | last post by:
This is mainly a speed question. In this example: I have a QuotesHdr table that has a few memo fields. If these memo fields are used extensively by some users, and if their are a large number of records in QuotesHdr, should I break out the memo fields into a separate table? The thinking here is that, for Quotes selection dropdowns that display all entries in QuotesHdr for selection, I would think that the entire record comes down over...
1
3796
by: PMADDIREDDY | last post by:
Is there any tool to break up large XML files into smaller ones.
13
29745
by: sevenz24 | last post by:
So i have my images set up like this : http://cgi.ebay.com/Tippmann-98-paintBall-Marker-Gun-Paint-Ball_W0QQitemZ250274334261QQihZ015QQcategoryZ47248QQssPageNameZWDVWQQrdZ1QQcmdZViewItem Scroll down to the bottom and you see several images, One large one and then several smaller ones, when you click on the smaller ones it places a larger version of that picture in place of where the other larger picture was, well i use this method on my...
0
9589
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9423
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10216
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10049
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
8873
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7413
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6675
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5310
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5448
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.