473,396 Members | 2,024 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,396 software developers and data experts.

Max columns limitation

Hi,

I'm using MySQL on Windows and I hit a limit somewhere
between 1300 and 1500 columns in a table. (There is
a good reason why I need that many columns - so please
bear with me.)

Is there any way to allow a table to have perhaps 2,000 to 5,000
columns without going to blob fields where I have to manage
the columns.

Relational joins over multiple tables won't do the trick since
I would need a view this wide anyway.

Is there any open source database that can handle this
monster?

John

Jul 20 '05 #1
8 7717

"John Small" <js****@atlantech.net> wrote in message
news:VS******************@fe39.usenetserver.com...
Hi,

I'm using MySQL on Windows and I hit a limit somewhere
between 1300 and 1500 columns in a table. (There is
a good reason why I need that many columns - so please
bear with me.)

Is there any way to allow a table to have perhaps 2,000 to 5,000
columns without going to blob fields where I have to manage
the columns.

Relational joins over multiple tables won't do the trick since
I would need a view this wide anyway.

Is there any open source database that can handle this
monster?

John


I am bearing with you. I know of no product that allows this many columns.
Please explain why you have so many columns. Are they attributes? If so,
what kind of entity needs 1500 attributes? Or do you have so many relations
that foreign keys are migrated? Are you doing a time series in a table? I'm
just curious. Best of luck....

Rich R.
Jul 20 '05 #2
John Small wrote:
Is there any way to allow a table to have perhaps 2,000 to 5,000
columns without going to blob fields where I have to manage
the columns.


The fact that you can't give a firm value for the number of columns
suggests to me that it's liable to grow, so even if you find an RDBMS
that stores 5,000 (or 10,000) columns, it won't be enough for long.

You need to create a second table to store all those values, referencing
the primary key of the first table. Join these two tables together and
fetch all the matching values, and then if you need to reassemble it
into an array in your application, do it in your application code. Not
every task can be done in a single SQL query.

If you're trying to deliver the dataset as a single row to some report
writer or other middleware product, I would look for another way to
represent the information. I just can't believe that creating a
5,000-column table is the best solution for anything!

Regards,
Bill K.
Jul 20 '05 #3

"Bill Karwin" <bi**@karwin.com> wrote in message
news:cp*********@enews4.newsguy.com...
John Small wrote:
Is there any way to allow a table to have perhaps 2,000 to 5,000
columns without going to blob fields where I have to manage
the columns.


The fact that you can't give a firm value for the number of columns
suggests to me that it's liable to grow, so even if you find an RDBMS
that stores 5,000 (or 10,000) columns, it won't be enough for long.

You need to create a second table to store all those values, referencing
the primary key of the first table. Join these two tables together and
fetch all the matching values, and then if you need to reassemble it
into an array in your application, do it in your application code. Not
every task can be done in a single SQL query.

If you're trying to deliver the dataset as a single row to some report
writer or other middleware product, I would look for another way to
represent the information. I just can't believe that creating a
5,000-column table is the best solution for anything!

Regards,
Bill K.


Yes, Bill, you are correct, If the columns grow, then something is wrong
with the design. And the second table would do it. But I would still love to
know the problem he is trying to solve.

Rich
Jul 20 '05 #4
Basically I'm storing heterogenius, hierarchical data and ultimately
it has to be flattened. I'm flattening it outside the database now.
What I really need is a deductive/associative database or a multidimensional
database but the customer wants it on the cheap. Sorry I can't give
you more details.

Thanks for your help!

"Rich R" <rr***@cshore.com> wrote in message
news:hX*******************@newssvr33.news.prodigy. com...

"Bill Karwin" <bi**@karwin.com> wrote in message
news:cp*********@enews4.newsguy.com...
John Small wrote:
> Is there any way to allow a table to have perhaps 2,000 to 5,000
> columns without going to blob fields where I have to manage
> the columns.


The fact that you can't give a firm value for the number of columns
suggests to me that it's liable to grow, so even if you find an RDBMS
that stores 5,000 (or 10,000) columns, it won't be enough for long.

You need to create a second table to store all those values, referencing
the primary key of the first table. Join these two tables together and
fetch all the matching values, and then if you need to reassemble it
into an array in your application, do it in your application code. Not
every task can be done in a single SQL query.

If you're trying to deliver the dataset as a single row to some report
writer or other middleware product, I would look for another way to
represent the information. I just can't believe that creating a
5,000-column table is the best solution for anything!

Regards,
Bill K.


Yes, Bill, you are correct, If the columns grow, then something is wrong
with the design. And the second table would do it. But I would still love
to
know the problem he is trying to solve.

Rich


Jul 20 '05 #5
John Small wrote:
Basically I'm storing heterogenius, hierarchical data and ultimately
it has to be flattened. I'm flattening it outside the database now.
What I really need is a deductive/associative database or a multidimensional
database but the customer wants it on the cheap. Sorry I can't give
you more details.


Hierarchical data?

So perhaps something like:

id, parent_id, data

Example data:
title1
|- title1A
|- title1B
| |- title1B1
| |- title1B2
|- title1C
title2

Data in database:
-----------------------------
| id | parent_id | data |
|---------------------------|
| 1 | 0 | title1 |
| 2 | 0 | title2 |
| 3 | 1 | title1A |
| 4 | 1 | title1B |
| 5 | 1 | title1C |
| 6 | 4 | title1B1 |
| 7 | 4 | title1B2 |
-----------------------------

The above example is good for some cases and bad for others. Of course
one solution could be also to save the whole path to the column, for
example in text-column, using indexes separated by commas.

-----------------------------
| id | parent_id | data |
|---------------------------|
| 1 | 0 | title1 |
| 2 | 0 | title2 |
| 3 | 1 | title1A |
| 4 | 1 | title1B |
| 5 | 1 | title1C |
| 6 | 1,4 | title1B1 |
| 7 | 1,4 | title1B2 |
-----------------------------

But again, this has it flaws. For example updating the hierarchy needs
much more work than the first example needed. But then again, it is much
easier to find all the parents for certain leaf.

The example given in here is simplified example. This can be used as it
is, but "proper" way would propably be to add another table, where you
save all the parents and their order for certain child.

So something like:
leaf_id | parent_id | order
---------------------------
3 | 1 | 1
4 | 1 | 1
5 | 1 | 1
6 | 1 | 1
6 | 4 | 2
7 | 1 | 1
7 | 4 | 2

The above contains the same information as previous, but this time the
data is separated to rows, as it should be in databases. From this table
you can easily search all parents for certain leaf,

select parent_id from table where leaf_id=6;

but you can also search all children and grant children etc. for certain
parent.
select leaf_id from table where parent_id=1;

I bulieve there are other solutions also, it greatly depends on the
details of the problem which is best to use.
Jul 20 '05 #6

"John Small" <js****@atlantech.net> wrote in message
news:dX***********@fe61.usenetserver.com...
Basically I'm storing heterogenius, hierarchical data and ultimately
it has to be flattened. I'm flattening it outside the database now.
What I really need is a deductive/associative database or a multidimensional database but the customer wants it on the cheap. Sorry I can't give
you more details.

Huh? Makes no sense to me. "heterogenius, hierarchical data and
"deductive/associative database or a multidimensional database". Give me an information artifact->a report with headings. Then I

may understand what you are talking about.

Rich R.

Rich R.
Jul 20 '05 #7
John Small wrote:
Basically I'm storing heterogenius, hierarchical data and ultimately
it has to be flattened. I'm flattening it outside the database now.


I recommend you continue to flatten the data outside the database.

You might waste a lot of time searching for an RDBMS that *might* handle
thousands of columns, but you'll certainly waste a lot of time trying to
get it to work reliably. It won't be "on the cheap" for your client
either way.

Just my opinion,
Bill K.
Jul 20 '05 #8
guzguz
1
wow .... 5,000 - 10,000 columns!!!!
are you from another world?
Apr 22 '06 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Agnes | last post by:
I need to limit the number of characters in the Textbox column in the datagrid. How can I do that ? Thanks
2
by: terpatwork | last post by:
Hi, (1) I have an access form that allows users to enter data, and when they click a button, the OnClick code that I've written uses a SQL INSERT statement to insert the data into the database. I...
3
by: Seong-Kook Shin | last post by:
C FAQ Q 13.2 says that sprintf() is guaranteed to work only for n <= 509: sprintf(dest, "%.*s", n, source); Does 509 appear in any C standard? If not, where it came from? I looked over C99...
2
by: Joseph Geretz | last post by:
I'm developing a Web Service using DIME to download and upload files from and to an IIS server. In order to increase the download filesize to unlimited, I have the following block in my App.config...
19
by: matt | last post by:
I've seen several posts that begin to address this problem, but have not found a simple, elegant solution that will accomplish this goal. The important part of this solution is that it must be...
3
by: SDRoy | last post by:
Hi Is there a limitation on how many rows can be fetched using a SqlDataReader in ASP.Net 2.0 ? -- Thanks, SDRoy
4
by: Ole Nielsby | last post by:
Here comes a generic class with a static property. Let's say they are exotic singleton pets. abstract class Pet {...} abstract class SharedPet<T>: Pet where T: SharedPet<T>, new() {...
13
by: Jo | last post by:
Hi. I'm getting the following error when creating a table with 250 columns .. I have tried creating it in a 32K tablespace , still the same issue. Is this a limitation in DB2? I am using DB2...
3
by: NoDBExperience | last post by:
I have a situation with my DB. I added a couple of columns to a certain table, which apparently affected many forms, would like to know if there is any code out there that would help me apply changes...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.