473,325 Members | 2,712 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,325 software developers and data experts.

Replacement for DataTable

Can anyone suggest a replacement, 3rd party or otherwise, object for
DataTable. We're running into performance issues with large dataset (>30k)
rows. Looking for something that can do binary serialization, typed column
row data, primary keys, filtering etc. SQL desktop engine or some other
database is not an option at this point.

Thanks

Briann
Jul 22 '07 #1
7 5149
Brian Richards wrote:
Can anyone suggest a replacement, 3rd party or otherwise, object for
DataTable. We're running into performance issues with large dataset (>30k)
rows. Looking for something that can do binary serialization, typed column
row data, primary keys, filtering etc. SQL desktop engine or some other
database is not an option at this point.
Why isn't a database an option? My very first suggestion would be just that.

--
Göran Andersson
_____
http://www.guffa.com
Jul 22 '07 #2
moo
On Jul 21, 6:27 pm, Göran Andersson <gu...@guffa.comwrote:
Brian Richards wrote:
Can anyone suggest a replacement, 3rd party or otherwise, object for
DataTable. We're running into performance issues with large dataset (>30k)
rows. Looking for something that can do binary serialization, typed column
row data, primary keys, filtering etc. SQL desktop engine or some other
database is not an option at this point.

Why isn't a database an option? My very first suggestion would be just that.

--
Göran Andersson
_____http://www.guffa.com

My database teacher would laugh at me for saying this, but have you
considered XML and XPath? I'm not sure of the performance
implications, but it's the alternative that springs most readily to
mind...

Jul 22 '07 #3
Datasets are keeping complete result sets and related data in memory, so you
have to deal with unavoidable performance issues.

If you create data sets and tables programmatically, you must implement some
kind of db engine to persist data and page through big tables / result sets.
Which doesn't make much sense considering availability of various db
engines, both commercial and free ones. All replacements will implement some
kind of db engine. XML won;t solve your performance issues and might make
situation worse, as parsing complete files will clog your memory with lots
of residuals. Also, it'll double memory consumption.

Now you have troubles with 30K rows. Consider what kind of issues you'll
have with 300K rows. Don't bump into brick wall.

Real question is: what kind of issues you have with 30K rows. This number is
not that big. Any specific details?
"Brian Richards" <br*******@pharsight.comwrote in message
news:uT**************@TK2MSFTNGP02.phx.gbl...
Can anyone suggest a replacement, 3rd party or otherwise, object for
DataTable. We're running into performance issues with large dataset (>30k)
rows. Looking for something that can do binary serialization, typed column
row data, primary keys, filtering etc. SQL desktop engine or some other
database is not an option at this point.

Thanks

Briann

Jul 22 '07 #4
My database teacher would laugh at me for saying this, but have you
considered XML and XPath? I'm not sure of the performance
implications, but it's the alternative that springs most readily to
mind...
I'm pretty sure XML is the exact reason that the DataTable performance is so
awful. Because it's XML internally.


Jul 22 '07 #5
* Brian Richards wrote, On 22-7-2007 2:38:
Can anyone suggest a replacement, 3rd party or otherwise, object for
DataTable. We're running into performance issues with large dataset (>30k)
rows. Looking for something that can do binary serialization, typed column
row data, primary keys, filtering etc. SQL desktop engine or some other
database is not an option at this point.

Thanks

Briann

Have you tried SQL Server CE (previously only available on Windows
Mobile, but work sin any .NET app now).

Jesse
Jul 22 '07 #6
Brian Richards wrote:
>My database teacher would laugh at me for saying this, but have you
considered XML and XPath? I'm not sure of the performance
implications, but it's the alternative that springs most readily to
mind...

I'm pretty sure XML is the exact reason that the DataTable performance is so
awful. Because it's XML internally.
I don't think that's the case, as DataTables work with data in native
format, and XML is only text.

--
Göran Andersson
_____
http://www.guffa.com
Jul 23 '07 #7
Brian Richards <br*******@pharsight.comwrote:
My database teacher would laugh at me for saying this, but have you
considered XML and XPath? I'm not sure of the performance
implications, but it's the alternative that springs most readily to
mind...

I'm pretty sure XML is the exact reason that the DataTable performance is so
awful. Because it's XML internally.
No it's not. Just because it can load from XML and save to XML doesn't
mean its internal storage is XML.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Jul 23 '07 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

53
by: Kerberos | last post by:
I followed Dan Cederholm's image replacement tutorial, to replace a header tag by a logo. The h1 is clickable if no CSS is applied but it I replace it by the logo, the area isn't clickable anymore...
3
by: Vibha Tripathi | last post by:
Hi Folks, I put a Regular Expression question on this list a couple days ago. I would like to rephrase my question as below: In the Python re.sub(regex, replacement, subject)...
20
by: Paul D. Boyle | last post by:
Hi all, There was a recent thread in this group which talked about the shortcomings of fgets(). I decided to try my hand at writing a replacement for fgets() using fgetc() and realloc() to read...
1
by: Julian Maisano | last post by:
Is there any C# class 'equivalent/replacement' for Delphi's TDataSet? I'm just thinking to translate an application mine, but it's based in a framework (also mine) that often uses TDataSets so I...
5
by: Stefan Turalski \(stic\) | last post by:
Hi, I'm wondering if there is a way to send a method parametrs by ref when DataTabel is a type of this value ? I done some sort of select over DataTable columns, just by removing them froma...
34
by: Craig Buchanan | last post by:
Which vb.net object is the best match for the vb6 collection class? Specifically, I would like to be able to access the Item property with an index or a key string. I wrote my own class that...
1
by: lawrence k | last post by:
Want to replace the limit clause in a query, but can't get it right. What's wrong with this: $pattern = "(.*)limit (.*)"; $replacement = '$1'; $replacement .= "LIMIT $limit"; $replacement .=...
3
by: =?Utf-8?B?RHVrZSAoQU4yNDcp?= | last post by:
I've added a web deployment project and want to use the config section replacement but I'm obviously not understanding something. I have set up an alternate appSettings file,...
11
by: endy_tj | last post by:
In vb6 I can use ADO's recordset to retrieve data from database, work on the data, then send the modifications back to the database. I use disconnected recordset, and call updatebatch to send...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.