473,326 Members | 2,173 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

How to "append" a datatable?

Hello!

I have huge unperformant "where like" querys to a mysql DB. So there
are some performance issues. To resolve my performance problem I thougt
to split the query into threads. I hold 10 connections to my database
and each connection queries in its own thread as the following:

conn1: select fname from root where p_key between 1 and 20000 fname
like '%somestring%';
conn2: select fname from root where p_key between 20001 and 40000 fname
like '%somestring%';
and so on ...

the root table contains today about 200000 rows growing

Now the problem is: How can I put the 10 datatable objects to one huge
object to bind to my grid object?

THX!
Christian Maier

PS Any other solution for my performance issue is welcome!

Apr 5 '06 #1
6 1542
do you present the hole data (the 200000 records) to the user once?
don't you page it?

if you page it... why not just get the data (100 at a time) to populate the
current datagrid... there's some tutorials on it.
--
Thank you in Advance.

Bruno Alexandre
(a Portuguese in Denmark)

"Christian Maier" <to*******@freesurf.fr> escreveu na mensagem
news:11*********************@g10g2000cwb.googlegro ups.com...
Hello!

I have huge unperformant "where like" querys to a mysql DB. So there
are some performance issues. To resolve my performance problem I thougt
to split the query into threads. I hold 10 connections to my database
and each connection queries in its own thread as the following:

conn1: select fname from root where p_key between 1 and 20000 fname
like '%somestring%';
conn2: select fname from root where p_key between 20001 and 40000 fname
like '%somestring%';
and so on ...

the root table contains today about 200000 rows growing

Now the problem is: How can I put the 10 datatable objects to one huge
object to bind to my grid object?

THX!
Christian Maier

PS Any other solution for my performance issue is welcome!

Apr 5 '06 #2
> do you present the hole data (the 200000 records) to the user once?
don't you page it?

the query is over 200000 records, the result should be <= 500 selects.
so the problem is not getting huge data into my datagrid, the problem
is the query it self. maybe there are only 10 selects from each thread.
but i do not know how to "union" all retured datatables out of my
threads.

christian

Apr 5 '06 #3
Hi Christian,

"Christian Maier" <to*******@freesurf.fr> schrieb im Newsbeitrag
news:11*********************@g10g2000cwb.googlegro ups.com...
I have huge unperformant "where like" querys to a mysql DB. So there
are some performance issues. To resolve my performance problem I thougt
to split the query into threads. I hold 10 connections to my database
and each connection queries in its own thread as the following:


in my opinion you have to find out what causes this performance leak!

Is it:
- your app (the way it handles the resulting datatables, etc.)
- the query
- the database connection (drivers, network, etc.)
- the database itself (old or buggy, heavy load, etc.)

If you know what causes the leak you'll have a greater chance solving it.

If something's wrong with the query, the database, the database connection
then I don't think it makes anything faster querying it with multiple
threads.

Just ideas....

Greets,
Kurt
Apr 5 '06 #4
I have found that if you don't invoke Table.Clear then the DataAdapter will
append data to it. Interestingly, however, if it were the same records,
then the records get replaced. If this doesn't work for you, try creating
the dataset as a file in the project and load data to it, that's how I do
it.

"Christian Maier" <to*******@freesurf.fr> wrote in message
news:11*********************@g10g2000cwb.googlegro ups.com...
Hello!

I have huge unperformant "where like" querys to a mysql DB. So there
are some performance issues. To resolve my performance problem I thougt
to split the query into threads. I hold 10 connections to my database
and each connection queries in its own thread as the following:

conn1: select fname from root where p_key between 1 and 20000 fname
like '%somestring%';
conn2: select fname from root where p_key between 20001 and 40000 fname
like '%somestring%';
and so on ...

the root table contains today about 200000 rows growing

Now the problem is: How can I put the 10 datatable objects to one huge
object to bind to my grid object?

THX!
Christian Maier

PS Any other solution for my performance issue is welcome!

Apr 6 '06 #5

+Vice schrieb:
I have found that if you don't invoke Table.Clear then the DataAdapter will
append data to it. Interestingly, however, if it were the same records,
then the records get replaced. If this doesn't work for you, try creating
the dataset as a file in the project and load data to it, that's how I do
it.


cool thank you for the hint!

Christian

Apr 6 '06 #6
> in my opinion you have to find out what causes this performance leak!
yea I did!
- your app (the way it handles the resulting datatables, etc.)
- the query
- the database connection (drivers, network, etc.)
- the database itself (old or buggy, heavy load, etc.) it is the query because if you select * where foo like '%something'
every index on foo is useless!
just if you query where foo like 'something%' you can tune the query
with an index.

when I select * where pkey between 1 and 10000 and foo like
'%something'
then the optimizer goes trough the primary key. but you are right, the
boost is in a milliseconds range.

I tried to make a hash table but you can imagine that there is not
enough memory allowed to allocate by the database.

I am thinking about select the whole data into a textfile and do in
instr() to find the lines I need but the file will be 50 MB at least I
do not know if this meight be faster.

thx!
Christian

If you know what causes the leak you'll have a greater chance solving it.

If something's wrong with the query, the database, the database connection
then I don't think it makes anything faster querying it with multiple
threads.

Just ideas....

Greets,
Kurt


Apr 6 '06 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: Grayson | last post by:
I have a need for an "Eval" function and found the perfect sample The problem is it doesn't like "IsDate" being fed into it. Any Ideas? seems like a missing reference, but I can't figure it out......
0
by: phmyhn | last post by:
I have two web pages, one is viewlarger.aspx, another one is shoppingcart.aspx. On the viewlarger.aspx, when clicking "add to cart" image button, the sub appends the id (passed from another page...
25
by: tsaar2003 | last post by:
Hi Pythonians, To begin with I'd like to apologize that I am not very experienced Python programmer so please forgive me if the following text does not make any sense. I have been missing...
1
by: pbd22 | last post by:
hi. for some reason i am not able to navigate the dom. i keep getting the 'no properties' error but i think i am doing everything right. this is an XML response from an AJAX call. The ajax...
2
by: mauricesmith42 | last post by:
Sorry i know this is rather large to be posting, but in order to understand the question you have to see all the code //#include <windows.h> //needed for opening folders #include...
1
by: =?Utf-8?B?TWlrZQ==?= | last post by:
This question isn't a true ASP.NET-related question, but I don't know where else to post it. Why do web browsers automatically append URL's with a forward-slash (/)? For example, if you type...
13
by: xzzy | last post by:
None of the following properly do the VB.net double quote conversion because all of the following in csharp convert to \" instead of just a double quote: " I have tried: char...
1
by: HarishAdea | last post by:
Hi, I am trying to run the JAVA pgm, but it is giving error as "selection does not contain a main type". The filename is "ScoreLeadSummary.java" when i try to run it or debug,it gives the pop...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.