473,324 Members | 2,370 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

Dataset with large data...PLEASE HELP

Hi ALL,

I am working in an windows based application using SQL Server 2000 as
database. There are few tables (refer parent tables) in the
application which are uploaded by a seprate application.

My application fetches data from parent tables and put data in
separate tables (chils tables) of application.
I am using dataset to fetch data from parent tables and insert/update
data in child table.
the problem is the records are so high (3 4 millions) then data is too
large and it takes hell lot of time to complete the process. Also
application server CPU utilization shoots out to max.

What will be the best way to achive this.
1. Should i use DataRepeater instead of dataset. or
2. Should i do processing in chunks. how can I do processing in
chunks???

or is there any other way i can process data.

Thanks
PAL

Jul 11 '07 #1
2 2765

<pa****@gmail.comwrote in message
news:11**********************@g4g2000hsf.googlegro ups.com...
Hi ALL,

I am working in an windows based application using SQL Server 2000 as
database. There are few tables (refer parent tables) in the
application which are uploaded by a seprate application.

My application fetches data from parent tables and put data in
separate tables (chils tables) of application.
I am using dataset to fetch data from parent tables and insert/update
data in child table.
the problem is the records are so high (3 4 millions) then data is too
large and it takes hell lot of time to complete the process. Also
application server CPU utilization shoots out to max.

What will be the best way to achive this.
1. Should i use DataRepeater instead of dataset. or
2. Should i do processing in chunks. how can I do processing in
chunks???

or is there any other way i can process data.
Use the SQL Command Object, data reader, dynamic SQL statements or calling
Stored Procedures.

Jul 12 '07 #2
I am not sure I understand what you are doing with data.

In any case, dataset loads everything into memory, that's why you get into
troubles.

If you just pump data from one table into another with some processing in
between, use Bulk Copy (SqlBulkCopy or bcp) for inserts. Then you can just
read from some data reader, which should solve your issues.

For updates you have to issue them individually, but even in this case you
can batch them up to command string limit and issue in batches. Say, save
all updates in some file. When reading is complete, read back updates and
run them.

<pa****@gmail.comwrote in message
news:11**********************@g4g2000hsf.googlegro ups.com...
Hi ALL,

I am working in an windows based application using SQL Server 2000 as
database. There are few tables (refer parent tables) in the
application which are uploaded by a seprate application.

My application fetches data from parent tables and put data in
separate tables (chils tables) of application.
I am using dataset to fetch data from parent tables and insert/update
data in child table.
the problem is the records are so high (3 4 millions) then data is too
large and it takes hell lot of time to complete the process. Also
application server CPU utilization shoots out to max.

What will be the best way to achive this.
1. Should i use DataRepeater instead of dataset. or
2. Should i do processing in chunks. how can I do processing in
chunks???

or is there any other way i can process data.

Thanks
PAL

Jul 12 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
by: James Morton | last post by:
I am writing a c# windows forms application. I read in an XML file to a dataset for read-only access (by the user and the application logic). This is not an MDI app but has LOTS of seperate...
4
by: Scott Lundstrom | last post by:
I have a large amount of data I'm dropping into a dataset and would like to find a way to at the very least show the user that the program is working at retrieving the data and not locked up. I...
14
by: Bihn | last post by:
I was reading about datareader which is said to be slimmer & faster then dataset. Since the datareader have to go fetching the dat from the database every time it need it, the data it gets then...
22
by: Arne | last post by:
How do I pass a dataset to a webservices? I need to submit a shoppingcart from a pocket PC to a webservice. What is the right datatype? II have tried dataset as a datatype, but I can't get it to...
2
by: Peter S. | last post by:
I am pulling some data from a source via ODBC and placing the information in a DataSet. The first pull is very large but once that is complete I plan to do nightly pulls to get any new data that...
12
by: Simon | last post by:
Hi all, I'm having a baffling problem with a windows service that I'm working on. Basically, I am using a typed dataset to insert a large number of rows into an SQL Server 2005 database. But...
1
by: palgre | last post by:
Hi ALL, I am working in an windows based application using SQL Server 2000 as database. There are few tables (refer parent tables) in the application which are uploaded by a seprate application....
2
by: =?Utf-8?B?c2lwcHl1Y29ubg==?= | last post by:
Have a complex process where I need to Import a large amount of data then run some transformations on this data then import into DataBase. The transformation involves multiple fields and multiple...
3
by: Ken Fine | last post by:
This is a question that someone familiar with ASP.NET and ADO.NET DataSets and DataTables should be able to answer fairly easily. The basic question is how I can efficiently match data from one...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.