Dataset with large data...PLEASE HELP

palgre

Hi ALL,

I am working in an windows based application using SQL Server 2000 as
database. There are few tables (refer parent tables) in the
application which are uploaded by a seprate application.

My application fetches data from parent tables and put data in
separate tables (chils tables) of application.
I am using dataset to fetch data from parent tables and insert/update
data in child table.
the problem is the records are so high (3 4 millions) then data is too
large and it takes hell lot of time to complete the process. Also
application server CPU utilization shoots out to max.

What will be the best way to achive this.
1. Should i use DataRepeater instead of dataset. or
2. Should i do processing in chunks. how can I do processing in
chunks???

or is there any other way i can process data.

Thanks
PAL

Jul 11 '07 #1

Subscribe Reply

2789

Mr. Arnold

<pa****@gmail.c omwrote in message
news:11******** **************@ g4g2000hsf.goog legroups.com...

Hi ALL,

I am working in an windows based application using SQL Server 2000 as
database. There are few tables (refer parent tables) in the
application which are uploaded by a seprate application.

My application fetches data from parent tables and put data in
separate tables (chils tables) of application.
I am using dataset to fetch data from parent tables and insert/update
data in child table.
the problem is the records are so high (3 4 millions) then data is too
large and it takes hell lot of time to complete the process. Also
application server CPU utilization shoots out to max.

What will be the best way to achive this.
1. Should i use DataRepeater instead of dataset. or
2. Should i do processing in chunks. how can I do processing in
chunks???

or is there any other way i can process data.

Use the SQL Command Object, data reader, dynamic SQL statements or calling
Stored Procedures.

Jul 12 '07 #2

AlexS

I am not sure I understand what you are doing with data.

In any case, dataset loads everything into memory, that's why you get into
troubles.

If you just pump data from one table into another with some processing in
between, use Bulk Copy (SqlBulkCopy or bcp) for inserts. Then you can just
read from some data reader, which should solve your issues.

For updates you have to issue them individually, but even in this case you
can batch them up to command string limit and issue in batches. Say, save
all updates in some file. When reading is complete, read back updates and
run them.

<pa****@gmail.c omwrote in message
news:11******** **************@ g4g2000hsf.goog legroups.com...

Hi ALL,

I am working in an windows based application using SQL Server 2000 as
database. There are few tables (refer parent tables) in the
application which are uploaded by a seprate application.

My application fetches data from parent tables and put data in
separate tables (chils tables) of application.
I am using dataset to fetch data from parent tables and insert/update
data in child table.
the problem is the records are so high (3 4 millions) then data is too
large and it takes hell lot of time to complete the process. Also
application server CPU utilization shoots out to max.

What will be the best way to achive this.
1. Should i use DataRepeater instead of dataset. or
2. Should i do processing in chunks. how can I do processing in
chunks???

or is there any other way i can process data.

Thanks
PAL

Jul 12 '07 #3

Similar topics

13077

Global Dataset in windows application

by: James Morton | last post by:

I am writing a c# windows forms application. I read in an XML file to a dataset for read-only access (by the user and the application logic). This is not an MDI app but has LOTS of seperate forms. They all need to get to tables within this DataSet. How can I make the DataSet "Global" for all forms within the application. I could just read in the file in every form to a newly instanced dataset (the XML is not that large) but this has...

C# / C Sharp

11800

Showing Progress while filling a dataset

by: Scott Lundstrom | last post by:

I have a large amount of data I'm dropping into a dataset and would like to find a way to at the very least show the user that the program is working at retrieving the data and not locked up. I have been unable to find a way to do this with a progress bar so I thought that I would try to disply a message with an animated gif, but I cannot get the gif to animate while the dataset is being filled which is not doing me any good. Any...

C# / C Sharp

2244

datareader vs. dataset

by: Bihn | last post by:

I was reading about datareader which is said to be slimmer & faster then dataset. Since the datareader have to go fetching the dat from the database every time it need it, the data it gets then should be up to date. However, both the IbuySpy and Duwamish samples and most, if not all, the shopping cart sample codes I've seen use dataset to implement the opration for ecommerce sites. So is the trip that the datareader need to go fetch the...

ASP.NET

25606

Passing a Dataset to a webservice

by: Arne | last post by:

How do I pass a dataset to a webservices? I need to submit a shoppingcart from a pocket PC to a webservice. What is the right datatype? II have tried dataset as a datatype, but I can't get it to compile. <WebMethod()> _ Public Function VerifySku(ByVal skus As XmlDataDocument) As DataSet Test program : Dim cartSet As DataSet cartSet = ws.VerifySku(cartSet)

.NET Framework

7880

Filling DataSet causes System OutOfMemory exception

by: Peter S. | last post by:

I am pulling some data from a source via ODBC and placing the information in a DataSet. The first pull is very large but once that is complete I plan to do nightly pulls to get any new data that gets put in the (remote) table. I can't seem to get past that initial (big) pull of data, as I get OutOfMemory exceptions. I took a look back at when this occurs and it seems to happen upon stuffing the DataSet with either the 2097153 record or...

C# / C Sharp

4161

Urgent Memory Leak Problem Using Type Dataset. Please help! :-(

by: Simon | last post by:

Hi all, I'm having a baffling problem with a windows service that I'm working on. Basically, I am using a typed dataset to insert a large number of rows into an SQL Server 2005 database. But there's a memory leak that seems to be to do with calling the data adapters update method. It's making the memory usage go through the roof and ultimately the service crashes after running out of memory.

C# / C Sharp

1152

Large DataSet problem : Please help

by: palgre | last post by:

Visual Basic .NET

1720

Working with Complex DataSet - Creating and Pass thru Transformati

by: =?Utf-8?B?c2lwcHl1Y29ubg==?= | last post by:

Have a complex process where I need to Import a large amount of data then run some transformations on this data then import into DataBase. The transformation involves multiple fields and multiple process - so the data needs to be read in 1 record at a time then run thru the transformation that may create new data value then everything is imported into a db to store. I have multiple questions 1)we used to have an internal data structure...

C# / C Sharp

2844

How can I make this more efficient? (combining DataSet results with the results of a DB lookup.)

by: Ken Fine | last post by:

This is a question that someone familiar with ASP.NET and ADO.NET DataSets and DataTables should be able to answer fairly easily. The basic question is how I can efficiently match data from one dataset to data in a second dataset, using a common key. I will first describe the problem in words and then I will show my code, which has most of the solution done already. I have built an ASP.NET that queries an Index Server and returns a...

ASP.NET

9643

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...

General

9480

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...

Windows Server

10319

Problem With Comparison Operator <=> in G++

by: Oralloy | last post by:

Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...

C / C++

9947

Discussion: How does Zigbee compare with other wireless protocols in smart home applications?

by: tracyyun | last post by:

Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...

General

8971

AI Job Threat for Devs

by: agi2029 | last post by:

Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...

Career Advice

7496

Access Europe - Using VBA to create a class based on a table - Wed 1 May

by: isladogs | last post by:

The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...

Microsoft Access / VBA

6737

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...

C# / C Sharp

5511

Windows Forms - .Net 8.0

by: adsilva | last post by:

A Windows Forms form does not have the event Unload, like VB6. What one acts like?

Visual Basic .NET

2877

Comprehensive Guide to Website Development in Toronto: Expert Insights from BSMN Consultancy

by: bsmnconsultancy | last post by:

In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

General