473,795 Members | 2,410 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Large DataSet problem : Please help

Hi ALL,

I am working in an windows based application using SQL Server 2000 as
database. There are few tables (refer parent tables) in the
application which are uploaded by a seprate application.

My application fetches data from parent tables and put data in
separate tables (chils tables) of application.
I am using dataset to fetch data from parent tables and insert/update
data in child table.
the problem is the records are so high (3 4 millions) then data is too
large and it takes hell lot of time to complete the process. Also
application server CPU utilization shoots out to max.

What will be the best way to achive this.
1. Should i use DataRepeater instead of dataset. or
2. Should i do processing in chunks. how can I do processing in
chunks???

or is there any other way i can process data.

Thanks
Sumit

Jul 11 '07 #1
1 1152
pa****@gmail.co m wrote in news:1184183030 .895020.221080
@w3g2000hsg.goo glegroups.com:

the problem is the records are so high (3 4 millions) then data is too
large and it takes hell lot of time to complete the process. Also
application server CPU utilization shoots out to max.

What will be the best way to achive this.
1. Should i use DataRepeater instead of dataset. or
2. Should i do processing in chunks. how can I do processing in
chunks???

More 10,000 rows in general is too much data for a dataset.

You should be processing in chunks - take a look at "Paging".

Also do you need ALL the data? Why are you sending so much data to the
client?
Jul 11 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
1855
by: MT | last post by:
I have an xsd file that I use to create a strongly typed dataset in my project. In the past, I have used the ReadXml method to load xml files into the generated class and read data in using this class. These files have now become big (order of 100MB) and this completely kills the readxml performance. I have read multiple posts that tell you not to use datasets for large files. I even tried calling BeginLoadTables before starting the...
1
1697
by: George Durzi | last post by:
I maintain a list of companies in my application. The first time the company table is accessed, I create a dataset and stick it in the Cache. When the company table is modified, I expire the cached item, recreate the DataSet, and stick it in the Cache again I get excellent performance when running this on my local machine. I can tell when an aspx is accessing a cached copy of the company list since it's very fast. However, it's painfully slow...
3
2278
by: Brent | last post by:
Hi, I'm wondering if it is good to use datasets for large amounts of data with many users. I'm talking tables with 130,000 records and 15 columns. And we want current data, so no cached data. Right now we are using an OleDbDataReader and then just doing reader.Read() to go through the necessary records. So would using a dataset be good, or bad for this compared to what we are doing? We're looking at it again to get the features of...
3
1538
by: spammy | last post by:
hi all, im currently writing an ASP.net front end to display data found in an XML file. the XML file is quite large (700k or so), and so takes a while to load. it will be changing around every second, so needs to be loaded at least at that rate. at the moment, the ASP application loads the xml every refresh - im thinking that this may cause problems when there are 20 clients each refreshing once per second. the solution im thinking of...
3
1946
by: vanvee | last post by:
Hi I have an application for my company's HR department where we store resumes for candidates we receive. I have an application that uses VB.Net and ADO.Net and data bindings (through code) to controls on a Windows form. The question I have is that once the database grows and contains a large number of records, I am worried the dataset is going to create too much overhead when the user clicks "show all records" (the full resume which...
22
25607
by: Arne | last post by:
How do I pass a dataset to a webservices? I need to submit a shoppingcart from a pocket PC to a webservice. What is the right datatype? II have tried dataset as a datatype, but I can't get it to compile. <WebMethod()> _ Public Function VerifySku(ByVal skus As XmlDataDocument) As DataSet Test program : Dim cartSet As DataSet cartSet = ws.VerifySku(cartSet)
1
2829
by: None | last post by:
Hi, I have developed webshop application using asp.net 1.1. I'm using DataGrid in one of the pages of my site. During the page load the DataGrid will be binded by around 7500 products(rows). At the same time i'm storing the DataSet object in the Session Variable for later refinement. I'm using StateServer mode for storeing the session values. But in my server the W3wp.exe is taking 90 - 100 % CPU
39
2860
by: cj | last post by:
I have a 2005 TCP/IP server that creates a new thread to handle each incoming TCP/IP request. Once the request has been answered by the thread the TCP/IP socket is disconnected and the sub/thread ends. This program originally written in 2003 and I rewrote it in 2005 earlier this year. I don't know how long this has been going on but we just noticed today that the program was using over a gig of virtual memory. Real memory was quite...
2
2790
by: palgre | last post by:
Hi ALL, I am working in an windows based application using SQL Server 2000 as database. There are few tables (refer parent tables) in the application which are uploaded by a seprate application. My application fetches data from parent tables and put data in separate tables (chils tables) of application. I am using dataset to fetch data from parent tables and insert/update data in child table.
0
9672
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9519
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10214
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
9042
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7538
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6780
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5437
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4113
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3723
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.