473,396 Members | 1,871 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,396 software developers and data experts.

Using Dataset for large amounts of data

Hi,
I'm wondering if it is good to use datasets for large amounts of data with
many users. I'm talking tables with 130,000 records and 15 columns. And we
want current data, so no cached data. Right now we are using an
OleDbDataReader and then just doing reader.Read() to go through the
necessary records. So would using a dataset be good, or bad for this
compared to what we are doing?

We're looking at it again to get the features of transactions out of it, but
are not sure if they can handle that much data very well?
Nov 18 '05 #1
3 2246
Holy cow!! Why would you bring back that much data? First thing I would
do, regardless of what you are trying to do, is find a way to greatly limit
that number of records you are returning somehow. Do your records have a
unique key of some sort? And are they ordered somehow, maybe by the unique
key (that would be prefered)?

DataReaders will perform faster than datasets. DataSets are disconnected
from the database and must store all their information in memory.
DataReaders remain connected to the database and pull records from the
database as you request them (i.e. reader.Read()).

HTH,

Raymond Lewallen

"Brent" <b@b.com> wrote in message
news:10*************@corp.supernews.com...
Hi,
I'm wondering if it is good to use datasets for large amounts of data with
many users. I'm talking tables with 130,000 records and 15 columns. And we
want current data, so no cached data. Right now we are using an
OleDbDataReader and then just doing reader.Read() to go through the
necessary records. So would using a dataset be good, or bad for this
compared to what we are doing?

We're looking at it again to get the features of transactions out of it, but are not sure if they can handle that much data very well?

Nov 18 '05 #2
Cor
Hi Raymond,

I agree completly with the first part of your message. With the second part
I do not (completly).
Holy cow!! Why would you bring back that much data? First thing I would
do, regardless of what you are trying to do, is find a way to greatly limit that number of records you are returning somehow. Do your records have a
unique key of some sort? And are they ordered somehow, maybe by the unique key (that would be prefered)? DataReaders will perform faster than datasets. DataSets are disconnected
from the database and must store all their information in memory.
DataReaders remain connected to the database and pull records from the
database as you request them (i.e. reader.Read()).


It is not said that DataReaders perform faster than using the dataadapter
which uses also the datareader.

Main thing is that it fixes very efficient the data in memory while with a
datareader you need to do that item by item.

The amount of data with a right selection, as you stated before, can often
be that few, that it is probably often almost the same amount of bytes as
the code needed to retrieve it using a datareader, so that can not be a
point. (and in a modern computer absolute not relevant).

Moreover when readed it the dataset becomes very handy to handle.

Just my thought to give the OP another opinion, while I can think about
situations where your statement is right.

Cor




Nov 18 '05 #3
Brent,

I posted arguments and sample code on DataSet vs DataReader on a response to
Cor on this post dated 4/7/2004 08:24 AM CST. Be sure and read that and it
my help further clarify what I failed to completely clarify in my initial
response post yesterday.

Raymond Lewallen

"Brent" <b@b.com> wrote in message
news:10*************@corp.supernews.com...
Hi,
I'm wondering if it is good to use datasets for large amounts of data with
many users. I'm talking tables with 130,000 records and 15 columns. And we
want current data, so no cached data. Right now we are using an
OleDbDataReader and then just doing reader.Read() to go through the
necessary records. So would using a dataset be good, or bad for this
compared to what we are doing?

We're looking at it again to get the features of transactions out of it, but are not sure if they can handle that much data very well?

Nov 18 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: michaaal | last post by:
If I use a form to pass data (say, for example, through a textbox) the data seems to be limited to somewhat smaller amounts. What should I do if I want to pass a large amount of data? For example...
10
by: Digety | last post by:
We are looking to store a large amount of user data that will be changed and accessed daily by a large number of people. We expect around 6-8 million subscribers to our service with each record...
0
by: CHosoi | last post by:
I want to transfer large amounts of data from an access database to a SQL server database. The schema within the Access database is vastly different to the SQL Server one. Mapping data from access...
1
by: Bart | last post by:
Dear all, I would like to encrypt a large amount of data by using public/private keys, but I read on MSDN: "Symmetric encryption is performed on streams and is therefore useful to encrypt large...
3
by: Brent | last post by:
Hi, I'm wondering if it is good to use datasets for large amounts of data with many users. I'm talking tables with 130,000 records and 15 columns. And we want current data, so no cached data....
1
by: JIM.H. | last post by:
Hello, Here is part of my code to retrieve some data from FoxPro and populate DataGrid. It works fine. What is the best and easiest way to import data from Dataset to a table in SQL Server. '...
7
by: =?Utf-8?B?TW9iaWxlTWFu?= | last post by:
Hello everyone: I am looking for everyone's thoughts on moving large amounts (actually, not very large, but large enough that I'm throwing exceptions using the default configurations). We're...
13
by: trpost | last post by:
I am looking for a way to send data from one page to another as POST data without using forms or cURL. I have a php script that is passing a list of cases from on page to another when a link is...
0
by: John Sheppard | last post by:
Hello there, I am wanting to pass my datasets by reference to other forms. As soon as I do that I have to reset any binding sources by setting their datasource property to the original...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.