473,785 Members | 2,919 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Using Dataset for large amounts of data

Hi,
I'm wondering if it is good to use datasets for large amounts of data with
many users. I'm talking tables with 130,000 records and 15 columns. And we
want current data, so no cached data. Right now we are using an
OleDbDataReader and then just doing reader.Read() to go through the
necessary records. So would using a dataset be good, or bad for this
compared to what we are doing?

We're looking at it again to get the features of transactions out of it, but
are not sure if they can handle that much data very well?
Nov 18 '05 #1
3 2275
Holy cow!! Why would you bring back that much data? First thing I would
do, regardless of what you are trying to do, is find a way to greatly limit
that number of records you are returning somehow. Do your records have a
unique key of some sort? And are they ordered somehow, maybe by the unique
key (that would be prefered)?

DataReaders will perform faster than datasets. DataSets are disconnected
from the database and must store all their information in memory.
DataReaders remain connected to the database and pull records from the
database as you request them (i.e. reader.Read()).

HTH,

Raymond Lewallen

"Brent" <b@b.com> wrote in message
news:10******** *****@corp.supe rnews.com...
Hi,
I'm wondering if it is good to use datasets for large amounts of data with
many users. I'm talking tables with 130,000 records and 15 columns. And we
want current data, so no cached data. Right now we are using an
OleDbDataReader and then just doing reader.Read() to go through the
necessary records. So would using a dataset be good, or bad for this
compared to what we are doing?

We're looking at it again to get the features of transactions out of it, but are not sure if they can handle that much data very well?

Nov 18 '05 #2
Cor
Hi Raymond,

I agree completly with the first part of your message. With the second part
I do not (completly).
Holy cow!! Why would you bring back that much data? First thing I would
do, regardless of what you are trying to do, is find a way to greatly limit that number of records you are returning somehow. Do your records have a
unique key of some sort? And are they ordered somehow, maybe by the unique key (that would be prefered)? DataReaders will perform faster than datasets. DataSets are disconnected
from the database and must store all their information in memory.
DataReaders remain connected to the database and pull records from the
database as you request them (i.e. reader.Read()).


It is not said that DataReaders perform faster than using the dataadapter
which uses also the datareader.

Main thing is that it fixes very efficient the data in memory while with a
datareader you need to do that item by item.

The amount of data with a right selection, as you stated before, can often
be that few, that it is probably often almost the same amount of bytes as
the code needed to retrieve it using a datareader, so that can not be a
point. (and in a modern computer absolute not relevant).

Moreover when readed it the dataset becomes very handy to handle.

Just my thought to give the OP another opinion, while I can think about
situations where your statement is right.

Cor




Nov 18 '05 #3
Brent,

I posted arguments and sample code on DataSet vs DataReader on a response to
Cor on this post dated 4/7/2004 08:24 AM CST. Be sure and read that and it
my help further clarify what I failed to completely clarify in my initial
response post yesterday.

Raymond Lewallen

"Brent" <b@b.com> wrote in message
news:10******** *****@corp.supe rnews.com...
Hi,
I'm wondering if it is good to use datasets for large amounts of data with
many users. I'm talking tables with 130,000 records and 15 columns. And we
want current data, so no cached data. Right now we are using an
OleDbDataReader and then just doing reader.Read() to go through the
necessary records. So would using a dataset be good, or bad for this
compared to what we are doing?

We're looking at it again to get the features of transactions out of it, but are not sure if they can handle that much data very well?

Nov 18 '05 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
2343
by: michaaal | last post by:
If I use a form to pass data (say, for example, through a textbox) the data seems to be limited to somewhat smaller amounts. What should I do if I want to pass a large amount of data? For example a list of 200 items?
10
2424
by: Digety | last post by:
We are looking to store a large amount of user data that will be changed and accessed daily by a large number of people. We expect around 6-8 million subscribers to our service with each record being approximately 2000-2500 bytes. The system needs to be running 24/7 and therefore cannot be shut down. What is the best way to implement this? We were thinking of setting up a cluster of servers to hold the information and another cluster...
0
1113
by: CHosoi | last post by:
I want to transfer large amounts of data from an access database to a SQL server database. The schema within the Access database is vastly different to the SQL Server one. Mapping data from access to SQL server is relatively straight forward. What I'm less sure about is the best approach to take to achieve this. Best option I've come up with so far is to use a .NET windows app which uses ADO.NET to read data from access into a typed...
1
5998
by: Bart | last post by:
Dear all, I would like to encrypt a large amount of data by using public/private keys, but I read on MSDN: "Symmetric encryption is performed on streams and is therefore useful to encrypt large amounts of data. Asymmetric encryption is performed on a small number of bytes and is therefore only useful for small amounts of data." There is not possibility to do it? I have tried to encrypt a 300kB file by RSA Algorithm, but I received...
3
224
by: Brent | last post by:
Hi, I'm wondering if it is good to use datasets for large amounts of data with many users. I'm talking tables with 130,000 records and 15 columns. And we want current data, so no cached data. Right now we are using an OleDbDataReader and then just doing reader.Read() to go through the necessary records. So would using a dataset be good, or bad for this compared to what we are doing? We're looking at it again to get the features of...
1
1874
by: JIM.H. | last post by:
Hello, Here is part of my code to retrieve some data from FoxPro and populate DataGrid. It works fine. What is the best and easiest way to import data from Dataset to a table in SQL Server. ' populate dataAdapter and dataset cmdSelect = Conn.CreateCommand cmdSelect.CommandText = sqlStr Da.SelectCommand = cmdSelect
7
10828
by: =?Utf-8?B?TW9iaWxlTWFu?= | last post by:
Hello everyone: I am looking for everyone's thoughts on moving large amounts (actually, not very large, but large enough that I'm throwing exceptions using the default configurations). We're doing a proof-of-concept on WCF whereby we have a Windows form client and a Server. Our server is a middle-tier that interfaces with our SQL 05 database server.
13
5117
by: trpost | last post by:
I am looking for a way to send data from one page to another as POST data without using forms or cURL. I have a php script that is passing a list of cases from on page to another when a link is clicked. This is working fine for the most part as a link, but sometimes the list gets very large and gets cut off. The reason it gets cut off appears to be a limitation on the amount of data that can be passesd in the URI. It looks like I can...
0
844
by: John Sheppard | last post by:
Hello there, I am wanting to pass my datasets by reference to other forms. As soon as I do that I have to reset any binding sources by setting their datasource property to the original datasource. I also find that I cannot then bind directly to my dataset without a binding source, so I have to duplicated all my binding sources on my second form. Is this normal or am I doing something wrong? How do you guys normally pass
0
9480
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10319
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9947
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7496
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6737
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5380
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5511
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4046
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
3645
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.