473,698 Members | 2,616 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Under the hood of DataReader

Hi,

My friend had a little confusion about the working of DataReader after
reading an article from MSDN. Following is a message from him...

<!-- Message starts -->
I was going thru DataReader in ADO.NET in MSDN and there is a confusion
regarding the buffering
of data in case of DATAREADER. The link for MSDN JAN 2004 is -

ms-help://MS.MSDNQTR.2004 JAN.1033/cpguide/html/cpconTheADONETD ataReader.htm
At this link, they say that :
Results are returned as the query executes, and are stored in the network
buffer on the client until you request them using the Read method of the
DataReader
That's in the starting part of the article. Later in the same article, it is
said that DataReader provides unbuffered stream....

The DataReader provides an unbuffered stream of data........... .......
DataReader is a good choice when retrieving large amounts of data because
the data is not cached in memory.
So, there is no buffering in the primary memory going on but there is some
network buffer. If there is, where is the network buffer - in the NIC or
some Router or other Gateway... ??
And whether at all, data in case of DataReader is buffered or not.. ??
<!-- Message ends -->

The way I understood it is as follows, but wanted to confirm it with you
foloks before I rant foolishly. Would appreciate further inputs or
corrections, which I most probably need.

- data is streamed to the client machine and gets queued up until it is
processed or the data reader is closed
- application (that's making the request) is given the starting pointer to
the data coming in
- data consumed by the application is destroyed as it is used (by calling
the Read method), similar to electricity...
- term 'buffer' in the unbuffered means not getting buffered in the
application's memory space, as with DataSet

thanks!
-Yasutaka
Nov 22 '05 #1
6 5450
First, let's make sure we understand our terms.
1. Server is the database server
2. Client is the software getting data (the one that opened the connection).
This can be a fat client or a web app.

The method of the data reader is simliar to any fast forward (firehose)
cursor on the server side. The information is streamed directly from the
database. On the client, the reader will require some form of buffer to hold
the data. It is a very short term buffer, as it is cleared when the
connection closes (NOTE: Until GC, the memory is not necessarily cleared,
but the data is no longer accessible).

When the DataReader is read, each "row" is destroyed from the stream
(buffer) as it is consumed. When closed, or the connection is closed, all
"rows" are destroyed. There is no built in persistence of the data, ala a
DataSet, so the data effectively dies if you do not store it in your app
somehow.

I know of a programmer who only uses DataReaders. He likes the perf. Problem
is his apps are much more complex and harder to maintain due to his refusal
to use a DataSet. I tend to err on the side of maintainability until I find
a perf problem. I would also caution anyone who decides that the ONLY way to
work with data is a DataReader, for performance reasons, as your app will
likely be maintained by someone who is not as educated (and perhaps not as
smart) as you.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

*************** *************** *************** *************** **********
Think Outside the Box!
*************** *************** *************** *************** **********
"Yasutaka Ito" <no****@nonexis tent.com> wrote in message
news:e6******** ********@TK2MSF TNGP12.phx.gbl. ..
Hi,

My friend had a little confusion about the working of DataReader after
reading an article from MSDN. Following is a message from him...

<!-- Message starts -->
I was going thru DataReader in ADO.NET in MSDN and there is a confusion
regarding the buffering
of data in case of DATAREADER. The link for MSDN JAN 2004 is -

ms-help://MS.MSDNQTR.2004 JAN.1033/cpguide/html/cpconTheADONETD ataReader.htm At this link, they say that :
Results are returned as the query executes, and are stored in the network
buffer on the client until you request them using the Read method of the
DataReader
That's in the starting part of the article. Later in the same article, it is said that DataReader provides unbuffered stream....

The DataReader provides an unbuffered stream of data........... .......
DataReader is a good choice when retrieving large amounts of data because
the data is not cached in memory.
So, there is no buffering in the primary memory going on but there is some
network buffer. If there is, where is the network buffer - in the NIC or
some Router or other Gateway... ??
And whether at all, data in case of DataReader is buffered or not.. ??
<!-- Message ends -->

The way I understood it is as follows, but wanted to confirm it with you
foloks before I rant foolishly. Would appreciate further inputs or
corrections, which I most probably need.

- data is streamed to the client machine and gets queued up until it is
processed or the data reader is closed
- application (that's making the request) is given the starting pointer to
the data coming in
- data consumed by the application is destroyed as it is used (by calling
the Read method), similar to electricity...
- term 'buffer' in the unbuffered means not getting buffered in the
application's memory space, as with DataSet

thanks!
-Yasutaka

Nov 22 '05 #2
> I know of a programmer who only uses DataReaders. He likes the perf.
Problem
is his apps are much more complex and harder to maintain due to his refusal to use a DataSet. I tend to err on the side of maintainability until I find a perf problem. I would also caution anyone who decides that the ONLY way to work with data is a DataReader, for performance reasons, as your app will
likely be maintained by someone who is not as educated (and perhaps not as
smart) as you.


I find DataReaders very simple to use, and mantain using two classes
implementing methods which create/destroy/query readers/dataconnections .
This way queries are done in few sentences and maintaining db code is a
matter of changing the class.

public class myDBConn {
var cStr = null; //connection str
public var dbLocation = "c:\\delo\\ ";
public var myConn : OleDbConnection = null; //connection

function myDBConn(db : String) { //constructor
cStr = "Provider=Micro soft.Jet.OLEDB. 4.0;Data Source=" + dbLocation + db
+ ";";
myConn = new OleDbConnection (cStr);
myConn.Open();
}

function myDBConn(db : String, dbPwd : String) {
cStr = "Provider=Micro soft.Jet.OLEDB. 4.0;Data Source=" + dbLocation + db
+ ";Jet OLEDB:Database Password=" + dbPwd + ";";
myConn = new OleDbConnection (cStr);
myConn.Open();
}

function Close() {
myConn.Close();
myConn = null;
}
} //class db connection
//recordset:
public class myRS {
var myCmd : OleDbCommand = new OleDbCommand();
public var reader : OleDbDataReader = null;

function myRS(dbConn : myDBConn) {
myCmd.Connectio n = dbConn.myConn;
myCmd.CommandTi meout = 20;
}

function Query(sql : String) {
myCmd.CommandTe xt = sql;
reader = myCmd.ExecuteRe ader();
}

function Exec(sql : String) {
myCmd.CommandTe xt = sql;
myCmd.ExecuteNo nQuery();
}

function Reset() {
try {
reader.Close();
} catch (e) { }
reader = null;
}

function Close() {
this.Reset();
myCmd.Dispose() ;
myCmd = null;
}

} //class recordset
Nov 22 '05 #3
Hi Jure,

What about the data retrieved through datareader?

--
Miha Markic [MVP C#] - RightHand .NET consulting & development
miha at rthand com
www.rthand.com

"Jure Spik" <sp***@siol.net > wrote in message
news:uD******** ******@TK2MSFTN GP10.phx.gbl...
I find DataReaders very simple to use, and mantain using two classes
implementing methods which create/destroy/query readers/dataconnections .
This way queries are done in few sentences and maintaining db code is a
matter of changing the class.

Nov 22 '05 #4
Miha, hello.

Of course this is not the way to go if you have a reason to implement
caching. Then I'd go for dataAdapter and a cached datatable.

"Miha Markic [MVP C#]" <miha at rthand com> wrote in message
news:#3******** ******@TK2MSFTN GP11.phx.gbl...
Hi Jure,

What about the data retrieved through datareader?

--
Miha Markic [MVP C#] - RightHand .NET consulting & development
miha at rthand com
www.rthand.com

"Jure Spik" <sp***@siol.net > wrote in message
news:uD******** ******@TK2MSFTN GP10.phx.gbl...
I find DataReaders very simple to use, and mantain using two classes
implementing methods which create/destroy/query readers/dataconnections .
This way queries are done in few sentences and maintaining db code is a
matter of changing the class.


Nov 22 '05 #5
Hi Jure,

Yes, that makes sense.
I think Cowboy had in mind this case.

--
Miha Markic [MVP C#] - RightHand .NET consulting & development
miha at rthand com
www.rthand.com

"Jure Spik" <sp***@siol.net > wrote in message
news:%2******** ********@TK2MSF TNGP12.phx.gbl. ..
Miha, hello.

Of course this is not the way to go if you have a reason to implement
caching. Then I'd go for dataAdapter and a cached datatable.

Nov 22 '05 #6
Let me clarify this a little: first of all, implementation details for
DataReader is provider-specific; we only provide guidelines of what would be
the expected behavior.

In order to describe the details of buffering, I'll pick the SQL Server
provider (SqlClient):

When you execute a SQL statement that returns rows, the server starts
sending rows to the client over the network in packets[1]. The client on the
other end consumes one packet at a time from the network. Every time you
call Read() on the reader, we see if we still have a full row of data in the
current 1-packet buffer; if we don't, we issue another network read and get
another packet and obtain the rest of the data for the row[2]. If you don't
call Read() for a while, we'll eventually have a packet on the client buffer
and a packet or two ready to be sent in the server buffer; at that point,
the server will stop producing data until we consume data from the client.
This means that there is no need to buffer the rest of the data because it
hasn't been sent by server yet.

Once you call Read() again, we discard the previous row and fetch the next
one from the packet buffer or the network, as described above.

Bottom line is that SqlDataReader will never buffer more than 1 packet worth
of data. You can even fine tune this further by using
CommandBehavior .SequentialAcce ss, which will cause the provider to fetch one
field at a time, instead of one row at a time.
[1] Default 8KB in size in 1.0/1.1, may (read will) change in future
versions.
[2] The the row is wide enough (i.e. many columns or very large values) a
single row may span many packets. In those cases, we may issue several
network reads for a single Read() call on the reader.
[note] "network" here may actually be shared memory if client and server are
in the same box, in that case there is no network, but the idea still
applies.
I hope this clarifies the internals a little. Let me know if you'd like to
dig into any other detail.

--
Pablo Castro
Program Manager - ADO.NET Team
Microsoft Corp.

This posting is provided "AS IS" with no warranties, and confers no rights.
"Cowboy (Gregory A. Beamer)" <No************ @comcast.netNoS pamM> wrote in
message news:e4******** ******@TK2MSFTN GP11.phx.gbl...
First, let's make sure we understand our terms.
1. Server is the database server
2. Client is the software getting data (the one that opened the connection). This can be a fat client or a web app.

The method of the data reader is simliar to any fast forward (firehose)
cursor on the server side. The information is streamed directly from the
database. On the client, the reader will require some form of buffer to hold the data. It is a very short term buffer, as it is cleared when the
connection closes (NOTE: Until GC, the memory is not necessarily cleared,
but the data is no longer accessible).

When the DataReader is read, each "row" is destroyed from the stream
(buffer) as it is consumed. When closed, or the connection is closed, all
"rows" are destroyed. There is no built in persistence of the data, ala a
DataSet, so the data effectively dies if you do not store it in your app
somehow.

I know of a programmer who only uses DataReaders. He likes the perf. Problem is his apps are much more complex and harder to maintain due to his refusal to use a DataSet. I tend to err on the side of maintainability until I find a perf problem. I would also caution anyone who decides that the ONLY way to work with data is a DataReader, for performance reasons, as your app will
likely be maintained by someone who is not as educated (and perhaps not as
smart) as you.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

*************** *************** *************** *************** **********
Think Outside the Box!
*************** *************** *************** *************** **********
"Yasutaka Ito" <no****@nonexis tent.com> wrote in message
news:e6******** ********@TK2MSF TNGP12.phx.gbl. ..
Hi,

My friend had a little confusion about the working of DataReader after
reading an article from MSDN. Following is a message from him...

<!-- Message starts -->
I was going thru DataReader in ADO.NET in MSDN and there is a confusion
regarding the buffering
of data in case of DATAREADER. The link for MSDN JAN 2004 is -

ms-help://MS.MSDNQTR.2004 JAN.1033/cpguide/html/cpconTheADONETD ataReader.htm
At this link, they say that :
Results are returned as the query executes, and are stored in the network buffer on the client until you request them using the Read method of the
DataReader
That's in the starting part of the article. Later in the same article, it is
said that DataReader provides unbuffered stream....

The DataReader provides an unbuffered stream of data........... .......
DataReader is a good choice when retrieving large amounts of data

because the data is not cached in memory.
So, there is no buffering in the primary memory going on but there is some network buffer. If there is, where is the network buffer - in the NIC or
some Router or other Gateway... ??
And whether at all, data in case of DataReader is buffered or not.. ??
<!-- Message ends -->

The way I understood it is as follows, but wanted to confirm it with you
foloks before I rant foolishly. Would appreciate further inputs or
corrections, which I most probably need.

- data is streamed to the client machine and gets queued up until it is
processed or the data reader is closed
- application (that's making the request) is given the starting pointer to the data coming in
- data consumed by the application is destroyed as it is used (by calling the Read method), similar to electricity...
- term 'buffer' in the unbuffered means not getting buffered in the
application's memory space, as with DataSet

thanks!
-Yasutaka


Nov 22 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
1318
by: Eric Pederson | last post by:
I didn't come across any illuminating discussion via Google, thus my question here (though it may be a neophyte question.) I am interested in the workings under the hood of Python's access of "files". What is actually happening at the various stages when I create a file object and "read" it? (1) >>> f = file("C:/GuidosParrot.txt","r") (2) >>> hesjustsleeping = f.read() At (1) have I loaded the file from hard drive into RAM when...
6
363
by: Yasutaka Ito | last post by:
Hi, My friend had a little confusion about the working of DataReader after reading an article from MSDN. Following is a message from him... <!-- Message starts --> I was going thru DataReader in ADO.NET in MSDN and there is a confusion regarding the buffering of data in case of DATAREADER. The link for MSDN JAN 2004 is -
2
1819
by: Richy Rich | last post by:
Hi, I've developed an application in C# which spawns a thread to receive datagrams. If I use the socket receive method, there are no problems when using the application under NT or XP. I recently moved over to using the UdpClient receive method. This was so I could record the IPEndPoint from
6
750
by: Ravi | last post by:
Hi, I am not able to understand why a datareader needs a connection to the DB all the time. Here is what I tried. Sqlcommand cmd = ("select * from table1",con) // where con is the connection object 1. DataReader dr = cmd.executereader(); 2.while(dr.read()) { // do something //} I have a break point in line 2. when i run my application and once it hits the break point (i.e line 2) I opened query analyzer and deleted all the rows in...
20
5527
by: Mark | last post by:
Hi all, quick question , a DataView is memory resident "view" of data in a data table therefore once populated you can close the connection to the database. Garbage collection can then be used to "clean up" the DataView once it is not referenced and will not effect the number of connections to the database. A DataReader on the other hand always maintains a connection to the database and must be explicitly closed (Do not rely on garbage...
2
1572
by: Andrei Pociu | last post by:
In a typical ASP .NET Web Application (website), I'm currently using a class where I declare some public static objects. For example there's the place where I initialize the SqlConnection. Also there's the place where I declare one public static DataReader, but I'm not sure this is the best way to do it. I use that DataReader in all the webforms and user controls, whenever I have to read data from the db. So, should I do it this way or...
1
3089
by: Brent | last post by:
I'm having a hard time wrapping my head around how to build a multi-dimensional array of n length out of a DataReader loop. Take this pseudo-code: ======================================= public string get_array(string sql) { //create db connection & open
3
1496
by: tpochep | last post by:
Hello. Is there any good information about python's internals and semantic? I mean "under the hood". For example, I want to understant what do these strings x = 10 y = 10 x = 20
7
2903
by: Diffident | last post by:
Hello All, I would like to use DataReader based accessing in my Data Access Layer (DAL). What is considered to be a best practice while returning from a DAL method that executes a query and returns N rows. DataReader object? Collection object? DataTable object? Returning a DataReader object is not a good practice...right? Thnks for all your suggestions!!
1
1668
by: MrAsm | last post by:
Hi, I'd like to know if C# object system is based on COM/DCOM, or is it a completely new technology, developed "from scratch"? In other words, does C# use COM "under the hood", or are C#/.NET and COM/DCOM completely separate fields? Thanks in advance, MrAsm
0
8610
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
9170
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
9031
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8902
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
7740
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
6528
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5862
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
4372
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
4623
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.