473,320 Members | 1,817 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

how big can disconnected dataset be?

Hello,

I have to read and write around one million records from
an external data source to Sql Server2k every night.
That's a lot of I/O. I am using VB6 for this (takes
hours). I am connecting to the external data source with
API's from its object library (as opposed to ODBC in which
case I would just use DTS to pull this data) and just
looping. I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4 gigs
of mem - may upgrade to 16 gigs if they let me have win2K
Advanced Server - currently 2 processors 1 gig each). My
idea is that if I could just read the data into memory (a
dataset) it would be much faster than writing each record
(180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?

While I'm at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets in
more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net for
multi-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They write
to 4 separate table in Sql Server2k. I would really like
to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?

Thanks,
Rich
Nov 20 '05 #1
10 1429

">
I have to read and write around one million records from
an external data source to Sql Server2k every night.
That's a lot of I/O.
Yes it is... =)
I am using VB6 for this (takes
hours). I am connecting to the external data source with
API's from its object library (as opposed to ODBC in which
case I would just use DTS to pull this data) and just
looping.
I'm assuming this is a proprietary data format?
I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4 gigs
of mem - may upgrade to 16 gigs if they let me have win2K
Advanced Server - currently 2 processors 1 gig each). My
idea is that if I could just read the data into memory (a
dataset) it would be much faster than writing each record
(180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?

Your still going to have a downturn on speed on the update/insert. I'm not
sure if it will cache data on your local disk or not depending on the amount
of memory. But you are still going to be subject to the speed of your SQL
Server. So you can have it all queued up, but its not necessarily going to
make it faster.
While I'm at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets in
more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net for
multi-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They write
to 4 separate table in Sql Server2k. I would really like
to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?

Many ways, I just don't know the benefit of it from what your giving us.
Yeah your reads and loading will be fast, but is this program going to run
on the SQL server itself? If your looking to reduce your amount of network
traffic, yeah this will solve your problem, if your looking to speed up
injection into your SQL server, not gonna do it... =(

You could use a more updated OLEDB driver for SQL Server, which may shave
some time off because of optimized programming, but again, don't know how
much of a gain you will actually get.

-CJ
Thanks,
Rich

Nov 20 '05 #2
Thanks. I forgot to mention, yes, reducing network
traffice, or rather, get all the data to the local server
and close the connection would be a good thing to
achieve. The other data source is Lotus Notes R5. I have
to pull data from 4 separate Notes DB's (nsf's). Not my
design. I could take it or leave it.

So you day that it would be possible to fill the datasets
(disconnected datasets) this way? Thinking outloud, I
have 4 tables in Sql Server and would have 4 datasets in
my VB.net app. Then, with multi-threading, I would start
pulling data into each of these datasets simultaneously.
I have 4 gigs of mem right now. The deal is that the old
way I would just write the data directly to the Sql
Tables. Here I load the data to memory as fast as I can,
close my connection to Lotus Notes (using Domino Object
Library here) and then start pushing the data into Sql
Server from the datasets. Wouldn't this be better, more
efficient than having 4 com based apps reading and writing
data 250,000+ times a piece?

-----Original Message-----

">
I have to read and write around one million records from
an external data source to Sql Server2k every night.
That's a lot of I/O.
Yes it is... =)
I am using VB6 for this (takes
hours). I am connecting to the external data source with API's from its object library (as opposed to ODBC in which case I would just use DTS to pull this data) and just
looping.


I'm assuming this is a proprietary data format?
I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4 gigs of mem - may upgrade to 16 gigs if they let me have win2K Advanced Server - currently 2 processors 1 gig each). My idea is that if I could just read the data into memory (a dataset) it would be much faster than writing each record (180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?


Your still going to have a downturn on speed on the

update/insert. I'm notsure if it will cache data on your local disk or not depending on the amountof memory. But you are still going to be subject to the speed of your SQLServer. So you can have it all queued up, but its not necessarily going tomake it faster.
While I'm at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets in more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net for multi-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They write to 4 separate table in Sql Server2k. I would really like to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?

Many ways, I just don't know the benefit of it from what

your giving us.Yeah your reads and loading will be fast, but is this program going to runon the SQL server itself? If your looking to reduce your amount of networktraffic, yeah this will solve your problem, if your looking to speed upinjection into your SQL server, not gonna do it... =(

You could use a more updated OLEDB driver for SQL Server, which may shavesome time off because of optimized programming, but again, don't know howmuch of a gain you will actually get.

-CJ
Thanks,
Rich

.

Nov 20 '05 #3

"Rich" <an*******@discussions.microsoft.com> wrote in message
news:14*****************************@phx.gbl...
Thanks. I forgot to mention, yes, reducing network
traffice, or rather, get all the data to the local server
and close the connection would be a good thing to
achieve. The other data source is Lotus Notes R5. I have
to pull data from 4 separate Notes DB's (nsf's). Not my
design. I could take it or leave it.

I coded notes (4.5 and 5) for 2 years... I can honestly say that I have
never looked on a project and said "You know what would be a good
solution... Notes..."

I know it has its advantages, but coming from a developer standpoint it
kinda sucks and is INCREDIBLY slow...
So you day that it would be possible to fill the datasets
(disconnected datasets) this way? Thinking outloud, I
have 4 tables in Sql Server and would have 4 datasets in
my VB.net app.
Yeah, that would be fine. I'm sure the Notes API is slowing you down as it
is... So this will reduce your overall network traffic (well, amount of
time used for network traffic, you'll have a big burst at the beginning and
then nothing).
Then, with multi-threading, I would start
pulling data into each of these datasets simultaneously.
I have 4 gigs of mem right now. The deal is that the old
way I would just write the data directly to the Sql
Tables. Here I load the data to memory as fast as I can,
close my connection to Lotus Notes (using Domino Object
Library here) and then start pushing the data into Sql
Server from the datasets. Wouldn't this be better, more
efficient than having 4 com based apps reading and writing
data 250,000+ times a piece?
For network traffic, yes, it would be better. I don't think you will see a
huge performance increase on your SQL server, but I'm sure you will see some
(as you now are not requesting data from a foreign source for each record).

Nonetheless... 180 fields, 1 million records, still going to take al ittl
etime to get in there. =)

-CJ

-----Original Message-----

">
I have to read and write around one million records from
an external data source to Sql Server2k every night.
That's a lot of I/O.


Yes it is... =)
I am using VB6 for this (takes
hours). I am connecting to the external data source with API's from its object library (as opposed to ODBC in which case I would just use DTS to pull this data) and just
looping.


I'm assuming this is a proprietary data format?
I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4 gigs of mem - may upgrade to 16 gigs if they let me have win2K Advanced Server - currently 2 processors 1 gig each). My idea is that if I could just read the data into memory (a dataset) it would be much faster than writing each record (180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?


Your still going to have a downturn on speed on the

update/insert. I'm not
sure if it will cache data on your local disk or not

depending on the amount
of memory. But you are still going to be subject to the

speed of your SQL
Server. So you can have it all queued up, but its not

necessarily going to
make it faster.
While I'm at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets in more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net for multi-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They write to 4 separate table in Sql Server2k. I would really like to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?


Many ways, I just don't know the benefit of it from what

your giving us.
Yeah your reads and loading will be fast, but is this

program going to run
on the SQL server itself? If your looking to reduce your

amount of network
traffic, yeah this will solve your problem, if your

looking to speed up
injection into your SQL server, not gonna do it... =(

You could use a more updated OLEDB driver for SQL Server,

which may shave
some time off because of optimized programming, but

again, don't know how
much of a gain you will actually get.

-CJ
Thanks,
Rich

.

Nov 20 '05 #4
Cor
Hi Rich,

I was making a message, but I see that my points of addition are only minors
on the text from CJ.

(A little thing, if your server goes down in the middle of an update with a
dataset, do you know what that does mean for you?)
You could use a more updated OLEDB driver for SQL Server, which may shave
some time off because of optimized programming, but again, don't know how
much of a gain you will actually get.


This is the only thing that I will make a real addition.

Have a look at the SQL provider.

http://msdn.microsoft.com/library/de...tproviders.asp

Cor
Nov 20 '05 #5

"Cor" <no*@non.com> wrote in message
news:OX****************@TK2MSFTNGP11.phx.gbl...
Hi Rich,

I was making a message, but I see that my points of addition are only minors on the text from CJ.

(A little thing, if your server goes down in the middle of an update with a dataset, do you know what that does mean for you?)

A runtime exception.

=)
You could use a more updated OLEDB driver for SQL Server, which may shave some time off because of optimized programming, but again, don't know how much of a gain you will actually get.


This is the only thing that I will make a real addition.

Have a look at the SQL provider.

http://msdn.microsoft.com/library/de...tproviders.asp
Excellent point.
Cor

Nov 20 '05 #6
Thank you all for your replies. At least now I can start
focusing on how to working multithreading. Plus, since I
am still relatively new to VB.Net I'm still having a
little trouble using external references - like the
proper/correct syntax (like do I still goto References and
select the library object I need for Domino Library - or
do I use an Import statement). Oh well, that will be for
my next post :).

Thanks all,
Rich

-----Original Message-----
Hello,

I have to read and write around one million records from
an external data source to Sql Server2k every night.
That's a lot of I/O. I am using VB6 for this (takes
hours). I am connecting to the external data source with
API's from its object library (as opposed to ODBC in whichcase I would just use DTS to pull this data) and just
looping. I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4 gigs
of mem - may upgrade to 16 gigs if they let me have win2K
Advanced Server - currently 2 processors 1 gig each). My
idea is that if I could just read the data into memory (a
dataset) it would be much faster than writing each record
(180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?

While I'm at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets in
more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net formulti-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They writeto 4 separate table in Sql Server2k. I would really like
to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?

Thanks,
Rich
.

Nov 20 '05 #7
I've spent 7 years coding Notes 4.5, 5.x, and 6, all I can can say is that,
there are advantages in certain design scanarios. Notes is an easy place to
code up an application quickly amd some requirements lend themselves well to
notes.

However, having said that, Notes is dying out and will continue to do so,
there are so many better things around to take you away from it.

Regards - OHM

CJ Taylor wrote:
"Rich" <an*******@discussions.microsoft.com> wrote in message
news:14*****************************@phx.gbl...
Thanks. I forgot to mention, yes, reducing network
traffice, or rather, get all the data to the local server
and close the connection would be a good thing to
achieve. The other data source is Lotus Notes R5. I have
to pull data from 4 separate Notes DB's (nsf's). Not my
design. I could take it or leave it.


I coded notes (4.5 and 5) for 2 years... I can honestly say that I
have never looked on a project and said "You know what would be a good
solution... Notes..."

I know it has its advantages, but coming from a developer standpoint
it kinda sucks and is INCREDIBLY slow...
So you day that it would be possible to fill the datasets
(disconnected datasets) this way? Thinking outloud, I
have 4 tables in Sql Server and would have 4 datasets in
my VB.net app.


Yeah, that would be fine. I'm sure the Notes API is slowing you down
as it is... So this will reduce your overall network traffic (well,
amount of time used for network traffic, you'll have a big burst at
the beginning and then nothing).
Then, with multi-threading, I would start
pulling data into each of these datasets simultaneously.
I have 4 gigs of mem right now. The deal is that the old
way I would just write the data directly to the Sql
Tables. Here I load the data to memory as fast as I can,
close my connection to Lotus Notes (using Domino Object
Library here) and then start pushing the data into Sql
Server from the datasets. Wouldn't this be better, more
efficient than having 4 com based apps reading and writing
data 250,000+ times a piece?


For network traffic, yes, it would be better. I don't think you will
see a huge performance increase on your SQL server, but I'm sure you
will see some (as you now are not requesting data from a foreign
source for each record).

Nonetheless... 180 fields, 1 million records, still going to take al
ittl etime to get in there. =)

-CJ

-----Original Message-----

">
I have to read and write around one million records from
an external data source to Sql Server2k every night.
That's a lot of I/O.

Yes it is... =)

I am using VB6 for this (takes
hours). I am connecting to the external data source

with
API's from its object library (as opposed to ODBC in

which
case I would just use DTS to pull this data) and just
looping.

I'm assuming this is a proprietary data format?

I was thinking that with VB.Net I could read
this data into a dataset in memory (currently have 4

gigs
of mem - may upgrade to 16 gigs if they let me have

win2K
Advanced Server - currently 2 processors 1 gig each).

My
idea is that if I could just read the data into memory

(a
dataset) it would be much faster than writing each

record
(180 fields per record). Then I could just do a
dataAdapter.Update or InsertInto to Sql Server once the
data is all in local memory. Any thoughts?
Your still going to have a downturn on speed on the

update/insert. I'm not
sure if it will cache data on your local disk or not

depending on the amount
of memory. But you are still going to be subject to the

speed of your SQL
Server. So you can have it all queued up, but its not

necessarily going to
make it faster.

While I'm at it, the records come from 4 different
sources. I was thinking about using multi-threading and
pull the data simultaneously. I am aware than Sql
Server2k only supports 4 gigs a mem. But if I have more
than 4 gigs of data can one VB.Net app manage datasets

in
more than 4 gigs of memory? Once I fill my datasets, I
would just do one InsertInto at a time. How is VB.Net

for
multi-threading? Again, in VB6 I invoke 4 separate apps
which simultaneously pull the data each night. They

write
to 4 separate table in Sql Server2k. I would really

like
to have all this in one app and read/write directly to
memory. Is VB.Net a way to do this?
Many ways, I just don't know the benefit of it from what

your giving us.
Yeah your reads and loading will be fast, but is this

program going to run
on the SQL server itself? If your looking to reduce your

amount of network
traffic, yeah this will solve your problem, if your

looking to speed up
injection into your SQL server, not gonna do it... =(

You could use a more updated OLEDB driver for SQL Server,

which may shave
some time off because of optimized programming, but

again, don't know how
much of a gain you will actually get.

-CJ

Thanks,
Rich
.


--
Best Regards - OHM
one.handed.man{at}BTInternet{dot}com
Nov 20 '05 #8
I've spent 7 years coding Notes 4.5, 5.x, and 6, all I can can say is that, there are advantages in certain design scanarios. Notes is an easy place to code up an application quickly amd some requirements lend themselves well to notes.
Thats why I made my second comment that I know it has its advantages (sweet
irony, I found a use today in a conversation.. I didn't bring it up though.
=))


However, having said that, Notes is dying out and will continue to do so,
there are so many better things around to take you away from it.

Exactly...

Btw,
Where the hell have you been? I don't think I've seen a post from you in
ages!
Regards - OHM

CJ Taylor wrote:
"Rich" <an*******@discussions.microsoft.com> wrote in message
news:14*****************************@phx.gbl...
Thanks. I forgot to mention, yes, reducing network
traffice, or rather, get all the data to the local server
and close the connection would be a good thing to
achieve. The other data source is Lotus Notes R5. I have
to pull data from 4 separate Notes DB's (nsf's). Not my
design. I could take it or leave it.


I coded notes (4.5 and 5) for 2 years... I can honestly say that I
have never looked on a project and said "You know what would be a good
solution... Notes..."

I know it has its advantages, but coming from a developer standpoint
it kinda sucks and is INCREDIBLY slow...
So you day that it would be possible to fill the datasets
(disconnected datasets) this way? Thinking outloud, I
have 4 tables in Sql Server and would have 4 datasets in
my VB.net app.


Yeah, that would be fine. I'm sure the Notes API is slowing you down
as it is... So this will reduce your overall network traffic (well,
amount of time used for network traffic, you'll have a big burst at
the beginning and then nothing).
Then, with multi-threading, I would start
pulling data into each of these datasets simultaneously.
I have 4 gigs of mem right now. The deal is that the old
way I would just write the data directly to the Sql
Tables. Here I load the data to memory as fast as I can,
close my connection to Lotus Notes (using Domino Object
Library here) and then start pushing the data into Sql
Server from the datasets. Wouldn't this be better, more
efficient than having 4 com based apps reading and writing
data 250,000+ times a piece?


For network traffic, yes, it would be better. I don't think you will
see a huge performance increase on your SQL server, but I'm sure you
will see some (as you now are not requesting data from a foreign
source for each record).

Nonetheless... 180 fields, 1 million records, still going to take al
ittl etime to get in there. =)

-CJ


-----Original Message-----

">
> I have to read and write around one million records from
> an external data source to Sql Server2k every night.
> That's a lot of I/O.

Yes it is... =)

> I am using VB6 for this (takes
> hours). I am connecting to the external data source
with
> API's from its object library (as opposed to ODBC in
which
> case I would just use DTS to pull this data) and just
> looping.

I'm assuming this is a proprietary data format?

> I was thinking that with VB.Net I could read
> this data into a dataset in memory (currently have 4
gigs
> of mem - may upgrade to 16 gigs if they let me have
win2K
> Advanced Server - currently 2 processors 1 gig each).
My
> idea is that if I could just read the data into memory
(a
> dataset) it would be much faster than writing each
record
> (180 fields per record). Then I could just do a
> dataAdapter.Update or InsertInto to Sql Server once the
> data is all in local memory. Any thoughts?
>

Your still going to have a downturn on speed on the
update/insert. I'm not
sure if it will cache data on your local disk or not
depending on the amount
of memory. But you are still going to be subject to the
speed of your SQL
Server. So you can have it all queued up, but its not
necessarily going to
make it faster.

> While I'm at it, the records come from 4 different
> sources. I was thinking about using multi-threading and
> pull the data simultaneously. I am aware than Sql
> Server2k only supports 4 gigs a mem. But if I have more
> than 4 gigs of data can one VB.Net app manage datasets
in
> more than 4 gigs of memory? Once I fill my datasets, I
> would just do one InsertInto at a time. How is VB.Net
for
> multi-threading? Again, in VB6 I invoke 4 separate apps
> which simultaneously pull the data each night. They
write
> to 4 separate table in Sql Server2k. I would really
like
> to have all this in one app and read/write directly to
> memory. Is VB.Net a way to do this?
>

Many ways, I just don't know the benefit of it from what
your giving us.
Yeah your reads and loading will be fast, but is this
program going to run
on the SQL server itself? If your looking to reduce your
amount of network
traffic, yeah this will solve your problem, if your
looking to speed up
injection into your SQL server, not gonna do it... =(

You could use a more updated OLEDB driver for SQL Server,
which may shave
some time off because of optimized programming, but
again, don't know how
much of a gain you will actually get.

-CJ

> Thanks,
> Rich
.


--
Best Regards - OHM
one.handed.man{at}BTInternet{dot}com

Nov 20 '05 #9
Ive been ultra busy in a new job. Ive got some work creating Labs for a a
Major company dealing with some very new stuff, SmartDocuments, SmartTags,
MapPoint etc. All good stuff

Regards - OHM

CJ Taylor wrote:
I've spent 7 years coding Notes 4.5, 5.x, and 6, all I can can say
is that, there are advantages in certain design scanarios. Notes is
an easy place to code up an application quickly amd some
requirements lend themselves well to notes.


Thats why I made my second comment that I know it has its advantages
(sweet irony, I found a use today in a conversation.. I didn't bring
it up though. =))


However, having said that, Notes is dying out and will continue to
do so, there are so many better things around to take you away from
it.


Exactly...

Btw,
Where the hell have you been? I don't think I've seen a post from
you in ages!
Regards - OHM

CJ Taylor wrote:
"Rich" <an*******@discussions.microsoft.com> wrote in message
news:14*****************************@phx.gbl...
Thanks. I forgot to mention, yes, reducing network
traffice, or rather, get all the data to the local server
and close the connection would be a good thing to
achieve. The other data source is Lotus Notes R5. I have
to pull data from 4 separate Notes DB's (nsf's). Not my
design. I could take it or leave it.
I coded notes (4.5 and 5) for 2 years... I can honestly say that I
have never looked on a project and said "You know what would be a
good solution... Notes..."

I know it has its advantages, but coming from a developer standpoint
it kinda sucks and is INCREDIBLY slow...

So you day that it would be possible to fill the datasets
(disconnected datasets) this way? Thinking outloud, I
have 4 tables in Sql Server and would have 4 datasets in
my VB.net app.

Yeah, that would be fine. I'm sure the Notes API is slowing you
down as it is... So this will reduce your overall network traffic
(well, amount of time used for network traffic, you'll have a big
burst at the beginning and then nothing).

Then, with multi-threading, I would start
pulling data into each of these datasets simultaneously.
I have 4 gigs of mem right now. The deal is that the old
way I would just write the data directly to the Sql
Tables. Here I load the data to memory as fast as I can,
close my connection to Lotus Notes (using Domino Object
Library here) and then start pushing the data into Sql
Server from the datasets. Wouldn't this be better, more
efficient than having 4 com based apps reading and writing
data 250,000+ times a piece?

For network traffic, yes, it would be better. I don't think you
will see a huge performance increase on your SQL server, but I'm
sure you will see some (as you now are not requesting data from a
foreign source for each record).

Nonetheless... 180 fields, 1 million records, still going to take al
ittl etime to get in there. =)

-CJ

> -----Original Message-----
>
> ">
>> I have to read and write around one million records from
>> an external data source to Sql Server2k every night.
>> That's a lot of I/O.
>
> Yes it is... =)
>
>> I am using VB6 for this (takes
>> hours). I am connecting to the external data source
with
>> API's from its object library (as opposed to ODBC in
which
>> case I would just use DTS to pull this data) and just
>> looping.
>
> I'm assuming this is a proprietary data format?
>
>> I was thinking that with VB.Net I could read
>> this data into a dataset in memory (currently have 4
gigs
>> of mem - may upgrade to 16 gigs if they let me have
win2K
>> Advanced Server - currently 2 processors 1 gig each).
My
>> idea is that if I could just read the data into memory
(a
>> dataset) it would be much faster than writing each
record
>> (180 fields per record). Then I could just do a
>> dataAdapter.Update or InsertInto to Sql Server once the
>> data is all in local memory. Any thoughts?
>>
>
> Your still going to have a downturn on speed on the
update/insert. I'm not
> sure if it will cache data on your local disk or not
depending on the amount
> of memory. But you are still going to be subject to the
speed of your SQL
> Server. So you can have it all queued up, but its not
necessarily going to
> make it faster.
>
>> While I'm at it, the records come from 4 different
>> sources. I was thinking about using multi-threading and
>> pull the data simultaneously. I am aware than Sql
>> Server2k only supports 4 gigs a mem. But if I have more
>> than 4 gigs of data can one VB.Net app manage datasets
in
>> more than 4 gigs of memory? Once I fill my datasets, I
>> would just do one InsertInto at a time. How is VB.Net
for
>> multi-threading? Again, in VB6 I invoke 4 separate apps
>> which simultaneously pull the data each night. They
write
>> to 4 separate table in Sql Server2k. I would really
like
>> to have all this in one app and read/write directly to
>> memory. Is VB.Net a way to do this?
>>
>
> Many ways, I just don't know the benefit of it from what
your giving us.
> Yeah your reads and loading will be fast, but is this
program going to run
> on the SQL server itself? If your looking to reduce your
amount of network
> traffic, yeah this will solve your problem, if your
looking to speed up
> injection into your SQL server, not gonna do it... =(
>
> You could use a more updated OLEDB driver for SQL Server,
which may shave
> some time off because of optimized programming, but
again, don't know how
> much of a gain you will actually get.
>
> -CJ
>
>> Thanks,
>> Rich
>
>
> .


--
Best Regards - OHM
one.handed.man{at}BTInternet{dot}com


--
Best Regards - OHM
one.handed.man{at}BTInternet{dot}com
Nov 20 '05 #10
Ive been ultra busy in a new job. Ive got some work creating Labs for a a
Major company dealing with some very new stuff, SmartDocuments, SmartTags,
MapPoint etc. All good stuff
That is fantastic. I was wondering what happened to you, but sounds like
you get to deal with a lot more exciting stuff than we do sometimes. Good
luck with that and try to stop in every now and then. =)

-CJ

Regards - OHM

CJ Taylor wrote:
I've spent 7 years coding Notes 4.5, 5.x, and 6, all I can can say
is that, there are advantages in certain design scanarios. Notes is
an easy place to code up an application quickly amd some
requirements lend themselves well to notes.


Thats why I made my second comment that I know it has its advantages
(sweet irony, I found a use today in a conversation.. I didn't bring
it up though. =))


However, having said that, Notes is dying out and will continue to
do so, there are so many better things around to take you away from
it.


Exactly...

Btw,
Where the hell have you been? I don't think I've seen a post from
you in ages!
Regards - OHM

CJ Taylor wrote:
"Rich" <an*******@discussions.microsoft.com> wrote in message
news:14*****************************@phx.gbl...
> Thanks. I forgot to mention, yes, reducing network
> traffice, or rather, get all the data to the local server
> and close the connection would be a good thing to
> achieve. The other data source is Lotus Notes R5. I have
> to pull data from 4 separate Notes DB's (nsf's). Not my
> design. I could take it or leave it.
>

I coded notes (4.5 and 5) for 2 years... I can honestly say that I
have never looked on a project and said "You know what would be a
good solution... Notes..."

I know it has its advantages, but coming from a developer standpoint
it kinda sucks and is INCREDIBLY slow...

> So you day that it would be possible to fill the datasets
> (disconnected datasets) this way? Thinking outloud, I
> have 4 tables in Sql Server and would have 4 datasets in
> my VB.net app.

Yeah, that would be fine. I'm sure the Notes API is slowing you
down as it is... So this will reduce your overall network traffic
(well, amount of time used for network traffic, you'll have a big
burst at the beginning and then nothing).

> Then, with multi-threading, I would start
> pulling data into each of these datasets simultaneously.
> I have 4 gigs of mem right now. The deal is that the old
> way I would just write the data directly to the Sql
> Tables. Here I load the data to memory as fast as I can,
> close my connection to Lotus Notes (using Domino Object
> Library here) and then start pushing the data into Sql
> Server from the datasets. Wouldn't this be better, more
> efficient than having 4 com based apps reading and writing
> data 250,000+ times a piece?

For network traffic, yes, it would be better. I don't think you
will see a huge performance increase on your SQL server, but I'm
sure you will see some (as you now are not requesting data from a
foreign source for each record).

Nonetheless... 180 fields, 1 million records, still going to take al
ittl etime to get in there. =)

-CJ

>
>
>
>
>> -----Original Message-----
>>
>> ">
>>> I have to read and write around one million records from
>>> an external data source to Sql Server2k every night.
>>> That's a lot of I/O.
>>
>> Yes it is... =)
>>
>>> I am using VB6 for this (takes
>>> hours). I am connecting to the external data source
> with
>>> API's from its object library (as opposed to ODBC in
> which
>>> case I would just use DTS to pull this data) and just
>>> looping.
>>
>> I'm assuming this is a proprietary data format?
>>
>>> I was thinking that with VB.Net I could read
>>> this data into a dataset in memory (currently have 4
> gigs
>>> of mem - may upgrade to 16 gigs if they let me have
> win2K
>>> Advanced Server - currently 2 processors 1 gig each).
> My
>>> idea is that if I could just read the data into memory
> (a
>>> dataset) it would be much faster than writing each
> record
>>> (180 fields per record). Then I could just do a
>>> dataAdapter.Update or InsertInto to Sql Server once the
>>> data is all in local memory. Any thoughts?
>>>
>>
>> Your still going to have a downturn on speed on the
> update/insert. I'm not
>> sure if it will cache data on your local disk or not
> depending on the amount
>> of memory. But you are still going to be subject to the
> speed of your SQL
>> Server. So you can have it all queued up, but its not
> necessarily going to
>> make it faster.
>>
>>> While I'm at it, the records come from 4 different
>>> sources. I was thinking about using multi-threading and
>>> pull the data simultaneously. I am aware than Sql
>>> Server2k only supports 4 gigs a mem. But if I have more
>>> than 4 gigs of data can one VB.Net app manage datasets
> in
>>> more than 4 gigs of memory? Once I fill my datasets, I
>>> would just do one InsertInto at a time. How is VB.Net
> for
>>> multi-threading? Again, in VB6 I invoke 4 separate apps
>>> which simultaneously pull the data each night. They
> write
>>> to 4 separate table in Sql Server2k. I would really
> like
>>> to have all this in one app and read/write directly to
>>> memory. Is VB.Net a way to do this?
>>>
>>
>> Many ways, I just don't know the benefit of it from what
> your giving us.
>> Yeah your reads and loading will be fast, but is this
> program going to run
>> on the SQL server itself? If your looking to reduce your
> amount of network
>> traffic, yeah this will solve your problem, if your
> looking to speed up
>> injection into your SQL server, not gonna do it... =(
>>
>> You could use a more updated OLEDB driver for SQL Server,
> which may shave
>> some time off because of optimized programming, but
> again, don't know how
>> much of a gain you will actually get.
>>
>> -CJ
>>
>>> Thanks,
>>> Rich
>>
>>
>> .

--
Best Regards - OHM
one.handed.man{at}BTInternet{dot}com


--
Best Regards - OHM
one.handed.man{at}BTInternet{dot}com

Nov 20 '05 #11

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Andrew | last post by:
I'm a long time VB6/ADO and Java developer new to ADO.NET. I'm trying to decide on best practices and I'd appreciate any assistance. I have one specific question and another more general...
1
by: rawCoder | last post by:
Hi All Is there a way to Execute a query (possibly including two tables and joins) on a totally disconnected dataset. By 'Totally Disconnted' , I mean there is no DB and there is no DBConnection...
4
by: Steve Le Monnier | last post by:
The ADO.NET DataSet is idea for application development, especially if you need disconnected data. DataReader objects are great in the connected environment but are forward only. What do you do...
5
by: J | last post by:
Private Sub Page_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load 'Put user code to initialize the page here ip() If Not IsPostBack Then ...
2
by: Sky | last post by:
Hello, The books on C# I have read talk of using the DataSets in disconnected mode, and Adding/Removing/Updating records from it, and letting it handle all the updating on the Database in one...
5
by: Jason Shohet | last post by:
I have an asp.net app, and suddenly, I lose connection to the web server. I'd like a way -- if the update doesn't go thru, to 'cache' the changes to the dataset. Then the next time the user hits...
1
by: Navin Mishra | last post by:
Hi, I've multiple related data tables in a disconnected dataset which could be accessed by mutltiple threads to read and write data. The documentation says that dataset and datatable are thread...
0
by: garethdjames | last post by:
For scalability we wish to use disconnected DataSets and hold them in the application cache. This means that multiple concurrent users will be reading the data (its read only) The DataSet is...
2
by: chandu | last post by:
Hi Guys, i am beginer in the worls of Asp.net and C#.net I have one problem that I have stored my changes of data into dataset using disconnected dataset and after completion of all changes i...
2
by: Gabriel | last post by:
Hello, I'm looking for documentation with "Best Practice" for ASP.NET application In which case use Connected or Disconnected mode Typed dataset or not ? I didn'd find anything pertinent...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.