Hi,
I have two databases D1 with 6 million records and D2 with 95 thousand
records. I need to check Common records from these two databases based on
UserID and need to insert into other database D3 and also need to create XML
files.
For this i followed below approach.
I have used two threads.
-->one thread to Pick the users from D1, filter out against D2 and will be
inserting into D3.
In this thread iam fetching 5000 records everytime and for each user
Calling an event using delegate. in this event iam filtering against D2
database and inserting user into D3 database as follows.
Public void Run()
{
While(!Stop)
{
--Here fetching 5000 records ranging from 240000 to 290000 against D1
database (Querying D1 database)
-- calling a event using delegate for each Record.
-- Filtering each record against D2 database (Querying D2 database)
-- If exists Inserting into D3 database or if not exists then
returning loop to process the next record.
}
}
-->other thread fetches all the records from D3 database and writes to Xml
files.
-- this thread fetches the records say 1000(after filtering out) at one
time (Querying D3 database)
-- here also i have created delegate which calls an event for each record.
-- for each user the data will be written to xml file
Now i have an issue with this, its killing the perfomance. can anyone please
help me on this as how to improve performance.
Iam using .Net 1.1 for development and SQL 2000 as backend.
Thanks,
-Veera 4 1542
"Veerabhadr aiah L M" <Veerabhadrai ah L M @discussions.mi crosoft.comwrot e
in message news:1C******** *************** ***********@mic rosoft.com...
Hi,
I have two databases D1 with 6 million records and D2 with 95 thousand
records. I need to check Common records from these two databases based on
UserID and need to insert into other database D3 and also need to create
XML
files.
For this i followed below approach.
I have used two threads.
-->one thread to Pick the users from D1, filter out against D2 and will be
inserting into D3.
In this thread iam fetching 5000 records everytime and for each user
Calling an event using delegate. in this event iam filtering against D2
database and inserting user into D3 database as follows.
Public void Run()
{
While(!Stop)
{
--Here fetching 5000 records ranging from 240000 to 290000 against
D1
database (Querying D1 database)
-- calling a event using delegate for each Record.
-- Filtering each record against D2 database (Querying D2 database)
-- If exists Inserting into D3 database or if not exists then
returning loop to process the next record.
}
}
-->other thread fetches all the records from D3 database and writes to Xml
files.
-- this thread fetches the records say 1000(after filtering out) at one
time (Querying D3 database)
-- here also i have created delegate which calls an event for each
record.
-- for each user the data will be written to xml file
Now i have an issue with this, its killing the perfomance. can anyone
please
help me on this as how to improve performance.
Iam using .Net 1.1 for development and SQL 2000 as backend.
Thanks,
-Veera
What kind of machine will this be running on? I would try to load all 95K
records from D2 into a hash table in memory so I never had to query D2
(that's going to be your bottleneck right there).
Also, is there a specific reason why you must save to D3, then re-load from
D3 and generate XML? Why not just generate the XML file at the same time you
insert into D3?
Are both DBs in the same server?
If so y ou can do it directly in the server and just save A LOT of
processing.
"Veerabhadr aiah L M" <Veerabhadrai ah L M @discussions.mi crosoft.comwrot e
in message news:1C******** *************** ***********@mic rosoft.com...
Hi,
I have two databases D1 with 6 million records and D2 with 95 thousand
records. I need to check Common records from these two databases based on
UserID and need to insert into other database D3 and also need to create
XML
files.
For this i followed below approach.
I have used two threads.
-->one thread to Pick the users from D1, filter out against D2 and will be
inserting into D3.
In this thread iam fetching 5000 records everytime and for each user
Calling an event using delegate. in this event iam filtering against D2
database and inserting user into D3 database as follows.
Public void Run()
{
While(!Stop)
{
--Here fetching 5000 records ranging from 240000 to 290000 against
D1
database (Querying D1 database)
-- calling a event using delegate for each Record.
-- Filtering each record against D2 database (Querying D2 database)
-- If exists Inserting into D3 database or if not exists then
returning loop to process the next record.
}
}
-->other thread fetches all the records from D3 database and writes to Xml
files.
-- this thread fetches the records say 1000(after filtering out) at one
time (Querying D3 database)
-- here also i have created delegate which calls an event for each
record.
-- for each user the data will be written to xml file
Now i have an issue with this, its killing the perfomance. can anyone
please
help me on this as how to improve performance.
Iam using .Net 1.1 for development and SQL 2000 as backend.
Thanks,
-Veera
--
Ignacio Machin http://www.laceupsolutions.com
Mobile & warehouse Solutions.
What kind of machine will this be running on? I would try to load all 95K
records from D2 into a hash table in memory so I never had to query D2
(that's going to be your bottleneck right there).
I would not do that. if both DBs are in the same server I would do the
entire query in the DB.
"Ignacio Machin ( .NET/ C# MVP )" <machin TA laceupsolutions .comwrote in
message news:eS******** ******@TK2MSFTN GP05.phx.gbl...
>
>What kind of machine will this be running on? I would try to load all 95K records from D2 into a hash table in memory so I never had to query D2 (that's going to be your bottleneck right there).
I would not do that. if both DBs are in the same server I would do the
entire query in the DB.
I guess I assumed he had some reason for not doing something that obvious in
the first place. Maybe that was a poor assumption on my part.
Assuming there is some reason to do the processing on the client, the OP
might also look into the SqlBulkCopy class for inserting a large number of
rows into D3 as well. If all he is doing is moving data, then yeah, using
Stored Procs or SSIS is definitely the way to go. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: grutta |
last post by:
I am writing a windows service that will recieve
notification when a USB Device is insterted into the
machine. I have used the RegisterDeviceNotification and
the RegisterServiceCtrlHandlerEx with a handler. The
handler portion seems to fail indicating that my
parameters are invalid. I am getting an error code 126
when i try to register the handler and 28 when i register
for event notification. Any ideas as to what the deal is?
I am...
|
by: news.onet.pl |
last post by:
I've launched some perfomance test for some program measuring
number of operations, net messages processed per second, etc.
Why is my point, is the question how Java initial and maximal
heap is connected with the swap space of Operating System.
Those Java utilize both memory and swap space for purpose
of its heap?
TIA,
|
by: James T. |
last post by:
Hello!
Let's say I have 2 functions, both return data from the database. Now I
would like to compare perfomance of these two functions. I would like to
know how long it takes to execute these functions.
How can I do that?
Thank you!
James
|
by: Al |
last post by:
Hi,
I've written a little app to process OLAP partitions on an Analysis Services
2000 database. I've been trying to multi-thread it to improve the overall
through-put. I've managed to multi-thread it, and I'm sure that the Thread
only block each other in one very small critical section, but my through-put
is worse by 30% when I use 2 "worker" threads. (This is on a 2 physical CPU
box).
If split the work load between 2 instances of the...
|
by: jambalapamba |
last post by:
Hi
I am removing node from a document depending on the style property and this is taking 7 seconds for checking complete document is there any ideas how can i improve the perfomance. Here is my part of code
foreach (IHTMLElement tempElement in document.all)
{
if (tempElement.style.visibility == "hidden" )
{
IHTMLDOMNode node = tempElement as...
| |
by: damodharan |
last post by:
Hello and thanks in advance.
I have two table in DB2 (ver 8). table details are
table one(Cust_det) has two fields (Cust_cin and cust_name).
table two(cust_add_det) has cust_cin and cust_address fields.
To join above two table which sql is better on perfomance point of view?
and how conditional where clause works?
|
by: MickJ |
last post by:
Hi,
I would like to write High perfomance server using C#. It would be desirable to hear offers and advices on this subject.
|
by: Alexander Adam |
last post by:
Hi!
I've got a pretty complex data structure that keeps instances of
structs allocated with the new operator.
Now my issue is that those structures are required to be allocated /
deallocated pretty often resulting
in a huge perfomance issue. Usually what I am doing is to create
arrays and initialize their sizes a few
times bigger than originally required to avoid another allocation on
the next run. Then I do simply reset them
|
by: ferozanna |
last post by:
I am using Asp.net 1.3 with C#
My application used by call center people
My applicaton is a three tier arch I have create data layer as class ibrary which goint to talke Ssqlserver 205 db
At time 2000 to 3000 user are using
How can i improve my application perfomance, Give me tips
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |