473,408 Members | 1,786 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,408 software developers and data experts.

C#.NET Bulk Processing few millions records

Hi All,

I need to do bulk processing several tables that contain few millions records to generate report.

there is 4 tables say A, B, C, D

A is the transaction table, I need to retrieve data from here where the date is between 1/1/200x to 31/12/200x. but the date is not indexed. about 3million rows to process
Table B is customer details. 1.4 million rows.
Table C is transaction details. 7 millions rows.
Table D is product details. 10k rows.

What I need to do is to create pivot report based on the criteria.
Simply joining the 4 tables is very very slow. I'm thinking to create store procedure in my SQL 2000 and process everything there using temporary table, but I'm afraid I will slow down the production environment.

Any good idea how to do it?
Aug 23 '07 #1
0 1180

Sign in to post your reply or Sign up for a free account.

Similar topics

7
by: iqbal | last post by:
Hi all, We have an application through which we are bulk inserting rows into a view. The definition of the view is such that it selects columns from a table on a remote server. I have added the...
9
by: adi | last post by:
Hi all, Hope there is a quick fix for this: I am inserting data from one table to another on the same DB. The insert is pretty simple as in: insert into datatable(field1, field2, field3)...
16
by: Philip Boonzaaier | last post by:
I want to be able to generate SQL statements that will go through a list of data, effectively row by row, enquire on the database if this exists in the selected table- If it exists, then the colums...
3
by: moonriver | last post by:
Right now I develop an application to retrieve over 30,000 records from a binary file and then load them into a SQL Server DB. So far I load those records one by one, but the performance is very...
1
by: RJN | last post by:
Hi We are writing a data processing application wherein we compare data in 2 sets of tables say TableA and TableB and if the data in TableB is different than TableA, then we do an...
2
by: Zarrin | last post by:
Hello, I read several articles of newsgroup about the bulk delete, and I found one way is to: -create a temporary table with all constraints of original table -insert rows to be retained into...
0
debasisdas
by: debasisdas | last post by:
We can fetch from a cursor into one or more collections: DECLARE TYPE NameList IS TABLE OF employees.last_name%TYPE; TYPE SalList IS TABLE OF employees.salary%TYPE; CURSOR c1 IS SELECT...
4
by: shreyask | last post by:
I have been working on doing bulk updates and inserts through sqlapi++ using a bulk udpate extention library, developed, a long time ago, internally. The database is oracle. The bulk inserts/updates...
0
by: bob laughland | last post by:
Hi All, I am using a combination of LINQ to SQL and bulk insert. In the process of performing 'one unit of work' I will be doing things like reading, and deleting records using LINQ to SQL and...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.