Im fairly new to programming so you will have to excuse my lack of knowledge and experience.
I am trying to find a way of reading from a database using threads. That is a database with say 1000000 rows and i want to be able to use say 5-10 threads to each read a set amount of rows within the database.
Although this may not be practical? Would anybody be able to advise as to the best way to go about this or just the theory behind it, or perhaps even a little code..............
4  2325 
Just subscribing, to see what is said. I am doing the EXACT same thing right now. lol
Cyberdaemon
right now my problem is:
A first chance exception of type 'System.InvalidOperationException' occurred in System.Data.dll
I can't find where the exception is being thrown. I have try statements all over my code and they return nothing...makes me sad. If any one has any ideas where to start looking. Any and all help would be appriciated! Thanks in advance.
Cyberdaemon
right now my problem is:
A first chance exception of type 'System.InvalidOperationException' occurred in System.Data.dll
I can't find where the exception is being thrown. I have try statements all over my code and they return nothing...makes me sad. If any one has any ideas where to start looking. Any and all help would be appriciated! Thanks in advance.
Cyberdaemon
How big is your code and could you post it here?
How big is your code and could you post it here?
- Private Shared Sub btnPullData_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnPullData.Click
-
''__IsLoaded()
-
-
Dim OnBaseAdapt As New SqlClient.SqlDataAdapter("", My.Settings.connectionstring1)
-
Dim _Continue As Boolean
-
-
Const BatchSize As Integer = 3500
-
Dim tmpString As String = ""
-
-
' Create the ID/P data table
-
Debug.Print("BATCHSIZE = " & BatchSize.ToString)
-
Debug.Print("Filling Onbase information...")
-
OnBaseAdapt.SelectCommand.CommandText = "<select logic>"
-
OnBaseAdapt.SelectCommand.CommandTimeout = 0
-
OnBaseAdapt.SelectCommand.CommandType = CommandType.Text
-
OnBaseAdapt.Fill(tblOnBase)
-
-
' Create the adapter for the tracking database
-
Dim TKRAdapt As New SqlClient.SqlDataAdapter("<select logic>", My.Settings.ConnectionString2)
-
TKRAdapt.SelectCommand.CommandTimeout = 0
-
TKRAdapt.SelectCommand.CommandType = CommandType.Text
-
-
' Create the database reader
-
Dim TKRReader As SqlClient.SqlDataReader
-
Dim tmpBatch As DataTable
-
-
' Create the batches, and let process on individual threads
-
TKRAdapt.SelectCommand.Connection.Open()
-
TKRReader = TKRAdapt.SelectCommand.ExecuteReader '(CommandBehavior.CloseConnection)
-
Debug.Print("Starting the Tracking Process")
-
While TKRReader.Read()
-
-
' Create a new batch table
-
tmpBatch = New DataTable
-
''''Debug.Print("Expanding Batch Table...")
-
For LoopVarC As Integer = 0 To 4
-
tmpBatch.Columns.Add()
-
Next
-
' Fill batch with the information from the tkr database at BATCHSIZE records at a time
-
''''Debug.Print("Creating Batch...")
-
-
For LoopVarI As Integer = 0 To BatchSize - 1
-
_Continue = TKRReader.HasRows
-
If _Continue = False Then
-
Exit While
-
End If
-
-
TKRReader.Read()
-
-
tmpBatch.Rows.Add()
-
For LoopVarL As Integer = 0 To 4
-
Try
-
tmpBatch.Rows(LoopVarI).Item(LoopVarL) = TKRReader.Item(LoopVarL)
-
Catch ex As InvalidOperationException
-
Exit For
-
End Try
-
Next
-
-
Next
-
-
''''Debug.Print("Batch Creation Complete! Starting Thread...")
-
-
' Pass to the processing thread and start again
-
Dim ThdAction As New Thread(AddressOf ThreadAction)
-
ThdAction.IsBackground = True
-
ThdAction.Start(tmpBatch)
-
Thread.Sleep(50)
-
''''Debug.Print("Thread Started, Create Next Batch")
-
End While
-
''''Debug.Print("All possible Batches Created, wait for processing to complete")
-
' close conneciton
-
TKRReader.Close()
-
TKRAdapt.SelectCommand.Connection.Close()
-
End Sub
-
-
Private Shared Sub ThreadAction(ByVal batch As Object)
-
Debug.Print(String.Format("{0} thread started", Thread.CurrentThread.ManagedThreadId.ToString))
-
' Create the Logic object
-
Dim CVobj As New CVLogic(CType(batch, DataTable))
-
Dim ProcessLck As New Object
-
Dim UpdateLck As New Object
-
-
Dim ST As Date = Date.Now
-
' Process the Data
-
SyncLock ProcessLck
-
Try
-
Debug.Print(String.Format("{0} : {1}", Thread.CurrentThread.ManagedThreadId.ToString, "Processing the Data @ Thread Action calling logic object"))
-
CVobj.Process()
-
Catch ex As InvalidOperationException
-
Debug.Print("cannot process!!")
-
End Try
-
End SyncLock
-
Dim TS As TimeSpan = Date.Now - ST
-
Debug.Print("{0} took {1} to complete PROCESS", Thread.CurrentThread.ManagedThreadId.ToString, TS.ToString)
-
Thread.Sleep(50)
-
-
' Update the data
-
SyncLock UpdateLck
-
''''''Debug.Print(String.Format("{0} : {1}", Thread.CurrentThread.ManagedThreadId.ToString, "Updating the Data @ Thread Action calling logic object"))
-
CVobj.Update()
-
End SyncLock
-
Thread.Sleep(50)
-
Thread.CurrentThread.Abort()
-
End Sub
-
-
-
End Class
-
-
Public Class CVLogic
-
Public tbl As DataTable ' Holds the CV read batch
-
-
Public Sub New(ByVal batch As DataTable) ' Initialize the internal datatable
-
tbl = New DataTable
-
tbl = batch
-
End Sub
-
-
' Process the Batch
-
Public Sub Process()
-
Dim BatchProcess As New Object
-
Dim TKR, OnB As DataTableReader
-
Dim sTRAKitemname As String = ""
-
-
' Generate the table reader
-
TKR = tbl.CreateDataReader
-
-
' SEARCH the tables
-
SyncLock BatchProcess
-
Debug.Print(String.Format("{0}: Starting Process", Thread.CurrentThread.ManagedThreadId.ToString))
-
While TKR.Read
-
Try
-
OnB = tblOnBase.CreateDataReader
-
Catch ex As InvalidOperationException
-
Debug.Print(String.Format("{0} : Cannot Get to OnBase data!", Thread.CurrentThread.ManagedThreadId.ToString))
-
Exit While
-
End Try
-
-
sTRAKitemname = String.Format("{0} {1} - {2} - {3}", TKR("column2").ToString, TKR("column3").ToString, ReformatDate(TKR("column4"), True).Trim, TKR("column1").ToString)
-
While OnB.Read
-
''''Debug.Print(String.Format("{0} <PROCESS> : Reading onbase", Thread.CurrentThread.ManagedThreadId.ToString))
-
If sTRAKitemname.Trim = OnB("itemname").ToString.Trim Then
-
Try
-
If Val(TKR("column5").ToString) = Val(OnB("numberpages").ToString) Then
-
''''Debug.Print(String.Format("{0} : LOADED", Thread.CurrentThread.ManagedThreadId.ToString))
-
Else
-
''''Debug.Print(String.Format("{0} : PARTIAL", Thread.CurrentThread.ManagedThreadId.ToString))
-
End If
-
Thread.Sleep(10)
-
Exit While
-
Catch ex As InvalidOperationException
-
Debug.Print(String.Format("{0} : Cannot Get to OnBase data!", Thread.CurrentThread.ManagedThreadId.ToString))
-
Exit While
-
End Try
-
End If
-
End While
-
End While
-
End SyncLock
-
-
End Sub
Good luck, it is a mess right now, I plan on cleaning it as well as the rest of the program once functional. My threading knowledge is, limited. Any suggestions would be helpful. Some information omitted due to massive length thus causing clutter. i.e. Selection logic for SQL. If you have ANY questions do not hesitate. Thank you in advance and may the force be with you!
Cyberdaemon
Sign in to post your reply or Sign up for a free account.
Similar topics
by: Ronan Viernes |
last post by:
Hi,
I have created a python script (see below) to count the maximum number
of threads per process (by starting new threads continuously until it
breaks).
######
#testThread.py
import...
|
by: Al Tobey |
last post by:
I was building perl 5.8.2 on RedHat Enterprise Linux 3.0 (AS) today and
noticed that it included in it's ccflags "-DTHREADS_HAVE_PIDS." I am
building with -Dusethreads. With newer Linux...
|
by: m |
last post by:
Hello,
I have an application that processes thousands of files each day. The
filenames and various related file information is retrieved, related
filenames are associate and placed in a linked...
|
by: Kovan Akrei |
last post by:
Hi,
I would like to know how to reuse an object of a thread (if it is possible)
in Csharp? I have the following program:
using System;
using System.Threading;
using System.Collections;
...
|
by: bygandhi |
last post by:
Hi -
I am writing a service which will check a process and its threads for
their state ( alive or dead ).
The process has 5 .net managed threads created using thread.start and
each have been...
|
by: [Yosi] |
last post by:
I would like to know how threads behavior in .NET .
When an application create 4 threads for example start all of them, the OS
task manager will execute all 4 thread in deterministic order manes,...
|
by: RahimAsif |
last post by:
Hi guys,
I would like some advice on thread programming using C#.
I am writing an application that communicates with a panel over
ethernet, collects data and writes it to a file. The way the...
|
by: mjheitland |
last post by:
Hi,
I like to know how many threads are used by a Threading.Timer object.
When I create a Threading.Timer object calling a short running method
every 5 seconds I expected to have one additional...
|
by: Darian |
last post by:
Is there a way to find all the thread names that are running in a
project?
For example, if I have 5 threads T1, T2, T3, T4, T5...and T2, T4, and
T5 are running...I want to be able to know that...
|
by: tdahsu |
last post by:
All,
I'd appreciate any help. I've got a list of files in a directory, and
I'd like to iterate through that list and process each one. Rather
than do that serially, I was thinking I should...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
| |
