With VB.NET 2005, and a Windows Form, running on a dual CPU box, I need to
take a recordset (e.g. 100,000 records) and spawn a thread to handle an
internet XML transaction routine for each of the records. This is a nice
use of threading because those internet requests are going against 3rd party
servers that often have 1 second latency problems and so handling them with
multiple threads is the fastest way to get through all the records in the
recordset. I'd like to set a variable/constant that controls the number
of threads that are allowed to run simultaneously and have the code keep
spawning threads up to the number. Then, I can play with that number and
find out what # of threads works best. But I'm not the threading expert
and I don't know what new options I have with .NET 2.0 framework. I also
can't find any example that shows threads being repeatedly spawned up to a
certain number in an effort to work through a single recordset. Obviously
each thread need to communicate back to the main routine when done (fire an
event??) and that main routine would need to then take the next record and
throw it to another thread, if # threads is less than max # threads. Can
someone please help me with the pseudocode for this, or better yet, point me
to a real code example that does this? 10 1666
HK,
The change that you get a threading sample with a recordset is almost equal
to 0,001%
Maybe if you first try to make a datatable from it that your change will be
higher.
As well do you have to know which protocol you use. Standard will HTTP 1.1
not handle more than 2 connections at the same time (this is tweakable).
So beside the threading you have a lot of others problems to do. Making your
program first without threading will probably a much better approach. Adding
than some threading is mostly not the big challenge.
Just my thought,
Cor
I don't understand your thinking that I won't get much benefit. I see it
differently but maybe I'm wrong. Maybe my spec wasn't clear.
HTTP connection limits aside, I would go FROM this:
-----------------------------
Do
send XML request
get XML response 1 second later after internet latency
Until All 100,000 Records Exhaused
-----------------------------
TO this:
-----------------------------
Do
if thread available
spawn Task with this record
end if
Until All 100,000 Records Exhaused
Sub Task
send XML request
get XML response 1 second later after internet latency
communicate that thread is finished back to main loop
End Sub
-----------------------------
Most of the time spent on each task is waiting for internet latency. If I
have 10 worker bees (threads) all taking the next record in the recordset, I
would probably have 10 responses every second rather than 1 response every
second and I could go through my 100,000 records 10x faster. Bandwidth is
not a bottleneck. Seems like an ideal threading app, and the companies on
the other end have agreed. But tell me if I'm wrong, of course.
I just need better pseudocode that what I wrote above because I don't know
how to track what thread is available and how to then spawn off the next one
with the next record.
"Cor Ligthert [MVP]" <no************@planet.nl> wrote in message
news:O9**************@TK2MSFTNGP09.phx.gbl... HK,
The change that you get a threading sample with a recordset is almost
equal to 0,001%
Maybe if you first try to make a datatable from it that your change will
be higher.
As well do you have to know which protocol you use. Standard will HTTP 1.1 not handle more than 2 connections at the same time (this is tweakable).
So beside the threading you have a lot of others problems to do. Making
your program first without threading will probably a much better approach.
Adding than some threading is mostly not the big challenge.
Just my thought,
Cor
HK wrote: I don't understand your thinking that I won't get much benefit. I see it differently but maybe I'm wrong. Maybe my spec wasn't clear.
HTTP connection limits aside, I would go FROM this: ----------------------------- Do send XML request get XML response 1 second later after internet latency Until All 100,000 Records Exhaused -----------------------------
TO this:
----------------------------- Do if thread available spawn Task with this record end if Until All 100,000 Records Exhaused
Sub Task send XML request get XML response 1 second later after internet latency communicate that thread is finished back to main loop End Sub -----------------------------
Most of the time spent on each task is waiting for internet latency. If I have 10 worker bees (threads) all taking the next record in the recordset, I would probably have 10 responses every second rather than 1 response every second and I could go through my 100,000 records 10x faster. Bandwidth is not a bottleneck. Seems like an ideal threading app, and the companies on the other end have agreed. But tell me if I'm wrong, of course.
I just need better pseudocode that what I wrote above because I don't know how to track what thread is available and how to then spawn off the next one with the next record. "Cor Ligthert [MVP]" <no************@planet.nl> wrote in message news:O9**************@TK2MSFTNGP09.phx.gbl...
HK,
The change that you get a threading sample with a recordset is almost
equal
to 0,001%
Maybe if you first try to make a datatable from it that your change will
be
higher.
As well do you have to know which protocol you use. Standard will HTTP 1.1 not handle more than 2 connections at the same time (this is tweakable).
So beside the threading you have a lot of others problems to do. Making
your
program first without threading will probably a much better approach.
Adding
than some threading is mostly not the big challenge.
Just my thought,
Cor
Let me see if I understand what you are trying to do:
You have 100,000 records on the local side. For each record you have to
make a web request. Your thinking you can make 10 web requests at once
in order to speed up the process.
Well I understand why you think threads would speed this up, but you'd
be better sending more than one request at a time. I'm not sure if you
have access to the server side stuff, but you should design your xml to
handle multiple requests in one send. Then you send in 100,000 requests
and only have the latency once.
Chris
"Chris" <no@spam.com> wrote in message
news:O9**************@TK2MSFTNGP10.phx.gbl... HK wrote: I don't understand your thinking that I won't get much benefit. I see
it differently but maybe I'm wrong. Maybe my spec wasn't clear.
HTTP connection limits aside, I would go FROM this: ----------------------------- Do send XML request get XML response 1 second later after internet latency Until All 100,000 Records Exhaused -----------------------------
TO this:
----------------------------- Do if thread available spawn Task with this record end if Until All 100,000 Records Exhaused
Sub Task send XML request get XML response 1 second later after internet latency communicate that thread is finished back to main loop End Sub -----------------------------
Most of the time spent on each task is waiting for internet latency. If
I have 10 worker bees (threads) all taking the next record in the
recordset, I would probably have 10 responses every second rather than 1 response
every second and I could go through my 100,000 records 10x faster. Bandwidth
is not a bottleneck. Seems like an ideal threading app, and the companies
on the other end have agreed. But tell me if I'm wrong, of course.
I just need better pseudocode that what I wrote above because I don't
know how to track what thread is available and how to then spawn off the next
one with the next record. "Cor Ligthert [MVP]" <no************@planet.nl> wrote in message news:O9**************@TK2MSFTNGP09.phx.gbl...
HK,
The change that you get a threading sample with a recordset is almost
equal
to 0,001%
Maybe if you first try to make a datatable from it that your change will
be
higher.
As well do you have to know which protocol you use. Standard will HTTP
1.1not handle more than 2 connections at the same time (this is tweakable).
So beside the threading you have a lot of others problems to do. Making
your
program first without threading will probably a much better approach.
Adding
than some threading is mostly not the big challenge.
Just my thought,
Cor
Let me see if I understand what you are trying to do:
You have 100,000 records on the local side. For each record you have to make a web request. Your thinking you can make 10 web requests at once in order to speed up the process.
Well I understand why you think threads would speed this up, but you'd be better sending more than one request at a time. I'm not sure if you have access to the server side stuff, but you should design your xml to handle multiple requests in one send. Then you send in 100,000 requests and only have the latency once.
Chris
The companies on the other side don't allow that. I don't control their
servers and they have already told me that other customers get around this
with threading and send multiple requests simultaneously.
So yes, I do plan to send 10 web requests simultaneously, each in its own
thread.
Can anyone help with code or pseudocode? Again, the part that troubles me
is communicating that the thread is done (I think that's best served by
raising an event) and then having some loop of code that says "do I have any
free threads? yes? ok, you take this record and you take that one. now
I'm at 10 active threads and need to wait for another few to come free".
That looping sounds a bit challenging but surely this concept of
"maintaining 10 active threads until done" has been done many times before
and thus I'm posting hoping for some code or pseudocode that does that part.
I do have it working in VB6 and need to do a project conversion to .NET 2.0.
That is why I keep wanting someone here to help with the pseudocode, or real
code, for that piece that gives me trouble...
Again, the part that troubles me is communicating that the thread is done (I
think that's best served by raising an event) and then having some loop of
code that says "do I have any free threads? yes? ok, you take this record
and you take that one. now
I'm at 10 active threads and need to wait for another few to come free".
That looping sounds a bit challenging but surely this concept of
"maintaining 10 active threads until done" has been done many times before
and thus I'm posting hoping for some code or pseudocode that does that part.
"Cor Ligthert [MVP]" <no************@planet.nl> wrote in message
news:%2****************@TK2MSFTNGP09.phx.gbl... HK,
I did not say it was not a good idea, however I think you first have to
get it working withouth the threads. Adding the thread is than mostly not the job.
This is a good purpose for a queu by the way, however try it first
withouth that and try to use a datatable instead of a recordset, than you don't
have to use ADO.
Cor
"HK" <re**************@notreal.com> schreef in bericht news:XM*****************@tornado.socal.rr.com...I don't understand your thinking that I won't get much benefit. I see it differently but maybe I'm wrong. Maybe my spec wasn't clear.
HTTP connection limits aside, I would go FROM this: ----------------------------- Do send XML request get XML response 1 second later after internet latency Until All 100,000 Records Exhaused -----------------------------
TO this:
----------------------------- Do if thread available spawn Task with this record end if Until All 100,000 Records Exhaused
Sub Task send XML request get XML response 1 second later after internet latency communicate that thread is finished back to main loop End Sub -----------------------------
Most of the time spent on each task is waiting for internet latency. If
I have 10 worker bees (threads) all taking the next record in the
recordset, I would probably have 10 responses every second rather than 1 response
every second and I could go through my 100,000 records 10x faster. Bandwidth
is not a bottleneck. Seems like an ideal threading app, and the companies
on the other end have agreed. But tell me if I'm wrong, of course.
I just need better pseudocode that what I wrote above because I don't
know how to track what thread is available and how to then spawn off the next one with the next record. "Cor Ligthert [MVP]" <no************@planet.nl> wrote in message news:O9**************@TK2MSFTNGP09.phx.gbl... HK,
The change that you get a threading sample with a recordset is almost equal to 0,001%
Maybe if you first try to make a datatable from it that your change
will be higher.
As well do you have to know which protocol you use. Standard will HTTP 1.1 not handle more than 2 connections at the same time (this is
tweakable). So beside the threading you have a lot of others problems to do. Making your program first without threading will probably a much better approach. Adding than some threading is mostly not the big challenge.
Just my thought,
Cor
HK,
I wrote already have a look at "queu" that is made for this kind of
problems.
Cor
HK,
You ask for to much code. Than you real have to hire somebody, it is not
just a snippet what you ask.
While as I wrote you real have no change for even parts for that with a
recordset. The recordset is almost not used anymore in Net.
Cor
HK,
I did not say it was not a good idea, however I think you first have to get
it working withouth the threads. Adding the thread is than mostly not the
job.
This is a good purpose for a queu by the way, however try it first withouth
that and try to use a datatable instead of a recordset, than you don't have
to use ADO.
Cor
"HK" <re**************@notreal.com> schreef in bericht
news:XM*****************@tornado.socal.rr.com... I don't understand your thinking that I won't get much benefit. I see it differently but maybe I'm wrong. Maybe my spec wasn't clear.
HTTP connection limits aside, I would go FROM this: ----------------------------- Do send XML request get XML response 1 second later after internet latency Until All 100,000 Records Exhaused -----------------------------
TO this:
----------------------------- Do if thread available spawn Task with this record end if Until All 100,000 Records Exhaused
Sub Task send XML request get XML response 1 second later after internet latency communicate that thread is finished back to main loop End Sub -----------------------------
Most of the time spent on each task is waiting for internet latency. If I have 10 worker bees (threads) all taking the next record in the recordset, I would probably have 10 responses every second rather than 1 response every second and I could go through my 100,000 records 10x faster. Bandwidth is not a bottleneck. Seems like an ideal threading app, and the companies on the other end have agreed. But tell me if I'm wrong, of course.
I just need better pseudocode that what I wrote above because I don't know how to track what thread is available and how to then spawn off the next one with the next record. "Cor Ligthert [MVP]" <no************@planet.nl> wrote in message news:O9**************@TK2MSFTNGP09.phx.gbl... HK,
The change that you get a threading sample with a recordset is almost equal to 0,001%
Maybe if you first try to make a datatable from it that your change will be higher.
As well do you have to know which protocol you use. Standard will HTTP 1.1 not handle more than 2 connections at the same time (this is tweakable).
So beside the threading you have a lot of others problems to do. Making your program first without threading will probably a much better approach. Adding than some threading is mostly not the big challenge.
Just my thought,
Cor
I asked for a few lines of pseudocode or for someone to point me to a real
code sample. Neither request is "too much" IMO.
The recordset concept is the lesser concern here but thanks for the tip.
"Cor Ligthert [MVP]" <no************@planet.nl> wrote in message
news:%2****************@TK2MSFTNGP14.phx.gbl... HK,
You ask for to much code. Than you real have to hire somebody, it is not just a snippet what you ask.
While as I wrote you real have no change for even parts for that with a recordset. The recordset is almost not used anymore in Net.
Cor
Hi,
I would recommend avoiding worker threads for this kind of work - you
can use the asynchronous methods (BeginXxx/EndXxx) available for most
communication classes (e.g. HttpWebRequest.BeginGetResponse, etc.)
You can write it like (pseudocode)
// start at most 10 processing threads
function doRequests()
{
lock (requests)
{
for (i = 0; i < 10 && requests.Available; i++)
beginProcessRequest(requests.Dequeue(), onRequestComplete);
}
... wait for the requests to complete
}
function beginProcessRequest(request, callback)
{
... do something that processes your request asynchronously and calls }
function onRequestComplete()
{
lock (requests)
{
... do whatever you need to do
if (requests.Available)
beginProcessRequest(requests.Dequeue, onRequestComplete);
}
}
If you insist on the thread approach, you can:
function doRequests()
{
for (i = 0; i < 10; i++)
spawnWorker(worker);
... wait for the requests to complete
}
function worker()
{
for (;;)
{
lock (requests)
{
if (!requests.Available)
return;
request = requests.Dequeue();
}
... process your request
}
}
If you have any more questions, don't hesitate to ask :)
Stefan
HK wrote: I asked for a few lines of pseudocode or for someone to point me to a real code sample. Neither request is "too much" IMO.
The recordset concept is the lesser concern here but thanks for the tip.
"Cor Ligthert [MVP]" <no************@planet.nl> wrote in message news:%2****************@TK2MSFTNGP14.phx.gbl...
HK,
You ask for to much code. Than you real have to hire somebody, it is not just a snippet what you ask.
While as I wrote you real have no change for even parts for that with a recordset. The recordset is almost not used anymore in Net.
Cor
This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: max khesin |
last post by:
I am working on a windoze service (deamon) that needs to spawn up to
NumCPU heavyweight processes on SMP and wait for results, which have to
be logged. So I have to do something like (total...
|
by: Ronan Viernes |
last post by:
Hi,
I have created a python script (see below) to count the maximum number
of threads per process (by starting new threads continuously until it
breaks).
######
#testThread.py
import...
|
by: chahnaz.ourzikene |
last post by:
Hi all,
This is the first i post in this newsgroup, i hope my english is not too
bad...
Let's get straight to the point ! I have a little probleme using threads in
my little training example :...
|
by: Leonardo Hyppolito |
last post by:
Hello,
I am trying to write a multithread program that simulates producers and
consumers. My program can have many producers and many consumers (each in a
separate thread). It has a storage...
|
by: Sinan Nalkaya |
last post by:
hello,
i need a function like that,
wait 5 seconds:
(during wait) do the function
but function waits for keyboard input so if you dont enter any it waits
forever.
i tried time.sleep() but when...
|
by: cj |
last post by:
I have to take all records in a SQL db that have an empty val_code field
and process and update the records with a code number in the val_code field.
The processing is done by sending select info...
|
by: Abubakar |
last post by:
Hi,
I created 2 functions which run in seperate threads. what they do simply is
that thread1 starts and does something while the thread 2 waits for it. Then
after thread1 is done, it resumes the...
|
by: bean330 |
last post by:
Hey, I'm somewhat new to C# and I need a little help, please!
I'm selecting a bunch of records, setting properties on a COM
executable and then calling a method on that executable to run. I...
|
by: manontheedge |
last post by:
I'm trying to learn multithreading in C. I've never done it before, and I've spent several hours reading about it, and looking at examples. I found this example and have been messing with it (I've...
|
by: CloudSolutions |
last post by:
Introduction:
For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
|
by: Faith0G |
last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome former...
|
by: ryjfgjl |
last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
| |