Threading...I thought this was supposed to be faster than doing things sequentially...

[Cliff]

I've got a number of SNMP devices scattered around the globe that I
want to get some information off..

I've got a couple of classes whcih get a quite complex table together
from SQL and SNMP devices and return a DataTable Object.

but as the devices are all in totally different locations (we have
about 20) and the main time lag is due to latency I want to go and get
the data simultaniuously from all the resources.

Each individual table get takes between .25 and 2 seconds so
sequentially, the process takes about 8-10 seconds for 10 devices.

Having converted it to be multi threaded, it now takes 10-15 seconds.
(even when compiled as release code)

I've added some timing code to make sure it is running, but it seems
to take ages to get the threads started. I need to get this down to
2/3 seconds or it won't work.

Is there any way to speed up actually creating the threads, as that
seems to be taking 3 seconds alone!

Any help appreciated!

Cliff Dabbs

Here's the code

ArrayList arr = new ArrayList();

DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting To Launch Threads");

foreach (ASAVPN avpn in asavpns)
{
Debug.WriteLine("Launching Thread " +
avpn.DeviceName);
Thread th = new Thread(GetTable);
arr.Add(th);
th.Start(avpn);
}

foreach (VPN3000 kvpn in v3kvpns)
{
Debug.WriteLine("Launching Thread " +
kvpn.DeviceName);
Thread th = new Thread(GetTable);
arr.Add(th);
th.Start(kvpn);
}

TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Launch of Threads " + ts.ToString());
//Wait until the threads end;
bool moveon = true;
do
{
Thread.Sleep(100);
moveon = true;
foreach (Thread th in arr)
if (th.ThreadState ==
System.Threading.ThreadState.Running)
moveon = false;
}
while (moveon == false);

DataView dv = new DataView(dt1);

this.dataGridView1.DataSource = dv;
}

public void GetTable(object data)
{
lock (data)
{
DataTable dt;
string devicename = ((SNMPDevice)(data)).DeviceName;
DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting Thread Work: " +
devicename);
if (data.GetType() == typeof(ASAVPN))
dt = ((ASAVPN)(data)).SessionTable;
else
dt = ((VPN3000)(data)).SessionTable;

TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Ending Thread Work " + devicename + "
" + ts.ToString());

lock (this.ds.Tables["vpninfo"])
{
this.ds.Tables["vpninfo"].Merge(dt, true,
MissingSchemaAction.Add);
}
}
}

[Brian Gideon]

On Mar 6, 11:56 am, "Cliff" <[e-mail]:

I've got a number of SNMP devices scattered around the globe that I
want to get some information off..

I've got a couple of classes whcih get a quite complex table together
from SQL and SNMP devices and return a DataTable Object.

but as the devices are all in totally different locations (we have
about 20) and the main time lag is due to latency I want to go and get
the data simultaniuously from all the resources.

Each individual table get takes between .25 and 2 seconds so
sequentially, the process takes about 8-10 seconds for 10 devices.

Having converted it to be multi threaded, it now takes 10-15 seconds.
(even when compiled as release code)

I've added some timing code to make sure it is running, but it seems
to take ages to get the threads started. I need to get this down to
2/3 seconds or it won't work.

Is there any way to speed up actually creating the threads, as that
seems to be taking 3 seconds alone!

Any help appreciated!

Cliff Dabbs

Here's the code

ArrayList arr = new ArrayList();

DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting To Launch Threads");

foreach (ASAVPN avpn in asavpns)
{
Debug.WriteLine("Launching Thread " +
avpn.DeviceName);
Thread th = new Thread(GetTable);
arr.Add(th);
th.Start(avpn);
}

foreach (VPN3000 kvpn in v3kvpns)
{
Debug.WriteLine("Launching Thread " +
kvpn.DeviceName);
Thread th = new Thread(GetTable);
arr.Add(th);
th.Start(kvpn);
}

TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Launch of Threads " + ts.ToString());

//Wait until the threads end;
bool moveon = true;
do
{
Thread.Sleep(100);
moveon = true;
foreach (Thread th in arr)
if (th.ThreadState ==
System.Threading.ThreadState.Running)
moveon = false;
}
while (moveon == false);

DataView dv = new DataView(dt1);

this.dataGridView1.DataSource = dv;
}

public void GetTable(object data)
{
lock (data)
{
DataTable dt;
string devicename = ((SNMPDevice)(data)).DeviceName;
DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting Thread Work: " +
devicename);
if (data.GetType() == typeof(ASAVPN))
dt = ((ASAVPN)(data)).SessionTable;
else
dt = ((VPN3000)(data)).SessionTable;

TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Ending Thread Work " + devicename + "
" + ts.ToString());

lock (this.ds.Tables["vpninfo"])
{
this.ds.Tables["vpninfo"].Merge(dt, true,
MissingSchemaAction.Add);
}
}
}

Hi,

I took a cursory look at the code and I noticed a couple of things.
First, it's better to use Thread.Join to wait for a thread to end.
Second, the threads are pointless the way the code is structured. You
have a lock around the entire GetTable method which causes the threads
to execute serially anyway.

Brian

[Brian Gideon]

On Mar 6, 12:26 pm, "Brian Gideon" <[e-mail]:

Second, the threads are pointless the way the code is structured. You
have a lock around the entire GetTable method which causes the threads
to execute serially anyway.

Err...I just noticed the lock is acquired on the data object. I'll
have to take a closer look when I get time.

[Jon Skeet [C# MVP]]

Cliff <Ju**@ballsdeep.netwrote:

I've got a number of SNMP devices scattered around the globe that I
want to get some information off..

I've got a couple of classes whcih get a quite complex table together
from SQL and SNMP devices and return a DataTable Object.

but as the devices are all in totally different locations (we have
about 20) and the main time lag is due to latency I want to go and get
the data simultaniuously from all the resources.

In the code you posted, it's not at all clear where the bit which will
introduce latency is. It looks like it's all "local" work, in which
case introducing threading is likely to slow things down a little bit
rather than speeding things up (unless you've got multiple processors).

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Cliff]

On 6 Mar, 18:26, "Brian Gideon" <[e-mail]:

On Mar 6, 11:56 am, "Cliff" <[e-mail]:

I've got a number of SNMP devices scattered around the globe that I
want to get some information off..

I've got a couple of classes whcih get a quite complex table together
from SQL and SNMP devices and return a DataTable Object.

but as the devices are all in totally different locations (we have
about 20) and the main time lag is due to latency I want to go and get
the data simultaniuously from all the resources.

Each individual table get takes between .25 and 2 seconds so
sequentially, the process takes about 8-10 seconds for 10 devices.

Having converted it to be multi threaded, it now takes 10-15 seconds.
(even when compiled as release code)

I've added some timing code to make sure it is running, but it seems
to take ages to get the threads started. I need to get this down to
2/3 seconds or it won't work.

Is there any way to speed up actually creating the threads, as that
seems to be taking 3 seconds alone!

Any help appreciated!

CliffDabbs

Here's the code

ArrayList arr = new ArrayList();

DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting To Launch Threads");

foreach (ASAVPN avpn in asavpns)
{
Debug.WriteLine("Launching Thread " +
avpn.DeviceName);
Thread th = new Thread(GetTable);
arr.Add(th);
th.Start(avpn);
}

foreach (VPN3000 kvpn in v3kvpns)
{
Debug.WriteLine("Launching Thread " +
kvpn.DeviceName);
Thread th = new Thread(GetTable);
arr.Add(th);
th.Start(kvpn);
}

TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Launch of Threads " + ts.ToString());

//Wait until the threads end;
bool moveon = true;
do
{
Thread.Sleep(100);
moveon = true;
foreach (Thread th in arr)
if (th.ThreadState ==
System.Threading.ThreadState.Running)
moveon = false;
}
while (moveon == false);

DataView dv = new DataView(dt1);

this.dataGridView1.DataSource = dv;
}

public void GetTable(object data)
{
lock (data)
{
DataTable dt;
string devicename = ((SNMPDevice)(data)).DeviceName;
DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting Thread Work: " +
devicename);
if (data.GetType() == typeof(ASAVPN))
dt = ((ASAVPN)(data)).SessionTable;
else
dt = ((VPN3000)(data)).SessionTable;

TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Ending Thread Work " + devicename + "
" + ts.ToString());

lock (this.ds.Tables["vpninfo"])
{
this.ds.Tables["vpninfo"].Merge(dt, true,
MissingSchemaAction.Add);
}
}
}

Hi,

I took a cursory look at the code and I noticed a couple of things.
First, it's better to use Thread.Join to wait for a thread to end.
Second, the threads are pointless the way the code is structured. You
have a lock around the entire GetTable method which causes the threads
to execute serially anyway.

Brian- Hide quoted text -

- Show quoted text -

Hi, thanks for the quick response.

I put the lock in becuase I was wondering if there were problems
around locking, and I wanted to cover all bases...but yes, I see your
point now.

although as "data" is effectivly a different object in every thread
doesn't that mean I'm not locking threads out...i.e. thread 1 locks
data 1, thread 2 locks data 2???

or doesn't it work like that?

anyway, the lock doesn't seem to change anything, I've taken it out
and I have the same problems.

About thread.join, How does that work when I'm waiting for multiple
threads to finish?

is this what you mean?

foreach (Thread th in arr)
th.Join;

as effectivly I need to join all the threads, but execution stops as
soon as you execute that command.

Thanks again...really need this help (as you can probably tell!)

Cliff.

[Jon Skeet [C# MVP]]

Cliff <Ju**@ballsdeep.netwrote:

<snip>

About thread.join, How does that work when I'm waiting for multiple
threads to finish?

is this what you mean?

foreach (Thread th in arr)
th.Join;

Yup.

as effectivly I need to join all the threads, but execution stops as
soon as you execute that command.

Execution stops on the main thread, yes - but seeing as the whole point
is to wait until all the other threads have finished, it's doing
*exactly* what you want it to.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Cliff]

On 6 Mar, 19:28, Jon Skeet [C# MVP] <[e-mail]:

Cliff<[e-mail]:

<snip>

About thread.join, How does that work when I'm waiting for multiple
threads to finish?

is this what you mean?

foreach (Thread th in arr)
th.Join;

Yup.

as effectivly I need to join all the threads, but execution stops as
soon as you execute that command.

Execution stops on the main thread, yes - but seeing as the whole point
is to wait until all the other threads have finished, it's doing
*exactly* what you want it to.

--
Jon Skeet - <[e-mail]>http://www.pobox.com/~skeet Blog:http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

The part that introduces latency is

if (data.GetType() == typeof(ASAVPN))
dt = ((ASAVPN)(data)).SessionTable;
else
dt = ((VPN3000)(data)).SessionTable;

Getting the session table requires round trips, and waits for data and
subsequent requests etc..

I guess I'm just astounded that setting up 10 threads takes 2-3
seconds. I remember in C++ (years ago as I've been using C# since it
came out) it would have been almost instant.

Cliff.

[Jon Skeet [C# MVP]]

Cliff <Ju**@ballsdeep.netwrote:

The part that introduces latency is

if (data.GetType() == typeof(ASAVPN))
dt = ((ASAVPN)(data)).SessionTable;
else
dt = ((VPN3000)(data)).SessionTable;

Getting the session table requires round trips, and waits for data and
subsequent requests etc..

Ah, right. Note that the "is" and "as" operators are generally easier
to read and more performant than calling GetType.

I guess I'm just astounded that setting up 10 threads takes 2-3
seconds. I remember in C++ (years ago as I've been using C# since it
came out) it would have been almost instant.

It doesn't take 2-3 seconds to set up 10 threads. That's not the issue.

Have you checked what happens across the network while those multiple
threads are (supposedly) accessing the data in parallel?

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Willy Denoyette [MVP]]

"Cliff" <Ju**@ballsdeep.netwrote in message
news:11**********************@t69g2000cwt.googlegr oups.com...

On 6 Mar, 19:28, Jon Skeet [C# MVP] <[e-mail]:

>Cliff<[e-mail]:

<snip>

About thread.join, How does that work when I'm waiting for multiple
threads to finish?

is this what you mean?

foreach (Thread th in arr)
th.Join;

Yup.

as effectivly I need to join all the threads, but execution stops as
soon as you execute that command.

Execution stops on the main thread, yes - but seeing as the whole point
is to wait until all the other threads have finished, it's doing
*exactly* what you want it to.

--
Jon Skeet - <[e-mail]>http://www.pobox.com/~skeet
Blog:http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

The part that introduces latency is

if (data.GetType() == typeof(ASAVPN))
dt = ((ASAVPN)(data)).SessionTable;
else
dt = ((VPN3000)(data)).SessionTable;

Getting the session table requires round trips, and waits for data and
subsequent requests etc..

I guess I'm just astounded that setting up 10 threads takes 2-3
seconds. I remember in C++ (years ago as I've been using C# since it
came out) it would have been almost instant.

Cliff.

Are you talking about the timespan between:

DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting To Launch Threads");
...
....
TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Launch of Threads " + ts.ToString());
or
DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting To Launch Threads");
...
....
TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Ending Thread Work " + devicename + "
" + ts.ToString());

Starting 10 threads should normally take much less than a second, all depends what the
threads are doing, and it's not possible to know what's done here at all...
dt = ((ASAVPN)(data)).SessionTable;
else
dt = ((VPN3000)(data)).SessionTable;

but *if* SessionTable is quite processor intensive, then, I'm not surprised it takes 2-3
seconds to start 10 threads.
Willy.

[Cliff]

On 6 Mar, 23:21, "Willy Denoyette [MVP]" <[e-mail]>
wrote:

"Cliff" <[e-mail] in message

news:11**********************@t69g2000cwt.googlegr oups.com...

On 6 Mar, 19:28, Jon Skeet [C# MVP] <[e-mail]:

>Cliff<[e-mail]:

<snip>

About thread.join, How does that work when I'm waiting for multiple
threads to finish?

is this what you mean?

foreach (Thread th in arr)
th.Join;

Yup.

as effectivly I need to join all the threads, but execution stops as
soon as you execute that command.

Execution stops on the main thread, yes - but seeing as the whole point
is to wait until all the other threads have finished, it's doing
*exactly* what you want it to.

--
Jon Skeet - <[e-mail]>http://www.pobox.com/~skeet
Blog:http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

The part that introduces latency is

if (data.GetType() == typeof(ASAVPN))
dt = ((ASAVPN)(data)).SessionTable;
else
dt = ((VPN3000)(data)).SessionTable;

Getting the session table requires round trips, and waits for data and
subsequent requests etc..

I guess I'm just astounded that setting up 10 threads takes 2-3
seconds. I remember in C++ (years ago as I've been using C# since it
came out) it would have been almost instant.

Cliff.

Are you talking about the timespan between:

DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting To Launch Threads");
..
...
TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Launch of Threads " + ts.ToString());

or
DateTime StartTime = DateTime.Now;
Debug.WriteLine("Starting To Launch Threads");
..
...
TimeSpan ts = DateTime.Now.Subtract(StartTime);
Debug.WriteLine("Ending Thread Work " + devicename + "
" + ts.ToString());

Starting 10 threads should normally take much less than a second, all depends what the
threads are doing, and it's not possible to know what's done here at all...
dt = ((ASAVPN)(data)).SessionTable;
else
dt = ((VPN3000)(data)).SessionTable;

but *if* SessionTable is quite processor intensive, then, I'm not surprised it takes 2-3
seconds to start 10 threads.

Willy.- Hide quoted text -

- Show quoted text -

The Session Tables are Calling SNMP Calls in 3rd party DLLs and
waiting for the response. As far as i know, that code requests
something, and synchronsly waits for a response, then requests the
next thing. Each call and response taking 100 - 500 ms resulting in a
few seconds operation. (I know some of them are longer, but the
majority are 2/3 seconds)

Hence I'm trying to remove the Synchronous part, and do things
asynchronously.

That is not a processor intensive task.

I've put this on a faster box now, and it seems to be working
better....

It also seems to work far faster the second time its run. maybe that
could be todo with caching?
Here's the readout from it actually running. I've put spaces and
comments round the important bits

Starting the process.
Starting To Launch Threads
Launching Thread HTCVEDT11
Starting Thread Work: HTCVEDT11
Launching Thread HTCVEDT12
Starting Thread Work: HTCVEDT12
Launching Thread NY3VEDT13
Starting Thread Work: NY3VEDT13
Launching Thread TOKP-VPNRAP01

Ending Thread Work HTCVEDT12
00:00:00.4397272
*One of the first threads ends before the threads have even
finished being setup*

The thread 0x874 has exited with code 0 (0x0).
Ending Thread Work NY3VEDT13 00:00:00.3497830
The thread 0xfb8 has exited with code 0 (0x0).
Starting Thread Work: TOKP-VPNRAP01
Launching Thread STCP-VPNRAP01
Starting Thread Work: STCP-VPNRAP01
Launching Thread DTCVEDTVPN1
Starting Thread Work: DTCVEDTVPN1
Launching Thread STCP-VPNRAP02
Starting Thread Work: STCP-VPNRAP02
Launching Thread DTCVEDTVPN2
Starting Thread Work: DTCVEDTVPN2
Launching Thread GTCVEDTVPN5
Starting Thread Work: GTCVEDTVPN5
Launching Thread GTCVEDTVPN6
Starting Thread Work: GTCVEDTVPN6
Launching Thread STCP-VPNRAP03
Starting Thread Work: STCP-VPNRAP03
Ending Thread Work STCP-VPNRAP02 00:00:00.4697086
The thread 0x338 has exited with code 0 (0x0).
Launching Thread STCP-VPNRAP04
Starting Thread Work: STCP-VPNRAP04
Ending Thread Work STCP-VPNRAP03 00:00:00.2298574
The thread 0xe48 has exited with code 0 (0x0).
Ending Thread Work STCP-VPNRAP04 00:00:00.2198636
The thread 0x180 has exited with code 0 (0x0).
Ending Thread Work GTCVEDTVPN6 00:00:00.7695226
The thread '<No Name>' (0x9d8) has exited with code 0 (0x0).
Launching Thread TOKP-VPNRAP02

Launching the threads took:
00:00:03.1880222
*Only here have all the threads been setup thats 3 seconds
into the task when some of the first threads have already finished.*

Starting Thread Work: TOKP-VPNRAP02
Ending Thread Work STCP-VPNRAP01 00:00:01.3891382
The thread 0xf20 has exited with code 0 (0x0).
Ending Thread Work TOKP-VPNRAP02 00:00:01.3291754
The thread 0xbe0 has exited with code 0 (0x0).
Ending Thread Work GTCVEDTVPN5 00:00:02.6983260
The thread '<No Name>' (0xc94) has exited with code 0 (0x0).
Ending Thread Work HTCVEDT11 00:00:08.9644386
The thread '<No Name>' (0xf24) has exited with code 0 (0x0).
Ending Thread Work DTCVEDTVPN1 00:00:07.8951020
The thread 0xbf0 has exited with code 0 (0x0).
Ending Thread Work TOKP-VPNRAP01 00:00:12.7121136
The thread 0xb70 has exited with code 0 (0x0).
Ending Thread Work DTCVEDTVPN2 00:00:18.3586106
The thread '<No Name>' (0x718) has exited with code 0 (0x0).
The thread 0x13c has exited with code 0 (0x0).
Waiting for the threads finsh 00:00:17.4091996 To Complete their work