FileVersionInfo.GetVersionInfo() performance in threaded app

[Magnus Koch]

Hi,
I'm working on a little deployment app to support .Net file deployments to
about 7 different locations on the network. It's somewhat intelligent in
that it only deploys changed files, and moves the old files to a backup
folder, so that most deployments can be done w/o booting all users off. The
network is pretty slow, so I built it so that each deployment (defined as
unique source/target folder combination) runs in a separate thread.

The issue I'm running into is that FileVersionInfo.GetVersionInfo seems to
either be running in a single thread below the surface, or have some other
issue with multithreading that I don't understand. Each call is at least
twice as long and, at regular intervals, the CPU utilization hits 100% and
stays there for 15-20 secs, with more than 50% Kernel Time, whatever that
might imply. The end result is that the app runs just as slow as in a single
thread, except it's even more annoying to watch.

The only obvious point of contention is that the threads may access the same
file in a source folder, but they are all local or on a fast connection, so
I can't see how it would hang it for that long.

Does anyone know what the issue is and how I can fix it? Would a separate
AppDomain help? Process?

Thanks,
Magnus

[Nak]

Hi there,

I think you might be missing 1 main point,

* Only 1 part of the hard drive can be accessed at any given time by the
drive heads.

I think the only case that this would be different is if you were to
have a RAID setup.

Nick.

--
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
"No matter. Whatever the outcome, you are changed."

Fergus - September 5th 2003
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

[Cor]

Hi Magsnus,
I was thinking about it, and there where some things in my head.
But first a little question.
Is the throughput time from 7 simultaneous connections longer or is the sum
from the total processor time higher (That would be normal)?
Cor

[One Handed Man [ OHM ]]

I think Nak has this hit firmly on the head. Unless the network speed is
slower than the disk access/read/write speed, then that's the answer.

OHM
"Nak" <[e-mail]> wrote in message
news:Os**************@TK2MSFTNGP10.phx.gbl...

Hi there,

I think you might be missing 1 main point,

* Only 1 part of the hard drive can be accessed at any given time by the drive heads.

I think the only case that this would be different is if you were to
have a RAID setup.

Nick.

--
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ "No matter. Whatever the outcome, you are changed."

Fergus - September 5th 2003
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

[Magnus]

Hi,
Thanks for you thoughts on this.

The network speed is significantly slower than the disk speed. Loading the
version info is subsecond on a local machine, and the files are typically
smaller than 100k. Running against a single remote machine, it runs in about
3-5 seconds. Running 7 at the time, they start hitting 20-90 seconds. The
only culprits I can see at this point are;
1. The GetVersionInfo method was not designed to run in multiple threads
(in the same process), e.g. by using a single instance of the file system
and/or by using sync locks to handle concurrent calls.
2. The appdomain and/or process is running some part in a single thread,
e.g. by pooling disk resources through a single pipe to the os.
3. The antivirus software may force the entire files to be loaded,
sequentially, before the info can be read from them.

/Magnus

"One Handed Man [ OHM ]" <te***************************@BTOpenworld.com>
wrote in message news:O2**************@TK2MSFTNGP10.phx.gbl...

I think Nak has this hit firmly on the head. Unless the network speed is
slower than the disk access/read/write speed, then that's the answer.

OHM
"Nak" <[e-mail]> wrote in message
news:Os**************@TK2MSFTNGP10.phx.gbl...

Hi there,

I think you might be missing 1 main point,

* Only 1 part of the hard drive can be accessed at any given time by

the

drive heads.

I think the only case that this would be different is if you were to
have a RAID setup.

Nick.

--

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

"No matter. Whatever the outcome, you are changed."

Fergus - September 5th 2003

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

[Magnus]

Hi,
The total runtime seems to be about the same, but I didn't time it exactly.
Note the times (from a different reply in the same thread):

"The network speed is significantly slower than the disk speed. Loading the
version info is subsecond on a local machine, and the files are typically
smaller than 100k. Running against a single remote machine, it runs in about
3-5 seconds. Running 7 at the time, they start hitting 20-90 seconds.

The only culprits I can see at this point are;
1. The GetVersionInfo method was not designed to run in multiple threads
(in the same process), e.g. by using a single instance of the file system
and/or by using sync locks to handle concurrent calls.
2. The appdomain and/or process is running some part in a single thread,
e.g. by pooling disk resources through a single pipe to the os.
3. The antivirus software may force the entire files to be loaded,
sequentially, before the info can be read from them."

Thanks,
Magnus

"Cor" <no*@non.com> wrote in message
news:3f***********************@reader22.wxs.nl...

Hi Magsnus,
I was thinking about it, and there where some things in my head.
But first a little question.
Is the throughput time from 7 simultaneous connections longer or is the sum from the total processor time higher (That would be normal)?
Cor

[Magnus]

Hi,
BTW, just thought of it. GetVersionInfo is a shared method, which would
imply either instantiating io objects all over the place or running with
synclocks and a single set of io objects. The latter seems the most likely
from the behavior. Can I get around that by using a separate AppDomain, or
do I need a separate process?
Thanks,
Magnus

"Magnus" <ma*********@hotmail.com> wrote in message
news:e9**************@TK2MSFTNGP09.phx.gbl...

Hi,
The total runtime seems to be about the same, but I didn't time it exactly. Note the times (from a different reply in the same thread):

"The network speed is significantly slower than the disk speed. Loading the version info is subsecond on a local machine, and the files are typically
smaller than 100k. Running against a single remote machine, it runs in about 3-5 seconds. Running 7 at the time, they start hitting 20-90 seconds.

The only culprits I can see at this point are;
1. The GetVersionInfo method was not designed to run in multiple threads
(in the same process), e.g. by using a single instance of the file system
and/or by using sync locks to handle concurrent calls.
2. The appdomain and/or process is running some part in a single thread,
e.g. by pooling disk resources through a single pipe to the os.
3. The antivirus software may force the entire files to be loaded,
sequentially, before the info can be read from them."

Thanks,
Magnus

"Cor" <no*@non.com> wrote in message
news:3f***********************@reader22.wxs.nl...

Hi Magsnus,
I was thinking about it, and there where some things in my head.
But first a little question.
Is the throughput time from 7 simultaneous connections longer or is the

sum

from the total processor time higher (That would be normal)?
Cor

[Nak]

Hi there Magnus,

The network speed is significantly slower than the disk speed. Loading the
version info is subsecond on a local machine, and the files are typically
smaller than 100k. Running against a single remote machine, it runs in about 3-5 seconds. Running 7 at the time, they start hitting 20-90 seconds. The
only culprits I can see at this point are;
It seems like you are answering your question there, it runs ok on a
local machine but slow over the network? If you have a slow network how can
multithreading make it any faster? You're just going to be using more
bandwidth at once if your multithreading, and if your'e network can't take
the load then it's going to slow down.
1. The GetVersionInfo method was not designed to run in multiple threads
(in the same process), e.g. by using a single instance of the file system
and/or by using sync locks to handle concurrent calls.
I'm not sure if the .NET version info function is like in comparrison to the
Win32 API equivilent. Have you tried it?
3. The antivirus software may force the entire files to be loaded,
sequentially, before the info can be read from them.
Have you tried disabling the anti-virus software to rule that out?

I think you will find it is either 1 of 2 things,

1. As mentioned previously your are overloading the hard drive.
2. You are overloading your network bandwidth. How is your network setup?
why is it so slow? Do you have 10 baseT network cards or something? Have
you got a *bad* router in the way?

Let's say that the function forces more information than is required to be
sent across the network, increasing the load beyond the capable threshold,
have you thought about attempting a quick web service that acquires the data
and sends only what is needed for you? Or some other host application?
Just ideas.

Nick.

--
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
"No matter. Whatever the outcome, you are changed."

Fergus - September 5th 2003
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\


/Magnus

"One Handed Man [ OHM ]" <te***************************@BTOpenworld.com>
wrote in message news:O2**************@TK2MSFTNGP10.phx.gbl...

I think Nak has this hit firmly on the head. Unless the network speed is
slower than the disk access/read/write speed, then that's the answer.

OHM
"Nak" <[e-mail]> wrote in message
news:Os**************@TK2MSFTNGP10.phx.gbl...

Hi there,

I think you might be missing 1 main point,

* Only 1 part of the hard drive can be accessed at any given time by

the

drive heads.

I think the only case that this would be different is if you were

to have a RAID setup.

Nick.

--

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

"No matter. Whatever the outcome, you are changed."

Fergus - September 5th 2003

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

[Magnus]

Hi,
It seems to work fine running in a separate appdomain. The bottlenecks I'm
trying to get around are latency and slow VPN connections to the remote
locations. The local pipe is big enough to handle all requests at the same
time.
/Magnus

"Nak" <[e-mail]> wrote in message
news:Ov**************@TK2MSFTNGP09.phx.gbl...

Hi there Magnus,

The network speed is significantly slower than the disk speed. Loading the

version info is subsecond on a local machine, and the files are typically smaller than 100k. Running against a single remote machine, it runs in about

3-5 seconds. Running 7 at the time, they start hitting 20-90 seconds. The only culprits I can see at this point are;

It seems like you are answering your question there, it runs ok on a
local machine but slow over the network? If you have a slow network how

can multithreading make it any faster? You're just going to be using more
bandwidth at once if your multithreading, and if your'e network can't take
the load then it's going to slow down.

1. The GetVersionInfo method was not designed to run in multiple threads
(in the same process), e.g. by using a single instance of the file system and/or by using sync locks to handle concurrent calls.
I'm not sure if the .NET version info function is like in comparrison to

the Win32 API equivilent. Have you tried it?

3. The antivirus software may force the entire files to be loaded,
sequentially, before the info can be read from them.
Have you tried disabling the anti-virus software to rule that out?

I think you will find it is either 1 of 2 things,

1. As mentioned previously your are overloading the hard drive.
2. You are overloading your network bandwidth. How is your network setup?
why is it so slow? Do you have 10 baseT network cards or something? Have
you got a *bad* router in the way?

Let's say that the function forces more information than is required to be
sent across the network, increasing the load beyond the capable threshold,
have you thought about attempting a quick web service that acquires the

data and sends only what is needed for you? Or some other host application?
Just ideas.

Nick.

--
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ "No matter. Whatever the outcome, you are changed."

Fergus - September 5th 2003
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

/Magnus

"One Handed Man [ OHM ]" <te***************************@BTOpenworld.com>
wrote in message news:O2**************@TK2MSFTNGP10.phx.gbl...

I think Nak has this hit firmly on the head. Unless the network speed is slower than the disk access/read/write speed, then that's the answer.

OHM
"Nak" <[e-mail]> wrote in message
news:Os**************@TK2MSFTNGP10.phx.gbl...
> Hi there,
>
> I think you might be missing 1 main point,
>
> * Only 1 part of the hard drive can be accessed at any given time

by the
> drive heads.
>
> I think the only case that this would be different is if you

were

to > have a RAID setup.
>
> Nick.
>
> --
>

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ > "No matter. Whatever the outcome, you are changed."
>
> Fergus - September 5th 2003
>

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

>
>

[Nak]

> It seems to work fine running in a separate appdomain. The bottlenecks I'm

trying to get around are latency and slow VPN connections to the remote
locations. The local pipe is big enough to handle all requests at the same
time.

That leaves you being pretty much buggered doesn't it? Sorry to hear that,
surely there must be some way of speeding up your VPN connections and such
like? even if it means new / different hardware or software changes...?

Nick.

--
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
"No matter. Whatever the outcome, you are changed."

Fergus - September 5th 2003
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

[Cor]

Hi Magnus,
I did some testing work.
I get the next figures.
Just relative figures absolute is not intresting of course.
request = fileninfo like you do on a netwerk sharedrive
1 thread with 600 requests 50x
2 threads with 300 requests 41x
6 threads with 100 request 36x
synclock had no influence at all.
So multithreading makes the throughput time shorter, there is maybe
something not right in your application I think.
I hope this helps a little bit.
Cor