alternative file.copy

[luis molina Micasoft]

it seems that when i do file.copy the svchost.exe is hanged, i mean if i make
40 threads of file.copy , 40 copys of files at same time the system is going
down and stop responding, this is when i'm working with cifs (shares). there
is another solution to copy files than file.copy in .net?

[Herfried K. Wagner [MVP]]

"luis molina Micasoft" <lu****************@discussions.microsoft.com>
schrieb:

it seems that when i do file.copy the svchost.exe is hanged, i
mean if i make 40 threads of file.copy

Reduce the number of threads...

--
Herfried K. Wagner [MVP]
<URL:http://dotnet.mvps.org/>

[Cor Ligthert]

Luis,

You make me curious, are you using threads to copy files, and then what is
the reason for that?

Cor

"luis molina Micasoft" <lu****************@discussions.microsoft.com>
....

it seems that when i do file.copy the svchost.exe is hanged, i mean if i
make
40 threads of file.copy , 40 copys of files at same time the system is
going
down and stop responding, this is when i'm working with cifs (shares).
there
is another solution to copy files than file.copy in .net?

[Jay B. Harlow [MVP - Outlook]]

Luis,
There are any number of alternatives to file.copy, some that work better
then others.

However! I'm not so sure that File.Copy is the problem as much as 40
File.Copy running at one time is.

To copy a file, you (or the Framework) needs open the source file, open the
destination file, read of chunk of data, write a chunk of data, repeat
reading & writing until there is no more data.

Depending on the size of the files & the size of the chunks, I would expect
40 copies to bring a system to its knees... I would expect adding network
shares to the mix would only compound the problem

Are you using the ThreadPool or creating the threads yourself.

I would consider using a thread safe Queue to hold the copy requests and
only start a handful of Workers. Each worker would get one copy request from
the Queue, copy the file, then get another request.

I would also consider using Asynchronous File I/O to copy the contents with
Stream.BeginRead & Stream.BeginWrite, allowing the overlapping of the reads
& writes, which may require reducing the number of Workers.

I'm not sure which ones, I suspect there are a handful of Performance
Counters that you could use to monitor the process and dynamically adjust
the number of Workers to match your system performance. System not being
stressed increase workers, system being stressed decrease workers... I know
MSDN magazine a few years ago had an article on how to use Performance
Counters to adjust number of workers...

Hope this helps
Jay

"luis molina Micasoft" <lu****************@discussions.microsoft.com> wrote
in message news:93**********************************@microsof t.com...

it seems that when i do file.copy the svchost.exe is hanged, i mean if i
make
40 threads of file.copy , 40 copys of files at same time the system is
going
down and stop responding, this is when i'm working with cifs (shares).
there
is another solution to copy files than file.copy in .net?

[luis molina Micasoft]

yes you got me, im not using a threadpool, and im trying to copy 20 or 40
files from uncs shares at same time with a independent thread for each one of
them, im gonna try with the threadpool and the asyncronous io, thanks for all.

"Jay B. Harlow [MVP - Outlook]" wrote:

Luis,
There are any number of alternatives to file.copy, some that work better
then others.

However! I'm not so sure that File.Copy is the problem as much as 40
File.Copy running at one time is.

To copy a file, you (or the Framework) needs open the source file, open the
destination file, read of chunk of data, write a chunk of data, repeat
reading & writing until there is no more data.

Depending on the size of the files & the size of the chunks, I would expect
40 copies to bring a system to its knees... I would expect adding network
shares to the mix would only compound the problem

Are you using the ThreadPool or creating the threads yourself.

I would consider using a thread safe Queue to hold the copy requests and
only start a handful of Workers. Each worker would get one copy request from
the Queue, copy the file, then get another request.

I would also consider using Asynchronous File I/O to copy the contents with
Stream.BeginRead & Stream.BeginWrite, allowing the overlapping of the reads
& writes, which may require reducing the number of Workers.

I'm not sure which ones, I suspect there are a handful of Performance
Counters that you could use to monitor the process and dynamically adjust
the number of Workers to match your system performance. System not being
stressed increase workers, system being stressed decrease workers... I know
MSDN magazine a few years ago had an article on how to use Performance
Counters to adjust number of workers...

Hope this helps
Jay

"luis molina Micasoft" <lu****************@discussions.microsoft.com> wrote
in message news:93**********************************@microsof t.com...

it seems that when i do file.copy the svchost.exe is hanged, i mean if i
make
40 threads of file.copy , 40 copys of files at same time the system is
going
down and stop responding, this is when i'm working with cifs (shares).
there
is another solution to copy files than file.copy in .net?

[Cor Ligthert]

Jay,

I am quiet sure that using threads for diskcopying is useless. It gives you
even more threadmanaging processtime, and you even have even to check in a
strange way that you are not using twice the same filename in multiple
threads with the change that it will blow up your program because you cannot
control a part of this process.

Cor

[Jay B. Harlow [MVP - Outlook]]

Cor,

I am quiet sure that using threads for diskcopying is useless. I'm suspect, for you, you are correct! :-|
you even have even to check in a strange way that you are not using twice
the same filename in multiple Which is what the Thread Safe Queue is for. Here I am referring to a
System.Collections.Queue object. Then using either Queue.Synchronized to
create a thread safe Queue or encapsulate the Queue in your own class with
SyncLock statements to make it Thread Safe.

Seeing as a Queue is a First In First Out (FIFO) construct you would put all
the copy requests (an object with source name & destination name properties)
into the Queue. The workers would then read a request & process the copy.
the change that it will blow up your program because you cannot control a
part of this process. My change (using a Thread Safe Queue) is a standard pattern for
multi-threading, its the pattern that ThreadPool is based on (as suggested
by the method ThreadPool.QueueUserWorkItem)

Asynchronous File I/O is a standard pattern within the Framework.

http://msdn.microsoft.com/library/de...nousfileio.asp

I hope you realize my real suggestion to Luis was to limit the number of
requests, I was then offering alternatives that he may not have considered.

I do agree that one needs to be more careful writing multi-threaded
applications or not using Multi-threading, however as you reread the Luis's
original question, you will find he is already using Multi-threading!

Hope this helps
Jay
"Cor Ligthert" <no************@planet.nl> wrote in message
news:uw**************@TK2MSFTNGP11.phx.gbl... Jay,

I am quiet sure that using threads for diskcopying is useless. It gives
you even more threadmanaging processtime, and you even have even to check
in a strange way that you are not using twice the same filename in
multiple threads with the change that it will blow up your program because
you cannot control a part of this process.

Cor

[Jay B. Harlow [MVP - Outlook]]

Luis,
Using the ThreadPool may not help per se.

My point is the number of requests going on. The ThreadPool will effectively
limit you to 25 requests. 25 requests may still be too many.

If I used the ThreadPool I would still try to limit the number of actual
copy requests going on...

Hope this helps
Jay

"luis molina Micasoft" <lu****************@discussions.microsoft.com> wrote
in message news:1A**********************************@microsof t.com...

yes you got me, im not using a threadpool, and im trying to copy 20 or 40
files from uncs shares at same time with a independent thread for each one
of
them, im gonna try with the threadpool and the asyncronous io, thanks for
all.

"Jay B. Harlow [MVP - Outlook]" wrote:

Luis,
There are any number of alternatives to file.copy, some that work better
then others.

However! I'm not so sure that File.Copy is the problem as much as 40
File.Copy running at one time is.

To copy a file, you (or the Framework) needs open the source file, open
the
destination file, read of chunk of data, write a chunk of data, repeat
reading & writing until there is no more data.

Depending on the size of the files & the size of the chunks, I would
expect
40 copies to bring a system to its knees... I would expect adding network
shares to the mix would only compound the problem

Are you using the ThreadPool or creating the threads yourself.

I would consider using a thread safe Queue to hold the copy requests and
only start a handful of Workers. Each worker would get one copy request
from
the Queue, copy the file, then get another request.

I would also consider using Asynchronous File I/O to copy the contents
with
Stream.BeginRead & Stream.BeginWrite, allowing the overlapping of the
reads
& writes, which may require reducing the number of Workers.

I'm not sure which ones, I suspect there are a handful of Performance
Counters that you could use to monitor the process and dynamically adjust
the number of Workers to match your system performance. System not being
stressed increase workers, system being stressed decrease workers... I
know
MSDN magazine a few years ago had an article on how to use Performance
Counters to adjust number of workers...

Hope this helps
Jay

"luis molina Micasoft" <lu****************@discussions.microsoft.com>
wrote
in message news:93**********************************@microsof t.com...

> it seems that when i do file.copy the svchost.exe is hanged, i mean if
> i
> make
> 40 threads of file.copy , 40 copys of files at same time the system is
> going
> down and stop responding, this is when i'm working with cifs (shares).
> there
> is another solution to copy files than file.copy in .net?

[Nak]

Hi Luis,

This is probably going to sound stupid, but arent multiple asynchronous
file operations slow because of hard drive technology? I'll give you an
example, copying a load of MP3's (50mb) from one drive to the other chugs
along quite happy, then when you throw just 1 more file copy to happen at
the same time the entire operation gets slowed down, and my understanding
for this is because the hard drive can't read / write that many sectors at
the same time.

Probably not helpful, but I thought I'd say it anyway! :-)

Nick.

"luis molina Micasoft" <lu****************@discussions.microsoft.com> wrote
in message news:93**********************************@microsof t.com...

it seems that when i do file.copy the svchost.exe is hanged, i mean if i
make
40 threads of file.copy , 40 copys of files at same time the system is
going
down and stop responding, this is when i'm working with cifs (shares).
there
is another solution to copy files than file.copy in .net?