473,236 Members | 1,694 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,236 software developers and data experts.

C# Write lines to file at high volume, multiple threads

Hi -

I am attempting to write lines to a file at high volume, multiple threads.
Here is my scenario:

(initial "WriteToFile" object created via a parent multithreaded process,
which receives files, can call custom code for each file received - much like
BizTalk)

1. Receive multiple files, each file received creates a "WriteToFile" object
2. Each "WriteToFile" object parses it's file, creates a line, and attempts
to write that line to a specified file. Most or all of the "WriteToFile"
objects will be attempting to write to the same file.
3. If a "WriteToFile" object can't write it's data to the file, it should
throw an exception, handle it, and try again.

Questions:
1. Would this be better handled as a Windows service which reads files from
a queue and processes them serially?

2. If I didn't want to use a queue or service approach, what could I use to
check whether a file is accessible or not? Ie: Is some other process writing
to it?

3. Any code suggestions for a high volume file-write service or process?

Thaks for your help in advance!
--
Dameon
May 11 '07 #1
2 11299
"Dameon" <Da****@discussions.microsoft.comwrote in message
news:9B**********************************@microsof t.com...
Hi -

I am attempting to write lines to a file at high volume, multiple threads.
Here is my scenario:

(initial "WriteToFile" object created via a parent multithreaded process,
which receives files, can call custom code for each file received - much
like
BizTalk)

1. Receive multiple files, each file received creates a "WriteToFile"
object
2. Each "WriteToFile" object parses it's file, creates a line, and
attempts
to write that line to a specified file. Most or all of the "WriteToFile"
objects will be attempting to write to the same file.
3. If a "WriteToFile" object can't write it's data to the file, it should
throw an exception, handle it, and try again.

Questions:
1. Would this be better handled as a Windows service which reads files
from
a queue and processes them serially?

2. If I didn't want to use a queue or service approach, what could I use
to
check whether a file is accessible or not? Ie: Is some other process
writing
to it?

3. Any code suggestions for a high volume file-write service or process?
I think if you want a high volume file writing, you want a single thread
writing to the file and the WriteToFile objects should parse it's file,
create a line and pass that line to the file writer thread by adding it to a
Queue of lines to be written.

May 11 '07 #2
I'd use file proxy object servicing the queue of requests

It can sit on separate thread and be waken up by monitor event by any other
posting thread. File access code then is encapsulated in proxy object. For
high volume you need to use asynch mode or file writing

If you have multiple files, you can create multiple proxies. Sending threads
can use global queue, which might distribute payload to corresponding proxy
queue. Could be done as service or as module in your application (or
plug-in).

As about locking, if only you write to the file, you can open it in
exclusive mode.

HTH
Alex

"Dameon" <Da****@discussions.microsoft.comwrote in message
news:9B**********************************@microsof t.com...
Hi -

I am attempting to write lines to a file at high volume, multiple threads.
Here is my scenario:

(initial "WriteToFile" object created via a parent multithreaded process,
which receives files, can call custom code for each file received - much
like
BizTalk)

1. Receive multiple files, each file received creates a "WriteToFile"
object
2. Each "WriteToFile" object parses it's file, creates a line, and
attempts
to write that line to a specified file. Most or all of the "WriteToFile"
objects will be attempting to write to the same file.
3. If a "WriteToFile" object can't write it's data to the file, it should
throw an exception, handle it, and try again.

Questions:
1. Would this be better handled as a Windows service which reads files
from
a queue and processes them serially?

2. If I didn't want to use a queue or service approach, what could I use
to
check whether a file is accessible or not? Ie: Is some other process
writing
to it?

3. Any code suggestions for a high volume file-write service or process?

Thaks for your help in advance!
--
Dameon

May 14 '07 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: Irmen de Jong | last post by:
Hi, Things like Twisted, medusa, etc.... that claim to be able to support hundreds of concurrent connections because of the async I/O framework they're based on.... can someone give a few...
5
by: nicolas_riesch | last post by:
Does someone know if the module pytz (http://sourceforge.net/projects/pytz/) is thread-safe ? I have not seen it explicitely stated, and just wanted to be sure, as I want to use it. That's...
18
by: jas | last post by:
Hi, I would like to start a new process and be able to read/write from/to it. I have tried things like... import subprocess as sp p = sp.Popen("cmd.exe", stdout=sp.PIPE)...
3
by: Evan Smith | last post by:
Scenario: new functionality was recently deployed in a 3-tier business application. Since deployment, CPU use has shot up to very high levels. Using event monitors to try to track down the...
3
by: lltaylor | last post by:
Hello All, I apologise for such a basic question, I am just getting to grips with c#, I usually develop in java. What I am trying to do is: I have a file which stores look-up information,...
2
by: Arsen V. | last post by:
Hello, Did anyone have some experience with the following: 1) Need to track high volume of impressions - 20,000,000+ per day 2) Backend is SQL Server 2000 3) Webfarm of IIS with ASP.NET 4)...
5
by: Ron Mexico | last post by:
I have written a graphing engine (very similar to what BigCharts.com offers). Basically, it's an ASPX page that accepts parameters and calls back-end business objects (dlls) to create a graph. ...
5
by: Sumana | last post by:
Hi All, We developed our project on VC++.Net console application to create image of disk and to write the image We are having problem with reading and writing the sector beyond 6GB Disk or...
5
by: Claudio Grondi | last post by:
I have just started to play around with the bsddb3 module interfacing the Berkeley Database. Beside the intended database file databaseFile.bdb I see in same directory also the __db.001...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.