Two threads read from one file? | | |
I've got two threads reading from one disk file (I'm using the Pthread
library).
I need each thread to have its own file seek pointer, so they can read from
independent parts of the file.
I tried using _sopen() once (for the entire process) then using fdopen() in
each thread to get unique file seek pointers, but it is not working well.
Any suggestions? | | | | re: Two threads read from one file?
noleander wrote:[color=blue]
> I've got two threads reading from one disk file (I'm using the Pthread
> library).
>
> I need each thread to have its own file seek pointer, so they can
> read from independent parts of the file.
>
> I tried using _sopen() once (for the entire process) then using
> fdopen() in each thread to get unique file seek pointers, but it is
> not working well.[/color]
What's wrong with just using fopen() twice? That would seem to be the
obvious solution.
-cd | | | | re: Two threads read from one file?
If you have two threads, and they both open a single file for read, and they
use fopen() or open() to open the file, they will share a single seek
pointer. That means that they cannot perform independent read operations.
The only way to get separate seek pointers for the threads is to use sopen()
once for the process, followed by fdopen() for each thread.
I am using the latter technique, but it is failing in some circumstances. I
was wondering if anyone else has tried this and has some lessons-learned.
"Carl Daniel [VC++ MVP]" wrote:
[color=blue]
> noleander wrote:[color=green]
> > I've got two threads reading from one disk file (I'm using the Pthread
> > library).
> >
> > I need each thread to have its own file seek pointer, so they can
> > read from independent parts of the file.
> >
> > I tried using _sopen() once (for the entire process) then using
> > fdopen() in each thread to get unique file seek pointers, but it is
> > not working well.[/color]
>
> What's wrong with just using fopen() twice? That would seem to be the
> obvious solution.
>
> -cd
>
>
>[/color] | | | | re: Two threads read from one file?
"noleander" <noleander@discussions.microsoft.com> wrote[color=blue]
> If you have two threads, and they both open a single file for read, and they
> use fopen() or open() to open the file, they will share a single seek
> pointer. That means that they cannot perform independent read operations.
>
> The only way to get separate seek pointers for the threads is to use sopen()
> once for the process, followed by fdopen() for each thread.
>
> I am using the latter technique, but it is failing in some circumstances. I
> was wondering if anyone else has tried this and has some lessons-learned.[/color]
You must be doing something wrong; fopen (here two seperate FILE*
on the same file) is working fine both sequentially (ie. w/o threads,
but "interleaved"), and also parallel in threads.
But: you must use FILE* as an auto variable, not global and also not on the heap.
[color=blue]
> "Carl Daniel [VC++ MVP]" wrote:
>[color=green]
> > noleander wrote:[color=darkred]
> > > I've got two threads reading from one disk file (I'm using the Pthread
> > > library).
> > >
> > > I need each thread to have its own file seek pointer, so they can
> > > read from independent parts of the file.
> > >
> > > I tried using _sopen() once (for the entire process) then using
> > > fdopen() in each thread to get unique file seek pointers, but it is
> > > not working well.[/color]
> >
> > What's wrong with just using fopen() twice? That would seem to be the
> > obvious solution.
> >
> > -cd[/color][/color] | | | | re: Two threads read from one file?
Not true.
If you have two threads in a single process, and they both call fopen() on a
single disk file (even with their own dedicated FILE* pointer) both thread's
FILE* will point to the same FILE (_iobuf) structure.
The only way to get two FILE* pointers is to use open() or sopen() first,
then call fdopen() in each thread with the fileID from open().
But even using this approach, it appears that the two FILE* are sharing a
single underlying seek pointer.
"um" wrote:
[color=blue]
> "noleander" <noleander@discussions.microsoft.com> wrote[color=green]
> > If you have two threads, and they both open a single file for read, and they
> > use fopen() or open() to open the file, they will share a single seek
> > pointer. That means that they cannot perform independent read operations.
> >
> > The only way to get separate seek pointers for the threads is to use sopen()
> > once for the process, followed by fdopen() for each thread.
> >
> > I am using the latter technique, but it is failing in some circumstances. I
> > was wondering if anyone else has tried this and has some lessons-learned.[/color]
>
> You must be doing something wrong; fopen (here two seperate FILE*
> on the same file) is working fine both sequentially (ie. w/o threads,
> but "interleaved"), and also parallel in threads.
> But: you must use FILE* as an auto variable, not global and also not on the heap.
>
>
>[color=green]
> > "Carl Daniel [VC++ MVP]" wrote:
> >[color=darkred]
> > > noleander wrote:
> > > > I've got two threads reading from one disk file (I'm using the Pthread
> > > > library).
> > > >
> > > > I need each thread to have its own file seek pointer, so they can
> > > > read from independent parts of the file.
> > > >
> > > > I tried using _sopen() once (for the entire process) then using
> > > > fdopen() in each thread to get unique file seek pointers, but it is
> > > > not working well.
> > >
> > > What's wrong with just using fopen() twice? That would seem to be the
> > > obvious solution.
> > >
> > > -cd[/color][/color]
>
>
>
>[/color] | | | | re: Two threads read from one file?
"noleander" <noleander@discussions.microsoft.com> wrote in message
news:782D102E-E158-47C4-8DFD-73E052EE4B6F@microsoft.com...[color=blue]
> Not true.
>
> If you have two threads in a single process, and they both call fopen() on a
> single disk file (even with their own dedicated FILE* pointer) both thread's
> FILE* will point to the same FILE (_iobuf) structure.
>
> The only way to get two FILE* pointers is to use open() or sopen() first,
> then call fdopen() in each thread with the fileID from open().
>
> But even using this approach, it appears that the two FILE* are sharing a
> single underlying seek pointer.[/color]
Sorry, I still can't believe it.
Best would be to post your test code.
[color=blue]
> "um" wrote:
>[color=green]
> > "noleander" <noleander@discussions.microsoft.com> wrote[color=darkred]
> > > If you have two threads, and they both open a single file for read, and[/color][/color][/color]
they[color=blue][color=green][color=darkred]
> > > use fopen() or open() to open the file, they will share a single seek
> > > pointer. That means that they cannot perform independent read operations.
> > >
> > > The only way to get separate seek pointers for the threads is to use[/color][/color][/color]
sopen()[color=blue][color=green][color=darkred]
> > > once for the process, followed by fdopen() for each thread.
> > >
> > > I am using the latter technique, but it is failing in some circumstances.[/color][/color][/color]
I[color=blue][color=green][color=darkred]
> > > was wondering if anyone else has tried this and has some lessons-learned.[/color]
> >
> > You must be doing something wrong; fopen (here two seperate FILE*
> > on the same file) is working fine both sequentially (ie. w/o threads,
> > but "interleaved"), and also parallel in threads.
> > But: you must use FILE* as an auto variable, not global and also not on the[/color][/color]
heap.[color=blue][color=green]
> >
> >
> >[color=darkred]
> > > "Carl Daniel [VC++ MVP]" wrote:
> > >
> > > > noleander wrote:
> > > > > I've got two threads reading from one disk file (I'm using the Pthread
> > > > > library).
> > > > >
> > > > > I need each thread to have its own file seek pointer, so they can
> > > > > read from independent parts of the file.
> > > > >
> > > > > I tried using _sopen() once (for the entire process) then using
> > > > > fdopen() in each thread to get unique file seek pointers, but it is
> > > > > not working well.
> > > >
> > > > What's wrong with just using fopen() twice? That would seem to be the
> > > > obvious solution.
> > > >
> > > > -cd[/color][/color][/color] | | | | re: Two threads read from one file?
noleander wrote:[color=blue]
> Not true.
>
> If you have two threads in a single process, and they both call fopen() on a
> single disk file (even with their own dedicated FILE* pointer) both thread's
> FILE* will point to the same FILE (_iobuf) structure.
>
> The only way to get two FILE* pointers is to use open() or sopen() first,
> then call fdopen() in each thread with the fileID from open().
>
> But even using this approach, it appears that the two FILE* are sharing a
> single underlying seek pointer.[/color]
This simple console program is a direct counter example to your claim.
Why do you think your claim is true?
#include <stdio.h>
#include <assert.h>
int main()
{
FILE* in1;
FILE* in2;
char buffer1[512] = {0};
char buffer2[512] = {0};
FILE* out = fopen("test.txt", "wb");
fputs("If this appears twice, file pointers are independent.\n", out);
fclose(out);
in1 = fopen("test.txt", "rb");
in2 = fopen("test.txt", "rb");
assert(in1 != in2);
fgets(buffer1, sizeof buffer1, in1);
fgets(buffer2, sizeof buffer2, in2);
fclose(in1);
fclose(in2);
puts(buffer1);
puts(buffer2);
return 0;
}
Output:
If this appears twice, file pointers are independent.
If this appears twice, file pointers are independent.
Tom | | | | re: Two threads read from one file?
As most of the respondents have already pointed out - you should be quite ok
with plain fopen(). _sopen() is primarily for file sharing (at least this
is what I remember doing in the past) among processes.
Anyway, are you sure that your problems are not related to rendering of data
(e.g. displaying, writing, etc.) instead of reading? Please see the code
below that demonstrates fopen() in a multithreaded environment. The program
first creates a file and then launches a number of threads (specified on a
command line) to read from it. Note that I 'protect' rendering of data being
read by a critical section - this is just to make sure that this look ok on
the screen. You can redirect the output to another file if you like.
Bogdan
#include "stdafx.h"
#include <stdio.h>
#include <stdlib.h>
#include <process.h>
#include <windows.h>
#define LINE_LEN 4
long offsets[] = {0, 4, 8, 12, 16, 20, 24, 28, 32};
const char* fname = "threadfp.txt";
CRITICAL_SECTION cs;
int num_reads = 100;
int createFile(char* fname)
{
FILE* fp = fopen(fname, "wb");
if (NULL == fp) {
printf("Failed to create %s\n", fname);
return errno;
}
for (int i = 0; i < sizeof(offsets) / sizeof(offsets[0]); i++) {
char c = i + '0';
for (int j = 0; j < LINE_LEN; j++) {
fwrite(&c, 1, 1, fp);
}
}
fclose(fp);
return 0;
}
void readFile(void* param)
{
char buf[LINE_LEN];
int i;
FILE* fp = fopen((const char*) param, "rb");
DWORD tid = ::GetCurrentThreadId();
srand((unsigned int) tid);
if (NULL != fp) {
for (int n = 0; n < num_reads; n++) {
i = rand() % (sizeof(offsets) / sizeof(offsets[0]));
if (0 == fseek(fp, offsets[i], SEEK_SET)) {
if (sizeof(buf) == fread(buf, 1, sizeof(buf), fp)) {
EnterCriticalSection(&cs);
printf("[tid=%04X, offset=%2ld]", tid, offsets[i]);
for (int i = 0; i < sizeof(buf); i++) {
printf("%c", buf[i]);
}
printf("\n");
fflush(stdout);
LeaveCriticalSection(&cs);
}
}
::Sleep(500);
}
}
}
int _tmain(int argc, _TCHAR* argv[])
{
if (argc > 2 && 0 == createFile(argv[1])) {
if (argc > 3) {
num_reads = atoi(argv[3]);
}
InitializeCriticalSection(&cs);
int n = atoi(argv[2]);
for (int i = 0; i < n; n++) {
_beginthread(readFile, 0, argv[1]);
}
while (1) {
::Sleep(1000);
}
}
return 0;
}
"noleander" <noleander@discussions.microsoft.com> wrote in message
news:1026966A-4D15-4896-8C19-7E1FE2DC19E1@microsoft.com...[color=blue]
> I've got two threads reading from one disk file (I'm using the Pthread
> library).
>
> I need each thread to have its own file seek pointer, so they can read
> from
> independent parts of the file.
>
> I tried using _sopen() once (for the entire process) then using fdopen()
> in
> each thread to get unique file seek pointers, but it is not working well.
>
> Any suggestions?
>[/color] | | | | re: Two threads read from one file?
Yet another comment...
Just remembered that you were using pthread library. Don't know much about
it in a Win32 environment. You have to make sure that stream io functions
in clib (e.g. fopen, fseek, fread, etc.) are thread safe. If you don't use
pthread you need to use _beginthread(ex) instead of CreateThread.
Bogdan
"noleander" <noleander@discussions.microsoft.com> wrote in message
news:EDEEC341-3AAA-4627-AEEC-406AB1706726@microsoft.com...[color=blue]
> If you have two threads, and they both open a single file for read, and
> they
> use fopen() or open() to open the file, they will share a single seek
> pointer. That means that they cannot perform independent read operations.
>
> The only way to get separate seek pointers for the threads is to use
> sopen()
> once for the process, followed by fdopen() for each thread.
>
> I am using the latter technique, but it is failing in some circumstances.
> I
> was wondering if anyone else has tried this and has some lessons-learned.
>
>
>
> "Carl Daniel [VC++ MVP]" wrote:
>[color=green]
>> noleander wrote:[color=darkred]
>> > I've got two threads reading from one disk file (I'm using the Pthread
>> > library).
>> >
>> > I need each thread to have its own file seek pointer, so they can
>> > read from independent parts of the file.
>> >
>> > I tried using _sopen() once (for the entire process) then using
>> > fdopen() in each thread to get unique file seek pointers, but it is
>> > not working well.[/color]
>>
>> What's wrong with just using fopen() twice? That would seem to be the
>> obvious solution.
>>
>> -cd
>>
>>
>>[/color][/color] |  | Similar .NET Framework bytes | | | /bytes/about
We are a network of experts and professionals in IT and software development that help one another with answers to tough questions and share insights.
Get the best answers to your questions from over 226,501 network members.
|