473,326 Members | 2,337 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Two threads read from one file?

I've got two threads reading from one disk file (I'm using the Pthread
library).

I need each thread to have its own file seek pointer, so they can read from
independent parts of the file.

I tried using _sopen() once (for the entire process) then using fdopen() in
each thread to get unique file seek pointers, but it is not working well.

Any suggestions?

Nov 17 '05 #1
8 4030
noleander wrote:
I've got two threads reading from one disk file (I'm using the Pthread
library).

I need each thread to have its own file seek pointer, so they can
read from independent parts of the file.

I tried using _sopen() once (for the entire process) then using
fdopen() in each thread to get unique file seek pointers, but it is
not working well.


What's wrong with just using fopen() twice? That would seem to be the
obvious solution.

-cd
Nov 17 '05 #2
If you have two threads, and they both open a single file for read, and they
use fopen() or open() to open the file, they will share a single seek
pointer. That means that they cannot perform independent read operations.

The only way to get separate seek pointers for the threads is to use sopen()
once for the process, followed by fdopen() for each thread.

I am using the latter technique, but it is failing in some circumstances. I
was wondering if anyone else has tried this and has some lessons-learned.

"Carl Daniel [VC++ MVP]" wrote:
noleander wrote:
I've got two threads reading from one disk file (I'm using the Pthread
library).

I need each thread to have its own file seek pointer, so they can
read from independent parts of the file.

I tried using _sopen() once (for the entire process) then using
fdopen() in each thread to get unique file seek pointers, but it is
not working well.


What's wrong with just using fopen() twice? That would seem to be the
obvious solution.

-cd

Nov 17 '05 #3
um
"noleander" <no*******@discussions.microsoft.com> wrote
If you have two threads, and they both open a single file for read, and they
use fopen() or open() to open the file, they will share a single seek
pointer. That means that they cannot perform independent read operations.

The only way to get separate seek pointers for the threads is to use sopen()
once for the process, followed by fdopen() for each thread.

I am using the latter technique, but it is failing in some circumstances. I
was wondering if anyone else has tried this and has some lessons-learned.
You must be doing something wrong; fopen (here two seperate FILE*
on the same file) is working fine both sequentially (ie. w/o threads,
but "interleaved"), and also parallel in threads.
But: you must use FILE* as an auto variable, not global and also not on the heap.
"Carl Daniel [VC++ MVP]" wrote:
noleander wrote:
I've got two threads reading from one disk file (I'm using the Pthread
library).

I need each thread to have its own file seek pointer, so they can
read from independent parts of the file.

I tried using _sopen() once (for the entire process) then using
fdopen() in each thread to get unique file seek pointers, but it is
not working well.


What's wrong with just using fopen() twice? That would seem to be the
obvious solution.

-cd


Nov 17 '05 #4
Not true.

If you have two threads in a single process, and they both call fopen() on a
single disk file (even with their own dedicated FILE* pointer) both thread's
FILE* will point to the same FILE (_iobuf) structure.

The only way to get two FILE* pointers is to use open() or sopen() first,
then call fdopen() in each thread with the fileID from open().

But even using this approach, it appears that the two FILE* are sharing a
single underlying seek pointer.
"um" wrote:
"noleander" <no*******@discussions.microsoft.com> wrote
If you have two threads, and they both open a single file for read, and they
use fopen() or open() to open the file, they will share a single seek
pointer. That means that they cannot perform independent read operations.

The only way to get separate seek pointers for the threads is to use sopen()
once for the process, followed by fdopen() for each thread.

I am using the latter technique, but it is failing in some circumstances. I
was wondering if anyone else has tried this and has some lessons-learned.


You must be doing something wrong; fopen (here two seperate FILE*
on the same file) is working fine both sequentially (ie. w/o threads,
but "interleaved"), and also parallel in threads.
But: you must use FILE* as an auto variable, not global and also not on the heap.
"Carl Daniel [VC++ MVP]" wrote:
noleander wrote:
> I've got two threads reading from one disk file (I'm using the Pthread
> library).
>
> I need each thread to have its own file seek pointer, so they can
> read from independent parts of the file.
>
> I tried using _sopen() once (for the entire process) then using
> fdopen() in each thread to get unique file seek pointers, but it is
> not working well.

What's wrong with just using fopen() twice? That would seem to be the
obvious solution.

-cd


Nov 17 '05 #5
um
"noleander" <no*******@discussions.microsoft.com> wrote in message
news:78**********************************@microsof t.com...
Not true.

If you have two threads in a single process, and they both call fopen() on a
single disk file (even with their own dedicated FILE* pointer) both thread's
FILE* will point to the same FILE (_iobuf) structure.

The only way to get two FILE* pointers is to use open() or sopen() first,
then call fdopen() in each thread with the fileID from open().

But even using this approach, it appears that the two FILE* are sharing a
single underlying seek pointer.
Sorry, I still can't believe it.
Best would be to post your test code.

"um" wrote:
"noleander" <no*******@discussions.microsoft.com> wrote
If you have two threads, and they both open a single file for read, and they use fopen() or open() to open the file, they will share a single seek
pointer. That means that they cannot perform independent read operations.

The only way to get separate seek pointers for the threads is to use sopen() once for the process, followed by fdopen() for each thread.

I am using the latter technique, but it is failing in some circumstances. I was wondering if anyone else has tried this and has some lessons-learned.


You must be doing something wrong; fopen (here two seperate FILE*
on the same file) is working fine both sequentially (ie. w/o threads,
but "interleaved"), and also parallel in threads.
But: you must use FILE* as an auto variable, not global and also not on the heap.

"Carl Daniel [VC++ MVP]" wrote:

> noleander wrote:
> > I've got two threads reading from one disk file (I'm using the Pthread
> > library).
> >
> > I need each thread to have its own file seek pointer, so they can
> > read from independent parts of the file.
> >
> > I tried using _sopen() once (for the entire process) then using
> > fdopen() in each thread to get unique file seek pointers, but it is
> > not working well.
>
> What's wrong with just using fopen() twice? That would seem to be the
> obvious solution.
>
> -cd

Nov 17 '05 #6
noleander wrote:
Not true.

If you have two threads in a single process, and they both call fopen() on a
single disk file (even with their own dedicated FILE* pointer) both thread's
FILE* will point to the same FILE (_iobuf) structure.

The only way to get two FILE* pointers is to use open() or sopen() first,
then call fdopen() in each thread with the fileID from open().

But even using this approach, it appears that the two FILE* are sharing a
single underlying seek pointer.


This simple console program is a direct counter example to your claim.
Why do you think your claim is true?

#include <stdio.h>
#include <assert.h>

int main()
{
FILE* in1;
FILE* in2;
char buffer1[512] = {0};
char buffer2[512] = {0};
FILE* out = fopen("test.txt", "wb");
fputs("If this appears twice, file pointers are independent.\n", out);
fclose(out);

in1 = fopen("test.txt", "rb");
in2 = fopen("test.txt", "rb");
assert(in1 != in2);
fgets(buffer1, sizeof buffer1, in1);
fgets(buffer2, sizeof buffer2, in2);
fclose(in1);
fclose(in2);

puts(buffer1);
puts(buffer2);

return 0;
}

Output:
If this appears twice, file pointers are independent.

If this appears twice, file pointers are independent.

Tom
Nov 17 '05 #7
As most of the respondents have already pointed out - you should be quite ok
with plain fopen(). _sopen() is primarily for file sharing (at least this
is what I remember doing in the past) among processes.
Anyway, are you sure that your problems are not related to rendering of data
(e.g. displaying, writing, etc.) instead of reading? Please see the code
below that demonstrates fopen() in a multithreaded environment. The program
first creates a file and then launches a number of threads (specified on a
command line) to read from it. Note that I 'protect' rendering of data being
read by a critical section - this is just to make sure that this look ok on
the screen. You can redirect the output to another file if you like.

Bogdan
#include "stdafx.h"

#include <stdio.h>

#include <stdlib.h>

#include <process.h>

#include <windows.h>

#define LINE_LEN 4

long offsets[] = {0, 4, 8, 12, 16, 20, 24, 28, 32};

const char* fname = "threadfp.txt";

CRITICAL_SECTION cs;

int num_reads = 100;

int createFile(char* fname)

{

FILE* fp = fopen(fname, "wb");

if (NULL == fp) {

printf("Failed to create %s\n", fname);

return errno;

}

for (int i = 0; i < sizeof(offsets) / sizeof(offsets[0]); i++) {

char c = i + '0';

for (int j = 0; j < LINE_LEN; j++) {

fwrite(&c, 1, 1, fp);

}

}

fclose(fp);

return 0;

}

void readFile(void* param)

{

char buf[LINE_LEN];

int i;

FILE* fp = fopen((const char*) param, "rb");

DWORD tid = ::GetCurrentThreadId();

srand((unsigned int) tid);

if (NULL != fp) {

for (int n = 0; n < num_reads; n++) {

i = rand() % (sizeof(offsets) / sizeof(offsets[0]));

if (0 == fseek(fp, offsets[i], SEEK_SET)) {

if (sizeof(buf) == fread(buf, 1, sizeof(buf), fp)) {

EnterCriticalSection(&cs);

printf("[tid=%04X, offset=%2ld]", tid, offsets[i]);

for (int i = 0; i < sizeof(buf); i++) {

printf("%c", buf[i]);

}

printf("\n");

fflush(stdout);

LeaveCriticalSection(&cs);

}

}

::Sleep(500);

}

}

}

int _tmain(int argc, _TCHAR* argv[])

{

if (argc > 2 && 0 == createFile(argv[1])) {

if (argc > 3) {

num_reads = atoi(argv[3]);

}

InitializeCriticalSection(&cs);

int n = atoi(argv[2]);

for (int i = 0; i < n; n++) {

_beginthread(readFile, 0, argv[1]);

}

while (1) {

::Sleep(1000);

}

}

return 0;

}


"noleander" <no*******@discussions.microsoft.com> wrote in message
news:10**********************************@microsof t.com...
I've got two threads reading from one disk file (I'm using the Pthread
library).

I need each thread to have its own file seek pointer, so they can read
from
independent parts of the file.

I tried using _sopen() once (for the entire process) then using fdopen()
in
each thread to get unique file seek pointers, but it is not working well.

Any suggestions?

Nov 17 '05 #8
Yet another comment...
Just remembered that you were using pthread library. Don't know much about
it in a Win32 environment. You have to make sure that stream io functions
in clib (e.g. fopen, fseek, fread, etc.) are thread safe. If you don't use
pthread you need to use _beginthread(ex) instead of CreateThread.

Bogdan

"noleander" <no*******@discussions.microsoft.com> wrote in message
news:ED**********************************@microsof t.com...
If you have two threads, and they both open a single file for read, and
they
use fopen() or open() to open the file, they will share a single seek
pointer. That means that they cannot perform independent read operations.

The only way to get separate seek pointers for the threads is to use
sopen()
once for the process, followed by fdopen() for each thread.

I am using the latter technique, but it is failing in some circumstances.
I
was wondering if anyone else has tried this and has some lessons-learned.

"Carl Daniel [VC++ MVP]" wrote:
noleander wrote:
> I've got two threads reading from one disk file (I'm using the Pthread
> library).
>
> I need each thread to have its own file seek pointer, so they can
> read from independent parts of the file.
>
> I tried using _sopen() once (for the entire process) then using
> fdopen() in each thread to get unique file seek pointers, but it is
> not working well.


What's wrong with just using fopen() twice? That would seem to be the
obvious solution.

-cd

Nov 17 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Eric B. | last post by:
Hi, I'm running into a slight programming issue here that I am hoping someone can help me with. I am developping a web-based PHP application, which means I have to consider multiple threads of...
7
by: Guyon Morée | last post by:
If I have multiple threads reading from the same file, would that be a problem? if yes, how would I solve it? Let's say I want to take it a step further and start writing to 1 file form...
9
by: Arafangion | last post by:
Hello, recently I've been trying to figure out how the heck to just destroy a thread. I have since rewritten the code in question, but what I was trying to do was to use the .Abort() method, but...
6
by: RahimAsif | last post by:
Hi guys, I would like some advice on thread programming using C#. I am writing an application that communicates with a panel over ethernet, collects data and writes it to a file. The way the...
6
by: James Radke | last post by:
Hello, I have a multithreaded windows NT service application (vb.net 2003) that I am working on (my first one), which reads a message queue and creates multiple threads to perform the processing...
7
by: Michael | last post by:
I'm writing an application that decodes a file containing binary records. Each record is a particular event type. Each record is translated into ASCII and then written to a file. Each file contains...
9
by: jdlists | last post by:
I have inheirted some existing code, that i will explain in a moment, have needed to extend and ultimately should be able to run in threads. I've done a bunch of work with python but very little...
16
by: WATYF | last post by:
Hi there... I have a huge text file that needs to be processed. At the moment, I'm loading it into memory in small chunks (x amount of lines) and processing it that way. I'd like the process to be...
4
by: tdahsu | last post by:
All, I'd appreciate any help. I've got a list of files in a directory, and I'd like to iterate through that list and process each one. Rather than do that serially, I was thinking I should...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.