Comparison issues using fstat and very large files in C++

[Lars B]

Hey guys,
I have written a C++ program that passes data from a file to an FPGA board and back again using software and DMA buffers.
In my program I need to compare the size of a given file against a software buffer of size 3MB. This is needed so as to see which function to use to read from the file. As the files used range from very large (>30GB) to very small (<3MB), I have enabled large file support and I obtain the file size by using the fstat function.

I make sure that large file support is enabled in the header file

// Larger file support for files greater then 20 gigabytes
#ifndef _LARGEFILE_SOURCE
#define _LARGEFILE_SOURCE
#endif
#ifndef _LARGEFILE64_SOURCE
#define _LARGEFILE64_SOURCE
#endif
 

The code that I am using to compare the files is this

#define BUF_SIZE   0x300000
..
..
..
..
..
struct stat  statbuf;
fstat(dat_fd, &statbuf); //where dat_fd is the file handler
 
off64_t file_size = (off64_t)statbuf.st_size;
off64_t temp_buf_size = (off64_t) BUF_SIZE; //software buffer
if(temp_buf_size > file_size)
{
     small_flag = true; //ie 1
}
else
{
     small_flag = false; //ie 0
}
 

To verify the workings of this comparison and code section, I write out the information to a debug file using this code and unsigned long long identifiers

Debug::writeDebug("file comparison results -> BUF_SIZE =%llu file_size =%llu small_flag =%d\n", 
temp_buf_size, file_size, small_flag);
 

This works fine for files in the 1-10MB range but fails completely for larger files in the 2-3GB range. Instead of the file size i expect to be printed, i get nonsensical answers and incorrect comparison results.

Here are the debug outputs:
for file size = 1024000 bytes

file comparison results -> BUF_SIZE =3145728 file_size =1024000 small_flag =1
 

for file size = 10240000 bytes

file comparison results -> BUF_SIZE =3145728 file_size =10240000 small_flag =0
 

for file size = 3659983208 bytes

file comparison results -> BUF_SIZE =3145728 file_size =18446744073074567528  small_flag =1
 

As you can see, my typecasting and comparison fails for large files somewhere. The comparison behaves properly if I use unsigned long (32-bit) instead of unsigned long long (64-bit / off64_t) but some of the files I use are greater than 4GB (the limit of an 32-bit number), thus I can't use it. I've double checked the fstat function and st_size is definitely either an off_t or off64_t, depending on whether large file support has been enabled (assuming that off_t = 32-bits and off64_t = 64-bits)

Does anyone have any idea why this simple comparison is not working?

Thanks
Lars

[Lars B]

After much searching through the code and a lot of debug statements, I have finally solved this problem. It turned out that the size of the value returned by off_t via the fstat function was only 32-bits, not the expected 64-bits as it would be if 64-bit support was enabled.

I checked through my makefile and it seems to show that it is being enabled correctly

CFLAGS  = -D_FILE_OFFSET_BITS=64 -Wall -Iadmxrc2 -Iadmxrc2/common -Iinclude
 
LDFLAGS =  -L. $(LIBPATH)
 
SRCS =    Debug.cpp \
                FileReader.cpp \
                Monitor.cpp \
                Control.cpp \
                Main.cpp
 
all: main
 
main: libcommon.a libnet.a $(SRCS:.cpp=.o)
 
      $(CXX) $(LDFLAGS) $(CFLAGS) $(SRCS:.cpp=.o) -lccgnu2 -ldl -ladmxrc2  -o fpc_main -L"./lib" -lcommon -lnet
libnet.a:
        make -C ./lib/net -f net.mak
 
libcommon.a: args.o time.o
        ar r ./lib/libcommon.a args.o time.o
 
args.o: admxrc2/common/args.c admxrc2/common/common.h
        $(CC) -c $(CFLAGS) -o $@ admxrc2/common/args.c
 
time.o: admxrc2/common/time.c admxrc2/common/common.h
        $(CC) -c $(CFLAGS) -o $@ admxrc2/common/time.c
 
clean:
        rm -f $(TARGET) *.o *.a
        make -C ./lib/net -f net.mak clean
 
 

But upon a clean compilation of the files, I noticed that 64-bit support was enabled for the C functions but not the C++ functions

[code]
gcc -c -D_FILE_OFFSET_B ITS=64 -Wall -Iadmxrc2 -Iadmxrc2/common -Iinclude -o args.o admxrc2/common/args.c
gcc -c -D_FILE_OFFSET_B ITS=64 -Wall -Iadmxrc2 -Iadmxrc2/common -Iinclude -o time.o admxrc2/common/time.c
ar r ./lib/libcommon.a args.o time.o
make -C ./lib/net -f net.mak
make[1]: Entering directory `/home/lib/net'
cc -c -o net_endpt.o net_endpt.c
cc -c -o net_error.o net_error.c
cc -c -o net_io.o net_io.c
cc -c -o net_tcp.o net_tcp.c
ar rv ../libnet.a net_endpt.o net_error.o net_io.o net_tcp.o
ar: creating ../libnet.a
a - net_endpt.o
a - net_error.o
a - net_io.o
a - net_tcp.o
make[1]: Leaving directory `/home/lib/net'
g++ -c -o Debug.o Debug.cpp
g++ -c -o FileReader.o FileReader.cpp
g++ -c -o Monitor.o Monitor.cpp
g++ -c -o Control.o Control.cpp
g++ -c -o Main.o Main.cpp
[\CODE]

To rectify this, I specified that the offset to be used was 64-bits in the FileReader.h file, thus making certain that all applicable functions called would return 64-bit values, not 32-bits.

#ifndef FILEREADER_H_
#define FILEREADER_H_
 
// Larger file support for files greater then 20 gigabytes
#ifndef _LARGEFILE_SOURCE
#define _LARGEFILE_SOURCE
#endif
#ifndef _LARGEFILE64_SOURCE
#define _LARGEFILE64_SOURCE
#endif
 
#define _FILE_OFFSET_BITS   64 //definition of offset size
 
#include <stdlib.h>
#include <unistd.h>
#include <features.h>
#include <ctype.h>
#include <cc++/thread.h>
 

This fixed the problem immediately. But I feel that this solution is inelegant, as I would prefer that the Makefile handles the large file support for bith the C files and the C++ files. How does one ensure that the file_offset is 64 for the C++ files as well as the C files in compilation? What must I include in the Makefile so that all the C++ files use large file support?

Thanks, Lars