By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,161 Members | 2,002 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,161 IT Pros & Developers. It's quick & easy.

Comparison issues using fstat and very large files in C++

P: 5
Hey guys,
I have written a C++ program that passes data from a file to an FPGA board and back again using software and DMA buffers.
In my program I need to compare the size of a given file against a software buffer of size 3MB. This is needed so as to see which function to use to read from the file. As the files used range from very large (>30GB) to very small (<3MB), I have enabled large file support and I obtain the file size by using the fstat function.

I make sure that large file support is enabled in the header file
Expand|Select|Wrap|Line Numbers
  1. // Larger file support for files greater then 20 gigabytes
  2. #ifndef _LARGEFILE_SOURCE
  3. #define _LARGEFILE_SOURCE
  4. #endif
  5. #ifndef _LARGEFILE64_SOURCE
  6. #define _LARGEFILE64_SOURCE
  7. #endif
  8.  

The code that I am using to compare the files is this
Expand|Select|Wrap|Line Numbers
  1. #define BUF_SIZE   0x300000
  2. ..
  3. ..
  4. ..
  5. ..
  6. ..
  7. struct stat  statbuf;
  8. fstat(dat_fd, &statbuf); //where dat_fd is the file handler
  9.  
  10. off64_t file_size = (off64_t)statbuf.st_size;
  11. off64_t temp_buf_size = (off64_t) BUF_SIZE; //software buffer
  12. if(temp_buf_size > file_size)
  13. {
  14.      small_flag = true; //ie 1
  15. }
  16. else
  17. {
  18.      small_flag = false; //ie 0
  19. }
  20.  
To verify the workings of this comparison and code section, I write out the information to a debug file using this code and unsigned long long identifiers
Expand|Select|Wrap|Line Numbers
  1. Debug::writeDebug("file comparison results -> BUF_SIZE =%llu file_size =%llu small_flag =%d\n", 
  2. temp_buf_size, file_size, small_flag);
  3.  
This works fine for files in the 1-10MB range but fails completely for larger files in the 2-3GB range. Instead of the file size i expect to be printed, i get nonsensical answers and incorrect comparison results.

Here are the debug outputs:
for file size = 1024000 bytes
Expand|Select|Wrap|Line Numbers
  1. file comparison results -> BUF_SIZE =3145728 file_size =1024000 small_flag =1
  2.  
for file size = 10240000 bytes
Expand|Select|Wrap|Line Numbers
  1. file comparison results -> BUF_SIZE =3145728 file_size =10240000 small_flag =0
  2.  
for file size = 3659983208 bytes
Expand|Select|Wrap|Line Numbers
  1. file comparison results -> BUF_SIZE =3145728 file_size =18446744073074567528  small_flag =1
  2.  
As you can see, my typecasting and comparison fails for large files somewhere. The comparison behaves properly if I use unsigned long (32-bit) instead of unsigned long long (64-bit / off64_t) but some of the files I use are greater than 4GB (the limit of an 32-bit number), thus I can't use it. I've double checked the fstat function and st_size is definitely either an off_t or off64_t, depending on whether large file support has been enabled (assuming that off_t = 32-bits and off64_t = 64-bits)

Does anyone have any idea why this simple comparison is not working?

Thanks
Lars
May 8 '07 #1
Share this Question
Share on Google+
1 Reply


P: 5
After much searching through the code and a lot of debug statements, I have finally solved this problem. It turned out that the size of the value returned by off_t via the fstat function was only 32-bits, not the expected 64-bits as it would be if 64-bit support was enabled.

I checked through my makefile and it seems to show that it is being enabled correctly

Expand|Select|Wrap|Line Numbers
  1. CFLAGS  = -D_FILE_OFFSET_BITS=64 -Wall -Iadmxrc2 -Iadmxrc2/common -Iinclude
  2.  
  3. LDFLAGS =  -L. $(LIBPATH)
  4.  
  5. SRCS =    Debug.cpp \
  6.                 FileReader.cpp \
  7.                 Monitor.cpp \
  8.                 Control.cpp \
  9.                 Main.cpp
  10.  
  11. all: main
  12.  
  13. main: libcommon.a libnet.a $(SRCS:.cpp=.o)
  14.  
  15.       $(CXX) $(LDFLAGS) $(CFLAGS) $(SRCS:.cpp=.o) -lccgnu2 -ldl -ladmxrc2  -o fpc_main -L"./lib" -lcommon -lnet
  16. libnet.a:
  17.         make -C ./lib/net -f net.mak
  18.  
  19. libcommon.a: args.o time.o
  20.         ar r ./lib/libcommon.a args.o time.o
  21.  
  22. args.o: admxrc2/common/args.c admxrc2/common/common.h
  23.         $(CC) -c $(CFLAGS) -o $@ admxrc2/common/args.c
  24.  
  25. time.o: admxrc2/common/time.c admxrc2/common/common.h
  26.         $(CC) -c $(CFLAGS) -o $@ admxrc2/common/time.c
  27.  
  28. clean:
  29.         rm -f $(TARGET) *.o *.a
  30.         make -C ./lib/net -f net.mak clean
  31.  
  32.  
But upon a clean compilation of the files, I noticed that 64-bit support was enabled for the C functions but not the C++ functions

[code]
gcc -c -D_FILE_OFFSET_BITS=64 -Wall -Iadmxrc2 -Iadmxrc2/common -Iinclude -o args.o admxrc2/common/args.c
gcc -c -D_FILE_OFFSET_BITS=64 -Wall -Iadmxrc2 -Iadmxrc2/common -Iinclude -o time.o admxrc2/common/time.c
ar r ./lib/libcommon.a args.o time.o
make -C ./lib/net -f net.mak
make[1]: Entering directory `/home/lib/net'
cc -c -o net_endpt.o net_endpt.c
cc -c -o net_error.o net_error.c
cc -c -o net_io.o net_io.c
cc -c -o net_tcp.o net_tcp.c
ar rv ../libnet.a net_endpt.o net_error.o net_io.o net_tcp.o
ar: creating ../libnet.a
a - net_endpt.o
a - net_error.o
a - net_io.o
a - net_tcp.o
make[1]: Leaving directory `/home/lib/net'
g++ -c -o Debug.o Debug.cpp
g++ -c -o FileReader.o FileReader.cpp
g++ -c -o Monitor.o Monitor.cpp
g++ -c -o Control.o Control.cpp
g++ -c -o Main.o Main.cpp
[\CODE]

To rectify this, I specified that the offset to be used was 64-bits in the FileReader.h file, thus making certain that all applicable functions called would return 64-bit values, not 32-bits.

Expand|Select|Wrap|Line Numbers
  1. #ifndef FILEREADER_H_
  2. #define FILEREADER_H_
  3.  
  4. // Larger file support for files greater then 20 gigabytes
  5. #ifndef _LARGEFILE_SOURCE
  6. #define _LARGEFILE_SOURCE
  7. #endif
  8. #ifndef _LARGEFILE64_SOURCE
  9. #define _LARGEFILE64_SOURCE
  10. #endif
  11.  
  12. #define _FILE_OFFSET_BITS   64 //definition of offset size
  13.  
  14. #include <stdlib.h>
  15. #include <unistd.h>
  16. #include <features.h>
  17. #include <ctype.h>
  18. #include <cc++/thread.h>
  19.  
This fixed the problem immediately. But I feel that this solution is inelegant, as I would prefer that the Makefile handles the large file support for bith the C files and the C++ files. How does one ensure that the file_offset is 64 for the C++ files as well as the C files in compilation? What must I include in the Makefile so that all the C++ files use large file support?

Thanks, Lars
May 10 '07 #2

Post your reply

Sign in to post your reply or Sign up for a free account.