473,320 Members | 1,713 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Comparison issues using fstat and very large files in C++

5
Hey guys,
I have written a C++ program that passes data from a file to an FPGA board and back again using software and DMA buffers.
In my program I need to compare the size of a given file against a software buffer of size 3MB. This is needed so as to see which function to use to read from the file. As the files used range from very large (>30GB) to very small (<3MB), I have enabled large file support and I obtain the file size by using the fstat function.

I make sure that large file support is enabled in the header file
Expand|Select|Wrap|Line Numbers
  1. // Larger file support for files greater then 20 gigabytes
  2. #ifndef _LARGEFILE_SOURCE
  3. #define _LARGEFILE_SOURCE
  4. #endif
  5. #ifndef _LARGEFILE64_SOURCE
  6. #define _LARGEFILE64_SOURCE
  7. #endif
  8.  

The code that I am using to compare the files is this
Expand|Select|Wrap|Line Numbers
  1. #define BUF_SIZE   0x300000
  2. ..
  3. ..
  4. ..
  5. ..
  6. ..
  7. struct stat  statbuf;
  8. fstat(dat_fd, &statbuf); //where dat_fd is the file handler
  9.  
  10. off64_t file_size = (off64_t)statbuf.st_size;
  11. off64_t temp_buf_size = (off64_t) BUF_SIZE; //software buffer
  12. if(temp_buf_size > file_size)
  13. {
  14.      small_flag = true; //ie 1
  15. }
  16. else
  17. {
  18.      small_flag = false; //ie 0
  19. }
  20.  
To verify the workings of this comparison and code section, I write out the information to a debug file using this code and unsigned long long identifiers
Expand|Select|Wrap|Line Numbers
  1. Debug::writeDebug("file comparison results -> BUF_SIZE =%llu file_size =%llu small_flag =%d\n", 
  2. temp_buf_size, file_size, small_flag);
  3.  
This works fine for files in the 1-10MB range but fails completely for larger files in the 2-3GB range. Instead of the file size i expect to be printed, i get nonsensical answers and incorrect comparison results.

Here are the debug outputs:
for file size = 1024000 bytes
Expand|Select|Wrap|Line Numbers
  1. file comparison results -> BUF_SIZE =3145728 file_size =1024000 small_flag =1
  2.  
for file size = 10240000 bytes
Expand|Select|Wrap|Line Numbers
  1. file comparison results -> BUF_SIZE =3145728 file_size =10240000 small_flag =0
  2.  
for file size = 3659983208 bytes
Expand|Select|Wrap|Line Numbers
  1. file comparison results -> BUF_SIZE =3145728 file_size =18446744073074567528  small_flag =1
  2.  
As you can see, my typecasting and comparison fails for large files somewhere. The comparison behaves properly if I use unsigned long (32-bit) instead of unsigned long long (64-bit / off64_t) but some of the files I use are greater than 4GB (the limit of an 32-bit number), thus I can't use it. I've double checked the fstat function and st_size is definitely either an off_t or off64_t, depending on whether large file support has been enabled (assuming that off_t = 32-bits and off64_t = 64-bits)

Does anyone have any idea why this simple comparison is not working?

Thanks
Lars
May 8 '07 #1
1 6262
Lars B
5
After much searching through the code and a lot of debug statements, I have finally solved this problem. It turned out that the size of the value returned by off_t via the fstat function was only 32-bits, not the expected 64-bits as it would be if 64-bit support was enabled.

I checked through my makefile and it seems to show that it is being enabled correctly

Expand|Select|Wrap|Line Numbers
  1. CFLAGS  = -D_FILE_OFFSET_BITS=64 -Wall -Iadmxrc2 -Iadmxrc2/common -Iinclude
  2.  
  3. LDFLAGS =  -L. $(LIBPATH)
  4.  
  5. SRCS =    Debug.cpp \
  6.                 FileReader.cpp \
  7.                 Monitor.cpp \
  8.                 Control.cpp \
  9.                 Main.cpp
  10.  
  11. all: main
  12.  
  13. main: libcommon.a libnet.a $(SRCS:.cpp=.o)
  14.  
  15.       $(CXX) $(LDFLAGS) $(CFLAGS) $(SRCS:.cpp=.o) -lccgnu2 -ldl -ladmxrc2  -o fpc_main -L"./lib" -lcommon -lnet
  16. libnet.a:
  17.         make -C ./lib/net -f net.mak
  18.  
  19. libcommon.a: args.o time.o
  20.         ar r ./lib/libcommon.a args.o time.o
  21.  
  22. args.o: admxrc2/common/args.c admxrc2/common/common.h
  23.         $(CC) -c $(CFLAGS) -o $@ admxrc2/common/args.c
  24.  
  25. time.o: admxrc2/common/time.c admxrc2/common/common.h
  26.         $(CC) -c $(CFLAGS) -o $@ admxrc2/common/time.c
  27.  
  28. clean:
  29.         rm -f $(TARGET) *.o *.a
  30.         make -C ./lib/net -f net.mak clean
  31.  
  32.  
But upon a clean compilation of the files, I noticed that 64-bit support was enabled for the C functions but not the C++ functions

[code]
gcc -c -D_FILE_OFFSET_BITS=64 -Wall -Iadmxrc2 -Iadmxrc2/common -Iinclude -o args.o admxrc2/common/args.c
gcc -c -D_FILE_OFFSET_BITS=64 -Wall -Iadmxrc2 -Iadmxrc2/common -Iinclude -o time.o admxrc2/common/time.c
ar r ./lib/libcommon.a args.o time.o
make -C ./lib/net -f net.mak
make[1]: Entering directory `/home/lib/net'
cc -c -o net_endpt.o net_endpt.c
cc -c -o net_error.o net_error.c
cc -c -o net_io.o net_io.c
cc -c -o net_tcp.o net_tcp.c
ar rv ../libnet.a net_endpt.o net_error.o net_io.o net_tcp.o
ar: creating ../libnet.a
a - net_endpt.o
a - net_error.o
a - net_io.o
a - net_tcp.o
make[1]: Leaving directory `/home/lib/net'
g++ -c -o Debug.o Debug.cpp
g++ -c -o FileReader.o FileReader.cpp
g++ -c -o Monitor.o Monitor.cpp
g++ -c -o Control.o Control.cpp
g++ -c -o Main.o Main.cpp
[\CODE]

To rectify this, I specified that the offset to be used was 64-bits in the FileReader.h file, thus making certain that all applicable functions called would return 64-bit values, not 32-bits.

Expand|Select|Wrap|Line Numbers
  1. #ifndef FILEREADER_H_
  2. #define FILEREADER_H_
  3.  
  4. // Larger file support for files greater then 20 gigabytes
  5. #ifndef _LARGEFILE_SOURCE
  6. #define _LARGEFILE_SOURCE
  7. #endif
  8. #ifndef _LARGEFILE64_SOURCE
  9. #define _LARGEFILE64_SOURCE
  10. #endif
  11.  
  12. #define _FILE_OFFSET_BITS   64 //definition of offset size
  13.  
  14. #include <stdlib.h>
  15. #include <unistd.h>
  16. #include <features.h>
  17. #include <ctype.h>
  18. #include <cc++/thread.h>
  19.  
This fixed the problem immediately. But I feel that this solution is inelegant, as I would prefer that the Makefile handles the large file support for bith the C files and the C++ files. How does one ensure that the file_offset is 64 for the C++ files as well as the C files in compilation? What must I include in the Makefile so that all the C++ files use large file support?

Thanks, Lars
May 10 '07 #2

Sign in to post your reply or Sign up for a free account.

Similar topics

5
by: democratix | last post by:
Hi, I've only got a couple years experience developing for Access but have recently been experimenting with HTML/javascript for gui and client-side scripting, mysql for database and php for...
5
by: mayamorning123 | last post by:
A comparison among six VSS remote tools including SourceOffSite , SourceAnyWhere, VSS Connect, SourceXT, VSS Remoting, VSS.NET To view the full article, please visit...
2
by: Martijn | last post by:
Hi, I have an open file handle from which I would like to retrieve the file descriptor for use with fstat. Is this possible or should I redesign my algorithm a bit (I have the feeling I am...
20
by: Daniel Billingsley | last post by:
I read this fantastic article this morning, which raises some interesting points... like, for example, the TREMENDOUS performance advantage of writing code with using() as opposed to just letting...
7
by: bcutting | last post by:
I am looking for a way to take a large number of images and find matches among them. These images may not be exact replicas. Images may have been resized, cropped, faded, color corrected, etc. ...
0
by: metaperl | last post by:
A Comparison of Python Class Objects and Init Files for Program Configuration ============================================================================= Terrence Brannon bauhaus@metaperl.com...
1
by: ansc1 | last post by:
Hello, I'm new to using php coding. I need help with the following: 1. There is a submit button on the form and is saves information to my database. After clicking on "Save Measurement" it...
16
by: yu_kuo | last post by:
Is there any comparison data on perfomance difference between std::string and c style string? Or maybe if there are source code which could be used to measuer on different compiler/platform, in a...
11
by: solarisss | last post by:
I have thousands of files whose exitence needs to be checked. I think fstat is too costly for this. Is there any better way for the same ?
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.