473,320 Members | 1,914 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

File Size Limit Exceeded - How to handle work around this?

I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.

Is this a compiler issue? I would like to see a C code example of how
to increase the limit or make it unlimited (if that is a wise thing to
do).

Thanks in advance!

Dec 13 '06 #1
9 11279
In article <11*********************@79g2000cws.googlegroups.c om>,
eastcoastguyz <ea***********@hotmail.comwrote:
>I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.
>Is this a compiler issue?
Probably not.
>I would like to see a C code example of how
to increase the limit or make it unlimited (if that is a wise thing to
do).
It is probably an operating system limitation (or a disk quota
limitation). ulimit (which is not part of C) with -a set to
unlimited, just means that the operating system will allow you to
write files as big as is supported by that particular file system.

Whether your operating system supports larger files at all would
be OS specific, as would be any special means to create such files.
This is a matter that should be taken to a resource that deals
with your specific OS. It is -possible- that they will tell you
there to change some flags to your compiles, but that would be
for deep OS implementation reasons, not for reasons directly related
to standard C.
--
If you lie to the compiler, it will get its revenge. -- Henry Spencer
Dec 14 '06 #2

eastcoastguyz wrote:
I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.

Is this a compiler issue? I would like to see a C code example of how
to increase the limit or make it unlimited (if that is a wise thing to
do).

Thanks in advance!
Write the file in blocks and maintain your own chain or use a database,
which will already have done that

Dec 14 '06 #3
I forgot to mention, the OS is Linux with the latest version of CentOS
using 'cc'.

eastcoastguyz wrote:
I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.

Is this a compiler issue? I would like to see a C code example of how
to increase the limit or make it unlimited (if that is a wise thing to
do).

Thanks in advance!
Dec 14 '06 #4
I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.

Is this a compiler issue? I would like to see a C code example of how
to increase the limit or make it unlimited (if that is a wise thing to
do).
It's been a long time, though, so this advice may be dated.

A long time ago, when the Linux 2.4 kernel was new (i.e., when 64-bit
file sizes were just starting to catch on), we used to have this same
problem. Back then, you had to compile with some special symbol
defined, LARGE_FILES or something like that. That caused the typical
file size types (offset_t and the like) to be 64 bit. You might find
something like that in the compiler documentation.

Of course, to narrow it to OS or C, you could have your program write
to stdout and redirect to a file. If it still doesn't work, you have
an OS problem. And if it isn't C, it could be your OS kernel, or your
shell, or your filesystem (e.g., NFS) that has some kind of limit in
it.

Michael

Dec 14 '06 #5

Michael wrote:
I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.

Is this a compiler issue? I would like to see a C code example of how
to increase the limit or make it unlimited (if that is a wise thing to
do).

It's been a long time, though, so this advice may be dated.

A long time ago, when the Linux 2.4 kernel was new (i.e., when 64-bit
file sizes were just starting to catch on), we used to have this same
problem. Back then, you had to compile with some special symbol
defined, LARGE_FILES or something like that. That caused the typical
file size types (offset_t and the like) to be 64 bit. You might find
something like that in the compiler documentation.

Of course, to narrow it to OS or C, you could have your program write
to stdout and redirect to a file. If it still doesn't work, you have
an OS problem. And if it isn't C, it could be your OS kernel, or your
shell, or your filesystem (e.g., NFS) that has some kind of limit in
it.

Michael
Thanks for the suggestion. I made another version of this C program,
where it writes to standard output and piped the output to a file. It
had no problem creating a file size over 3GB, exceeding the 2GB limit I
encountered.
This system is using gcc version 3.4.6 20060404 (Red Hat 3.4.6-3) which
is what came with the latest version of CentOS. The kernel is:
[root@localhost proc]# cat /proc/version
Linux version 2.6.9-42.0.3.EL (buildsvn@build-i386) (gcc version 3.4.6
20060404

Dec 14 '06 #6
Hello,
I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.
Try to compile your program with the following flags:
-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64

HTH,
Loic.

Dec 14 '06 #7


On Wed, 13 Dec 2006, lo******@gmx.net wrote:
Hello,
>>>I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.

Try to compile your program with the following flags:
-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64
Probably he meant:

-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
Dec 14 '06 #8

Kohn Emil Dan wrote:
On Wed, 13 Dec 2006, lo******@gmx.net wrote:
Hello,
>>I wrote a simple program to continue to create a very large file (on
purpose), and even though there is plenty of disk space on that device
the program aborted with the error message "File Size Limit Exceeded".
The file size was 2147483647. I checked ulimit -a and its set to
unlimited.
Try to compile your program with the following flags:
-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64

Probably he meant:

-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
Yes! This worked. Thanks to everyone who posted in response to my
request for help.

Does anyone know if it is required to do these compiler options
regardless of the hardware the same OS runs on? Thanks!

Dec 14 '06 #9
eastcoastguyz wrote:
Kohn Emil Dan wrote:
>On Wed, 13 Dec 2006, lo******@gmx.net wrote:
>>Hello,

>I wrote a simple program to continue to create a very large file (on
>purpose), and even though there is plenty of disk space on that device
>the program aborted with the error message "File Size Limit Exceeded".
>The file size was 2147483647. I checked ulimit -a and its set to
>unlimited.
Try to compile your program with the following flags:
-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64
Probably he meant:

-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

Yes! This worked. Thanks to everyone who posted in response to my
request for help.

Does anyone know if it is required to do these compiler options
regardless of the hardware the same OS runs on? Thanks!
<OT>
I think the only macro you need defined is _GNU_SOURCE which will
enable a number of macros and the ...LARGEFILE64... ones are
among them. I think this is the preferred way to do this. I
couldn't find _GNU_SOURCE in my header files on 64 bit Gentoo
</OT>
but it may be because I didn't pay too much attention. You
shouldn't have posted the question here and you should not
consider advices received from this group. The standard C doesn't
know about 64 vs 32 bit, files larger than 2 GB or specific
extensions. This is a question for a group that deals with the
combination of compiler/OS that you're using.

--
Ioan - Ciprian Tandau
tandau _at_ freeshell _dot_ org (hope it's not too late)
(... and that it still works...)
Dec 14 '06 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Michael Ferrier | last post by:
Hello, I have a Php program made up of a small script "A.php" that includes a large script "B.php". This program runs fine on dozens of web servers, but fails with no error message on one...
1
by: guru.slt | last post by:
my c program fails to write file that is bigger than 2G bytes. I used gcc compiler. After writing for a while, It says: "filesize limit exceed". How to solve this? But, if I compile the c code...
2
by: steve | last post by:
I am setting up a huge database in mysql, and I get the above error in Linux. I believe it is related to the size of one of my tables, which is 4,294,966,772 bytes in size. Can someone help. How...
0
by: Daniel | last post by:
Is this a bug in .net file io? is there a work around? when writing a large file with using(StreamWriter ... if the network drive of the share is removed then the file is never closed. I tried...
8
by: Peter Ballard | last post by:
Hi all, I've got a C program which outputs all its data using a statement of the form: putchar(ch, outfile); This has worked fine for years until it had to output more than 2GB of data...
1
by: Amod | last post by:
Hi, I m copying data from a video stream on the hard disk using a C++ programme in win32 API. my current file system is FAT32 .. the file size limit of FAT32 is 4GB. I want to automatically split...
1
by: Amod | last post by:
Hi, I m copying data from a video stream on the hard disk using a C++ programme in win32 API. my current file system is FAT32 .. the file size limit of FAT32 is 4GB. I want to automatically split...
5
by: bobh | last post by:
HI, I understand AccessXP file size has a 2 gig limit but I trying to understand the following I have a tab delimited data file that is 1.3 gigs big and I try to import it into an empty...
0
by: gdetre | last post by:
Dear all, I'm trying to get a large, machine-generated regular expression (many thousands of characters) to work in Python on a Mac (running Leopard), and I keep banging my head against this...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.