473,320 Members | 2,112 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

file size limit exceeded

Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

Just wondering.

Regards,

Peter Ballard
Adelaide, AUSTRALIA
pb******@ozemail.com.au
http://www.ozemail.com.au/~pballard/
Nov 13 '05 #1
8 18716
On 30 Jul 2003 20:37:23 -0700, pb******@ozemail.com.au (Peter Ballard)
wrote in comp.lang.c:
Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

Just wondering.


Please use a proper signature block, like mine, that begins with a
line with the three characters "-- ", so proper newsreaders will
automatically snip it when replying.

As to file size limitations, they have nothing at all to do with the C
standard and everything to do with your compiler and/or operating
system. You need to ask in a group like
news:comp.os.linux.development.apps.

--
Jack Klein
Home: http://JK-Technology.Com
FAQs for
comp.lang.c http://www.eskimo.com/~scs/C-faq/top.html
comp.lang.c++ http://www.parashift.com/c++-faq-lite/
alt.comp.lang.learn.c-c++ ftp://snurse-l.org/pub/acllc-c++/faq
Nov 13 '05 #2

"Peter Ballard" <pb******@ozemail.com.au> wrote in message
news:9d*************************@posting.google.co m...
Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here?


(snip)

The C library routines allow fseek() calls to be made on the file. The
system doesn't know that you aren't doing any fseek().

The position argument to fseek() is an int. To prevent problems that could
occur, the system limits file size.

-- glen
Nov 13 '05 #3
In <zv2Wa.16407$Oz4.6184@rwcrnsc54> "Glen Herrmannsfeldt" <ga*@ugcs.caltech.edu> writes:

"Peter Ballard" <pb******@ozemail.com.au> wrote in message
news:9d*************************@posting.google.c om...
Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here?
(snip)

The C library routines allow fseek() calls to be made on the file. The
system doesn't know that you aren't doing any fseek().


So what? fseek() is allowed to fail.
The position argument to fseek() is an int. To prevent problems that could
occur, the system limits file size.


Nonsense! That's what fsetpos() is for.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de
Nov 13 '05 #4
In <9d*************************@posting.google.com> pb******@ozemail.com.au (Peter Ballard) writes:
I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?


Just a guess. Your system has large file support, but it is not enabled
by default. When fopen() calls open(), it doesn't specify whatever magic
is necessary for enabling the large file support. When bash opens a file,
it does.

The magic is actually documented in the open(2) man page.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de
Nov 13 '05 #5
In <vi************@corp.supernews.com> Derk Gwen <de******@HotPOP.com> writes:
# Now since it works, it's not really urgent. But I'm curious: what's
# going on here? I've read that the "file size limit exceeded" error is
# due to my OS (which is Linux, 2.4 I think), but then why should the OS
# care whether the file is opened and closed inside my C program (when
# it didn't work) or in bash (when it did work)? Is the problem due to
# some pointer size limit in my machine's implementation of C (the
# program was compiled using gcc)? Or something else?

Are you sure it's the kernel or stdio? You can bypass stdio and use
open and write calls to check if it also fails at 2GB.


How do you suppose stdio is actually performing its I/O?

Next time, engage your brain before posting.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de
Nov 13 '05 #6
In article <9d*************************@posting.google.com> ,
pb******@ozemail.com.au says...
Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?


That's OT for this newsgroup (comp.os.linux.development.* for example)
however, a google search for "LFS" "Large file summit", 2GB, etc.
will probably lead you to the right answer as well.

Nov 13 '05 #7
Randy Howard <ra**********@FOOmegapathdslBAR.net> wrote in message news:<MP************************@news.megapathdsl. net>...
In article <9d*************************@posting.google.com> ,
pb******@ozemail.com.au says...
Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?
That's OT for this newsgroup (comp.os.linux.development.* for example)


Probably. But in my defence, my problem was unique to my C program.
(bash was fine making >2GB; as were test programs in two other
languages which I tried since my 1st post).
however, a google search for "LFS" "Large file summit", 2GB, etc.
will probably lead you to the right answer as well.


Thanks to all who responded. This URL seems to cover the issue fairly
well:

http://www.suse.de/~aj/linux_lfs.html

--
Regards,

Peter Ballard
Adelaide, AUSTRALIA
pb******@ozemail.com.au
http://www.ozemail.com.au/~pballard/
Nov 13 '05 #8
In article <9d**************************@posting.google.com >,
pb******@ozemail.com.au says...
That's OT for this newsgroup (comp.os.linux.development.* for example)


Probably. But in my defence, my problem was unique to my C program.


It could have also happened with any language using 32bit file offsets.
Still doesn't make it a c.l.c. issue. :-)
however, a google search for "LFS" "Large file summit", 2GB, etc.
will probably lead you to the right answer as well.


Thanks to all who responded. This URL seems to cover the issue fairly
well:

http://www.suse.de/~aj/linux_lfs.html


Indeed. With the correct platform-dependant invocations, you can make
Linux C programs generate (or interact with) files over several terabytes.
Again, that's not really the c.l.c. problem domain.

Nov 13 '05 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Don | last post by:
Hi all, I need to read an uploaded file into chunks, so I can examine the acculative size between chunks to determine if a maximum size is being exceeded. I'm currently using the PHP function...
2
by: steve | last post by:
I am setting up a huge database in mysql, and I get the above error in Linux. I believe it is related to the size of one of my tables, which is 4,294,966,772 bytes in size. Can someone help. How...
9
by: Jenny | last post by:
Hi - I've inherited responsibility for our church's website. It has a sizeable main menu, which uses mouseover image swaps. These images use a pre-load script but the script is only included in the...
6
by: Tom Kaminski [MVP] | last post by:
Is there anyway to gracefully catch a file upload limit error? When the limit is exceeded, IE displays a generic "Cannot find server or DNS error" message. -- Tom Kaminski IIS MVP...
6
by: tshad | last post by:
I have an upload file input as: <input id="MyFile" style="width:300px" type="File" runat="Server"> This works fine, but I find that if my page doesn't pass validation during postback, the page...
13
by: Prabhat | last post by:
Hi Friends, I have code that will "BinaryWrite" a EXE file of 20mb to client using the ADO Stream. After the download the EXE file works fine without any problem. When I verified the EXE that...
9
by: eastcoastguyz | last post by:
I wrote a simple program to continue to create a very large file (on purpose), and even though there is plenty of disk space on that device the program aborted with the error message "File Size...
6
Jacotheron
by: Jacotheron | last post by:
I need a PHP script that can upload music files (mp3). The script is for a home project I have started a while ago. I have a MySQL database of all the music that I have. Other computers on the...
1
KevinADC
by: KevinADC | last post by:
Note: You may skip to the end of the article if all you want is the perl code. Introduction Many websites have a form or a link you can use to download a file. You click a form button or click...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.