file size limit exceeded

[Peter Ballard]

Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

Just wondering.

Regards,

Peter Ballard
Adelaide, AUSTRALIA
pb******@ozemai l.com.au
http://www.ozemail.com.au/~pballard/

[Jack Klein]

On 30 Jul 2003 20:37:23 -0700, pb******@ozemai l.com.au (Peter Ballard)
wrote in comp.lang.c:

Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

Just wondering.

Please use a proper signature block, like mine, that begins with a
line with the three characters "-- ", so proper newsreaders will
automatically snip it when replying.

As to file size limitations, they have nothing at all to do with the C
standard and everything to do with your compiler and/or operating
system. You need to ask in a group like
news:comp.os.li nux.development .apps.

--
Jack Klein
Home: http://JK-Technology.Com
FAQs for
comp.lang.c http://www.eskimo.com/~scs/C-faq/top.html
comp.lang.c++ http://www.parashift.com/c++-faq-lite/
alt.comp.lang.l earn.c-c++ ftp://snurse-l.org/pub/acllc-c++/faq

[Glen Herrmannsfeldt]

"Peter Ballard" <pb******@ozema il.com.au> wrote in message
news:9d******** *************** **@posting.goog le.com...

Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here?

(snip)

The C library routines allow fseek() calls to be made on the file. The
system doesn't know that you aren't doing any fseek().

The position argument to fseek() is an int. To prevent problems that could
occur, the system limits file size.

-- glen

[Dan Pop]

In <zv2Wa.16407$Oz 4.6184@rwcrnsc5 4> "Glen Herrmannsfeldt" <ga*@ugcs.calte ch.edu> writes:

"Peter Ballard" <pb******@ozema il.com.au> wrote in message
news:9d******* *************** ***@posting.goo gle.com...

Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here?
(snip)

The C library routines allow fseek() calls to be made on the file. The
system doesn't know that you aren't doing any fseek().

So what? fseek() is allowed to fail.
The position argument to fseek() is an int. To prevent problems that could
occur, the system limits file size.

Nonsense! That's what fsetpos() is for.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de

[Dan Pop]

In <9d************ *************@p osting.google.c om> pb******@ozemai l.com.au (Peter Ballard) writes:

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

Just a guess. Your system has large file support, but it is not enabled
by default. When fopen() calls open(), it doesn't specify whatever magic
is necessary for enabling the large file support. When bash opens a file,
it does.

The magic is actually documented in the open(2) man page.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de

[Dan Pop]

In <vi************ @corp.supernews .com> Derk Gwen <de******@HotPO P.com> writes:

# Now since it works, it's not really urgent. But I'm curious: what's
# going on here? I've read that the "file size limit exceeded" error is
# due to my OS (which is Linux, 2.4 I think), but then why should the OS
# care whether the file is opened and closed inside my C program (when
# it didn't work) or in bash (when it did work)? Is the problem due to
# some pointer size limit in my machine's implementation of C (the
# program was compiled using gcc)? Or something else?

Are you sure it's the kernel or stdio? You can bypass stdio and use
open and write calls to check if it also fails at 2GB.

How do you suppose stdio is actually performing its I/O?

Next time, engage your brain before posting.

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Da*****@ifh.de

[Randy Howard]

In article <9d************ *************@p osting.google.c om>,
pb******@ozemai l.com.au says...

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

That's OT for this newsgroup (comp.os.linux. development.* for example)
however, a google search for "LFS" "Large file summit", 2GB, etc.
will probably lead you to the right answer as well.

[Peter Ballard]

Randy Howard <ra**********@F OOmegapathdslBA R.net> wrote in message news:<MP******* *************** **@news.megapat hdsl.net>...

In article <9d************ *************@p osting.google.c om>,
pb******@ozemai l.com.au says...

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?
That's OT for this newsgroup (comp.os.linux. development.* for example)

Probably. But in my defence, my problem was unique to my C program.
(bash was fine making >2GB; as were test programs in two other
languages which I tried since my 1st post).
however, a google search for "LFS" "Large file summit", 2GB, etc.
will probably lead you to the right answer as well.

Thanks to all who responded. This URL seems to cover the issue fairly
well:

http://www.suse.de/~aj/linux_lfs.html

--
Regards,

Peter Ballard
Adelaide, AUSTRALIA
pb******@ozemai l.com.au
http://www.ozemail.com.au/~pballard/

[Randy Howard]

In article <9d************ **************@ posting.google. com>,
pb******@ozemai l.com.au says...

That's OT for this newsgroup (comp.os.linux. development.* for example)

Probably. But in my defence, my problem was unique to my C program.

It could have also happened with any language using 32bit file offsets.
Still doesn't make it a c.l.c. issue. :-)

however, a google search for "LFS" "Large file summit", 2GB, etc.
will probably lead you to the right answer as well.

Thanks to all who responded. This URL seems to cover the issue fairly
well:

http://www.suse.de/~aj/linux_lfs.html

Indeed. With the correct platform-dependant invocations, you can make
Linux C programs generate (or interact with) files over several terabytes.
Again, that's not really the c.l.c. problem domain.