Greetings,
/var/adm/wtmp files were getting out of hand
and I wanted to filter the file rather than zeroing it out.
I found some code that would only keep the last x days
but it required creating a temporary file in /tmp
and then copying it back to /var/adm. I rewrote the code
to load the contents of the wtmp file into memory and then
write the filtered records in memory directly back to /var/adm/wtmp,
eliminating the need for a temporary file.
This worked great until I encountered a wtmp file that was
370 MB. malloc fails setting errno to ENOMEM (not enough
storage space). My system has 16GB of memory and 9GB of paging
(33% used). So, I should have plenty of memory.
My question: is there a workaround for this type of malloc error.
Thanks,
Maja 6 4137
"Maja" <ma********@yahoo.com> wrote in message
news:11**********************@c13g2000cwb.googlegr oups.com...
<snip> This worked great until I encountered a wtmp file that was 370 MB. malloc fails setting errno to ENOMEM (not enough storage space). My system has 16GB of memory and 9GB of paging (33% used). So, I should have plenty of memory.
Hmmm... By the sound of it (you do not provide the actual call to malloc
which fails), i'd say you're right and this is a bug somewhere. However, in
order to make a specific diagnosis, i'd need to know which implementation of
'malloc' you are using (there are quite a few around) and check it out.
Something i'm not planning to do.
The call *should* succeed, AFAICT, so if it doesn't, it's an implementation
issue,you can best pass on to the vendor of your compiler or ask this
question again on a newsgroup that is dedicated to your OS/compiler.
HTH (a tiny bit)
regards,
dandelion.
Maja wrote: Greetings,
/var/adm/wtmp files were getting out of hand and I wanted to filter the file rather than zeroing it out. I found some code that would only keep the last x days but it required creating a temporary file in /tmp and then copying it back to /var/adm. I rewrote the code to load the contents of the wtmp file into memory and then write the filtered records in memory directly back to /var/adm/wtmp, eliminating the need for a temporary file.
This worked great until I encountered a wtmp file that was 370 MB. malloc fails setting errno to ENOMEM (not enough storage space). My system has 16GB of memory and 9GB of paging (33% used). So, I should have plenty of memory.
My question: is there a workaround for this type of malloc error. Thanks, Maja
Your answer is system specific. You probably would get a better answer
in (looking at what you have) comp.unix.programmer. That said, I think
the temporary file is good, because you may some day hit a file big
enough (say, > 2 or 4 GB) that you'd likely have problems with, at least
on a 32 bit machine.
[OT] -- I can reproduce the behavior on linux if I set "ulimit -v" to
something less than the amount I'm trying to malloc. Perhaps
you have some limits imposed that way... If that doesn't help,
perhaps you have some issues in your standard library malloc...
-David
On Fri, 10 Dec 2004 06:56:55 -0800, Maja wrote: Greetings,
/var/adm/wtmp files were getting out of hand and I wanted to filter the file rather than zeroing it out. I found some code that would only keep the last x days but it required creating a temporary file in /tmp and then copying it back to /var/adm. I rewrote the code to load the contents of the wtmp file into memory and then write the filtered records in memory directly back to /var/adm/wtmp, eliminating the need for a temporary file.
This worked great until I encountered a wtmp file that was 370 MB. malloc fails setting errno to ENOMEM (not enough storage space). My system has 16GB of memory and 9GB of paging (33% used). So, I should have plenty of memory.
The limitations on what malloc() can allocate are down to your platform,
well beyond the scope of the C language itself. Perhaps there are
configurable per-process limits etc. Also, if you are only storing
filtered records in memory the memory used would most likely be less,
perhaps a lot less, than 370MB. Perhaps you have a bug in your code and
you are requesting much more memory than you thought. That's easy to log.
My question: is there a workaround for this type of malloc error.
Request less memory or configure your system to allow more memory to be
requested.
However I question the whole approach of storing the data in memory. If
your program has a problem after it starts to write data back to wtmp the
data not written will be lost. Writing to a temporary file is a much
better approach. Just write to a temporary file in /var/adm and rename the
file back to wtmp when complete. This could well be MORE efficient than
storing large amounts of data in memory before writing, if efficiency is
your concern.
Lawrence
"Maja" <ma********@yahoo.com> writes:
[...] This worked great until I encountered a wtmp file that was 370 MB. malloc fails setting errno to ENOMEM (not enough storage space). My system has 16GB of memory and 9GB of paging (33% used). So, I should have plenty of memory.
The amount of memory available on your system isn't necessarily the
same as the amount of memory available to your program.
--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
We must do something. This is something. Therefore, we must do this.
It should have occured to me I might be up against
a system limit. There's a -bmaxdata compile flag
I used to allow the program to allocate memory over
256 meg.
Many thanks for pointing me in the right direction.
Maja
In article <41***********************@dreader16.news.xs4all.n l>, "dandelion" <da*******@meadow.net> writes: "Maja" <ma********@yahoo.com> wrote in message news:11**********************@c13g2000cwb.googlegr oups.com...
This worked great until I encountered a wtmp file that was 370 MB. malloc fails setting errno to ENOMEM (not enough storage space). My system has 16GB of memory and 9GB of paging (33% used). So, I should have plenty of memory. Hmmm... By the sound of it (you do not provide the actual call to malloc which fails), i'd say you're right and this is a bug somewhere.
Nonsense. malloc is allowed to fail for any reason. Assuming a bug
here is like assuming that someone stole your car if you can't find
your keys.
Off topic: The reference to wtmp implies this is a POSIX system. Most
POSIX systems (maybe all; I'm not going to check the standard to see
if this is required) provide a mechanism to limit how much memory a
process can allocate. This is a Good Thing, and has been part of Unix
for a long time. Consult the documentation for the setrlimit system
call and the ulimit command (really commands, since various shells
provide various flavors of ulimit builtins).
The call *should* succeed, AFAICT,
You can't tell. None of us can; we don't have nearly enough
information. Certainly the amount of physical and virtual memory in
the system doesn't suffice. (In fact, depending on the OS, it may
not even be relevant.)
And, once again, we have an excellent demonstration why answering OT
questions here is a bad idea.
so if it doesn't, it's an implementation issue,
Or it isn't, since this is a hosted implementation, and it may just
be reporting - correctly - that the hosting OS has denied its request.
you can best pass on to the vendor of your compiler or ask this question again on a newsgroup that is dedicated to your OS/compiler.
Indeed.
--
Michael Wojcik mi************@microfocus.com
The antics which have been drawn together in this book are huddled here
for mutual protection like sheep. If they had half a wit apiece each
would bound off in many directions, to unsimplify the target. -- Walt Kelly This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Brian Blais |
last post by:
Hello,
I saw on a couple of recent posts people saying that casting the return
value of malloc is bad, like:
d=(double *) malloc(50*sizeof(double));
why is this bad? I had always thought...
|
by: H.A. Sujith |
last post by:
If malloc fails what should I do?
1. Exit imediately.
2. Print an error message (or put a log entry) and exit.
3. Print an error message (or put a log entry) and continue
execution (after...
|
by: Chess Saurus |
last post by:
I'm getting a little bit tired of writing
if (a = malloc(...) == NULL) {
// error code
}
I mean, is it really possible that a malloc call could fail, except in
the case of running out of...
|
by: Rano |
last post by:
/*
Hello,
I've got some troubles with a stupid program...
In fact, I just start with the C language and sometime
I don't understand how I really have to use malloc.
I've readden the FAQ...
|
by: James Dow Allen |
last post by:
The gcc compiler treats malloc() specially! I have no
particular question, but it might be fun to hear from
anyone who knows about gcc's special behavior.
Some may find this post interesting;...
|
by: Jorge Peixoto de Morais Neto |
last post by:
I was reading the code of FFmpeg and it seems that they use malloc
just too much. The problems and dangers of malloc are widely known.
Malloc also has some overhead (although I don't know what is...
|
by: desktop |
last post by:
I have read in Bjarne Stroustrup that using malloc and free should be
avoided in C++ because they deal with uninitialized memory and one
should instead use new and delete.
But why is that a...
|
by: niranjan.singh |
last post by:
This is regarding to test an SDK memory stuff.
In what situation malloc gets fail.
any comment/reply pls....
regards
|
by: Marty James |
last post by:
Howdy,
I was reflecting recently on malloc.
Obviously, for tiny allocations like 20 bytes to strcpy a filename or
something, there's no point putting in a check on the return value of
malloc....
|
by: taylorcarr |
last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
| | |