473,839 Members | 1,352 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Weird. malloc() fails in spite of having enough virtual memory...

Hi folks,

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.

I attached the output of tracking memory usage (using vmstat) until
the malloc failure is encountered.

Any idea on why malloc(for 1K chunk) eventually fails despite it has
enough VM at that moment??

Only 20MB difference in free VM between (when a child myxmlreader
starts) and (when it dies due to malloc failure). So, still enough
VM remains.
(A new request comes in and a child process starts reading a big xml
file)
root 5471 18.6 0.4 75556 7888 d0 D 2:30AM 0:07.66 ./
myxmlreader
root 5468 0.0 0.2 71036 3280 d0 I 2:30AM 0:00.06 ./
myxmlreader
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
1 1 0 718380 1415840 2 0 0 0 2 0 0 1277 10288 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 1 0 719108 1414164 2 0 0 0 2 0 0 1277 10293 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
3 0 0 720632 1412600 2 0 0 0 2 0 0 1277 10299 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
3 0 0 722328 1410924 2 0 0 0 2 0 0 1277 10307 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
3 0 0 725316 1406360 2 0 0 0 2 0 0 1277 10317 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 0 0 727044 1404392 2 0 0 0 2 0 0 1277 10324 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 0 0 730300 1401128 2 0 0 0 2 0 0 1277 10336 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
3 0 0 732288 1399156 2 0 0 0 2 0 0 1277 10346 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 1 0 734096 1397312 2 0 0 0 2 0 0 1277 10357 668 14
86 0
root@% vmstat
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr ad0 in sy cs us sy
id
2 1 0 712780 1352132 2 0 0 0 2 0 0 1277 10365 668 14
86 0
(At this moment, the child process has processed reading/saving one
thousand and several hundreds of xml object and finally malloc has
failed , the child process has died)

root@% ps -aux | grep myxmlreader
root 5468 0.0 0.2 71036 3192 d0 S 2:30AM 0:00.06 ./
myxmlreader

Jul 29 '07 #1
6 3375
it********@gmai l.com wrote:
Hi folks,

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.

I attached the output of tracking memory usage (using vmstat) until
the malloc failure is encountered.

Any idea on why malloc(for 1K chunk) eventually fails despite it has
enough VM at that moment??

Only 20MB difference in free VM between (when a child myxmlreader
starts) and (when it dies due to malloc failure). So, still enough
VM remains.
<snip>

It's hard to say what is the cause of your problem without a detailed look
at your code. To many allocations and deallocations can cause memory
fragmentation to become a real issue. One strategy is to allocate a fixed
number of containers and reuse them, (i.e. read and save, read again...),
until the entire file has been traversed.

You don't need to deallocate the containers after using them once. Why not
reuse them for the next object?

Jul 29 '07 #2

<it********@gma il.comwrote in message
news:11******** *************@j 4g2000prf.googl egroups.com...
Hi folks,

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.
Sounds very much like you are not freeing the memory. If you have 1 GB
installed, then you will run out at about the 1000 mark if each is a few
hundred K.
However it could be that the request is for a corrupt amount. Try replacing
malloc() with a wrapper that prints out the amout requested on failure.

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

Jul 29 '07 #3
On Jul 29, 3:10 pm, itsolut...@gmai l.com wrote:
Hi folks,

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.

As pointed out elsethread, fragmentation may be an issue (there is
lots of free memory, but no one chunk that is big enough).
However, since it appears that you only allocate a few megabytes in
total
this does not seem likely. Have you tried not freeing?

However, malloc may be failing for a reason that has nothing to
do with the amount of free memory. It is very common for a malloc
failure to be due to memory corruption. Are you sure that
nowhere in your system is something that
corrupts memory? Does malloc return NULL or does it fail
with a segfault?

- William Hughes

Jul 29 '07 #4
"Malcolm McLean" <re*******@btin ternet.comwrite s:
<it********@gma il.comwrote in message
news:11******** *************@j 4g2000prf.googl egroups.com...
>Hi folks,

Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(wit h its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.
Sounds very much like you are not freeing the memory. If you have 1 GB
installed, then you will run out at about the 1000 mark if each is a
few hundred K.
Did you never hear of virtual memory? How do you think that might affect things?
However it could be that the request is for a corrupt amount. Try
replacing malloc() with a wrapper that prints out the amout requested
on failure.
--
Jul 29 '07 #5
In article <11************ *********@j4g20 00prf.googlegro ups.com>,
<it********@gma il.comwrote:
>Could you shed some light on this issue?
Without looking at your code, it's hard to say, but some possibilities
are:

- you have a memory limit set, perhaps system-wide or for the user
- you are leaking memory elsewhere (e.g. in the database API)
- you have a bug which is corrupting malloc's data structures

If possible, use a malloc debugger to verify that you are doing the
mallocs and frees you expect. Try running it on a different OS, and
see if you get the same behaviour. Cut out some of the code, and see
whether it still fails.

-- Richard
--
"Considerat ion shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.
Jul 29 '07 #6
it********@gmai l.com wrote:
Could you shed some light on this issue?

my program is running on Freebsd as a daemon. When user sends a
request, it forks itself and lets its child process handles the
request.
And its main role is just to read a big xml file and save each object
into its embedded DB(such as gdbm).

Each object data can be up to several K bytes. So, whenever reading
each object from XML file, I call malloc() for making
containers(with its unit 1K) which will contain the data.
After reading each object , I immediately save data in containers
into the internal DB via DB API. And then, I free those containers.
So, there should be no memory explosion issue.

But, I always see malloc() failure as it reads(and saves into DB)
some number(like around one or 2 thousands ) of object. It's
weird. I always free the memory chunk returned by malloc once each
object is read/saved.
Probably pretty much off-topic:

Check the system docs for how malloc() is implemented. It is possible
for some systems to return an allocation failure for things like not
being able to satisfy the allocation out of swap at the specific moment.
That is, there is often no way to distinguish the specific reason the
allocation failed.

Someone else already mentioned fragmentation. Take a look at "ulimit",
as well, to see what the data segment of a process can be per process.
This value can often grow to the hard limit and stay there, chewing up
VM even though it has been freed.

If your allocations are implemented with brk()/sbrk() or similar system
calls, you can also swap in a library that uses mmap() or other more
modern ways of getting (and releasing) resources.

--
clvrmnky <mailto:sp***** *@clevermonkey. org>

Direct replies will be blacklisted. Replace "spamtrap" with my name to
contact me directly.
Jul 30 '07 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
660
by: Steve Zimmerman | last post by:
Esteemed contributors to clc: Thank you for all the responses. Experiments 2 and 3 below are identical, except that experiment 2 does not call free(), while experiment 3 does. With such a trivial program, is it safe not to call free()? Are there dangers in experiment 2 that experiment 3 avoids?
25
5086
by: H.A. Sujith | last post by:
If malloc fails what should I do? 1. Exit imediately. 2. Print an error message (or put a log entry) and exit. 3. Print an error message (or put a log entry) and continue execution (after possibly recovering from the error). Printing an error message might be difficult in a graphical environment. --
27
4285
by: Chess Saurus | last post by:
I'm getting a little bit tired of writing if (a = malloc(...) == NULL) { // error code } I mean, is it really possible that a malloc call could fail, except in the case of running out of virtual memory? -Chess
6
4165
by: Maja | last post by:
Greetings, /var/adm/wtmp files were getting out of hand and I wanted to filter the file rather than zeroing it out. I found some code that would only keep the last x days but it required creating a temporary file in /tmp and then copying it back to /var/adm. I rewrote the code to load the contents of the wtmp file into memory and then write the filtered records in memory directly back to /var/adm/wtmp, eliminating the need for a...
10
1852
by: Bonj | last post by:
Hello. I hope somebody can help me on this, because I'm running out of options to turn to. I have almost solved my regular expression function. Basically it works OK if unicode is defined. It doesn't work OK in ANSI mode however, as it has to use MultiByteToWideChar and WideCharToMultiByte. I've discovered that the regular expression part is working fine. As far as I can tell the regular expression code is correctly parsing what it...
12
3285
by: gooch | last post by:
I originally posted the following code in a group discussing threads but after further research I think I have a c question about the code. I know there are a couple of non standard c includes here and the POSIX stuff is non standard but this is how I stumbled onto this question. #include <INTEGRITY.h> #include <stdlib.h> #include <stdio.h> #include <pthread.h> #include <signal.h>
6
5499
by: sam_cit | last post by:
Hi Everyone, I wanted to know as to how malloc() works, if my understanding is correct, it is implementation specific of the vendor who provides the library(alloc.h). If so, is there any standard as to how it should be implemented? If there is no space in the RAM, will malloc() return NULL or will it allocate a memory in a new page using virtual memory and return the
21
8397
by: llothar | last post by:
Hello, i need to manage a heap in shared memory. Does anybody know about a portable (win32+mac+posix) c implementation for this.
71
19155
by: desktop | last post by:
I have read in Bjarne Stroustrup that using malloc and free should be avoided in C++ because they deal with uninitialized memory and one should instead use new and delete. But why is that a problem? I cannot see why using malloc instead of new does not give the same result.
0
9855
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, weíll explore What is ONU, What Is Router, ONU & Routerís main usage, and What is the difference between ONU and Router. Letís take a closer look ! Part I. Meaning of...
0
9697
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10908
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10587
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10649
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10295
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
5682
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4487
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3136
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.