about pollsys and dtrace.

[Sophie000]

I have a single-threading implemented program (written by somebody else). In this program, one server listens and accepts connection set up request from multiple clients, puts client information in data base, and provides this information to other processes.

Here is the problem: although the tasks the server are doing are very simple (as I just mentioned), the CPU of the server was quickly used up when the number of clients goes up. My task is to find out if the program has some bugs which leads to the CPU exhaustion; or if some implementation of the program is not efficient.

By using dtrace, I found most of the CPU was consumed by the following system calls: pollsys, recvfrom, sendto. I know the recvfrom, sendto are used to receive data from remote clients and send data to clients, but I am not familiar with pollsys and did not find enough information from internet. It seems like The pollsys syscall is normally used
for checking whether there is any data that can be read from or write to a file descriptor. Then how do I know who call pollsys and in which case the pollsys is called. Is it called periodically or called by signal interrupt, or due to some events?

I am a newbi in unix and its tracing tools. I wonder if anybody can answer my questions and also give me some suggestion like how to narrow down my focus and dig out the reason of the problem.

Thanks.

[gpraghuram]

Its better to use select call instead of pollsys.
Try using select


Raghuram

[Sophie000]

but I saw a select() call with a timeout was included in a while loop.

Do you mean pollsys will not be called if select() mode is used?

Thanks.

[gpraghuram]

but I saw a select() call with a timeout was included in a while loop.

Do you mean pollsys will not be called if select() mode is used?

Thanks.

select calls last parameter is the timeout value.
I dont think so that pollsys will be called inside select.


Raghuram

[Sophie000]

Since I found pollsys consumed most of the CPU, I want to see in which case it is called. To do that, I now used truss -t pollsys -T pollsys -p PID, and pstack PID to see how is the pollsys called. Here is the result:
(I do not understand what lwp# /thread # mean,but it seems pollsys is called by select() and something else I do not understand).
What does it mean? Thanks.

----------------- lwp# 1 / thread# 1 --------------------
fffffd7ffe76e3ca pollsys (fffffd7fffdfb7f0, 16, fffffd7fffdfbaf0, 0)
fffffd7ffe718ee4 pselect () + 154
fffffd7ffe7191b2 select () + 72
000000000046918d inf_man () + 53d
000000000046dbf4 main () + 234
000000000044bdcc ???????? ()
----------------- lwp# 2 / thread# 2 --------------------
fffffd7ffe76d85a sigtimedwait (fffffd7ffe3fdd10, fffffd7ffe3fdd20, fffffd7ffe3fdea0)
fffffd7fff0e71c9 sigtimedwait () + 9
fffffd7fff095307 signalRun () + 87
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 3 / thread# 3 --------------------
fffffd7ffe76d79a nanosleep (fffffd7ffe1fdf90, fffffd7ffe1fdf80)
0000000000458dea events_poller () + 1a
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 4 / thread# 4 --------------------
fffffd7ffe76d79a nanosleep (fffffd7fea1fdf70, fffffd7fea1fdf60)
0000000000470a75 brand_data_poller () + c5
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 5 / thread# 5 --------------------
fffffd7ffe76e3ca pollsys (fffffd7fe9ff9d10, 1, 0, 0)
fffffd7ffe718ee4 pselect () + 154
fffffd7ffe7191b2 select () + 72
000000000044ca44 resp_service_req () + 1a4
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 6 / thread# 6 --------------------
fffffd7ffe76e3ca pollsys (fffffd7fe9df7db0, 1, 0, 0)
fffffd7ffe718ee4 pselect () + 154
fffffd7ffe7191b2 select () + 72
fffffd7fff2ede79 process_snmp_request () + a9
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 7 / thread# 7 --------------------
fffffd7ffe76e3ca pollsys (fffffd7fe9bf7db0, 1, 0, 0)
fffffd7ffe718ee4 pselect () + 154
fffffd7ffe7191b2 select () + 72
000000000048be9e process_licensing_request () + ae
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 8 / thread# 8 --------------------
fffffd7ffe76e3ca pollsys (66edb0, 24, fffffd7fe99fdf10, 0)
fffffd7ffe714822 poll () + 52
fffffd7ffeee50b0 PollerInvokePoll () + 30
fffffd7ffeee4f4b PollerStep () + 17b
fffffd7ffeee5a32 _RunnerStart () + 62
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()

[gpraghuram]

Since I found pollsys consumed most of the CPU, I want to see in which case it is called. To do that, I now used truss -t pollsys -T pollsys -p PID, and pstack PID to see how is the pollsys called. Here is the result:
(I do not understand what lwp# /thread # mean,but it seems pollsys is called by select() and something else I do not understand).
What does it mean? Thanks.

----------------- lwp# 1 / thread# 1 --------------------
fffffd7ffe76e3ca pollsys (fffffd7fffdfb7f0, 16, fffffd7fffdfbaf0, 0)
fffffd7ffe718ee4 pselect () + 154
fffffd7ffe7191b2 select () + 72
000000000046918d inf_man () + 53d
000000000046dbf4 main () + 234
000000000044bdcc ???????? ()
----------------- lwp# 2 / thread# 2 --------------------
fffffd7ffe76d85a sigtimedwait (fffffd7ffe3fdd10, fffffd7ffe3fdd20, fffffd7ffe3fdea0)
fffffd7fff0e71c9 sigtimedwait () + 9
fffffd7fff095307 signalRun () + 87
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 3 / thread# 3 --------------------
fffffd7ffe76d79a nanosleep (fffffd7ffe1fdf90, fffffd7ffe1fdf80)
0000000000458dea events_poller () + 1a
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 4 / thread# 4 --------------------
fffffd7ffe76d79a nanosleep (fffffd7fea1fdf70, fffffd7fea1fdf60)
0000000000470a75 brand_data_poller () + c5
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 5 / thread# 5 --------------------
fffffd7ffe76e3ca pollsys (fffffd7fe9ff9d10, 1, 0, 0)
fffffd7ffe718ee4 pselect () + 154
fffffd7ffe7191b2 select () + 72
000000000044ca44 resp_service_req () + 1a4
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 6 / thread# 6 --------------------
fffffd7ffe76e3ca pollsys (fffffd7fe9df7db0, 1, 0, 0)
fffffd7ffe718ee4 pselect () + 154
fffffd7ffe7191b2 select () + 72
fffffd7fff2ede79 process_snmp_request () + a9
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 7 / thread# 7 --------------------
fffffd7ffe76e3ca pollsys (fffffd7fe9bf7db0, 1, 0, 0)
fffffd7ffe718ee4 pselect () + 154
fffffd7ffe7191b2 select () + 72
000000000048be9e process_licensing_request () + ae
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()
----------------- lwp# 8 / thread# 8 --------------------
fffffd7ffe76e3ca pollsys (66edb0, 24, fffffd7fe99fdf10, 0)
fffffd7ffe714822 poll () + 52
fffffd7ffeee50b0 PollerInvokePoll () + 30
fffffd7ffeee4f4b PollerStep () + 17b
fffffd7ffeee5a32 _RunnerStart () + 62
fffffd7ffe76b47b _thr_setup () + 5b
fffffd7ffe76b6b0 _lwp_start ()

lwp means light-weight-process.
I havent debugged in this way.
Raghuram

[Sophie000]

The result showed that pollsys is called by select() and another user function.
From Beej's tutorial on socket programming, I learned that CPU exhaustion
due to polling can be avoided by using select() mode.
But here I saw that select() also use pollsys.