 |
C++Talk.NET
C++ language newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
Jaydeep Chovatia
Guest
|
 Posted: Tue Jul 17, 2012 5:16 am Post subject: futex high system utilization |
 |
|
Hi,
In my c++ (Linux) application I am seeing high CPU utilization (almost 90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run “strace” command and found that around 80% of time is being spent in “futex” system call, please see snippet of strace here:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
80.66 100.007743 1567 63806 22761 futex
9.15 11.340274 96925 117 4 restart_syscall
5.60 6.939598 4401 1577 poll
Initially I though this is because of pthread_lock, unlock, condition timedwait, condition wait, etc. system calls I am using in the application, to prove that I have overridden pthread_lock, unlock, condition timedwait, condition wait, unlock, etc. using LD_PRELOAD functionality and concluded that not many threads are waiting here (mutex->__data.__nusers remains less than 5 most of the time), so it doesn’t seem that these sys calls are part of the problem.
Now I am running out of clue about how to find out code location/system call which is causing this futex high utilization, any help on this would be appreciated.
Thank you,
Jaydeep |
|
| Back to top |
|
 |
Jorgen Grahn
Guest
|
| Posted: Tue Jul 17, 2012 8:41 am Post subject: Re: futex high system utilization |
|
|
On Tue, 2012-07-17, Jaydeep Chovatia wrote:
| Quote: | Hi,
In my c++ (Linux) application I am seeing high CPU utilization
(almost 90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried
to run ?strace? command and found that around 80% of time is being
spent in ?futex? system call,
|
I'm not sure you can use strace to find that out -- strace makes
syscalls much more expensive.
| Quote: | please see snippet of strace here:
|
Please break your paragraphs into lines, so I don't have to do it for
you.
| Quote: | % time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
80.66 100.007743 1567 63806 22761 futex
9.15 11.340274 96925 117 4 restart_syscall
5.60 6.939598 4401 1577 poll
Initially I though this is because of pthread_lock, unlock,
condition timedwait, condition wait, etc. system calls I am using in
the application, to prove that I have overridden pthread_lock, unlock,
condition timedwait, condition wait, unlock, etc. using LD_PRELOAD
functionality and concluded that not many threads are waiting here
(mutex->__data.__nusers remains less than 5 most of the time), so it
doesn?t seem that these sys calls are part of the problem.
Now I am running out of clue about how to find out code
location/system call which is causing this futex high utilization, any
help on this would be appreciated.
|
This is a question which you should ask in some Linux newsgroup.
I have only one suggestion to offer: try ltrace too. It traces
library calls instead of system calls. Both strace and ltrace are
very useful on Linux.
/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o . |
|
| Back to top |
|
|
Miquel van Smoorenburg
Guest
|
| Posted: Tue Jul 17, 2012 9:38 am Post subject: Re: futex high system utilization |
|
|
In article <2e7545e9-cd42-4a1e-9a99-be76aeabd317 (AT) googlegroups (DOT) com>,
Jaydeep Chovatia <chovatia.jaydeep (AT) gmail (DOT) com> wrote:
| Quote: | In my c++ (Linux) application I am seeing high CPU utilization (almost
90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run
|
Is this a machine that has been running for a few weeks at least,
and runs ntpd? If so the system might be suffering from a
leap-second-adjustment bug .. as root run date -s "`date`" or
reboot, then see if your problem is gone.
Mike. |
|
| Back to top |
|
|
Jorgen Grahn
Guest
|
| Posted: Tue Jul 17, 2012 1:48 pm Post subject: Re: futex high system utilization |
|
|
On Tue, 2012-07-17, Miquel van Smoorenburg wrote:
| Quote: | In article <2e7545e9-cd42-4a1e-9a99-be76aeabd317 (AT) googlegroups (DOT) com>,
Jaydeep Chovatia <chovatia.jaydeep (AT) gmail (DOT) com> wrote:
In my c++ (Linux) application I am seeing high CPU utilization (almost
90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run
Is this a machine that has been running for a few weeks at least,
and runs ntpd? If so the system might be suffering from a
leap-second-adjustment bug .. as root run date -s "`date`" or
reboot, then see if your problem is gone.
|
Do you have any URL for that? I was aware of the recent leap second,
but haven't heard about that triggering ntpd bugs. Also I'm always
interested in new ways to botch time calculations in software :-)
/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o . |
|
| Back to top |
|
|
Scott Lurndal
Guest
|
| Posted: Tue Jul 17, 2012 2:56 pm Post subject: Re: futex high system utilization |
|
|
Jaydeep Chovatia <chovatia.jaydeep (AT) gmail (DOT) com> writes:
| Quote: | Hi,
In my c++ (Linux) application I am seeing high CPU utilization (almost 90%)=
and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run =93strace=
=94 command and found that around 80% of time is being spent in =93futex=94=
system call, please see snippet of strace here:
[omitted]
Initially I though this is because of pthread_lock, unlock, condition timed=
wait, condition wait, etc. system calls I am using in the application, to p=
rove that I have overridden pthread_lock, unlock, condition timedwait, cond=
ition wait, unlock, etc. using LD_PRELOAD functionality and concluded that =
not many threads are waiting here (mutex->__data.__nusers remains less than=
5 most of the time), so it doesn=92t seem that these sys calls are part of=
the problem.
|
If the .__nusers remains greater than zero much of the time, they you have
contention issues, and contention issues will lead to wasted cycles. What
is the granulariy of the data being protected by a pthread_mutex_t?
| Quote: |
Now I am running out of clue about how to find out code location/system cal=
l which is causing this futex high utilization, any help on this would be a=
ppreciated.
|
Compile and link with the -pg option to the gcc toolchain. Then run gprof
on the resulting gmon.out file. This should pinpoint the hotspots in
your application. It is likely that you need to either reduce the number
of threads, or increase the granularity of the critical section by introducing
additional locks or using gcc or C++11 atomics.
scott |
|
| Back to top |
|
|
Miquel van Smoorenburg
Guest
|
| Posted: Tue Jul 17, 2012 7:38 pm Post subject: Re: futex high system utilization |
|
|
In article <slrnk0b29i.rk7.grahn+nntp (AT) frailea (DOT) sa.invalid>,
Jorgen Grahn <grahn+nntp (AT) snipabacken (DOT) se> wrote:
| Quote: | On Tue, 2012-07-17, Miquel van Smoorenburg wrote:
In article <2e7545e9-cd42-4a1e-9a99-be76aeabd317 (AT) googlegroups (DOT) com>,
Jaydeep Chovatia <chovatia.jaydeep (AT) gmail (DOT) com> wrote:
In my c++ (Linux) application I am seeing high CPU utilization (almost
90%) and it that sys-cpu:user-cpu ratio is 8:2. I then tried to run
Is this a machine that has been running for a few weeks at least,
and runs ntpd? If so the system might be suffering from a
leap-second-adjustment bug .. as root run date -s "`date`" or
reboot, then see if your problem is gone.
Do you have any URL for that? I was aware of the recent leap second,
but haven't heard about that triggering ntpd bugs. Also I'm always
interested in new ways to botch time calculations in software 
|
It isn't a bug in ntpd, it's just that if you run ntpd it would
have told the kernel to handle the extra 2012-06-30 23:59:60 leapsecond,
and that triggered a bug in the kernels timer implementation.
It crashed older kernels, and on newer kernels it caused applications
using futexes to use 100% cpu ..
See for example
http://www.devand.com/index.php/technology/12-tech/11-leap
or Google for "linux leap second bug 2012".
Mike. |
|
| Back to top |
|
|
Powered by phpBB © 2001, 2006 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
