<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"><meta name="Generator" content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
pre
{mso-style-priority:99;
mso-style-link:"HTML con formato previo Car";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Texto de globo Car";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.TextodegloboCar
{mso-style-name:"Texto de globo Car";
mso-style-priority:99;
mso-style-link:"Texto de globo";
font-family:"Tahoma","sans-serif";
mso-fareast-language:ES-CL;}
span.EstiloCorreo20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.HTMLconformatoprevioCar
{mso-style-name:"HTML con formato previo Car";
mso-style-priority:99;
mso-style-link:"HTML con formato previo";
font-family:"Courier New";
mso-fareast-language:ES-CL;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 3.0cm 70.85pt 3.0cm;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang="ES-CL" link="blue" vlink="purple"><div class="WordSection1"><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I used the dstat commad and posted the results in another answer.</span></p>
<pre><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I already do the gcore and bt this is the output : </span><i><span style="color:black"><a href="http://pastebin.freeswitch.org/20112"><span lang="EN-US">http://pastebin.freeswitch.org/20112</span></a></span></i><span lang="EN-US" style="color:black"></span></pre>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The thread 7 seems to get al the cpu load.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Regards,</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ricardo.-</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><b><span lang="ES" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De:</span></b><span lang="ES" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a>] <b>En nombre de </b>Anthony Minessale<br>
<b>Enviado el:</b> martes, 23 de octubre de 2012 19:38<br><b>Para:</b> FreeSWITCH Users Help<br><b>Asunto:</b> Re: [Freeswitch-users] CPU Spikes every 20 minutes</span></p><p class="MsoNormal"> </p><p class="MsoNormal">No, </p>
<div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">If you have these kinds of problems my only advice is what I already said in my last post about newer OS.</p></div><div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">
You can try top -H -p <current pid of FS process></p></div><div><p class="MsoNormal"> </p></div><div><p class="MsoNormal">watch for any high cpu usage in a particular thread, gcore it and post a bt.</p></div><div><p class="MsoNormal">
</p></div><div><p class="MsoNormal"> </p><div><p class="MsoNormal">On Tue, Oct 23, 2012 at 8:03 AM, Ricardo Martinez <<a href="mailto:rmartinez@redvoiss.net" target="_blank">rmartinez@redvoiss.net</a>> wrote:</p><div>
<div><p class="MsoNormal" style><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Is there an equivalent command in Centos 5.5?</span></p><p class="MsoNormal" style>
<span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal" style><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ricardo.-</span></p>
<p class="MsoNormal" style><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p><p class="MsoNormal" style><b><span lang="ES" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De:</span></b><span lang="ES" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>] <b>En nombre de </b>Anthony Minessale<br>
<b>Enviado el:</b> lunes, 22 de octubre de 2012 19:37</span></p><div><div><p class="MsoNormal"><br><b>Para:</b> FreeSWITCH Users Help<br><b>Asunto:</b> Re: [Freeswitch-users] CPU Spikes every 20 minutes</p></div></div><div>
<div><p class="MsoNormal" style> </p><div><p class="MsoNormal" style>iotop doesn't exist yet in in 5.5</p></div><div><p class="MsoNormal" style> </p></div><p class="MsoNormal" style>Try cent 5.7 </p><div><p class="MsoNormal" style>
</p></div><div><p class="MsoNormal" style>or cent 6.3 or debian 6 if you want some more modern kernels to get iotop</p></div><div><p class="MsoNormal" style> </p></div><div><p class="MsoNormal" style> </p></div><div><p class="MsoNormal" style="margin-bottom:12.0pt">
<br><br></p><div><p class="MsoNormal" style>On Mon, Oct 22, 2012 at 4:59 PM, Ken Rice <<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>> wrote:</p><div><p class="MsoNormal" style><span style="font-size:11.0pt;font-family:"Courier New"">Keeping in mind that you are using a sangoma transcoder board, that means you are handling the equivalent of 440 concurrent calls worth of media for people not using sangoma transcoding hardware (yes theres 4 call legs in that scenario, 1 from endpoint A to FS, 1 from FS to D100, 1 From D100 back to FS, 1 from FS back out to the other end point). That doesn’t sound like that big of a problem... You will have a fair bit of network traffic that way (the D100 shows up as another network interface as you are probably aware)...<br>
<br>I would be looking at something like iotop or something else to see where that load is coming from<br><br>Also, you have plenty of CPU free... So there is either a real problem here, or this is one of those things where the number of threads handling RTP and SIP are hanging around the run queue long enough to show up in the load average... </span></p>
<div><div><p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Courier New""><br><br><br><br><br>On 10/22/12 3:53 PM, "Ricardo Martinez" <<a href="http://rmartinez@redvoiss.net" target="_blank">rmartinez@redvoiss.net</a>> wrote:</span></p>
</div></div><blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"><div><div><p class="MsoNormal" style><span style="font-size:11.0pt;font-family:"Courier New"">Ken.<br>There is an average of 220 concurrent calls.<br>
Machine : DL380 G7 – 2 Quad Core – 8 Gig RAM.<br>Centos 5.5 64bits<br> <br>Ricarod.-<br> <br><br></span><b><span style="font-size:10.0pt;font-family:"Courier New"">De:</span></b><span style="font-size:10.0pt;font-family:"Courier New""> <a href="http://freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">mailto:freeswitch-users-bounces@lists.freeswitch.org</a>] <b>En nombre de </b>Ken Rice<br>
<b>Enviado el:</b> lunes, 22 de octubre de 2012 17:47<br><b>Para:</b> FreeSWITCH Users Help<br><b>Asunto:</b> Re: [Freeswitch-users] CPU Spikes every 20 minutes<br></span> <br><span style="font-size:11.0pt;font-family:"Courier New"">How many concurrent calls are you doing on that box? <br>
<br><br>On 10/22/12 3:33 PM, "Ricardo Martinez" <<a href="http://rmartinez@redvoiss.net" target="_blank">rmartinez@redvoiss.net</a>> wrote:<br>Hi Ken.<br>I mounted the db directory in RAM as you mention, but the “load average” still is high.<br>
You can see this here : <br> <br></span><span style="font-size:9.0pt;font-family:"Courier New"">Filesystem Size Used Avail Use% Mounted on<br>/dev/mapper/VolGroup00-LogVol00<br> 132G 8.6G 117G 7% /<br>
proc 0 0 0 - /proc<br>sysfs 0 0 0 - /sys<br>devpts 0 0 0 - /dev/pts<br>/dev/cciss/c0d0p1 104M 20M 79M 21% /boot<br>
tmpfs 4.2G 0 4.2G 0% /dev/shm<br>none 0 0 0 - /proc/sys/fs/binfmt_misc<br>sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs<br><b>tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db<br>
</b></span><span style="font-size:11.0pt;font-family:"Courier New""> <br>This is a resume from uptime and total calls for my freeswitch :<br> <br></span><span style="font-size:9.0pt;font-family:"Courier New"">17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440<br>
17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510<br>17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488<br>
17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484<br>17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470<br>
17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454<br>17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416<br>
17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384<br>17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366<br>
17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388<br>17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442<br>
17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436<br>17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418<br>
17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422<br>17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410<br>
17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450<br>17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536<br>
17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530<br>17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496<br>
17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492<br>17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496<br>
17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466<br>17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452<br>
17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430<br>17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422<br>
17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500<br>17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498<br>
17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516<br>17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490<br>
17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436<br>17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372<br>
17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342<br>17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370<br>
17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382<br>17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366<br>
17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358<br>17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378<br>
17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386<br>17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402<br>
17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496<br>17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532<br>
17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506<br>17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482<br>
17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492<br>17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482<br>
17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458<br>17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470<br>
17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422<br>17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452<br>
17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536<br>17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524<br>
17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536<br>17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508<br>
17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478<br>17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408<br>
17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396<br>17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438<br>
17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470<br>17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478<br>
17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480<br>17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490<br>
17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470<br>17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450<br>
17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524<br>17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536<br>
17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528<br>17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524<br>
17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564<br>17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510<br>
17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476<br>17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498<br>
</span><span style="font-size:11.0pt;font-family:"Courier New""> <br>What else could be happening.<br>Did I mention that I’m using two transcoding cards from Sangoma?<br> <br>Regards,<br>Ricardo.-<br> <br> <br> <br>
<br></span><b><span style="font-size:10.0pt;font-family:"Courier New"">De:</span></b><span style="font-size:10.0pt;font-family:"Courier New""> <a href="http://freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">mailto:freeswitch-users-bounces@lists.freeswitch.org</a>] <b>En nombre de </b>Ken Rice<br>
<b>Enviado el:</b> lunes, 22 de octubre de 2012 16:50<br><b>Para:</b> FreeSWITCH Users Help<br><b>Asunto:</b> Re: [Freeswitch-users] CPU Spikes every 20 minutes<br></span> <br><span style="font-size:11.0pt;font-family:"Courier New"">Load Average is the number of threads/processes waiting in the kernel’s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH...<br>
<br>Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to see what the current load avg is...<br><br>Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type ‘show channels’ from fs_cli it doesn’t query the structures in RAM to show you that output, it pulls it from the sqlite db...<br>
<br>That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive.<br>Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch<br>
<br>Shutdown freeswitch<br>rm –rf /usr/local/freeswitch/db/*<br>mount –t tmpfs tmpfs /usr/local/freeswitch/db<br>Startup freeswitch<br><br>Once you have done that stick it back in rotation... <br><br>Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail’s DSN settings you’ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder)<br>
<br>K<br><br><br>On 10/22/12 2:30 PM, "Ricardo Martinez" <<a href="http://rmartinez@redvoiss.net" target="_blank">rmartinez@redvoiss.net</a>> wrote:<br>Hi Ken.<br>Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter?<br>
Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?.<br> <br>Thanks<br>Ricardo.-<br> <br><br></span><b><span style="font-size:10.0pt;font-family:"Courier New"">De:</span></b><span style="font-size:10.0pt;font-family:"Courier New""> <a href="http://freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">mailto:freeswitch-users-bounces@lists.freeswitch.org</a>] <b>En nombre de </b>Ken Rice<br>
<b>Enviado el:</b> lunes, 22 de octubre de 2012 16:18<br><b>Para:</b> FreeSWITCH Users Help<br><b>Asunto:</b> Re: [Freeswitch-users] CPU Spikes every 20 minutes<br></span> </p></div></div><div><div><p class="MsoNormal" style>
<span style="font-size:11.0pt;font-family:"Courier New"">Could it be just a normal sqlite running a vacuum?<br><br>Sqlite does this every so often or if its calling a sync.... <br><br>Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about<br>
<br>On 10/22/12 1:52 PM, "Ricardo Martinez" <<a href="http://rmartinez@redvoiss.net" target="_blank">rmartinez@redvoiss.net</a>> wrote:<br>This is the gdb from the proccess consuming more cpu :<br> <br>(gdb) thread 7<br>
[Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0<br>(gdb) bt<br>#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0<br>#1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229<br>
#2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229<br>#3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770<br>#4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520<br>
#5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062<br>#6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360<br>#7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236<br>
#8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289<br>#9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0)<br> at ./src/legacy.c:78<br>
#10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38)<br> at src/switch_core_db.c:93<br>#11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0)<br>
at src/switch_core_sqldb.c:445<br>#12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, </span></p></div></div><p class="MsoNormal" style><span style="font-size:11.0pt;font-family:"Courier New""> sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname='<a href="http://transcoder.red.net" target="_blank">transcoder.red.net</a> <<a href="http://transcoder.red.net" target="_blank">http://transcoder.red.net</a>> <<a href="http://transcoder.red.net" target="_blank">http://transcoder.red.net</a>> <<a href="http://transcoder.red.net" target="_blank">http://transcoder.red.net</a>> <<a href="http://transcoder.red.net" target="_blank">http://transcoder.red.net</a>> ';\ndelete from calls where (call"..., retries=1)</span></p>
<div><div><p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Courier New""><br> at src/switch_core_sqldb.c:787<br>#13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>)<br>
at src/switch_core_sqldb.c:1026<br>#14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0<br>#15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6<br> </span></p></div></div><div><div><p class="MsoNormal" style>
<b><span style="font-size:10.0pt;font-family:"Courier New"">De:</span></b><span style="font-size:10.0pt;font-family:"Courier New""> Ricardo Martinez [<a href="mailto:rmartinez@redvoiss.net" target="_blank">mailto:rmartinez@redvoiss.net</a>] <br>
<b>Enviado el:</b> lunes, 22 de octubre de 2012 15:50<br><b>Para:</b> 'FreeSWITCH Users Help'<br><b>Asunto:</b> RE: [Freeswitch-users] CPU Spikes every 20 minutes</span></p></div></div><div><div><p class="MsoNormal" style>
<br><span style="font-size:11.0pt;font-family:"Courier New"">AS far as i can see there is no java running.<br>This is the top at the moment of the spike : <br> <br>top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48<br>
Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie<br>Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st<br>Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers<br>
Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached<br> <br> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br>31772 root 15 0 870m 481m 6880 S <a href="tel:212.4%C2%A0%206.0%C2%A0%C2%A0%209523" target="_blank">212.4 6.0 9523</a>:51 freeswitch <br>
4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server <br>29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top <br>
1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init <br> 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 <br>
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 <br> 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 <br>
5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 <br> 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 <br>
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 <br> 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 <br>
9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 <br> 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 <br>
11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 <br> 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 <br>
13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 <br> 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 <br>
15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 <br> 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 <br>
17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 <br> 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 <br>
19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 <br> 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 <br> <br>
<br>Regards,<br>Ricardo.- <br> <br></span><b><span style="font-size:10.0pt;font-family:"Courier New"">De:</span></b><span style="font-size:10.0pt;font-family:"Courier New""> <a href="http://freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">mailto:freeswitch-users-bounces@lists.freeswitch.org</a>] <b>En nombre de </b>Guillermo Ruiz Camauer<br>
<b>Enviado el:</b> lunes, 22 de octubre de 2012 15:38<br><b>Para:</b> FreeSWITCH Users Help<br><b>Asunto:</b> Re: [Freeswitch-users] CPU Spikes every 20 minutes<br></span> <br>Do you have anything like Java running that could be doing garbage collection?<br>
<span style="font-size:11.0pt;font-family:"Courier New""><br></span> <br><span style="font-size:11.0pt;font-family:"Courier New""><br></span>What other processes are running on the machine? Does TOP show which process is causing the spike?<br>
<span style="font-size:11.0pt;font-family:"Courier New""><br></span> <br><span style="font-size:11.0pt;font-family:"Courier New""><br></span>Guillermo<br><span style="font-size:11.0pt;font-family:"Courier New""><br>
</span>On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <<a href="http://rmartinez@redvoiss.net" target="_blank">rmartinez@redvoiss.net</a>> wrote:<br>Hi Gabe.<br>I'm mean between the spikes there is nothing different as regarding to the<br>
calls... the cps maintains the same.. there are no scripts running or<br>crontabs running, the simultaneous calls are the same also. So, for some<br>reason the "load average" start to grow (every 20 or 25 minutes) without<br>
any clear cause.<br>I have debug with gdb the moment the server is with 12.5 of "load average"<br>here : <a href="http://pastebin.freeswitch.org/20112" target="_blank">http://pastebin.freeswitch.org/20112</a><br>
<br>Can we detect a problem from here?<br><br>Regards,<br>Ricardo.<br><br><br><br>-----Mensaje original-----<br>De: <a href="http://freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
[<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">mailto:freeswitch-users-bounces@lists.freeswitch.org</a>] En nombre de<br>Gabriel Gunderson<br>Enviado el: lunes, 22 de octubre de 2012 13:31<br>
Para: FreeSWITCH Users Help<br>Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes<br><span style="font-size:11.0pt;font-family:"Courier New""><br></span><br>On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <<a href="http://rmartinez@redvoiss.net" target="_blank">rmartinez@redvoiss.net</a>><br>
wrote:<br>> The problem is that I'm having weird spikes of "load average" cpu<br>> every 20 minutes (aprox), without any reason, reaching near of 12.5 of<br>> load average for this machine.<br><br>
What have you done to verify that it happens 'without any reason'?<br><br><br>Gabe<br><br>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br>
<a href="http://consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br><a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br><br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br><br>FreeSWITCH-users mailing list<br><a href="http://FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br><br>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="http://consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>FreeSWITCH-users mailing list<br><a href="http://FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br><br> </p></div></div></blockquote><div><div><p class="MsoNormal" style><br><span style="font-size:11.0pt;font-family:"Courier New"">-- <br>Ken<br><u><span style="color:blue"><a href="http://www.FreeSWITCH.org" target="_blank">http://www.FreeSWITCH.org</a><br>
<a href="http://www.ClueCon.com" target="_blank">http://www.ClueCon.com</a><br><a href="http://www.OSTAG.org" target="_blank">http://www.OSTAG.org</a><br></span></u><a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch</span></p>
</div></div></div><p class="MsoNormal" style="margin-bottom:12.0pt"><br>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>FreeSWITCH-users mailing list<br><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a></p>
</div><p class="MsoNormal" style><br><br clear="all"></p><div><p class="MsoNormal" style> </p></div><p class="MsoNormal" style>-- <br>Anthony Minessale II<br><br>FreeSWITCH <a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org/</a><br>
ClueCon <a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com/</a><br>Twitter: <a href="http://twitter.com/FreeSWITCH_wire" target="_blank">http://twitter.com/FreeSWITCH_wire</a><br><br>AIM: anthm<br><a href="mailto:MSN%3Aanthony_minessale@hotmail.com" target="_blank">MSN:anthony_minessale@hotmail.com</a><br>
GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com" target="_blank">PAYPAL:anthony.minessale@gmail.com</a><br>IRC: <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch<br><br>FreeSWITCH Developer Conference<br>
<a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a><br><a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org" target="_blank">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:<a href="tel:%2B19193869900" target="_blank">+19193869900</a></p></div></div></div></div></div><p class="MsoNormal" style="margin-bottom:12.0pt"><br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br><a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br><br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br><br>FreeSWITCH-users mailing list<br><a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a></p></div><p class="MsoNormal"><br><br clear="all"></p><div><p class="MsoNormal"> </p></div><p class="MsoNormal">-- <br>Anthony Minessale II<br>
<br>FreeSWITCH <a href="http://www.freeswitch.org/">http://www.freeswitch.org/</a><br>ClueCon <a href="http://www.cluecon.com/">http://www.cluecon.com/</a><br>Twitter: <a href="http://twitter.com/FreeSWITCH_wire">http://twitter.com/FreeSWITCH_wire</a><br>
<br>AIM: anthm<br><a href="mailto:MSN%3Aanthony_minessale@hotmail.com">MSN:anthony_minessale@hotmail.com</a><br>GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net">irc.freenode.net</a> #freeswitch<br><br>FreeSWITCH Developer Conference<br><a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a><br><a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:+19193869900</p></div><div><p class="MsoNormal"> </p></div></div></body></html>