<div dir="ltr">List, Steve<br><br>I will clarify what i'm asking here before I take Anothny's suggestion and join a "computer tuning" club as a way to "move forward".<br><a href="http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg">http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg</a><br>
<br>What is there to read on this subject? Links, textbook names - everything is appreciated. <br>What are the tools that show useful data and what i can do with FS to make the work easier? Compile with some flags to get more info on running threads?<br>
<br>Thanks,<br>Boris Ratner.<br><br><div class="gmail_quote">On Sat, Mar 9, 2013 at 12:51 AM, Steven Ayre <span dir="ltr"><<a href="mailto:steveayre@gmail.com" target="_blank">steveayre@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
After stopping the load FS still hogs 22.1% of memory.<br> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br>
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch </blockquote><div><br></div></div>Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks.<br>
<br>-Steve<div class="HOEnZb"><div class="h5"><br><br><br><br>On 8 March 2013 18:15, bratner bratner <<a href="mailto:ratner2@gmail.com" target="_blank">ratner2@gmail.com</a>> wrote:<br>> Here is sipp output and additional numbers for a test I ran with -nosql<br>
> param.<br>
><br>> The test ran 180CPS for ~3500seconds and the rest with 210cps.<br>><br>> Trouble (as in higher system cpu% ) started to appear around 8591seconds<br>> into the test.<br>> As you can see below the problem started just before 9124sec into the test <br>
> 210cps 5sec calls<br>> should not give you a lot more then 1050 concurrent calls.<br>><br>> ------------------------------ Scenario Screen -------- [1-9]: Change Screen<br>> --<br>> Call-rate(length) Port Total-time Total-calls Remote-host<br>
> 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 <br>> 192.96.201.164:5060(UDP)<br>><br>> 0 new calls during 0.000 s period 0 ms scheduler resolution<br>> 0 calls (limit 2000) Peak was 2000 calls, after 9124 s<br>
> 0 Running, 4640 Paused, 0 Woken up<br>> 20 dead call msg (discarded) 0 out-of-call msg (discarded) <br>> 1 open sockets <br>><br>> Messages Retrans Timeout <br>
> Unexpected-Msg<br>> INVITE ----------> 1834024 74 0 <br>> 100 <---------- 1834024 0 0 0 <br>> 180 <---------- 1834024 0 0 0 <br>
> 183 <---------- 0 0 0 0 <br>> 500 <---------- 0 0 0 0 <br>> 502 <---------- 0 0 0 0 <br>
> 503 <---------- 0 0 0 0 <br>> 408 <---------- 0 0 0 0 <br>> 480 <---------- 0 0 0 0 <br>
> 200 <---------- E-RTD1 1834024 81 0 0 <br>><br>> ACK ----------> 1834024 81 <br>> Pause [ 5000ms] 1834024 0 <br>
> BYE ----------> 1834024 7646 0 <br>> 503 <---------- 0 0 0 0 <br>> 200 <---------- 1834024 0 0 0 <br>
><br>> ------------------------------ Test Terminated<br>> --------------------------------<br>><br>><br>> ----------------------------- Statistics Screen ------- [1-9]: Change Screen<br>> --<br>> Start Time | 2013-03-08 15:22:18:204 1362756138.204833 <br>
> Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 <br>> Current Time | 2013-03-08 17:54:55:535 1362765295.535437 <br>> -------------------------+---------------------------+--------------------------<br>
> Counter Name | Periodic value | Cumulative value<br>> -------------------------+---------------------------+--------------------------<br>> Elapsed Time | 00:00:00:000 | 02:32:37:330 <br>
> Call Rate | 0.000 cps | 200.279 cps <br>> -------------------------+---------------------------+--------------------------<br>> Incoming call created | 0 | 0 <br>
> OutGoing call created | 0 | 1834024 <br>> Total Call created | | 1834024 <br>> Current Call | 0 | <br>
> -------------------------+---------------------------+--------------------------<br>> Successful call | 0 | 1834024 <br>> Failed call | 0 | 0 <br>
> -------------------------+---------------------------+--------------------------<br>> Response Time 1 | 00:00:00:000 | 00:00:00:149 <br>> Call Length | 00:00:00:000 | 00:00:05:158 <br>
> ------------------------------ Test Terminated<br>> --------------------------------<br>><br>><br>> After stopping the load FS still hogs 22.1% of memory.<br>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br>
> 15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch <br>><br>
><br>
> The symptoms of the crash are the same, just now with higher CPS and takes<br>> more time (more calls ) before crashing.<br>><br>> I will appreciate any suggestion.<br>><br>> Regards,<br>> Boris Ratner.<br>
><br>><br>><br>> On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <<a href="mailto:ratner2@gmail.com" target="_blank">ratner2@gmail.com</a>> wrote:<br>>><br>>> The original test was done on git master at the date mentioned. The sqlite<br>
>> core.db file was on /run/shm which is a tmpfs on unbuntu 12.04.<br>>> I will be recompiling from git master and test running with -nosql.<br>>><br>>> Testing my existing setup with -nosql seems more stable now running at<br>
>> 210CPS for some time (500k calls already passed) with ~35% idle cpu.<br>>> But the free mem is slowly going down. I will let it run untill the kernel<br>>> will kill it to see how many calls it can handle.<br>
>><br>>> During my tests i did not run FS with RT priority but according to htop<br>>> some of the threads are scheduled as RT.<br>>> My setup is doing bypass-media , thus FS handling only call establishment<br>
>> and teardown on both legs.<br>>><br>>> cat /proc/<FS pid>/status<br>>><br>>> Name: freeswitch<br>>> State: S (sleeping)<br>>> Tgid: 15995<br>>> Pid: 15995<br>
>> PPid: 1<br>>> TracerPid: 0<br>>> Uid: 999 999 999 999<br>>> Gid: 999 999 999 999<br>>> FDSize: 64<br>>> Groups:<br>>> VmPeak: 5002808 kB<br>
>> VmSize: 5002088 kB<br>>> VmLck: 0 kB<br>>> VmPin: 0 kB<br>>> VmHWM: 625900 kB<br>>> VmRSS: 624156 kB <-- this is going up<br>>> VmData: 4855788 kB<br>>> VmStk: 136 kB<br>
>> VmExe: 20 kB<br>>> VmLib: 18288 kB<br>>> VmPTE: 2352 kB<br>>> VmSwap: 0 kB<br>>> Threads: 1866<br>>> SigQ: 0/18446744073709551615<br>>> SigPnd: 0000000000000000<br>
>> ShdPnd: 0000000000000000<br>>> SigBlk: 0000000000000000<br>>> SigIgn: 0000000010003006<br>>> SigCgt: 0000000180014209<br>>> CapInh: 0000000000000000<br>>> CapPrm: 0000000000000000<br>
>> CapEff: 0000000000000000<br>>> CapBnd: ffffffffffffffff<br>>> Cpus_allowed: ffffff<br>>> Cpus_allowed_list: 0-23<br>>> Mems_allowed: 00000000,00000003<br>>> Mems_allowed_list: 0-1<br>
>> voluntary_ctxt_switches: 1803<br>>> nonvoluntary_ctxt_switches: 23<br>>><br>>><br>>> output of 'top -H' at 180CPS<br>>><br>>><br>>> top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07,<br>
>> 65.03<br>>> Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie<br>>> Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, <br>>> 0.0%st<br>>> Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers<br>
>> Swap: 0k total, 0k used, 0k free, 1165868k cached<br>>><br>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br>
>> 16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch <br>>> 16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch <br>
>> 16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch <br>>> 16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch <br>
>> 16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch <br>>> 19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top <br>
>> 18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch <br>>> 16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch <br>
>> 16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch <br>>> 16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch <br>
>> 16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch <br>>> 16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch <br>
>> <more of the above><br>>> ....<br>>> ....<br>>><br>>><br>>> Thanks to all of you ,<br>>> Boris Ratner.<br>>><br>>> On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko <<a href="mailto:dvl36.ripe.nick@gmail.com" target="_blank">dvl36.ripe.nick@gmail.com</a>><br>
>> wrote:<br>>>><br>>>> I can't reproduce such cps load on my ARMv5TE system. )<br>>>> bratner, please give us 'top -H'. I guess freeswitch running at realtime<br>>>> priority. <br>
>>><br>>>><br>>>> 2013/3/8 Ken Rice <<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>><br>>>>><br>>>>> Sqlite is probably getting hammered... Trust me... Mount the fs db dir<br>
>>>> as tmpfs or use the –nosql flag when starting freeswitch<br>>>>><br>>>>> I routinely run dialer traffic at much higher CPS then that<br>>>>><br>>>>><br>>>>><br>
>>>> On 3/7/13 7:58 PM, "Dmitry Lysenko" <<a href="mailto:dvl36.ripe.nick@gmail.com" target="_blank">dvl36.ripe.nick@gmail.com</a>> wrote:<br>>>>><br>>>>> bi, bo and wa field is low, so it seems that is not disk subsystem.<br>
>>>><br>>>>><br>>>>> 2013/3/8 Ken Rice <<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>><br>>>>><br>>>>> You are probably hammering the disk subsystem... Keep in mind that FS<br>
>>>> uses multiple sqlite databases by default... Mount the fs db dir as tmpfs<br>>>>> and try again<br>>>>><br>>>>><br>>>>><br>>>>> On 3/7/13 7:35 PM, "Dmitry Lysenko" <<a href="mailto:dvl36.ripe.nick@gmail.com" target="_blank">dvl36.ripe.nick@gmail.com</a><br>
>>>> <<a href="http://dvl36.ripe.nick@gmail.com" target="_blank">http://dvl36.ripe.nick@gmail.com</a>> > wrote:<br>>>>><br>>>>> Hm... But what about huge interrupt and context switching number?<br>
>>>><br>>>>><br>>>>> ________________________________<br>>>>><br>>>>> _________________________________________________________________________<br>>>>> Professional FreeSWITCH Consulting Services:<br>
>>>> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>>>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>><br>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
>>>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>>>>><br>>>>> Official FreeSWITCH Sites<br>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>>>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>>>>><br>
>>>> FreeSWITCH-users mailing list<br>
>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>>>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>><br>>>>><br>>>>> --<br>>>>> Ken<br>>>>> <a href="http://www.FreeSWITCH.org" target="_blank">http://www.FreeSWITCH.org</a><br>>>>> <a href="http://www.ClueCon.com" target="_blank">http://www.ClueCon.com</a><br>
>>>> <a href="http://www.OSTAG.org" target="_blank">http://www.OSTAG.org</a><br>>>>> <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch<br>>>>><br>>>>><br>
>>>> _________________________________________________________________________<br>
>>>> Professional FreeSWITCH Consulting Services:<br>>>>> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>>>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>><br>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>>>>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>>>>><br>>>>> Official FreeSWITCH Sites<br>
>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>>>>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>>>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>><br>>>>> FreeSWITCH-users mailing list<br>>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>>>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>><br>>>><br>>>><br>>>> _________________________________________________________________________<br>>>> Professional FreeSWITCH Consulting Services:<br>>>> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>>>><br>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>>>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
>>><br>>>> Official FreeSWITCH Sites<br>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>>>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>>>><br>>>> FreeSWITCH-users mailing list<br>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>>>><br>>><br>><br>><br>> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>><br>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>><br>> Official FreeSWITCH Sites<br>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>><br>> FreeSWITCH-users mailing list<br>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
</div></div><br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div>