[Freeswitch-users] Apparent performance bottleneck and FS freeze

Mon Jul 22 23:49:55 MSD 2013

Disabled in all profiles:
<param name="manage-presence" value="false"/>

On 13-07-22 03:40 PM, Anthony Minessale wrote:
> Did you turn of the presence in the sofia profile, that is enabled by 
> default and very costly.  comment out manage-presence
>
>
> On Mon, Jul 22, 2013 at 2:32 PM, Victor Chukalovskiy 
> <victor.chukalovskiy at gmail.com <mailto:victor.chukalovskiy at gmail.com>> 
> wrote:
>
>     Michael, Ken,
>
>     Thank you for the response. Firstly, let me clarify setup for each
>     of DBs in use:
>
>     *core_db:* sqlite, /dev/shm/core.db (RAM) - using default 50 max
>     handles
>     *mod_sofia db**:* sqlite, /opt/fs/db (HDD)
>     *mod_lcr db:* Postgres
>     *mod_limit:* hash backend
>
>     Note: I believe we rarely use mod_sofia DB's...this is because we
>     don't use any registrations, acl only. When I list sofia
>     per-profile db files on production box, most of them has not been
>     edited for a couple days. Are my conclusions correct here, or do I
>     miss something? Maybe they are accessed a lot but only for reading?
>
>     I understand recommendations from Ken and from Michaels are along
>     the lines of increasing core db and sofia db throughput. However
>     here is what puzzles me:
>     -the db used most extensively is core_db. It' already in the ram.
>     -we don't have much load on sofia dbs (or am I mistaken in my logic?)
>     -error log started with:
>
>
>     [CRIT] switch_time.c:967 Virtual Migration Detected! Syncing Clock
>     [WARNING] switch_scheduler.c:115 Task was executed late by 19
>     seconds 1 heartbeat (core)
>
>     Does it mean that there were abnormal events prior hitting 50 max
>     handles limit?
>
>     Thank you,
>     -Victor
>
>
>     On 13-07-22 02:52 PM, Anthony Minessale wrote:
>>     Check the odbcinst.ini settings from the wiki or use native pg mode.
>>     increase the max number of pg connections on your server and turn
>>     up FS to match in max-db-handles in switch.conf.xml
>>
>>     You are already beating it up pretty good esp for a fake server
>>     though.  2500 is a lot.
>>
>>
>>
>>     On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice <krice at freeswitch.org
>>     <mailto:krice at freeswitch.org>> wrote:
>>
>>         Mount the entire freeswitch/db dir as tmpfs... most of these
>>         errors are for
>>         the sofia DB getting backed up...
>>
>>         The coredb is a completely different sqlite db from the sofia db
>>
>>         On 7/22/13 12:54 PM, "Victor Chukalovskiy"
>>         <victor.chukalovskiy at gmail.com
>>         <mailto:victor.chukalovskiy at gmail.com>>
>>         wrote:
>>
>>         > Hello,
>>         >
>>         > Need some help finding the bottleneck here. Running
>>         1.2.stable on a
>>         > decent multi-core vmware platform. No RTP, SIP signaling
>>         only. Around
>>         > 2500 sessions and 30 average CPS load. Essentially, it's a
>>         multi-profile
>>         > .xml config that uses mod_lcr extensively (postgres DB).
>>         Yesterday it
>>         > stopped processing calls with the following log output:
>>         >
>>         > 2013-07-21 06:00:37.642678 [CRIT] switch_time.c:967 Virtual
>>         Migration
>>         > Detected! Syncing Clock
>>         > 2013-07-21 06:01:07.141734 [WARNING] switch_scheduler.c:115
>>         Task was
>>         > executed late by 19 seconds 1 heartbeat (core)
>>         > 2013-07-21 06:02:32.402668 [CRIT] switch_time.c:967 Virtual
>>         Migration
>>         > Detected! Syncing Clock
>>         > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115
>>         Task was
>>         > executed late by 115 seconds 1 heartbeat (core)
>>         > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115
>>         Task was
>>         > executed late by 74 seconds 2 check_ip (core)
>>         > 2013-07-21 06:05:15.102683 [CRIT] switch_time.c:967 Virtual
>>         Migration
>>         > Detected! Syncing Clock
>>         > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115
>>         Task was
>>         > executed late by 22 seconds 1 heartbeat (core)
>>         > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115
>>         Task was
>>         > executed late by 2 seconds 2 check_ip (core)
>>         > 2013-07-21 06:05:49.322672 [WARNING] mod_lcr.c:448 Max
>>         handles 50
>>         > exceeded, blocking....
>>         > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max
>>         handles 50
>>         > exceeded, blocking....
>>         > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max
>>         handles 50
>>         > exceeded, blocking....
>>         > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max
>>         handles 50
>>         > exceeded, blocking....
>>         > ................................
>>         > 2013-07-21 06:05:53.962672 [WARNING] sofia_glue.c:6605 Max
>>         handles 50
>>         > exceeded, blocking....
>>         > 2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448 Error connecting
>>         > 2013-07-21 06:05:54.422671 [WARNING] mod_lcr.c:1779 LCR
>>         lookup failed
>>         > for 14389955082
>>         > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max
>>         handles 50
>>         > exceeded, blocking....
>>         > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max
>>         handles 50
>>         > exceeded, blocking....
>>         > ................................
>>         > 2013-07-21 06:05:55.962758 [WARNING] sofia_glue.c:6605 Max
>>         handles 50
>>         > exceeded, blocking....
>>         > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error
>>         connecting
>>         > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error
>>         Opening DB
>>         > 2013-07-21 06:05:56.062671 [WARNING] sofia_glue.c:6605 Max
>>         handles 50
>>         > exceeded, blocking....
>>         > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error
>>         connecting
>>         > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error
>>         Opening DB
>>         >
>>         > FreeSWITCH stayed unusable since then until it was restarted.
>>         >
>>         > FYI core DB is already in the RAM:
>>         > <param name="core-db-name" value="/dev/shm/core.db" />
>>         >
>>         >
>>         > Thank you,
>>         > Victor
>>         >
>>         >
>>         _________________________________________________________________________
>>         > Professional FreeSWITCH Consulting Services:
>>         > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>         > http://www.freeswitchsolutions.com
>>         >
>>         > 
>>         > 
>>         >
>>         > Official FreeSWITCH Sites
>>         > http://www.freeswitch.org
>>         > http://wiki.freeswitch.org
>>         > http://www.cluecon.com
>>         >
>>         > FreeSWITCH-users mailing list
>>         > FreeSWITCH-users at lists.freeswitch.org
>>         <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>         > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>         >
>>         UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>         > http://www.freeswitch.org
>>
>>         --
>>         Ken
>>         http://www.FreeSWITCH.org
>>         http://www.ClueCon.com
>>         http://www.OSTAG.org
>>         irc.freenode.net <http://irc.freenode.net> #freeswitch
>>
>>
>>
>>         _________________________________________________________________________
>>         Professional FreeSWITCH Consulting Services:
>>         consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>         http://www.freeswitchsolutions.com
>>
>>         
>>         
>>
>>         Official FreeSWITCH Sites
>>         http://www.freeswitch.org
>>         http://wiki.freeswitch.org
>>         http://www.cluecon.com
>>
>>         FreeSWITCH-users mailing list
>>         FreeSWITCH-users at lists.freeswitch.org
>>         <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>         http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>         UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>         http://www.freeswitch.org
>>
>>
>>
>>
>>     -- 
>>     Anthony Minessale II
>>
>>     FreeSWITCH http://www.freeswitch.org/
>>     ClueCon http://www.cluecon.com/
>>     Twitter: http://twitter.com/FreeSWITCH_wire
>>
>>     AIM: anthm
>>     MSN:anthony_minessale at hotmail.com
>>     <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>     GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>     <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>     IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>
>>     FreeSWITCH Developer Conference
>>     sip:888 at conference.freeswitch.org
>>     <mailto:sip%3A888 at conference.freeswitch.org>
>>     googletalk:conf+888 at conference.freeswitch.org
>>     <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>     pstn:+19193869900
>>
>>
>>     _________________________________________________________________________
>>     Professional FreeSWITCH Consulting Services:
>>     consulting at freeswitch.org  <mailto:consulting at freeswitch.org>
>>     http://www.freeswitchsolutions.com
>>
>>     
>>     
>>
>>     Official FreeSWITCH Sites
>>     http://www.freeswitch.org
>>     http://wiki.freeswitch.org
>>     http://www.cluecon.com
>>
>>     FreeSWITCH-users mailing list
>>     FreeSWITCH-users at lists.freeswitch.org  <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>     http://www.freeswitch.org
>
>
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     http://www.freeswitchsolutions.com
>
>     
>     
>
>     Official FreeSWITCH Sites
>     http://www.freeswitch.org
>     http://wiki.freeswitch.org
>     http://www.cluecon.com
>
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
>
>
>
>
> -- 
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com 
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com 
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org 
> <mailto:sip%3A888 at conference.freeswitch.org>
> googletalk:conf+888 at conference.freeswitch.org 
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:+19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130722/4bb7eced/attachment-0001.html