[Freeswitch-users] Apparent performance bottleneck and FS freeze

Victor Chukalovskiy victor.chukalovskiy at gmail.com
Mon Jul 22 23:32:28 MSD 2013


Michael, Ken,

Thank you for the response. Firstly, let me clarify setup for each of 
DBs in use:

*core_db:* sqlite, /dev/shm/core.db (RAM) - using default 50 max handles
*mod_sofia db**:* sqlite, /opt/fs/db (HDD)
*mod_lcr db:* Postgres
*mod_limit:* hash backend

Note: I believe we rarely use mod_sofia DB's...this is because we don't 
use any registrations, acl only. When I list sofia per-profile db files 
on production box, most of them has not been edited for a couple days. 
Are my conclusions correct here, or do I miss something? Maybe they are 
accessed a lot but only for reading?

I understand recommendations from Ken and from Michaels are along the 
lines of increasing core db and sofia db throughput. However here is 
what puzzles me:
-the db used most extensively is core_db. It' already in the ram.
-we don't have much load on sofia dbs (or am I mistaken in my logic?)
-error log started with:

[CRIT] switch_time.c:967 Virtual Migration Detected! Syncing Clock
[WARNING] switch_scheduler.c:115 Task was executed late by 19 seconds 1 
heartbeat (core)

Does it mean that there were abnormal events prior hitting 50 max 
handles limit?

Thank you,
-Victor

On 13-07-22 02:52 PM, Anthony Minessale wrote:
> Check the odbcinst.ini settings from the wiki or use native pg mode.
> increase the max number of pg connections on your server and turn up 
> FS to match in max-db-handles in switch.conf.xml
>
> You are already beating it up pretty good esp for a fake server 
> though.  2500 is a lot.
>
>
>
> On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice <krice at freeswitch.org 
> <mailto:krice at freeswitch.org>> wrote:
>
>     Mount the entire freeswitch/db dir as tmpfs... most of these
>     errors are for
>     the sofia DB getting backed up...
>
>     The coredb is a completely different sqlite db from the sofia db
>
>     On 7/22/13 12:54 PM, "Victor Chukalovskiy"
>     <victor.chukalovskiy at gmail.com <mailto:victor.chukalovskiy at gmail.com>>
>     wrote:
>
>     > Hello,
>     >
>     > Need some help finding the bottleneck here. Running 1.2.stable on a
>     > decent multi-core vmware platform. No RTP, SIP signaling only.
>     Around
>     > 2500 sessions and 30 average CPS load. Essentially, it's a
>     multi-profile
>     > .xml config that uses mod_lcr extensively (postgres DB).
>     Yesterday it
>     > stopped processing calls with the following log output:
>     >
>     > 2013-07-21 06:00:37.642678 [CRIT] switch_time.c:967 Virtual
>     Migration
>     > Detected! Syncing Clock
>     > 2013-07-21 06:01:07.141734 [WARNING] switch_scheduler.c:115 Task was
>     > executed late by 19 seconds 1 heartbeat (core)
>     > 2013-07-21 06:02:32.402668 [CRIT] switch_time.c:967 Virtual
>     Migration
>     > Detected! Syncing Clock
>     > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was
>     > executed late by 115 seconds 1 heartbeat (core)
>     > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was
>     > executed late by 74 seconds 2 check_ip (core)
>     > 2013-07-21 06:05:15.102683 [CRIT] switch_time.c:967 Virtual
>     Migration
>     > Detected! Syncing Clock
>     > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was
>     > executed late by 22 seconds 1 heartbeat (core)
>     > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was
>     > executed late by 2 seconds 2 check_ip (core)
>     > 2013-07-21 06:05:49.322672 [WARNING] mod_lcr.c:448 Max handles 50
>     > exceeded, blocking....
>     > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max
>     handles 50
>     > exceeded, blocking....
>     > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max
>     handles 50
>     > exceeded, blocking....
>     > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max
>     handles 50
>     > exceeded, blocking....
>     > ................................
>     > 2013-07-21 06:05:53.962672 [WARNING] sofia_glue.c:6605 Max
>     handles 50
>     > exceeded, blocking....
>     > 2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448 Error connecting
>     > 2013-07-21 06:05:54.422671 [WARNING] mod_lcr.c:1779 LCR lookup
>     failed
>     > for 14389955082
>     > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max
>     handles 50
>     > exceeded, blocking....
>     > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max
>     handles 50
>     > exceeded, blocking....
>     > ................................
>     > 2013-07-21 06:05:55.962758 [WARNING] sofia_glue.c:6605 Max
>     handles 50
>     > exceeded, blocking....
>     > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
>     > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB
>     > 2013-07-21 06:05:56.062671 [WARNING] sofia_glue.c:6605 Max
>     handles 50
>     > exceeded, blocking....
>     > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
>     > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB
>     >
>     > FreeSWITCH stayed unusable since then until it was restarted.
>     >
>     > FYI core DB is already in the RAM:
>     > <param name="core-db-name" value="/dev/shm/core.db" />
>     >
>     >
>     > Thank you,
>     > Victor
>     >
>     >
>     _________________________________________________________________________
>     > Professional FreeSWITCH Consulting Services:
>     > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     > http://www.freeswitchsolutions.com
>     >
>     > 
>     > 
>     >
>     > Official FreeSWITCH Sites
>     > http://www.freeswitch.org
>     > http://wiki.freeswitch.org
>     > http://www.cluecon.com
>     >
>     > FreeSWITCH-users mailing list
>     > FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     >
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     > http://www.freeswitch.org
>
>     --
>     Ken
>     http://www.FreeSWITCH.org
>     http://www.ClueCon.com
>     http://www.OSTAG.org
>     irc.freenode.net <http://irc.freenode.net> #freeswitch
>
>
>
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     http://www.freeswitchsolutions.com
>
>     
>     
>
>     Official FreeSWITCH Sites
>     http://www.freeswitch.org
>     http://wiki.freeswitch.org
>     http://www.cluecon.com
>
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
>
>
>
>
> -- 
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com 
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com 
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org 
> <mailto:sip%3A888 at conference.freeswitch.org>
> googletalk:conf+888 at conference.freeswitch.org 
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:+19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130722/dad3a062/attachment-0001.html 


Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list