[Freeswitch-users] Apparent performance bottleneck and FS freeze
Victor Chukalovskiy
victor.chukalovskiy at gmail.com
Mon Jul 22 23:32:28 MSD 2013
Michael, Ken,
Thank you for the response. Firstly, let me clarify setup for each of
DBs in use:
*core_db:* sqlite, /dev/shm/core.db (RAM) - using default 50 max handles
*mod_sofia db**:* sqlite, /opt/fs/db (HDD)
*mod_lcr db:* Postgres
*mod_limit:* hash backend
Note: I believe we rarely use mod_sofia DB's...this is because we don't
use any registrations, acl only. When I list sofia per-profile db files
on production box, most of them has not been edited for a couple days.
Are my conclusions correct here, or do I miss something? Maybe they are
accessed a lot but only for reading?
I understand recommendations from Ken and from Michaels are along the
lines of increasing core db and sofia db throughput. However here is
what puzzles me:
-the db used most extensively is core_db. It' already in the ram.
-we don't have much load on sofia dbs (or am I mistaken in my logic?)
-error log started with:
[CRIT] switch_time.c:967 Virtual Migration Detected! Syncing Clock
[WARNING] switch_scheduler.c:115 Task was executed late by 19 seconds 1
heartbeat (core)
Does it mean that there were abnormal events prior hitting 50 max
handles limit?
Thank you,
-Victor
On 13-07-22 02:52 PM, Anthony Minessale wrote:
> Check the odbcinst.ini settings from the wiki or use native pg mode.
> increase the max number of pg connections on your server and turn up
> FS to match in max-db-handles in switch.conf.xml
>
> You are already beating it up pretty good esp for a fake server
> though. 2500 is a lot.
>
>
>
> On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice <krice at freeswitch.org
> <mailto:krice at freeswitch.org>> wrote:
>
> Mount the entire freeswitch/db dir as tmpfs... most of these
> errors are for
> the sofia DB getting backed up...
>
> The coredb is a completely different sqlite db from the sofia db
>
> On 7/22/13 12:54 PM, "Victor Chukalovskiy"
> <victor.chukalovskiy at gmail.com <mailto:victor.chukalovskiy at gmail.com>>
> wrote:
>
> > Hello,
> >
> > Need some help finding the bottleneck here. Running 1.2.stable on a
> > decent multi-core vmware platform. No RTP, SIP signaling only.
> Around
> > 2500 sessions and 30 average CPS load. Essentially, it's a
> multi-profile
> > .xml config that uses mod_lcr extensively (postgres DB).
> Yesterday it
> > stopped processing calls with the following log output:
> >
> > 2013-07-21 06:00:37.642678 [CRIT] switch_time.c:967 Virtual
> Migration
> > Detected! Syncing Clock
> > 2013-07-21 06:01:07.141734 [WARNING] switch_scheduler.c:115 Task was
> > executed late by 19 seconds 1 heartbeat (core)
> > 2013-07-21 06:02:32.402668 [CRIT] switch_time.c:967 Virtual
> Migration
> > Detected! Syncing Clock
> > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was
> > executed late by 115 seconds 1 heartbeat (core)
> > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was
> > executed late by 74 seconds 2 check_ip (core)
> > 2013-07-21 06:05:15.102683 [CRIT] switch_time.c:967 Virtual
> Migration
> > Detected! Syncing Clock
> > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was
> > executed late by 22 seconds 1 heartbeat (core)
> > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was
> > executed late by 2 seconds 2 check_ip (core)
> > 2013-07-21 06:05:49.322672 [WARNING] mod_lcr.c:448 Max handles 50
> > exceeded, blocking....
> > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max
> handles 50
> > exceeded, blocking....
> > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max
> handles 50
> > exceeded, blocking....
> > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max
> handles 50
> > exceeded, blocking....
> > ................................
> > 2013-07-21 06:05:53.962672 [WARNING] sofia_glue.c:6605 Max
> handles 50
> > exceeded, blocking....
> > 2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448 Error connecting
> > 2013-07-21 06:05:54.422671 [WARNING] mod_lcr.c:1779 LCR lookup
> failed
> > for 14389955082
> > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max
> handles 50
> > exceeded, blocking....
> > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max
> handles 50
> > exceeded, blocking....
> > ................................
> > 2013-07-21 06:05:55.962758 [WARNING] sofia_glue.c:6605 Max
> handles 50
> > exceeded, blocking....
> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB
> > 2013-07-21 06:05:56.062671 [WARNING] sofia_glue.c:6605 Max
> handles 50
> > exceeded, blocking....
> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB
> >
> > FreeSWITCH stayed unusable since then until it was restarted.
> >
> > FYI core DB is already in the RAM:
> > <param name="core-db-name" value="/dev/shm/core.db" />
> >
> >
> > Thank you,
> > Victor
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > http://www.freeswitchsolutions.com
> >
> >
> >
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
>
> --
> Ken
> http://www.FreeSWITCH.org
> http://www.ClueCon.com
> http://www.OSTAG.org
> irc.freenode.net <http://irc.freenode.net> #freeswitch
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> <mailto:sip%3A888 at conference.freeswitch.org>
> googletalk:conf+888 at conference.freeswitch.org
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:+19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130722/dad3a062/attachment-0001.html
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list