Michael, Ken, Thank you for the response. Firstly, let me clarify setup for each of DBs in use: *core_db:* sqlite, /dev/shm/core.db (RAM) - using default 50 max handles *mod_sofia db**:* sqlite, /opt/fs/db (HDD) *mod_lcr db:* Postgres *mod_limit:* hash backend Note: I believe we rarely use mod_sofia DB's...this is because we don't use any registrations, acl only. When I list sofia per-profile db files on production box, most of them has not been edited for a couple days. Are my conclusions correct here, or do I miss something? Maybe they are accessed a lot but only for reading? I understand recommendations from Ken and from Michaels are along the lines of increasing core db and sofia db throughput. However here is what puzzles me: -the db used most extensively is core_db. It' already in the ram. -we don't have much load on sofia dbs (or am I mistaken in my logic?) -error log started with: [CRIT] switch_time.c:967 Virtual Migration Detected! Syncing Clock [WARNING] switch_scheduler.c:115 Task was executed late by 19 seconds 1 heartbeat (core) Does it mean that there were abnormal events prior hitting 50 max handles limit? Thank you, -Victor On 13-07-22 02:52 PM, Anthony Minessale wrote:
Check the odbcinst.ini settings from the wiki or use native pg mode. increase the max number of pg connections on your server and turn up FS to match in max-db-handles in switch.conf.xml
You are already beating it up pretty good esp for a fake server though. 2500 is a lot.
On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice <krice@freeswitch.org <mailto:krice@freeswitch.org>> wrote:
Mount the entire freeswitch/db dir as tmpfs... most of these errors are for the sofia DB getting backed up...
The coredb is a completely different sqlite db from the sofia db
On 7/22/13 12:54 PM, "Victor Chukalovskiy" <victor.chukalovskiy@gmail.com <mailto:victor.chukalovskiy@gmail.com>> wrote:
> Hello, > > Need some help finding the bottleneck here. Running 1.2.stable on a > decent multi-core vmware platform. No RTP, SIP signaling only. Around > 2500 sessions and 30 average CPS load. Essentially, it's a multi-profile > .xml config that uses mod_lcr extensively (postgres DB). Yesterday it > stopped processing calls with the following log output: > > 2013-07-21 06:00:37.642678 [CRIT] switch_time.c:967 Virtual Migration > Detected! Syncing Clock > 2013-07-21 06:01:07.141734 [WARNING] switch_scheduler.c:115 Task was > executed late by 19 seconds 1 heartbeat (core) > 2013-07-21 06:02:32.402668 [CRIT] switch_time.c:967 Virtual Migration > Detected! Syncing Clock > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was > executed late by 115 seconds 1 heartbeat (core) > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was > executed late by 74 seconds 2 check_ip (core) > 2013-07-21 06:05:15.102683 [CRIT] switch_time.c:967 Virtual Migration > Detected! Syncing Clock > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was > executed late by 22 seconds 1 heartbeat (core) > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was > executed late by 2 seconds 2 check_ip (core) > 2013-07-21 06:05:49.322672 [WARNING] mod_lcr.c:448 Max handles 50 > exceeded, blocking.... > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles 50 > exceeded, blocking.... > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles 50 > exceeded, blocking.... > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles 50 > exceeded, blocking.... > ................................ > 2013-07-21 06:05:53.962672 [WARNING] sofia_glue.c:6605 Max handles 50 > exceeded, blocking.... > 2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448 Error connecting > 2013-07-21 06:05:54.422671 [WARNING] mod_lcr.c:1779 LCR lookup failed > for 14389955082 > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max handles 50 > exceeded, blocking.... > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max handles 50 > exceeded, blocking.... > ................................ > 2013-07-21 06:05:55.962758 [WARNING] sofia_glue.c:6605 Max handles 50 > exceeded, blocking.... > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB > 2013-07-21 06:05:56.062671 [WARNING] sofia_glue.c:6605 Max handles 50 > exceeded, blocking.... > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB > > FreeSWITCH stayed unusable since then until it was restarted. > > FYI core DB is already in the RAM: > <param name="core-db-name" value="/dev/shm/core.db" /> > > > Thank you, > Victor > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net <http://irc.freenode.net> #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org