Michael, Ken,

Thank you for the response. Firstly, let me clarify setup for each of DBs in use:

core_db: sqlite, /dev/shm/core.db (RAM) - using default 50 max handles
mod_sofia db: sqlite, /opt/fs/db (HDD)
mod_lcr db: Postgres
mod_limit: hash backend

Note: I believe we rarely use mod_sofia DB's...this is because we don't use any registrations, acl only. When I list sofia per-profile db files on production box, most of them has not been edited for a couple days. Are my conclusions correct here, or do I miss something? Maybe they are accessed a lot but only for reading?

I understand recommendations from Ken and from Michaels are along the lines of increasing core db and sofia db throughput. However here is what puzzles me:
-the db used most extensively is core_db. It' already in the ram.
-we don't have much load on sofia dbs (or am I mistaken in my logic?)
-error log started with:

[CRIT] switch_time.c:967 Virtual Migration Detected! Syncing Clock

[WARNING] switch_scheduler.c:115 Task was executed late by 19 seconds 1 heartbeat (core)

Does it mean that there were abnormal events prior hitting 50 max handles limit?

Thank you,
-Victor

On 13-07-22 02:52 PM, Anthony Minessale wrote:

Check the odbcinst.ini settings from the wiki or use native pg mode.

increase the max number of pg connections on your server and turn up FS to match in max-db-handles in switch.conf.xml

You are already beating it up pretty good esp for a fake server though.  2500 is a lot.

On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice <krice@freeswitch.org> wrote:

Mount the entire freeswitch/db dir as tmpfs... most of these errors are for
the sofia DB getting backed up...

The coredb is a completely different sqlite db from the sofia db

On 7/22/13 12:54 PM, "Victor Chukalovskiy" <victor.chukalovskiy@gmail.com>
wrote:

> Hello,
>
> Need some help finding the bottleneck here. Running 1.2.stable on a
> decent multi-core vmware platform. No RTP, SIP signaling only. Around
> 2500 sessions and 30 average CPS load. Essentially, it's a multi-profile
> .xml config that uses mod_lcr extensively (postgres DB). Yesterday it
> stopped processing calls with the following log output:
>
> 2013-07-21 06:00:37.642678 [CRIT] switch_time.c:967 Virtual Migration
> Detected! Syncing Clock
> 2013-07-21 06:01:07.141734 [WARNING] switch_scheduler.c:115 Task was
> executed late by 19 seconds 1 heartbeat (core)
> 2013-07-21 06:02:32.402668 [CRIT] switch_time.c:967 Virtual Migration
> Detected! Syncing Clock
> 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was
> executed late by 115 seconds 1 heartbeat (core)
> 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was
> executed late by 74 seconds 2 check_ip (core)
> 2013-07-21 06:05:15.102683 [CRIT] switch_time.c:967 Virtual Migration
> Detected! Syncing Clock
> 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was
> executed late by 22 seconds 1 heartbeat (core)
> 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was
> executed late by 2 seconds 2 check_ip (core)
> 2013-07-21 06:05:49.322672 [WARNING] mod_lcr.c:448 Max handles 50
> exceeded, blocking....
> 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles 50
> exceeded, blocking....
> 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles 50
> exceeded, blocking....
> 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles 50
> exceeded, blocking....
> ................................
> 2013-07-21 06:05:53.962672 [WARNING] sofia_glue.c:6605 Max handles 50
> exceeded, blocking....
> 2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448 Error connecting
> 2013-07-21 06:05:54.422671 [WARNING] mod_lcr.c:1779 LCR lookup failed
> for 14389955082
> 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max handles 50
> exceeded, blocking....
> 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max handles 50
> exceeded, blocking....
> ................................
> 2013-07-21 06:05:55.962758 [WARNING] sofia_glue.c:6605 Max handles 50
> exceeded, blocking....
> 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
> 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB
> 2013-07-21 06:05:56.062671 [WARNING] sofia_glue.c:6605 Max handles 50
> exceeded, blocking....
> 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
> 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB
>
> FreeSWITCH stayed unusable since then until it was restarted.
>
> FYI core DB is already in the RAM:
> <param name="core-db-name" value="/dev/shm/core.db" />
>
>
> Thank you,
> Victor
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting@freeswitch.org
> http://www.freeswitchsolutions.com
>
> FreeSWITCH-powered IP PBX: The CudaTel Communication Server
> http://www.cudatel.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users@lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

--
Ken
http://www.FreeSWITCH.org
http://www.ClueCon.com
http://www.OSTAG.org
irc.freenode.net #freeswitch

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

--
Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm
MSN:anthony_minessale@hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com
IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference
sip:888@conference.freeswitch.org
googletalk:conf+888@conference.freeswitch.org
pstn:+19193869900

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org