[Freeswitch-users] Apparent performance bottleneck and FS freeze

[Victor Chukalovskiy]

Hello,

Need some help finding the bottleneck here. Running 1.2.stable on a 
decent multi-core vmware platform. No RTP, SIP signaling only. Around 
2500 sessions and 30 average CPS load. Essentially, it's a multi-profile 
.xml config that uses mod_lcr extensively (postgres DB). Yesterday it 
stopped processing calls with the following log output:

2013-07-21 06:00:37.642678 [CRIT] switch_time.c:967 Virtual Migration 
Detected! Syncing Clock
2013-07-21 06:01:07.141734 [WARNING] switch_scheduler.c:115 Task was 
executed late by 19 seconds 1 heartbeat (core)
2013-07-21 06:02:32.402668 [CRIT] switch_time.c:967 Virtual Migration 
Detected! Syncing Clock
2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was 
executed late by 115 seconds 1 heartbeat (core)
2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was 
executed late by 74 seconds 2 check_ip (core)
2013-07-21 06:05:15.102683 [CRIT] switch_time.c:967 Virtual Migration 
Detected! Syncing Clock
2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was 
executed late by 22 seconds 1 heartbeat (core)
2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was 
executed late by 2 seconds 2 check_ip (core)
2013-07-21 06:05:49.322672 [WARNING] mod_lcr.c:448 Max handles 50 
exceeded, blocking....
2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles 50 
exceeded, blocking....
2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles 50 
exceeded, blocking....
2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles 50 
exceeded, blocking....
................................
2013-07-21 06:05:53.962672 [WARNING] sofia_glue.c:6605 Max handles 50 
exceeded, blocking....
2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448 Error connecting
2013-07-21 06:05:54.422671 [WARNING] mod_lcr.c:1779 LCR lookup failed 
for 14389955082
2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max handles 50 
exceeded, blocking....
2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max handles 50 
exceeded, blocking....
................................
2013-07-21 06:05:55.962758 [WARNING] sofia_glue.c:6605 Max handles 50 
exceeded, blocking....
2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB
2013-07-21 06:05:56.062671 [WARNING] sofia_glue.c:6605 Max handles 50 
exceeded, blocking....
2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB

FreeSWITCH stayed unusable since then until it was restarted.

FYI core DB is already in the RAM:
<param name="core-db-name" value="/dev/shm/core.db" />


Thank you,
Victor