[Freeswitch-users] Apparent performance bottleneck and FS freeze

Tue Jul 23 01:25:40 MSD 2013

In all profiles it's commented out:

Assuming default is off, it's off.

BTW, sofia_glue.c only got 3017 lines. Please clarify.

On 13-07-22 04:42 PM, Anthony Minessale wrote:
> do you have manage-shared-appearance off too cos that re-enables 
> presence?
>
> edit sofia_glue.c on line 6614 and add a print to see what the sql is 
> that is being executed.
>
> switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_ERROR, "SQL: [%s]\n", 
> sql);
>
> then make mod_sofia-install
>
> and relaunch and try to catch a sample of what it's doing.
>
>
>
>
>
> On Mon, Jul 22, 2013 at 3:27 PM, Victor Chukalovskiy 
> <victor.chukalovskiy at gmail.com <mailto:victor.chukalovskiy at gmail.com>> 
> wrote:
>
>     Anthony,
>
>     Thank you, this is a very useful command. I see a bunch of records
>     like:
>
>     db="sofia_reg_<profile_name>"
>             Type: CORE_DB
>             Last used: 10
>             Flags: Unlocked, Detached(0)
>             Creator: *sofia_glue.c:6605*
>             Last User: *sofia_glue.c:6605*
>
>     Appears that these DBs are in use and user/creator is
>     sofia_glue.c:6605. Same as in error log:
>
>     [WARNING] *sofia_glue.c:6605* Max handles 50
>
>     And this is not a cored.db... it's sitting on the disk drive
>     still. Is there a way to setup sip_profiles not to create any
>     sofia_reg_...DBs?
>
>     Re: vmware setup. It's good to know it's not optimal, however it
>     gives rack space, administrator's time, and $ savings compared to
>     dedicated servers.
>
>     Should it be Ok to stick with VM setup provided enough CPU and RAM
>     is allocated to each FS guest? Are there reasons not to run FS on
>     VM other than CPU and RAM allocation?
>
>     Thank you,
>     Victor
>
>
>>     Its a large number of handles to have open all in sofia if you
>>     are not doing anything
>>
>>     db_cache status
>>
>>     from cli can show where those handles come from and what db they
>>     are using.
>>     Clearly the box is being overloaded past its limits.
>>
>>     This is why we don't endorse virtual servers.
>>
>>
>>
>>     On Mon, Jul 22, 2013 at 2:49 PM, Victor Chukalovskiy
>>     <victor.chukalovskiy at gmail.com
>>     <mailto:victor.chukalovskiy at gmail.com>> wrote:
>>
>>         Disabled in all profiles:
>>         <param name="manage-presence" value="false"/>
>>
>>
>>         On 13-07-22 03:40 PM, Anthony Minessale wrote:
>>>         Did you turn of the presence in the sofia profile, that is
>>>         enabled by default and very costly.  comment out manage-presence
>>>
>>>
>>>         On Mon, Jul 22, 2013 at 2:32 PM, Victor Chukalovskiy
>>>         <victor.chukalovskiy at gmail.com
>>>         <mailto:victor.chukalovskiy at gmail.com>> wrote:
>>>
>>>             Michael, Ken,
>>>
>>>             Thank you for the response. Firstly, let me clarify
>>>             setup for each of DBs in use:
>>>
>>>             *core_db:* sqlite, /dev/shm/core.db (RAM) - using
>>>             default 50 max handles
>>>             *mod_sofia db**:* sqlite, /opt/fs/db (HDD)
>>>             *mod_lcr db:* Postgres
>>>             *mod_limit:* hash backend
>>>
>>>             Note: I believe we rarely use mod_sofia DB's...this is
>>>             because we don't use any registrations, acl only. When I
>>>             list sofia per-profile db files on production box, most
>>>             of them has not been edited for a couple days. Are my
>>>             conclusions correct here, or do I miss something? Maybe
>>>             they are accessed a lot but only for reading?
>>>
>>>             I understand recommendations from Ken and from Michaels
>>>             are along the lines of increasing core db and sofia db
>>>             throughput. However here is what puzzles me:
>>>             -the db used most extensively is core_db. It' already in
>>>             the ram.
>>>             -we don't have much load on sofia dbs (or am I mistaken
>>>             in my logic?)
>>>             -error log started with:
>>>
>>>
>>>             [CRIT] switch_time.c:967 Virtual Migration Detected!
>>>             Syncing Clock
>>>             [WARNING] switch_scheduler.c:115 Task was executed late
>>>             by 19 seconds 1 heartbeat (core)
>>>
>>>             Does it mean that there were abnormal events prior
>>>             hitting 50 max handles limit?
>>>
>>>             Thank you,
>>>             -Victor
>>>
>>>
>>>             On 13-07-22 02:52 PM, Anthony Minessale wrote:
>>>>             Check the odbcinst.ini settings from the wiki or use
>>>>             native pg mode.
>>>>             increase the max number of pg connections on your
>>>>             server and turn up FS to match in max-db-handles in
>>>>             switch.conf.xml
>>>>
>>>>             You are already beating it up pretty good esp for a
>>>>             fake server though.  2500 is a lot.
>>>>
>>>>
>>>>
>>>>             On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice
>>>>             <krice at freeswitch.org <mailto:krice at freeswitch.org>> wrote:
>>>>
>>>>                 Mount the entire freeswitch/db dir as tmpfs... most
>>>>                 of these errors are for
>>>>                 the sofia DB getting backed up...
>>>>
>>>>                 The coredb is a completely different sqlite db from
>>>>                 the sofia db
>>>>
>>>>                 On 7/22/13 12:54 PM, "Victor Chukalovskiy"
>>>>                 <victor.chukalovskiy at gmail.com
>>>>                 <mailto:victor.chukalovskiy at gmail.com>>
>>>>                 wrote:
>>>>
>>>>                 > Hello,
>>>>                 >
>>>>                 > Need some help finding the bottleneck here.
>>>>                 Running 1.2.stable on a
>>>>                 > decent multi-core vmware platform. No RTP, SIP
>>>>                 signaling only. Around
>>>>                 > 2500 sessions and 30 average CPS load.
>>>>                 Essentially, it's a multi-profile
>>>>                 > .xml config that uses mod_lcr extensively
>>>>                 (postgres DB). Yesterday it
>>>>                 > stopped processing calls with the following log
>>>>                 output:
>>>>                 >
>>>>                 > 2013-07-21 06:00:37.642678 [CRIT]
>>>>                 switch_time.c:967 Virtual Migration
>>>>                 > Detected! Syncing Clock
>>>>                 > 2013-07-21 06:01:07.141734 [WARNING]
>>>>                 switch_scheduler.c:115 Task was
>>>>                 > executed late by 19 seconds 1 heartbeat (core)
>>>>                 > 2013-07-21 06:02:32.402668 [CRIT]
>>>>                 switch_time.c:967 Virtual Migration
>>>>                 > Detected! Syncing Clock
>>>>                 > 2013-07-21 06:04:42.499744 [WARNING]
>>>>                 switch_scheduler.c:115 Task was
>>>>                 > executed late by 115 seconds 1 heartbeat (core)
>>>>                 > 2013-07-21 06:04:42.499744 [WARNING]
>>>>                 switch_scheduler.c:115 Task was
>>>>                 > executed late by 74 seconds 2 check_ip (core)
>>>>                 > 2013-07-21 06:05:15.102683 [CRIT]
>>>>                 switch_time.c:967 Virtual Migration
>>>>                 > Detected! Syncing Clock
>>>>                 > 2013-07-21 06:05:44.961004 [WARNING]
>>>>                 switch_scheduler.c:115 Task was
>>>>                 > executed late by 22 seconds 1 heartbeat (core)
>>>>                 > 2013-07-21 06:05:44.961004 [WARNING]
>>>>                 switch_scheduler.c:115 Task was
>>>>                 > executed late by 2 seconds 2 check_ip (core)
>>>>                 > 2013-07-21 06:05:49.322672 [WARNING]
>>>>                 mod_lcr.c:448 Max handles 50
>>>>                 > exceeded, blocking....
>>>>                 > 2013-07-21 06:05:50.962987 [WARNING]
>>>>                 sofia_glue.c:6605 Max handles 50
>>>>                 > exceeded, blocking....
>>>>                 > 2013-07-21 06:05:50.962987 [WARNING]
>>>>                 sofia_glue.c:6605 Max handles 50
>>>>                 > exceeded, blocking....
>>>>                 > 2013-07-21 06:05:50.962987 [WARNING]
>>>>                 sofia_glue.c:6605 Max handles 50
>>>>                 > exceeded, blocking....
>>>>                 > ................................
>>>>                 > 2013-07-21 06:05:53.962672 [WARNING]
>>>>                 sofia_glue.c:6605 Max handles 50
>>>>                 > exceeded, blocking....
>>>>                 > 2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448
>>>>                 Error connecting
>>>>                 > 2013-07-21 06:05:54.422671 [WARNING]
>>>>                 mod_lcr.c:1779 LCR lookup failed
>>>>                 > for 14389955082
>>>>                 > 2013-07-21 06:05:54.962671 [WARNING]
>>>>                 sofia_glue.c:6605 Max handles 50
>>>>                 > exceeded, blocking....
>>>>                 > 2013-07-21 06:05:54.962671 [WARNING]
>>>>                 sofia_glue.c:6605 Max handles 50
>>>>                 > exceeded, blocking....
>>>>                 > ................................
>>>>                 > 2013-07-21 06:05:55.962758 [WARNING]
>>>>                 sofia_glue.c:6605 Max handles 50
>>>>                 > exceeded, blocking....
>>>>                 > 2013-07-21 06:05:56.062671 [ERR]
>>>>                 sofia_glue.c:6605 Error connecting
>>>>                 > 2013-07-21 06:05:56.062671 [ERR]
>>>>                 sofia_glue.c:6606 Error Opening DB
>>>>                 > 2013-07-21 06:05:56.062671 [WARNING]
>>>>                 sofia_glue.c:6605 Max handles 50
>>>>                 > exceeded, blocking....
>>>>                 > 2013-07-21 06:05:56.062671 [ERR]
>>>>                 sofia_glue.c:6605 Error connecting
>>>>                 > 2013-07-21 06:05:56.062671 [ERR]
>>>>                 sofia_glue.c:6606 Error Opening DB
>>>>                 >
>>>>                 > FreeSWITCH stayed unusable since then until it
>>>>                 was restarted.
>>>>                 >
>>>>                 > FYI core DB is already in the RAM:
>>>>                 > <param name="core-db-name"
>>>>                 value="/dev/shm/core.db" />
>>>>                 >
>>>>                 >
>>>>                 > Thank you,
>>>>                 > Victor
>>>>                 >
>>>>                 >
>>>>                 _________________________________________________________________________
>>>>                 > Professional FreeSWITCH Consulting Services:
>>>>                 > consulting at freeswitch.org
>>>>                 <mailto:consulting at freeswitch.org>
>>>>                 > http://www.freeswitchsolutions.com
>>>>                 >
>>>>                 > FreeSWITCH-powered IP PBX: The CudaTel
>>>>                 Communication Server
>>>>                 > 
>>>>                 >
>>>>                 > Official FreeSWITCH Sites
>>>>                 > http://www.freeswitch.org
>>>>                 > http://wiki.freeswitch.org
>>>>                 > http://www.cluecon.com
>>>>                 >
>>>>                 > FreeSWITCH-users mailing list
>>>>                 > FreeSWITCH-users at lists.freeswitch.org
>>>>                 <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>                 >
>>>>                 http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>                 >
>>>>                 UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>                 > http://www.freeswitch.org
>>>>
>>>>                 --
>>>>                 Ken
>>>>                 http://www.FreeSWITCH.org
>>>>                 http://www.ClueCon.com
>>>>                 http://www.OSTAG.org
>>>>                 irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>>
>>>>
>>>>
>>>>                 _________________________________________________________________________
>>>>                 Professional FreeSWITCH Consulting Services:
>>>>                 consulting at freeswitch.org
>>>>                 <mailto:consulting at freeswitch.org>
>>>>                 http://www.freeswitchsolutions.com
>>>>
>>>>                 FreeSWITCH-powered IP PBX: The CudaTel
>>>>                 Communication Server
>>>>                 
>>>>
>>>>                 Official FreeSWITCH Sites
>>>>                 http://www.freeswitch.org
>>>>                 http://wiki.freeswitch.org
>>>>                 http://www.cluecon.com
>>>>
>>>>                 FreeSWITCH-users mailing list
>>>>                 FreeSWITCH-users at lists.freeswitch.org
>>>>                 <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>                 http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>                 UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>                 http://www.freeswitch.org
>>>>
>>>>
>>>>
>>>>
>>>>             -- 
>>>>             Anthony Minessale II
>>>>
>>>>             FreeSWITCH http://www.freeswitch.org/
>>>>             ClueCon http://www.cluecon.com/
>>>>             Twitter: http://twitter.com/FreeSWITCH_wire
>>>>
>>>>             AIM: anthm
>>>>             MSN:anthony_minessale at hotmail.com
>>>>             <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>>>             GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>             <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>>>             IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>>
>>>>             FreeSWITCH Developer Conference
>>>>             sip:888 at conference.freeswitch.org
>>>>             <mailto:sip%3A888 at conference.freeswitch.org>
>>>>             googletalk:conf+888 at conference.freeswitch.org
>>>>             <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>>>             pstn:+19193869900
>>>>
>>>>
>>>>             _________________________________________________________________________
>>>>             Professional FreeSWITCH Consulting Services:
>>>>             consulting at freeswitch.org  <mailto:consulting at freeswitch.org>
>>>>             http://www.freeswitchsolutions.com
>>>>
>>>>             
>>>>             
>>>>
>>>>             Official FreeSWITCH Sites
>>>>             http://www.freeswitch.org
>>>>             http://wiki.freeswitch.org
>>>>             http://www.cluecon.com
>>>>
>>>>             FreeSWITCH-users mailing list
>>>>             FreeSWITCH-users at lists.freeswitch.org  <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>             http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>             UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>             http://www.freeswitch.org
>>>
>>>
>>>             _________________________________________________________________________
>>>             Professional FreeSWITCH Consulting Services:
>>>             consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>             http://www.freeswitchsolutions.com
>>>
>>>             
>>>             
>>>
>>>             Official FreeSWITCH Sites
>>>             http://www.freeswitch.org
>>>             http://wiki.freeswitch.org
>>>             http://www.cluecon.com
>>>
>>>             FreeSWITCH-users mailing list
>>>             FreeSWITCH-users at lists.freeswitch.org
>>>             <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>             http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>             UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>             http://www.freeswitch.org
>>>
>>>
>>>
>>>
>>>         -- 
>>>         Anthony Minessale II
>>>
>>>         FreeSWITCH http://www.freeswitch.org/
>>>         ClueCon http://www.cluecon.com/
>>>         Twitter: http://twitter.com/FreeSWITCH_wire
>>>
>>>         AIM: anthm
>>>         MSN:anthony_minessale at hotmail.com
>>>         <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>>         GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>         <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>>         IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>
>>>         FreeSWITCH Developer Conference
>>>         sip:888 at conference.freeswitch.org
>>>         <mailto:sip%3A888 at conference.freeswitch.org>
>>>         googletalk:conf+888 at conference.freeswitch.org
>>>         <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>>         pstn:+19193869900
>>>
>>>
>>>         _________________________________________________________________________
>>>         Professional FreeSWITCH Consulting Services:
>>>         consulting at freeswitch.org  <mailto:consulting at freeswitch.org>
>>>         http://www.freeswitchsolutions.com
>>>
>>>         
>>>         
>>>
>>>         Official FreeSWITCH Sites
>>>         http://www.freeswitch.org
>>>         http://wiki.freeswitch.org
>>>         http://www.cluecon.com
>>>
>>>         FreeSWITCH-users mailing list
>>>         FreeSWITCH-users at lists.freeswitch.org  <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>         http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>         UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>         http://www.freeswitch.org
>>
>>
>>         _________________________________________________________________________
>>         Professional FreeSWITCH Consulting Services:
>>         consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>         http://www.freeswitchsolutions.com
>>
>>         
>>         
>>
>>         Official FreeSWITCH Sites
>>         http://www.freeswitch.org
>>         http://wiki.freeswitch.org
>>         http://www.cluecon.com
>>
>>         FreeSWITCH-users mailing list
>>         FreeSWITCH-users at lists.freeswitch.org
>>         <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>         http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>         UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>         http://www.freeswitch.org
>>
>>
>>
>>
>>     -- 
>>     Anthony Minessale II
>>
>>     FreeSWITCH http://www.freeswitch.org/
>>     ClueCon http://www.cluecon.com/
>>     Twitter: http://twitter.com/FreeSWITCH_wire
>>
>>     AIM: anthm
>>     MSN:anthony_minessale at hotmail.com
>>     <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>     GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>     <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>     IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>
>>     FreeSWITCH Developer Conference
>>     sip:888 at conference.freeswitch.org
>>     <mailto:sip%3A888 at conference.freeswitch.org>
>>     googletalk:conf+888 at conference.freeswitch.org
>>     <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>     pstn:+19193869900
>>
>>
>>     _________________________________________________________________________
>>     Professional FreeSWITCH Consulting Services:
>>     consulting at freeswitch.org  <mailto:consulting at freeswitch.org>
>>     http://www.freeswitchsolutions.com
>>
>>     
>>     
>>
>>     Official FreeSWITCH Sites
>>     http://www.freeswitch.org
>>     http://wiki.freeswitch.org
>>     http://www.cluecon.com
>>
>>     FreeSWITCH-users mailing list
>>     FreeSWITCH-users at lists.freeswitch.org  <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>     http://www.freeswitch.org
>
>
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     http://www.freeswitchsolutions.com
>
>     
>     
>
>     Official FreeSWITCH Sites
>     http://www.freeswitch.org
>     http://wiki.freeswitch.org
>     http://www.cluecon.com
>
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
>
>
>
>
> -- 
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com 
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com 
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org 
> <mailto:sip%3A888 at conference.freeswitch.org>
> googletalk:conf+888 at conference.freeswitch.org 
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:+19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130722/16e76f99/attachment-0001.html 