[Freeswitch-users] Apparent performance bottleneck and FS freeze
Victor Chukalovskiy
victor.chukalovskiy at gmail.com
Tue Jul 23 00:27:06 MSD 2013
Anthony,
Thank you, this is a very useful command. I see a bunch of records like:
db="sofia_reg_<profile_name>"
Type: CORE_DB
Last used: 10
Flags: Unlocked, Detached(0)
Creator: *sofia_glue.c:6605*
Last User: *sofia_glue.c:6605*
Appears that these DBs are in use and user/creator is sofia_glue.c:6605.
Same as in error log:
[WARNING] *sofia_glue.c:6605* Max handles 50
And this is not a cored.db... it's sitting on the disk drive still. Is
there a way to setup sip_profiles not to create any sofia_reg_...DBs?
Re: vmware setup. It's good to know it's not optimal, however it gives
rack space, administrator's time, and $ savings compared to dedicated
servers.
Should it be Ok to stick with VM setup provided enough CPU and RAM is
allocated to each FS guest? Are there reasons not to run FS on VM other
than CPU and RAM allocation?
Thank you,
Victor
> Its a large number of handles to have open all in sofia if you are not
> doing anything
>
> db_cache status
>
> from cli can show where those handles come from and what db they are
> using.
> Clearly the box is being overloaded past its limits.
>
> This is why we don't endorse virtual servers.
>
>
>
> On Mon, Jul 22, 2013 at 2:49 PM, Victor Chukalovskiy
> <victor.chukalovskiy at gmail.com <mailto:victor.chukalovskiy at gmail.com>>
> wrote:
>
> Disabled in all profiles:
> <param name="manage-presence" value="false"/>
>
>
> On 13-07-22 03:40 PM, Anthony Minessale wrote:
>> Did you turn of the presence in the sofia profile, that is
>> enabled by default and very costly. comment out manage-presence
>>
>>
>> On Mon, Jul 22, 2013 at 2:32 PM, Victor Chukalovskiy
>> <victor.chukalovskiy at gmail.com
>> <mailto:victor.chukalovskiy at gmail.com>> wrote:
>>
>> Michael, Ken,
>>
>> Thank you for the response. Firstly, let me clarify setup for
>> each of DBs in use:
>>
>> *core_db:* sqlite, /dev/shm/core.db (RAM) - using default 50
>> max handles
>> *mod_sofia db**:* sqlite, /opt/fs/db (HDD)
>> *mod_lcr db:* Postgres
>> *mod_limit:* hash backend
>>
>> Note: I believe we rarely use mod_sofia DB's...this is
>> because we don't use any registrations, acl only. When I list
>> sofia per-profile db files on production box, most of them
>> has not been edited for a couple days. Are my conclusions
>> correct here, or do I miss something? Maybe they are accessed
>> a lot but only for reading?
>>
>> I understand recommendations from Ken and from Michaels are
>> along the lines of increasing core db and sofia db
>> throughput. However here is what puzzles me:
>> -the db used most extensively is core_db. It' already in the ram.
>> -we don't have much load on sofia dbs (or am I mistaken in my
>> logic?)
>> -error log started with:
>>
>>
>> [CRIT] switch_time.c:967 Virtual Migration Detected! Syncing
>> Clock
>> [WARNING] switch_scheduler.c:115 Task was executed late by 19
>> seconds 1 heartbeat (core)
>>
>> Does it mean that there were abnormal events prior hitting 50
>> max handles limit?
>>
>> Thank you,
>> -Victor
>>
>>
>> On 13-07-22 02:52 PM, Anthony Minessale wrote:
>>> Check the odbcinst.ini settings from the wiki or use native
>>> pg mode.
>>> increase the max number of pg connections on your server and
>>> turn up FS to match in max-db-handles in switch.conf.xml
>>>
>>> You are already beating it up pretty good esp for a fake
>>> server though. 2500 is a lot.
>>>
>>>
>>>
>>> On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice
>>> <krice at freeswitch.org <mailto:krice at freeswitch.org>> wrote:
>>>
>>> Mount the entire freeswitch/db dir as tmpfs... most of
>>> these errors are for
>>> the sofia DB getting backed up...
>>>
>>> The coredb is a completely different sqlite db from the
>>> sofia db
>>>
>>> On 7/22/13 12:54 PM, "Victor Chukalovskiy"
>>> <victor.chukalovskiy at gmail.com
>>> <mailto:victor.chukalovskiy at gmail.com>>
>>> wrote:
>>>
>>> > Hello,
>>> >
>>> > Need some help finding the bottleneck here. Running
>>> 1.2.stable on a
>>> > decent multi-core vmware platform. No RTP, SIP
>>> signaling only. Around
>>> > 2500 sessions and 30 average CPS load. Essentially,
>>> it's a multi-profile
>>> > .xml config that uses mod_lcr extensively (postgres
>>> DB). Yesterday it
>>> > stopped processing calls with the following log output:
>>> >
>>> > 2013-07-21 06:00:37.642678 [CRIT] switch_time.c:967
>>> Virtual Migration
>>> > Detected! Syncing Clock
>>> > 2013-07-21 06:01:07.141734 [WARNING]
>>> switch_scheduler.c:115 Task was
>>> > executed late by 19 seconds 1 heartbeat (core)
>>> > 2013-07-21 06:02:32.402668 [CRIT] switch_time.c:967
>>> Virtual Migration
>>> > Detected! Syncing Clock
>>> > 2013-07-21 06:04:42.499744 [WARNING]
>>> switch_scheduler.c:115 Task was
>>> > executed late by 115 seconds 1 heartbeat (core)
>>> > 2013-07-21 06:04:42.499744 [WARNING]
>>> switch_scheduler.c:115 Task was
>>> > executed late by 74 seconds 2 check_ip (core)
>>> > 2013-07-21 06:05:15.102683 [CRIT] switch_time.c:967
>>> Virtual Migration
>>> > Detected! Syncing Clock
>>> > 2013-07-21 06:05:44.961004 [WARNING]
>>> switch_scheduler.c:115 Task was
>>> > executed late by 22 seconds 1 heartbeat (core)
>>> > 2013-07-21 06:05:44.961004 [WARNING]
>>> switch_scheduler.c:115 Task was
>>> > executed late by 2 seconds 2 check_ip (core)
>>> > 2013-07-21 06:05:49.322672 [WARNING] mod_lcr.c:448 Max
>>> handles 50
>>> > exceeded, blocking....
>>> > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605
>>> Max handles 50
>>> > exceeded, blocking....
>>> > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605
>>> Max handles 50
>>> > exceeded, blocking....
>>> > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605
>>> Max handles 50
>>> > exceeded, blocking....
>>> > ................................
>>> > 2013-07-21 06:05:53.962672 [WARNING] sofia_glue.c:6605
>>> Max handles 50
>>> > exceeded, blocking....
>>> > 2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448 Error
>>> connecting
>>> > 2013-07-21 06:05:54.422671 [WARNING] mod_lcr.c:1779
>>> LCR lookup failed
>>> > for 14389955082
>>> > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605
>>> Max handles 50
>>> > exceeded, blocking....
>>> > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605
>>> Max handles 50
>>> > exceeded, blocking....
>>> > ................................
>>> > 2013-07-21 06:05:55.962758 [WARNING] sofia_glue.c:6605
>>> Max handles 50
>>> > exceeded, blocking....
>>> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605
>>> Error connecting
>>> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606
>>> Error Opening DB
>>> > 2013-07-21 06:05:56.062671 [WARNING] sofia_glue.c:6605
>>> Max handles 50
>>> > exceeded, blocking....
>>> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605
>>> Error connecting
>>> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606
>>> Error Opening DB
>>> >
>>> > FreeSWITCH stayed unusable since then until it was
>>> restarted.
>>> >
>>> > FYI core DB is already in the RAM:
>>> > <param name="core-db-name" value="/dev/shm/core.db" />
>>> >
>>> >
>>> > Thank you,
>>> > Victor
>>> >
>>> >
>>> _________________________________________________________________________
>>> > Professional FreeSWITCH Consulting Services:
>>> > consulting at freeswitch.org
>>> <mailto:consulting at freeswitch.org>
>>> > http://www.freeswitchsolutions.com
>>> >
>>> > FreeSWITCH-powered IP PBX: The CudaTel Communication
>>> Server
>>> >
>>> >
>>> > Official FreeSWITCH Sites
>>> > http://www.freeswitch.org
>>> > http://wiki.freeswitch.org
>>> > http://www.cluecon.com
>>> >
>>> > FreeSWITCH-users mailing list
>>> > FreeSWITCH-users at lists.freeswitch.org
>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> >
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> >
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> > http://www.freeswitch.org
>>>
>>> --
>>> Ken
>>> http://www.FreeSWITCH.org
>>> http://www.ClueCon.com
>>> http://www.OSTAG.org
>>> irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>>
>>>
>>> --
>>> Anthony Minessale II
>>>
>>> FreeSWITCH http://www.freeswitch.org/
>>> ClueCon http://www.cluecon.com/
>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>
>>> AIM: anthm
>>> MSN:anthony_minessale at hotmail.com
>>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>
>>> FreeSWITCH Developer Conference
>>> sip:888 at conference.freeswitch.org
>>> <mailto:sip%3A888 at conference.freeswitch.org>
>>> googletalk:conf+888 at conference.freeswitch.org
>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>> pstn:+19193869900
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>>
>> --
>> Anthony Minessale II
>>
>> FreeSWITCH http://www.freeswitch.org/
>> ClueCon http://www.cluecon.com/
>> Twitter: http://twitter.com/FreeSWITCH_wire
>>
>> AIM: anthm
>> MSN:anthony_minessale at hotmail.com
>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>
>> FreeSWITCH Developer Conference
>> sip:888 at conference.freeswitch.org
>> <mailto:sip%3A888 at conference.freeswitch.org>
>> googletalk:conf+888 at conference.freeswitch.org
>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>> pstn:+19193869900
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> <mailto:sip%3A888 at conference.freeswitch.org>
> googletalk:conf+888 at conference.freeswitch.org
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:+19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130722/26925bb1/attachment-0001.html
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list