[Freeswitch-users] Apparent performance bottleneck and FS freeze
Victor Chukalovskiy
victor.chukalovskiy at gmail.com
Tue Jul 23 21:05:09 MSD 2013
Anthony, thanks a lot. Here is what I got:
[ERR] sofia_glue.c:6614 SQL: [select
call_id,sip_user,sip_host,contact,status,rpid,expires,user_agent,server_user,server_host,profile_name,network_ip,0
from sip_registrations where expires > 0 and expires <= 1374597711]
[ERR] sofia_glue.c:6614 SQL: [select call_id from
sip_shared_appearance_dialogs where hostname='test' and
profile_name='test' and expires <= 1374597711]
Is there a way to disable those two selects to save a few DB calls?
Keep in mind that manage-shared-appearance and manage-presence are
already off.
As for the performance tweaking, I'm going to:
-move ALL sqlite DBs to RAM
-bump max handles. Does it sound reasonable and safe to go from 50 to
100 handles?
@Cal, saw your msg, thanks. I was indeed looking at master while Anthony
meant stable.
.Victor
On 13-07-22 04:42 PM, Anthony Minessale wrote:
> do you have manage-shared-appearance off too cos that re-enables
> presence?
>
> edit sofia_glue.c on line 6614 and add a print to see what the sql is
> that is being executed.
>
> switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_ERROR, "SQL: [%s]\n",
> sql);
>
> then make mod_sofia-install
>
> and relaunch and try to catch a sample of what it's doing.
>
>
>
>
>
> On Mon, Jul 22, 2013 at 3:27 PM, Victor Chukalovskiy
> <victor.chukalovskiy at gmail.com <mailto:victor.chukalovskiy at gmail.com>>
> wrote:
>
> Anthony,
>
> Thank you, this is a very useful command. I see a bunch of records
> like:
>
> db="sofia_reg_<profile_name>"
> Type: CORE_DB
> Last used: 10
> Flags: Unlocked, Detached(0)
> Creator: *sofia_glue.c:6605*
> Last User: *sofia_glue.c:6605*
>
> Appears that these DBs are in use and user/creator is
> sofia_glue.c:6605. Same as in error log:
>
> [WARNING] *sofia_glue.c:6605* Max handles 50
>
> And this is not a cored.db... it's sitting on the disk drive
> still. Is there a way to setup sip_profiles not to create any
> sofia_reg_...DBs?
>
> Re: vmware setup. It's good to know it's not optimal, however it
> gives rack space, administrator's time, and $ savings compared to
> dedicated servers.
>
> Should it be Ok to stick with VM setup provided enough CPU and RAM
> is allocated to each FS guest? Are there reasons not to run FS on
> VM other than CPU and RAM allocation?
>
> Thank you,
> Victor
>
>
>> Its a large number of handles to have open all in sofia if you
>> are not doing anything
>>
>> db_cache status
>>
>> from cli can show where those handles come from and what db they
>> are using.
>> Clearly the box is being overloaded past its limits.
>>
>> This is why we don't endorse virtual servers.
>>
>>
>>
>> On Mon, Jul 22, 2013 at 2:49 PM, Victor Chukalovskiy
>> <victor.chukalovskiy at gmail.com
>> <mailto:victor.chukalovskiy at gmail.com>> wrote:
>>
>> Disabled in all profiles:
>> <param name="manage-presence" value="false"/>
>>
>>
>> On 13-07-22 03:40 PM, Anthony Minessale wrote:
>>> Did you turn of the presence in the sofia profile, that is
>>> enabled by default and very costly. comment out manage-presence
>>>
>>>
>>> On Mon, Jul 22, 2013 at 2:32 PM, Victor Chukalovskiy
>>> <victor.chukalovskiy at gmail.com
>>> <mailto:victor.chukalovskiy at gmail.com>> wrote:
>>>
>>> Michael, Ken,
>>>
>>> Thank you for the response. Firstly, let me clarify
>>> setup for each of DBs in use:
>>>
>>> *core_db:* sqlite, /dev/shm/core.db (RAM) - using
>>> default 50 max handles
>>> *mod_sofia db**:* sqlite, /opt/fs/db (HDD)
>>> *mod_lcr db:* Postgres
>>> *mod_limit:* hash backend
>>>
>>> Note: I believe we rarely use mod_sofia DB's...this is
>>> because we don't use any registrations, acl only. When I
>>> list sofia per-profile db files on production box, most
>>> of them has not been edited for a couple days. Are my
>>> conclusions correct here, or do I miss something? Maybe
>>> they are accessed a lot but only for reading?
>>>
>>> I understand recommendations from Ken and from Michaels
>>> are along the lines of increasing core db and sofia db
>>> throughput. However here is what puzzles me:
>>> -the db used most extensively is core_db. It' already in
>>> the ram.
>>> -we don't have much load on sofia dbs (or am I mistaken
>>> in my logic?)
>>> -error log started with:
>>>
>>>
>>> [CRIT] switch_time.c:967 Virtual Migration Detected!
>>> Syncing Clock
>>> [WARNING] switch_scheduler.c:115 Task was executed late
>>> by 19 seconds 1 heartbeat (core)
>>>
>>> Does it mean that there were abnormal events prior
>>> hitting 50 max handles limit?
>>>
>>> Thank you,
>>> -Victor
>>>
>>>
>>> On 13-07-22 02:52 PM, Anthony Minessale wrote:
>>>> Check the odbcinst.ini settings from the wiki or use
>>>> native pg mode.
>>>> increase the max number of pg connections on your
>>>> server and turn up FS to match in max-db-handles in
>>>> switch.conf.xml
>>>>
>>>> You are already beating it up pretty good esp for a
>>>> fake server though. 2500 is a lot.
>>>>
>>>>
>>>>
>>>> On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice
>>>> <krice at freeswitch.org <mailto:krice at freeswitch.org>> wrote:
>>>>
>>>> Mount the entire freeswitch/db dir as tmpfs... most
>>>> of these errors are for
>>>> the sofia DB getting backed up...
>>>>
>>>> The coredb is a completely different sqlite db from
>>>> the sofia db
>>>>
>>>> On 7/22/13 12:54 PM, "Victor Chukalovskiy"
>>>> <victor.chukalovskiy at gmail.com
>>>> <mailto:victor.chukalovskiy at gmail.com>>
>>>> wrote:
>>>>
>>>> > Hello,
>>>> >
>>>> > Need some help finding the bottleneck here.
>>>> Running 1.2.stable on a
>>>> > decent multi-core vmware platform. No RTP, SIP
>>>> signaling only. Around
>>>> > 2500 sessions and 30 average CPS load.
>>>> Essentially, it's a multi-profile
>>>> > .xml config that uses mod_lcr extensively
>>>> (postgres DB). Yesterday it
>>>> > stopped processing calls with the following log
>>>> output:
>>>> >
>>>> > 2013-07-21 06:00:37.642678 [CRIT]
>>>> switch_time.c:967 Virtual Migration
>>>> > Detected! Syncing Clock
>>>> > 2013-07-21 06:01:07.141734 [WARNING]
>>>> switch_scheduler.c:115 Task was
>>>> > executed late by 19 seconds 1 heartbeat (core)
>>>> > 2013-07-21 06:02:32.402668 [CRIT]
>>>> switch_time.c:967 Virtual Migration
>>>> > Detected! Syncing Clock
>>>> > 2013-07-21 06:04:42.499744 [WARNING]
>>>> switch_scheduler.c:115 Task was
>>>> > executed late by 115 seconds 1 heartbeat (core)
>>>> > 2013-07-21 06:04:42.499744 [WARNING]
>>>> switch_scheduler.c:115 Task was
>>>> > executed late by 74 seconds 2 check_ip (core)
>>>> > 2013-07-21 06:05:15.102683 [CRIT]
>>>> switch_time.c:967 Virtual Migration
>>>> > Detected! Syncing Clock
>>>> > 2013-07-21 06:05:44.961004 [WARNING]
>>>> switch_scheduler.c:115 Task was
>>>> > executed late by 22 seconds 1 heartbeat (core)
>>>> > 2013-07-21 06:05:44.961004 [WARNING]
>>>> switch_scheduler.c:115 Task was
>>>> > executed late by 2 seconds 2 check_ip (core)
>>>> > 2013-07-21 06:05:49.322672 [WARNING]
>>>> mod_lcr.c:448 Max handles 50
>>>> > exceeded, blocking....
>>>> > 2013-07-21 06:05:50.962987 [WARNING]
>>>> sofia_glue.c:6605 Max handles 50
>>>> > exceeded, blocking....
>>>> > 2013-07-21 06:05:50.962987 [WARNING]
>>>> sofia_glue.c:6605 Max handles 50
>>>> > exceeded, blocking....
>>>> > 2013-07-21 06:05:50.962987 [WARNING]
>>>> sofia_glue.c:6605 Max handles 50
>>>> > exceeded, blocking....
>>>> > ................................
>>>> > 2013-07-21 06:05:53.962672 [WARNING]
>>>> sofia_glue.c:6605 Max handles 50
>>>> > exceeded, blocking....
>>>> > 2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448
>>>> Error connecting
>>>> > 2013-07-21 06:05:54.422671 [WARNING]
>>>> mod_lcr.c:1779 LCR lookup failed
>>>> > for 14389955082
>>>> > 2013-07-21 06:05:54.962671 [WARNING]
>>>> sofia_glue.c:6605 Max handles 50
>>>> > exceeded, blocking....
>>>> > 2013-07-21 06:05:54.962671 [WARNING]
>>>> sofia_glue.c:6605 Max handles 50
>>>> > exceeded, blocking....
>>>> > ................................
>>>> > 2013-07-21 06:05:55.962758 [WARNING]
>>>> sofia_glue.c:6605 Max handles 50
>>>> > exceeded, blocking....
>>>> > 2013-07-21 06:05:56.062671 [ERR]
>>>> sofia_glue.c:6605 Error connecting
>>>> > 2013-07-21 06:05:56.062671 [ERR]
>>>> sofia_glue.c:6606 Error Opening DB
>>>> > 2013-07-21 06:05:56.062671 [WARNING]
>>>> sofia_glue.c:6605 Max handles 50
>>>> > exceeded, blocking....
>>>> > 2013-07-21 06:05:56.062671 [ERR]
>>>> sofia_glue.c:6605 Error connecting
>>>> > 2013-07-21 06:05:56.062671 [ERR]
>>>> sofia_glue.c:6606 Error Opening DB
>>>> >
>>>> > FreeSWITCH stayed unusable since then until it
>>>> was restarted.
>>>> >
>>>> > FYI core DB is already in the RAM:
>>>> > <param name="core-db-name"
>>>> value="/dev/shm/core.db" />
>>>> >
>>>> >
>>>> > Thank you,
>>>> > Victor
>>>> >
>>>> >
>>>> _________________________________________________________________________
>>>> > Professional FreeSWITCH Consulting Services:
>>>> > consulting at freeswitch.org
>>>> <mailto:consulting at freeswitch.org>
>>>> > http://www.freeswitchsolutions.com
>>>> >
>>>> > FreeSWITCH-powered IP PBX: The CudaTel
>>>> Communication Server
>>>> >
>>>> >
>>>> > Official FreeSWITCH Sites
>>>> > http://www.freeswitch.org
>>>> > http://wiki.freeswitch.org
>>>> > http://www.cluecon.com
>>>> >
>>>> > FreeSWITCH-users mailing list
>>>> > FreeSWITCH-users at lists.freeswitch.org
>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>> >
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> >
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> > http://www.freeswitch.org
>>>>
>>>> --
>>>> Ken
>>>> http://www.FreeSWITCH.org
>>>> http://www.ClueCon.com
>>>> http://www.OSTAG.org
>>>> irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> <mailto:consulting at freeswitch.org>
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> FreeSWITCH-powered IP PBX: The CudaTel
>>>> Communication Server
>>>>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Anthony Minessale II
>>>>
>>>> FreeSWITCH http://www.freeswitch.org/
>>>> ClueCon http://www.cluecon.com/
>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>
>>>> AIM: anthm
>>>> MSN:anthony_minessale at hotmail.com
>>>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>>
>>>> FreeSWITCH Developer Conference
>>>> sip:888 at conference.freeswitch.org
>>>> <mailto:sip%3A888 at conference.freeswitch.org>
>>>> googletalk:conf+888 at conference.freeswitch.org
>>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>>> pstn:+19193869900
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>> http://www.freeswitchsolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>>
>>>
>>> --
>>> Anthony Minessale II
>>>
>>> FreeSWITCH http://www.freeswitch.org/
>>> ClueCon http://www.cluecon.com/
>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>
>>> AIM: anthm
>>> MSN:anthony_minessale at hotmail.com
>>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>
>>> FreeSWITCH Developer Conference
>>> sip:888 at conference.freeswitch.org
>>> <mailto:sip%3A888 at conference.freeswitch.org>
>>> googletalk:conf+888 at conference.freeswitch.org
>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>> pstn:+19193869900
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>>
>> --
>> Anthony Minessale II
>>
>> FreeSWITCH http://www.freeswitch.org/
>> ClueCon http://www.cluecon.com/
>> Twitter: http://twitter.com/FreeSWITCH_wire
>>
>> AIM: anthm
>> MSN:anthony_minessale at hotmail.com
>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>
>> FreeSWITCH Developer Conference
>> sip:888 at conference.freeswitch.org
>> <mailto:sip%3A888 at conference.freeswitch.org>
>> googletalk:conf+888 at conference.freeswitch.org
>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>> pstn:+19193869900
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> <mailto:sip%3A888 at conference.freeswitch.org>
> googletalk:conf+888 at conference.freeswitch.org
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:+19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130723/d77ed517/attachment-0001.html
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list