[Freeswitch-users] Apparent performance bottleneck and FS freeze

Anthony Minessale anthony.minessale at gmail.com
Wed Jul 24 02:13:57 MSD 2013 

last_used is seconds typically unattached handles unused for more than 30
sec are released.





On Tue, Jul 23, 2013 at 3:23 PM, Victor Chukalovskiy <
victor.chukalovskiy at gmail.com> wrote:

>  Yes, these are the only ones I see printed on our sandbox. Will try if I
> can get any other ones. I understand it's time consuming for you to
> respond, thanks again for your help.
>
> Today it got stuck again on production server and I ran db_cache status:
>
>
> db="sofia_reg_<profile_name>"
>         Type: CORE_DB
>         Last used: 150861
>         Flags: Locked, Attached(1)
>         Creator: sofia_glue.c:6380
>         Last User: src/switch_core_sqldb.c:1816
>
> .....
> <same as above for each profile, we got a few dozens>
> .....
> db="/dev/shm/core.db"
>         Type: CORE_DB
>         Last used: 150862
>         Flags: Locked, Attached(1)
>         Creator: src/switch_core_sqldb.c:1816
>         Last User:
>
> In what units is "last used"?
>
> VM runs on iSCSI SAN Storage.
>
>
>
> On 13-07-23 03:01 PM, Anthony Minessale wrote:
>
> Is that the only ones? how frequent are they?  Those are only for
> expiration and run very infrequently.  You may wish to study it longer
> because that should not be enough to slow everything down.  Your server is
> not on a fake HDD too is it?
>
>  I generally do not help on load testing or virtual server setups because
> it can consume a lot of time.
>
>
>
> On Tue, Jul 23, 2013 at 12:05 PM, Victor Chukalovskiy <
> victor.chukalovskiy at gmail.com> wrote:
>
>>  Anthony, thanks a lot. Here is what I got:
>>
>> [ERR] sofia_glue.c:6614 SQL: [select
>> call_id,sip_user,sip_host,contact,status,rpid,expires,user_agent,server_user,server_host,profile_name,network_ip,0
>> from sip_registrations where expires > 0 and expires <= 1374597711]
>> [ERR] sofia_glue.c:6614 SQL: [select call_id from
>> sip_shared_appearance_dialogs where hostname='test' and profile_name='test'
>> and expires <= 1374597711]
>>
>> Is there a way to disable those two selects to save a few DB calls?
>> Keep in mind that manage-shared-appearance and manage-presence are
>> already off.
>>
>> As for the performance tweaking, I'm going to:
>> -move ALL sqlite DBs to RAM
>> -bump max handles. Does it sound reasonable and safe to go from 50 to 100
>> handles?
>>
>> @Cal, saw your msg, thanks. I was indeed looking at master while Anthony
>> meant stable.
>>
>> .Victor
>>
>>
>> On 13-07-22 04:42 PM, Anthony Minessale wrote:
>>
>> do you have manage-shared-appearance off too cos that re-enables
>> presence?
>>
>>  edit sofia_glue.c on line 6614 and add a print to see what the sql is
>> that is being executed.
>>
>>  switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_ERROR, "SQL: [%s]\n",
>> sql);
>>
>>  then make mod_sofia-install
>>
>>  and relaunch and try to catch a sample of what it's doing.
>>
>>
>>
>>
>>
>> On Mon, Jul 22, 2013 at 3:27 PM, Victor Chukalovskiy <
>> victor.chukalovskiy at gmail.com> wrote:
>>
>>>  Anthony,
>>>
>>> Thank you, this is a very useful command. I see a bunch of records like:
>>>
>>> db="sofia_reg_<profile_name>"
>>>         Type: CORE_DB
>>>         Last used: 10
>>>         Flags: Unlocked, Detached(0)
>>>         Creator: *sofia_glue.c:6605*
>>>         Last User: *sofia_glue.c:6605*
>>>
>>> Appears that these DBs are in use and user/creator is sofia_glue.c:6605.
>>> Same as in error log:
>>>
>>> [WARNING] *sofia_glue.c:6605* Max handles 50
>>>
>>>  And this is not a cored.db... it's sitting on the disk drive still. Is
>>> there a way to setup sip_profiles not to create any sofia_reg_...DBs?
>>>
>>> Re: vmware setup. It's good to know it's not optimal, however it gives
>>> rack space, administrator's time, and $ savings compared to dedicated
>>> servers.
>>>
>>> Should it be Ok to stick with VM setup provided enough CPU and RAM is
>>> allocated to each FS guest? Are there reasons not to run FS on VM other
>>> than CPU and RAM allocation?
>>>
>>> Thank you,
>>> Victor
>>>
>>>
>>>   Its a large number of handles to have open all in sofia if you are
>>> not doing anything
>>>
>>>  db_cache status
>>>
>>>  from cli can show where those handles come from and what db they are
>>> using.
>>> Clearly the box is being overloaded past its limits.
>>>
>>>  This is why we don't endorse virtual servers.
>>>
>>>
>>>
>>> On Mon, Jul 22, 2013 at 2:49 PM, Victor Chukalovskiy <
>>> victor.chukalovskiy at gmail.com> wrote:
>>>
>>>>  Disabled in all profiles:
>>>> <param name="manage-presence" value="false"/>
>>>>
>>>>
>>>> On 13-07-22 03:40 PM, Anthony Minessale wrote:
>>>>
>>>> Did you turn of the presence in the sofia profile, that is enabled by
>>>> default and very costly.  comment out manage-presence
>>>>
>>>>
>>>> On Mon, Jul 22, 2013 at 2:32 PM, Victor Chukalovskiy <
>>>> victor.chukalovskiy at gmail.com> wrote:
>>>>
>>>>>  Michael, Ken,
>>>>>
>>>>> Thank you for the response. Firstly, let me clarify setup for each of
>>>>> DBs in use:
>>>>>
>>>>> *core_db:* sqlite, /dev/shm/core.db (RAM) - using default 50 max
>>>>> handles
>>>>> *mod_sofia db**:* sqlite, /opt/fs/db (HDD)
>>>>> *mod_lcr db:* Postgres
>>>>> *mod_limit:* hash backend
>>>>>
>>>>> Note: I believe we rarely use mod_sofia DB's...this is because we
>>>>> don't use any registrations, acl only. When I list sofia per-profile db
>>>>> files on production box, most of them has not been edited for a couple
>>>>> days. Are my conclusions correct here, or do I miss something? Maybe they
>>>>> are accessed a lot but only for reading?
>>>>>
>>>>> I understand recommendations from Ken and from Michaels are along the
>>>>> lines of increasing core db and sofia db throughput. However here is what
>>>>> puzzles me:
>>>>> -the db used most extensively is core_db. It' already in the ram.
>>>>> -we don't have much load on sofia dbs (or am I mistaken in my logic?)
>>>>> -error log started with:
>>>>>
>>>>>
>>>>> [CRIT] switch_time.c:967 Virtual Migration Detected! Syncing Clock
>>>>>  [WARNING] switch_scheduler.c:115 Task was executed late by 19
>>>>> seconds 1 heartbeat (core)
>>>>>
>>>>>  Does it mean that there were abnormal events prior hitting 50 max
>>>>> handles limit?
>>>>>
>>>>> Thank you,
>>>>> -Victor
>>>>>
>>>>>
>>>>> On 13-07-22 02:52 PM, Anthony Minessale wrote:
>>>>>
>>>>> Check the odbcinst.ini settings from the wiki or use native pg mode.
>>>>> increase the max number of pg connections on your server and turn up
>>>>> FS to match in max-db-handles in switch.conf.xml
>>>>>
>>>>>  You are already beating it up pretty good esp for a fake server
>>>>> though.  2500 is a lot.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice <krice at freeswitch.org>wrote:
>>>>>
>>>>>> Mount the entire freeswitch/db dir as tmpfs... most of these errors
>>>>>> are for
>>>>>> the sofia DB getting backed up...
>>>>>>
>>>>>> The coredb is a completely different sqlite db from the sofia db
>>>>>>
>>>>>> On 7/22/13 12:54 PM, "Victor Chukalovskiy" <
>>>>>> victor.chukalovskiy at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hello,
>>>>>> >
>>>>>> > Need some help finding the bottleneck here. Running 1.2.stable on a
>>>>>> > decent multi-core vmware platform. No RTP, SIP signaling only.
>>>>>> Around
>>>>>> > 2500 sessions and 30 average CPS load. Essentially, it's a
>>>>>> multi-profile
>>>>>> > .xml config that uses mod_lcr extensively (postgres DB). Yesterday
>>>>>> it
>>>>>> > stopped processing calls with the following log output:
>>>>>> >
>>>>>> > 2013-07-21 06:00:37.642678 [CRIT] switch_time.c:967 Virtual
>>>>>> Migration
>>>>>> > Detected! Syncing Clock
>>>>>> > 2013-07-21 06:01:07.141734 [WARNING] switch_scheduler.c:115 Task was
>>>>>> > executed late by 19 seconds 1 heartbeat (core)
>>>>>> > 2013-07-21 06:02:32.402668 [CRIT] switch_time.c:967 Virtual
>>>>>> Migration
>>>>>> > Detected! Syncing Clock
>>>>>> > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was
>>>>>> > executed late by 115 seconds 1 heartbeat (core)
>>>>>> > 2013-07-21 06:04:42.499744 [WARNING] switch_scheduler.c:115 Task was
>>>>>> > executed late by 74 seconds 2 check_ip (core)
>>>>>> > 2013-07-21 06:05:15.102683 [CRIT] switch_time.c:967 Virtual
>>>>>> Migration
>>>>>> > Detected! Syncing Clock
>>>>>> > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was
>>>>>> > executed late by 22 seconds 1 heartbeat (core)
>>>>>> > 2013-07-21 06:05:44.961004 [WARNING] switch_scheduler.c:115 Task was
>>>>>> > executed late by 2 seconds 2 check_ip (core)
>>>>>> > 2013-07-21 06:05:49.322672 [WARNING] mod_lcr.c:448 Max handles 50
>>>>>> > exceeded, blocking....
>>>>>> > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles
>>>>>> 50
>>>>>> > exceeded, blocking....
>>>>>> > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles
>>>>>> 50
>>>>>> > exceeded, blocking....
>>>>>> > 2013-07-21 06:05:50.962987 [WARNING] sofia_glue.c:6605 Max handles
>>>>>> 50
>>>>>> > exceeded, blocking....
>>>>>> > ................................
>>>>>> > 2013-07-21 06:05:53.962672 [WARNING] sofia_glue.c:6605 Max handles
>>>>>> 50
>>>>>> > exceeded, blocking....
>>>>>> > 2013-07-21 06:05:54.422671 [ERR] mod_lcr.c:448 Error connecting
>>>>>> > 2013-07-21 06:05:54.422671 [WARNING] mod_lcr.c:1779 LCR lookup
>>>>>> failed
>>>>>> > for 14389955082
>>>>>> > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max handles
>>>>>> 50
>>>>>> > exceeded, blocking....
>>>>>> > 2013-07-21 06:05:54.962671 [WARNING] sofia_glue.c:6605 Max handles
>>>>>> 50
>>>>>> > exceeded, blocking....
>>>>>> > ................................
>>>>>> > 2013-07-21 06:05:55.962758 [WARNING] sofia_glue.c:6605 Max handles
>>>>>> 50
>>>>>> > exceeded, blocking....
>>>>>> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
>>>>>> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB
>>>>>> > 2013-07-21 06:05:56.062671 [WARNING] sofia_glue.c:6605 Max handles
>>>>>> 50
>>>>>> > exceeded, blocking....
>>>>>> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6605 Error connecting
>>>>>> > 2013-07-21 06:05:56.062671 [ERR] sofia_glue.c:6606 Error Opening DB
>>>>>> >
>>>>>> > FreeSWITCH stayed unusable since then until it was restarted.
>>>>>> >
>>>>>> > FYI core DB is already in the RAM:
>>>>>> > <param name="core-db-name" value="/dev/shm/core.db" />
>>>>>> >
>>>>>> >
>>>>>> > Thank you,
>>>>>> > Victor
>>>>>> >
>>>>>> >
>>>>>> _________________________________________________________________________
>>>>>> > Professional FreeSWITCH Consulting Services:
>>>>>> > consulting at freeswitch.org
>>>>>> > http://www.freeswitchsolutions.com
>>>>>> >
>>>>>> > 
>>>>>> > 
>>>>>> >
>>>>>> > Official FreeSWITCH Sites
>>>>>> > http://www.freeswitch.org
>>>>>> > http://wiki.freeswitch.org
>>>>>> > http://www.cluecon.com
>>>>>> >
>>>>>> > FreeSWITCH-users mailing list
>>>>>> > FreeSWITCH-users at lists.freeswitch.org
>>>>>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> > UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> > http://www.freeswitch.org
>>>>>>
>>>>>>  --
>>>>>> Ken
>>>>>> http://www.FreeSWITCH.org
>>>>>> http://www.ClueCon.com
>>>>>> http://www.OSTAG.org
>>>>>> irc.freenode.net #freeswitch
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> 
>>>>>> 
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>  --
>>>>> Anthony Minessale II
>>>>>
>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>> ClueCon http://www.cluecon.com/
>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>
>>>>> AIM: anthm
>>>>> MSN:anthony_minessale at hotmail.com
>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>> IRC: irc.freenode.net #freeswitch
>>>>>
>>>>> FreeSWITCH Developer Conference
>>>>> sip:888 at conference.freeswitch.org
>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>> pstn:+19193869900
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>>>>
>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server
>>>>>
>>>>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>
>>>>
>>>>  --
>>>> Anthony Minessale II
>>>>
>>>> FreeSWITCH http://www.freeswitch.org/
>>>> ClueCon http://www.cluecon.com/
>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>
>>>> AIM: anthm
>>>> MSN:anthony_minessale at hotmail.com
>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>> IRC: irc.freenode.net #freeswitch
>>>>
>>>> FreeSWITCH Developer Conference
>>>> sip:888 at conference.freeswitch.org
>>>> googletalk:conf+888 at conference.freeswitch.org
>>>> pstn:+19193869900
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>>>
>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server
>>>>
>>>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>>>
>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>>
>>>  --
>>> Anthony Minessale II
>>>
>>> FreeSWITCH http://www.freeswitch.org/
>>> ClueCon http://www.cluecon.com/
>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>
>>> AIM: anthm
>>> MSN:anthony_minessale at hotmail.com
>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>> IRC: irc.freenode.net #freeswitch
>>>
>>> FreeSWITCH Developer Conference
>>> sip:888 at conference.freeswitch.org
>>> googletalk:conf+888 at conference.freeswitch.org
>>> pstn:+19193869900
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>>
>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server
>>>
>>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>>
>>  --
>> Anthony Minessale II
>>
>> FreeSWITCH http://www.freeswitch.org/
>> ClueCon http://www.cluecon.com/
>> Twitter: http://twitter.com/FreeSWITCH_wire
>>
>> AIM: anthm
>> MSN:anthony_minessale at hotmail.com
>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>> IRC: irc.freenode.net #freeswitch
>>
>> FreeSWITCH Developer Conference
>> sip:888 at conference.freeswitch.org
>> googletalk:conf+888 at conference.freeswitch.org
>> pstn:+19193869900
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>
>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server
>>
>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
>>
>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
>
>  --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> IRC: irc.freenode.net #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> googletalk:conf+888 at conference.freeswitch.org
> pstn:+19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>
> FreeSWITCH-powered IP PBX: The CudaTel Communication Server
>
> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
>
> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>


-- 
Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
googletalk:conf+888 at conference.freeswitch.org
pstn:+19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130723/193b11d8/attachment-0001.html

Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list