[Freeswitch-users] Apparent performance bottleneck and FS freeze
Victor Chukalovskiy
victor.chukalovskiy at gmail.com
Wed Jul 24 00:23:44 MSD 2013
Yes, these are the only ones I see printed on our sandbox. Will try if I
can get any other ones. I understand it's time consuming for you to
respond, thanks again for your help.
Today it got stuck again on production server and I ran db_cache status:
db="sofia_reg_<profile_name>"
Type: CORE_DB
Last used: 150861
Flags: Locked, Attached(1)
Creator: sofia_glue.c:6380
Last User: src/switch_core_sqldb.c:1816
.....
<same as above for each profile, we got a few dozens>
.....
db="/dev/shm/core.db"
Type: CORE_DB
Last used: 150862
Flags: Locked, Attached(1)
Creator: src/switch_core_sqldb.c:1816
Last User:
In what units is "last used"?
VM runs on iSCSI SAN Storage.
On 13-07-23 03:01 PM, Anthony Minessale wrote:
> Is that the only ones? how frequent are they? Those are only for
> expiration and run very infrequently. You may wish to study it longer
> because that should not be enough to slow everything down. Your
> server is not on a fake HDD too is it?
>
> I generally do not help on load testing or virtual server setups
> because it can consume a lot of time.
>
>
>
> On Tue, Jul 23, 2013 at 12:05 PM, Victor Chukalovskiy
> <victor.chukalovskiy at gmail.com <mailto:victor.chukalovskiy at gmail.com>>
> wrote:
>
> Anthony, thanks a lot. Here is what I got:
>
> [ERR] sofia_glue.c:6614 SQL: [select
> call_id,sip_user,sip_host,contact,status,rpid,expires,user_agent,server_user,server_host,profile_name,network_ip,0
> from sip_registrations where expires > 0 and expires <= 1374597711]
> [ERR] sofia_glue.c:6614 SQL: [select call_id from
> sip_shared_appearance_dialogs where hostname='test' and
> profile_name='test' and expires <= 1374597711]
>
> Is there a way to disable those two selects to save a few DB calls?
> Keep in mind that manage-shared-appearance and manage-presence are
> already off.
>
> As for the performance tweaking, I'm going to:
> -move ALL sqlite DBs to RAM
> -bump max handles. Does it sound reasonable and safe to go from 50
> to 100 handles?
>
> @Cal, saw your msg, thanks. I was indeed looking at master while
> Anthony meant stable.
>
> .Victor
>
>
> On 13-07-22 04:42 PM, Anthony Minessale wrote:
>> do you have manage-shared-appearance off too cos that re-enables
>> presence?
>>
>> edit sofia_glue.c on line 6614 and add a print to see what the
>> sql is that is being executed.
>>
>> switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_ERROR, "SQL:
>> [%s]\n", sql);
>>
>> then make mod_sofia-install
>>
>> and relaunch and try to catch a sample of what it's doing.
>>
>>
>>
>>
>>
>> On Mon, Jul 22, 2013 at 3:27 PM, Victor Chukalovskiy
>> <victor.chukalovskiy at gmail.com
>> <mailto:victor.chukalovskiy at gmail.com>> wrote:
>>
>> Anthony,
>>
>> Thank you, this is a very useful command. I see a bunch of
>> records like:
>>
>> db="sofia_reg_<profile_name>"
>> Type: CORE_DB
>> Last used: 10
>> Flags: Unlocked, Detached(0)
>> Creator: *sofia_glue.c:6605*
>> Last User: *sofia_glue.c:6605*
>>
>> Appears that these DBs are in use and user/creator is
>> sofia_glue.c:6605. Same as in error log:
>>
>> [WARNING] *sofia_glue.c:6605* Max handles 50
>>
>> And this is not a cored.db... it's sitting on the disk drive
>> still. Is there a way to setup sip_profiles not to create any
>> sofia_reg_...DBs?
>>
>> Re: vmware setup. It's good to know it's not optimal, however
>> it gives rack space, administrator's time, and $ savings
>> compared to dedicated servers.
>>
>> Should it be Ok to stick with VM setup provided enough CPU
>> and RAM is allocated to each FS guest? Are there reasons not
>> to run FS on VM other than CPU and RAM allocation?
>>
>> Thank you,
>> Victor
>>
>>
>>> Its a large number of handles to have open all in sofia if
>>> you are not doing anything
>>>
>>> db_cache status
>>>
>>> from cli can show where those handles come from and what db
>>> they are using.
>>> Clearly the box is being overloaded past its limits.
>>>
>>> This is why we don't endorse virtual servers.
>>>
>>>
>>>
>>> On Mon, Jul 22, 2013 at 2:49 PM, Victor Chukalovskiy
>>> <victor.chukalovskiy at gmail.com
>>> <mailto:victor.chukalovskiy at gmail.com>> wrote:
>>>
>>> Disabled in all profiles:
>>> <param name="manage-presence" value="false"/>
>>>
>>>
>>> On 13-07-22 03:40 PM, Anthony Minessale wrote:
>>>> Did you turn of the presence in the sofia profile, that
>>>> is enabled by default and very costly. comment out
>>>> manage-presence
>>>>
>>>>
>>>> On Mon, Jul 22, 2013 at 2:32 PM, Victor Chukalovskiy
>>>> <victor.chukalovskiy at gmail.com
>>>> <mailto:victor.chukalovskiy at gmail.com>> wrote:
>>>>
>>>> Michael, Ken,
>>>>
>>>> Thank you for the response. Firstly, let me clarify
>>>> setup for each of DBs in use:
>>>>
>>>> *core_db:* sqlite, /dev/shm/core.db (RAM) - using
>>>> default 50 max handles
>>>> *mod_sofia db**:* sqlite, /opt/fs/db (HDD)
>>>> *mod_lcr db:* Postgres
>>>> *mod_limit:* hash backend
>>>>
>>>> Note: I believe we rarely use mod_sofia DB's...this
>>>> is because we don't use any registrations, acl
>>>> only. When I list sofia per-profile db files on
>>>> production box, most of them has not been edited
>>>> for a couple days. Are my conclusions correct here,
>>>> or do I miss something? Maybe they are accessed a
>>>> lot but only for reading?
>>>>
>>>> I understand recommendations from Ken and from
>>>> Michaels are along the lines of increasing core db
>>>> and sofia db throughput. However here is what
>>>> puzzles me:
>>>> -the db used most extensively is core_db. It'
>>>> already in the ram.
>>>> -we don't have much load on sofia dbs (or am I
>>>> mistaken in my logic?)
>>>> -error log started with:
>>>>
>>>>
>>>> [CRIT] switch_time.c:967 Virtual Migration
>>>> Detected! Syncing Clock
>>>> [WARNING] switch_scheduler.c:115 Task was executed
>>>> late by 19 seconds 1 heartbeat (core)
>>>>
>>>> Does it mean that there were abnormal events prior
>>>> hitting 50 max handles limit?
>>>>
>>>> Thank you,
>>>> -Victor
>>>>
>>>>
>>>> On 13-07-22 02:52 PM, Anthony Minessale wrote:
>>>>> Check the odbcinst.ini settings from the wiki or
>>>>> use native pg mode.
>>>>> increase the max number of pg connections on your
>>>>> server and turn up FS to match in max-db-handles
>>>>> in switch.conf.xml
>>>>>
>>>>> You are already beating it up pretty good esp for
>>>>> a fake server though. 2500 is a lot.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 22, 2013 at 1:44 PM, Ken Rice
>>>>> <krice at freeswitch.org
>>>>> <mailto:krice at freeswitch.org>> wrote:
>>>>>
>>>>> Mount the entire freeswitch/db dir as tmpfs...
>>>>> most of these errors are for
>>>>> the sofia DB getting backed up...
>>>>>
>>>>> The coredb is a completely different sqlite db
>>>>> from the sofia db
>>>>>
>>>>> On 7/22/13 12:54 PM, "Victor Chukalovskiy"
>>>>> <victor.chukalovskiy at gmail.com
>>>>> <mailto:victor.chukalovskiy at gmail.com>>
>>>>> wrote:
>>>>>
>>>>> > Hello,
>>>>> >
>>>>> > Need some help finding the bottleneck here.
>>>>> Running 1.2.stable on a
>>>>> > decent multi-core vmware platform. No RTP,
>>>>> SIP signaling only. Around
>>>>> > 2500 sessions and 30 average CPS load.
>>>>> Essentially, it's a multi-profile
>>>>> > .xml config that uses mod_lcr extensively
>>>>> (postgres DB). Yesterday it
>>>>> > stopped processing calls with the following
>>>>> log output:
>>>>> >
>>>>> > 2013-07-21 06:00:37.642678 [CRIT]
>>>>> switch_time.c:967 Virtual Migration
>>>>> > Detected! Syncing Clock
>>>>> > 2013-07-21 06:01:07.141734 [WARNING]
>>>>> switch_scheduler.c:115 Task was
>>>>> > executed late by 19 seconds 1 heartbeat (core)
>>>>> > 2013-07-21 06:02:32.402668 [CRIT]
>>>>> switch_time.c:967 Virtual Migration
>>>>> > Detected! Syncing Clock
>>>>> > 2013-07-21 06:04:42.499744 [WARNING]
>>>>> switch_scheduler.c:115 Task was
>>>>> > executed late by 115 seconds 1 heartbeat (core)
>>>>> > 2013-07-21 06:04:42.499744 [WARNING]
>>>>> switch_scheduler.c:115 Task was
>>>>> > executed late by 74 seconds 2 check_ip (core)
>>>>> > 2013-07-21 06:05:15.102683 [CRIT]
>>>>> switch_time.c:967 Virtual Migration
>>>>> > Detected! Syncing Clock
>>>>> > 2013-07-21 06:05:44.961004 [WARNING]
>>>>> switch_scheduler.c:115 Task was
>>>>> > executed late by 22 seconds 1 heartbeat (core)
>>>>> > 2013-07-21 06:05:44.961004 [WARNING]
>>>>> switch_scheduler.c:115 Task was
>>>>> > executed late by 2 seconds 2 check_ip (core)
>>>>> > 2013-07-21 06:05:49.322672 [WARNING]
>>>>> mod_lcr.c:448 Max handles 50
>>>>> > exceeded, blocking....
>>>>> > 2013-07-21 06:05:50.962987 [WARNING]
>>>>> sofia_glue.c:6605 Max handles 50
>>>>> > exceeded, blocking....
>>>>> > 2013-07-21 06:05:50.962987 [WARNING]
>>>>> sofia_glue.c:6605 Max handles 50
>>>>> > exceeded, blocking....
>>>>> > 2013-07-21 06:05:50.962987 [WARNING]
>>>>> sofia_glue.c:6605 Max handles 50
>>>>> > exceeded, blocking....
>>>>> > ................................
>>>>> > 2013-07-21 06:05:53.962672 [WARNING]
>>>>> sofia_glue.c:6605 Max handles 50
>>>>> > exceeded, blocking....
>>>>> > 2013-07-21 06:05:54.422671 [ERR]
>>>>> mod_lcr.c:448 Error connecting
>>>>> > 2013-07-21 06:05:54.422671 [WARNING]
>>>>> mod_lcr.c:1779 LCR lookup failed
>>>>> > for 14389955082
>>>>> > 2013-07-21 06:05:54.962671 [WARNING]
>>>>> sofia_glue.c:6605 Max handles 50
>>>>> > exceeded, blocking....
>>>>> > 2013-07-21 06:05:54.962671 [WARNING]
>>>>> sofia_glue.c:6605 Max handles 50
>>>>> > exceeded, blocking....
>>>>> > ................................
>>>>> > 2013-07-21 06:05:55.962758 [WARNING]
>>>>> sofia_glue.c:6605 Max handles 50
>>>>> > exceeded, blocking....
>>>>> > 2013-07-21 06:05:56.062671 [ERR]
>>>>> sofia_glue.c:6605 Error connecting
>>>>> > 2013-07-21 06:05:56.062671 [ERR]
>>>>> sofia_glue.c:6606 Error Opening DB
>>>>> > 2013-07-21 06:05:56.062671 [WARNING]
>>>>> sofia_glue.c:6605 Max handles 50
>>>>> > exceeded, blocking....
>>>>> > 2013-07-21 06:05:56.062671 [ERR]
>>>>> sofia_glue.c:6605 Error connecting
>>>>> > 2013-07-21 06:05:56.062671 [ERR]
>>>>> sofia_glue.c:6606 Error Opening DB
>>>>> >
>>>>> > FreeSWITCH stayed unusable since then until
>>>>> it was restarted.
>>>>> >
>>>>> > FYI core DB is already in the RAM:
>>>>> > <param name="core-db-name"
>>>>> value="/dev/shm/core.db" />
>>>>> >
>>>>> >
>>>>> > Thank you,
>>>>> > Victor
>>>>> >
>>>>> >
>>>>> _________________________________________________________________________
>>>>> > Professional FreeSWITCH Consulting Services:
>>>>> > consulting at freeswitch.org
>>>>> <mailto:consulting at freeswitch.org>
>>>>> > http://www.freeswitchsolutions.com
>>>>> >
>>>>> > FreeSWITCH-powered IP PBX: The CudaTel
>>>>> Communication Server
>>>>> >
>>>>> >
>>>>> > Official FreeSWITCH Sites
>>>>> > http://www.freeswitch.org
>>>>> > http://wiki.freeswitch.org
>>>>> > http://www.cluecon.com
>>>>> >
>>>>> > FreeSWITCH-users mailing list
>>>>> > FreeSWITCH-users at lists.freeswitch.org
>>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>> >
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> > http://www.freeswitch.org
>>>>>
>>>>> --
>>>>> Ken
>>>>> http://www.FreeSWITCH.org
>>>>> http://www.ClueCon.com
>>>>> http://www.OSTAG.org
>>>>> irc.freenode.net <http://irc.freenode.net>
>>>>> #freeswitch
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> <mailto:consulting at freeswitch.org>
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> FreeSWITCH-powered IP PBX: The CudaTel
>>>>> Communication Server
>>>>>
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Anthony Minessale II
>>>>>
>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>> ClueCon http://www.cluecon.com/
>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>
>>>>> AIM: anthm
>>>>> MSN:anthony_minessale at hotmail.com
>>>>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>>>> IRC: irc.freenode.net <http://irc.freenode.net>
>>>>> #freeswitch
>>>>>
>>>>> FreeSWITCH Developer Conference
>>>>> sip:888 at conference.freeswitch.org
>>>>> <mailto:sip%3A888 at conference.freeswitch.org>
>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>>>> pstn:+19193869900
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> <mailto:consulting at freeswitch.org>
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> FreeSWITCH-powered IP PBX: The CudaTel
>>>> Communication Server
>>>>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Anthony Minessale II
>>>>
>>>> FreeSWITCH http://www.freeswitch.org/
>>>> ClueCon http://www.cluecon.com/
>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>
>>>> AIM: anthm
>>>> MSN:anthony_minessale at hotmail.com
>>>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>>
>>>> FreeSWITCH Developer Conference
>>>> sip:888 at conference.freeswitch.org
>>>> <mailto:sip%3A888 at conference.freeswitch.org>
>>>> googletalk:conf+888 at conference.freeswitch.org
>>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>>> pstn:+19193869900
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>> http://www.freeswitchsolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>>
>>>
>>> --
>>> Anthony Minessale II
>>>
>>> FreeSWITCH http://www.freeswitch.org/
>>> ClueCon http://www.cluecon.com/
>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>
>>> AIM: anthm
>>> MSN:anthony_minessale at hotmail.com
>>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>
>>> FreeSWITCH Developer Conference
>>> sip:888 at conference.freeswitch.org
>>> <mailto:sip%3A888 at conference.freeswitch.org>
>>> googletalk:conf+888 at conference.freeswitch.org
>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>> pstn:+19193869900
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>>
>> --
>> Anthony Minessale II
>>
>> FreeSWITCH http://www.freeswitch.org/
>> ClueCon http://www.cluecon.com/
>> Twitter: http://twitter.com/FreeSWITCH_wire
>>
>> AIM: anthm
>> MSN:anthony_minessale at hotmail.com
>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>
>> FreeSWITCH Developer Conference
>> sip:888 at conference.freeswitch.org
>> <mailto:sip%3A888 at conference.freeswitch.org>
>> googletalk:conf+888 at conference.freeswitch.org
>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>> pstn:+19193869900
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> <mailto:sip%3A888 at conference.freeswitch.org>
> googletalk:conf+888 at conference.freeswitch.org
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:+19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130723/99df058f/attachment-0001.html
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list