[Freeswitch-users] CPU Spikes every 20 minutes

Tue Oct 23 00:53:52 MSD 2012

Re: [Freeswitch-users] CPU Spikes every 20 minutes

Ken.

There is an average of  220 concurrent calls.

Machine : DL380 G7 – 2 Quad Core – 8 Gig RAM.

Centos 5.5 64bits

Ricarod.-

*De:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
freeswitch-users-bounces at lists.freeswitch.org] *En nombre de *Ken Rice
*Enviado el:* lunes, 22 de octubre de 2012 17:47
*Para:* FreeSWITCH Users Help
*Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes

How many concurrent calls are you doing on that box?

On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez at redvoiss.net> wrote:

Hi Ken.
I mounted the db directory in RAM as you mention, but the “load average”
still is high.
You can see this here :

Filesystem             Size   Used  Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       132G   8.6G   117G   7% /
proc                      0      0      0   -  /proc
sysfs                     0      0      0   -  /sys
devpts                    0      0      0   -  /dev/pts
/dev/cciss/c0d0p1      104M    20M    79M  21% /boot
tmpfs                  4.2G      0   4.2G   0% /dev/shm
none                      0      0      0   -  /proc/sys/fs/binfmt_misc
sunrpc                    0      0      0   -  /var/lib/nfs/rpc_pipefs
*tmpfs                  4.2G   1.3M   4.2G   1% /usr/local/freeswitch/db
*
This is a resume from uptime and total calls for my freeswitch :

17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77,
4.50, 4.66 , Total Calls : 209, Total sessions: 440
17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79,
4.92, 4.80 , Total Calls : 245, Total sessions: 510
17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25,
4.84, 4.77 , Total Calls : 235, Total sessions: 488
17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63,
4.94, 4.81 , Total Calls : 233, Total sessions: 484
17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42,
4.92, 4.80 , Total Calls : 226, Total sessions: 470
17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90,
4.84, 4.78 , Total Calls : 218, Total sessions: 454
17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43,
4.76, 4.75 , Total Calls : 198, Total sessions: 416
17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99,
4.68, 4.72 , Total Calls : 183, Total sessions: 384
17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59,
4.60, 4.70 , Total Calls : 172, Total sessions: 366
17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31,
4.54, 4.68 , Total Calls : 184, Total sessions: 388
17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96,
4.47, 4.65 , Total Calls : 207, Total sessions: 442
17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04,
4.48, 4.65 , Total Calls : 203, Total sessions: 436
17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72,
4.40, 4.63 , Total Calls : 198, Total sessions: 418
17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47,
4.32, 4.60 , Total Calls : 198, Total sessions: 422
17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23,
4.47, 4.65 , Total Calls : 194, Total sessions: 410
17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30,
5.31, 4.92 , Total Calls : 211, Total sessions: 450
17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11,
5.32, 4.92 , Total Calls : 251, Total sessions: 536
17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74,
5.49, 4.98 , Total Calls : 250, Total sessions: 530
17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48,
5.70, 5.05 , Total Calls : 236, Total sessions: 496
17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45,
5.76, 5.07 , Total Calls : 233, Total sessions: 492
17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57,
5.84, 5.10 , Total Calls : 235, Total sessions: 496
17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24,
6.04, 5.17 , Total Calls : 222, Total sessions: 466
17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46,
6.16, 5.21 , Total Calls : 215, Total sessions: 452
17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07,
6.36, 5.28 , Total Calls : 208, Total sessions: 430
17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22,
6.47, 5.32 , Total Calls : 202, Total sessions: 422
17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52,
6.61, 5.37 , Total Calls : 240, Total sessions: 500
17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96,
6.78, 5.44 , Total Calls : 239, Total sessions: 498
17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29,
7.35, 5.63 , Total Calls : 250, Total sessions: 516
17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78,
7.36, 5.64 , Total Calls : 236, Total sessions: 490
17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92,
7.29, 5.63 , Total Calls : 209, Total sessions: 436
17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97,
7.18, 5.60 , Total Calls : 180, Total sessions: 372
17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01,
7.06, 5.57 , Total Calls : 161, Total sessions: 342
17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57,
7.24, 5.64 , Total Calls : 175, Total sessions: 370
17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12,
7.22, 5.64 , Total Calls : 181, Total sessions: 382
17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19,
7.30, 5.67 , Total Calls : 178, Total sessions: 366
17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10,
7.35, 5.69 , Total Calls : 173, Total sessions: 358
17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57,
7.51, 5.76 , Total Calls : 182, Total sessions: 378
17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36,
7.53, 5.77 , Total Calls : 187, Total sessions: 386
17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21,
7.77, 5.86 , Total Calls : 190, Total sessions: 402
17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04,
7.81, 5.88 , Total Calls : 239, Total sessions: 496
17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27,
7.93, 5.93 , Total Calls : 257, Total sessions: 532
17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73,
8.09, 5.99 , Total Calls : 242, Total sessions: 506
17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67,
8.16, 6.03 , Total Calls : 233, Total sessions: 482
17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38,
8.17, 6.04 , Total Calls : 238, Total sessions: 492
17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67,
8.30, 6.10 , Total Calls : 236, Total sessions: 482
17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09,
8.46, 6.16 , Total Calls : 222, Total sessions: 458
17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37,
8.39, 6.15 , Total Calls : 221, Total sessions: 470
17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78,
8.33, 6.14 , Total Calls : 200, Total sessions: 422
17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64,
8.77, 6.30 , Total Calls : 214, Total sessions: 452
17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98,
8.93, 6.36 , Total Calls : 255, Total sessions: 536
17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23,
9.06, 6.42 , Total Calls : 252, Total sessions: 524
17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73,
9.46, 6.56 , Total Calls : 260, Total sessions: 536
17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03,
9.42, 6.56 , Total Calls : 246, Total sessions: 508
17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11,
9.52, 6.61 , Total Calls : 230, Total sessions: 478
17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94,
9.58, 6.64 , Total Calls : 194, Total sessions: 408
17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54,
9.59, 6.66 , Total Calls : 189, Total sessions: 396
17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94,
9.55, 6.66 , Total Calls : 202, Total sessions: 438
17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30,
9.49, 6.66 , Total Calls : 221, Total sessions: 470
17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88,
9.46, 6.67 , Total Calls : 220, Total sessions: 478
17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25,
9.80, 6.79 , Total Calls : 219, Total sessions: 480
17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67,
9.76, 6.79 , Total Calls : 222, Total sessions: 490
17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05,
9.69, 6.79 , Total Calls : 214, Total sessions: 470
17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53,
9.85, 6.85 , Total Calls : 210, Total sessions: 450
17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81,
9.97, 6.91 , Total Calls : 250, Total sessions: 524
17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70,
9.80, 6.87 , Total Calls : 258, Total sessions: 536
17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68,
9.64, 6.83 , Total Calls : 255, Total sessions: 528
17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83,
9.49, 6.80 , Total Calls : 246, Total sessions: 524
17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04,
9.35, 6.77 , Total Calls : 262, Total sessions: 564
17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32,
9.21, 6.74 , Total Calls : 238, Total sessions: 510
17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57,
9.06, 6.70 , Total Calls : 220, Total sessions: 476
17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88,
8.91, 6.67 , Total Calls : 228, Total sessions: 498

What else could be happening.
Did I mention that I’m using two transcoding cards from Sangoma?

Regards,
Ricardo.-

*De:* freeswitch-users-bounces at lists.freeswitch.org [
mailto:freeswitch-users-bounces at lists.freeswitch.org<freeswitch-users-bounces at lists.freeswitch.org>]
*En nombre de *Ken Rice
*Enviado el:* lunes, 22 de octubre de 2012 16:50
*Para:* FreeSWITCH Users Help
*Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes

Load Average is the number of threads/processes waiting in the kernel’s run
queue at sample time... Supposedly number of cores is the how much your
load avg can get to before your boxes is exhausted but this isnt exactly
true on modern hardware with highly multi-threaded software like
FreeSWITCH...

Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to
see what the current load avg is...

Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and
everything in between.... When you type ‘show channels’ from fs_cli it
doesn’t query the structures in RAM to show you that output, it pulls it
from the sqlite db...

That being said, you can test quickly to see if sqlite is hammering the
disk at some point by shifting freeswitch/db dir to a ram drive.
Example code for linux such as Centos or Debian assuming freeswitch is
installed to /usr/local/freeswitch

Shutdown freeswitch
rm –rf /usr/local/freeswitch/db/*
mount –t tmpfs tmpfs /usr/local/freeswitch/db
Startup freeswitch

Once you have done that stick it back in rotation...

Note this will not affect anything but voicemail if you have voicemail
running on that box and not using an external DB via mod_voicemail’s DSN
settings you’ll lose the voicemail DB... (I would assume this is not the
case since you are using this primarily as a Transcoder)

K

On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez at redvoiss.net> wrote:
Hi Ken.
Maybe that is the problem.  I was sure that the “load average” can’t go
more than #cores in your system.  In my case I have 8 cores, so a “load
average” of 12 is high from my point of view.  Where did you get the number
100 for this parameter?
Anyway, what is running a sqlite in freeswitch.  Does this have to do with
the core.db file? Or with the cdr_sqlite module?.

Thanks
Ricardo.-

*De:* freeswitch-users-bounces at lists.freeswitch.org [
mailto:freeswitch-users-bounces at lists.freeswitch.org<freeswitch-users-bounces at lists.freeswitch.org>]
*En nombre de *Ken Rice
*Enviado el:* lunes, 22 de octubre de 2012 16:18
*Para:* FreeSWITCH Users Help
*Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes

Could it be just a normal sqlite running a vacuum?

Sqlite does this every so often or if its calling a sync....

Also freeswitch with a load avg of 12 for a short period of time isnt
anything to really worry about. Due to the number of threads that’s
possible with FS (ie: 1 per call leg) its not unusual to see a box humming
along and the load avg well above 100 in my experience... This is due to
the number of threads waiting in the run queue when it was sampled... Now,
that’s not saying that this is not specifically indicative or a problem but
something else to think about

On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez at redvoiss.net> wrote:
This is the gdb from the proccess consuming more cpu :

(gdb) thread 7
[Switching to thread 7 (Thread 31781)]#0  0x0000003c03c0e047 in fsync ()
from /lib64/libpthread.so.0
(gdb) bt
#0  0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0
#1  0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value
optimized out>) at ./src/os_unix.c:1229
#2  0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at
./src/pager.c:2229
#3  0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0,
nTrunc=0) at ./src/pager.c:3770
#4  0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>,
zMaster=0x0) at ./src/btree.c:6520
#5  0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at
./src/vdbeaux.c:1062
#6  0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at
./src/vdbe.c:2360
#7  0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at
./src/vdbeapi.c:236
#8  sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289
#9  0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717
"COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0)
    at ./src/legacy.c:78
#10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070,
sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38)
    at src/switch_core_db.c:93
#11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0,
sql=0x2aede199e717 "COMMIT", err=0x0)
    at src/switch_core_sqldb.c:445
#12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans
(dbh=0x1a2dce0,
    sql=0x5f03bd0 "delete from calls where
(caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or
callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname='
transcoder.red.net <http://transcoder.red.net>  <http://transcoder.red.net>
 <http://transcoder.red.net> ';\ndelete from calls where (call"...,
retries=1)
    at src/switch_core_sqldb.c:787
#13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized
out>, obj=<value optimized out>)
    at src/switch_core_sqldb.c:1026
#14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0
#15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6

*De:* Ricardo Martinez [mailto:rmartinez at redvoiss.net<rmartinez at redvoiss.net>]

*Enviado el:* lunes, 22 de octubre de 2012 15:50
*Para:* 'FreeSWITCH Users Help'
*Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes

AS far as i can see there is no java running.
This is the top at the moment of the spike :

top - 15:48:29 up 17 days, 14:55,  3 users,  load average: 14.07, 8.87, 6.48
Tasks: 158 total,   1 running, 157 sleeping,   0 stopped,   0 zombie
Cpu(s): 12.6%us,  9.0%sy,  0.0%ni, 66.1%id,  2.7%wa,  0.3%hi,  9.2%si,
0.0%st
Mem:   8171500k total,  8041556k used,   129944k free,   177688k buffers
Swap: 10223608k total,        0k used, 10223608k free,  6951988k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
COMMAND
31772 root      15   0  870m 481m 6880 S 212.4  6.0   9523:51
freeswitch
 4210 root      19   0 36344 2940 1444 S  1.0  0.0  56:21.08
sngtc_server
29586 root      15   0 12740 1132  808 R  0.3  0.0   0:00.02
top
    1 root      15   0 10348  704  588 S  0.0  0.0   0:01.13
init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.41
migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00
ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00
watchdog/0
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.83
migration/1
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.00
ksoftirqd/1
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1

    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.14
migration/2
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.00
ksoftirqd/2
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00
watchdog/2
   11 root      RT  -5     0    0    0 S  0.0  0.0   0:00.07
migration/3
   12 root      34  19     0    0    0 S  0.0  0.0   0:00.00
ksoftirqd/3
   13 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00
watchdog/3
   14 root      RT  -5     0    0    0 S  0.0  0.0   0:00.06
migration/4
   15 root      34  19     0    0    0 S  0.0  0.0   0:00.00
ksoftirqd/4
   16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00
watchdog/4
   17 root      RT  -5     0    0    0 S  0.0  0.0   0:00.06
migration/5
   18 root      34  19     0    0    0 S  0.0  0.0   0:00.00
ksoftirqd/5
   19 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00
watchdog/5
   20 root      RT  -5     0    0    0 S  0.0  0.0   0:00.07
migration/6

Regards,
Ricardo.-

*De:* freeswitch-users-bounces at lists.freeswitch.org [
mailto:freeswitch-users-bounces at lists.freeswitch.org<freeswitch-users-bounces at lists.freeswitch.org>]
*En nombre de *Guillermo Ruiz Camauer
*Enviado el:* lunes, 22 de octubre de 2012 15:38
*Para:* FreeSWITCH Users Help
*Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes

Do you have anything like Java running that could be doing garbage
collection?

What other processes are running on the machine?  Does TOP show which
process is causing the spike?

Guillermo

On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez at redvoiss.net>
wrote:
Hi Gabe.
I'm mean between the spikes there is nothing different as regarding to the
calls... the cps maintains the same.. there are no scripts running or
crontabs running, the simultaneous calls are the same also.  So, for some
reason the "load average" start to grow (every 20 or 25 minutes) without
any clear cause.
I have debug with gdb the moment the server is with 12.5 of "load average"
here : http://pastebin.freeswitch.org/20112

Can we detect a problem from here?

Regards,
Ricardo.

-----Mensaje original-----
De: freeswitch-users-bounces at lists.freeswitch.org
[mailto:freeswitch-users-bounces at lists.freeswitch.org<freeswitch-users-bounces at lists.freeswitch.org>]
En nombre de
Gabriel Gunderson
Enviado el: lunes, 22 de octubre de 2012 13:31
Para: FreeSWITCH Users Help
Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes

On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez at redvoiss.net>
wrote:
> The problem is that I'm having weird spikes of  "load average" cpu
> every 20 minutes (aprox), without any reason, reaching near of 12.5 of
> load average for this machine.

What have you done to verify that it happens 'without any reason'?

Gabe

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

-- 
Ken
*http://www.FreeSWITCH.org
http://www.ClueCon.com
http://www.OSTAG.org
*irc.freenode.net #freeswitch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20121022/dd17f8bb/attachment-0001.html 