[Freeswitch-users] CPU Spikes every 20 minutes
Ken Rice
krice at freeswitch.org
Tue Oct 23 00:47:25 MSD 2012
How many concurrent calls are you doing on that box?
On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez at redvoiss.net> wrote:
> Hi Ken.
> I mounted the db directory in RAM as you mention, but the ³load average² still
> is high.
> You can see this here :
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/VolGroup00-LogVol00
> 132G 8.6G 117G 7% /
> proc 0 0 0 - /proc
> sysfs 0 0 0 - /sys
> devpts 0 0 0 - /dev/pts
> /dev/cciss/c0d0p1 104M 20M 79M 21% /boot
> tmpfs 4.2G 0 4.2G 0% /dev/shm
> none 0 0 0 - /proc/sys/fs/binfmt_misc
> sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs
> tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db
>
> This is a resume from uptime and total calls for my freeswitch :
>
> 17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50,
> 4.66 , Total Calls : 209, Total sessions: 440
> 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92,
> 4.80 , Total Calls : 245, Total sessions: 510
> 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84,
> 4.77 , Total Calls : 235, Total sessions: 488
> 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94,
> 4.81 , Total Calls : 233, Total sessions: 484
> 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92,
> 4.80 , Total Calls : 226, Total sessions: 470
> 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84,
> 4.78 , Total Calls : 218, Total sessions: 454
> 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76,
> 4.75 , Total Calls : 198, Total sessions: 416
> 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68,
> 4.72 , Total Calls : 183, Total sessions: 384
> 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60,
> 4.70 , Total Calls : 172, Total sessions: 366
> 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54,
> 4.68 , Total Calls : 184, Total sessions: 388
> 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47,
> 4.65 , Total Calls : 207, Total sessions: 442
> 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48,
> 4.65 , Total Calls : 203, Total sessions: 436
> 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40,
> 4.63 , Total Calls : 198, Total sessions: 418
> 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32,
> 4.60 , Total Calls : 198, Total sessions: 422
> 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47,
> 4.65 , Total Calls : 194, Total sessions: 410
> 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31,
> 4.92 , Total Calls : 211, Total sessions: 450
> 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32,
> 4.92 , Total Calls : 251, Total sessions: 536
> 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49,
> 4.98 , Total Calls : 250, Total sessions: 530
> 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70,
> 5.05 , Total Calls : 236, Total sessions: 496
> 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76,
> 5.07 , Total Calls : 233, Total sessions: 492
> 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84,
> 5.10 , Total Calls : 235, Total sessions: 496
> 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04,
> 5.17 , Total Calls : 222, Total sessions: 466
> 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16,
> 5.21 , Total Calls : 215, Total sessions: 452
> 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36,
> 5.28 , Total Calls : 208, Total sessions: 430
> 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47,
> 5.32 , Total Calls : 202, Total sessions: 422
> 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61,
> 5.37 , Total Calls : 240, Total sessions: 500
> 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78,
> 5.44 , Total Calls : 239, Total sessions: 498
> 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35,
> 5.63 , Total Calls : 250, Total sessions: 516
> 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36,
> 5.64 , Total Calls : 236, Total sessions: 490
> 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29,
> 5.63 , Total Calls : 209, Total sessions: 436
> 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18,
> 5.60 , Total Calls : 180, Total sessions: 372
> 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06,
> 5.57 , Total Calls : 161, Total sessions: 342
> 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24,
> 5.64 , Total Calls : 175, Total sessions: 370
> 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22,
> 5.64 , Total Calls : 181, Total sessions: 382
> 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30,
> 5.67 , Total Calls : 178, Total sessions: 366
> 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35,
> 5.69 , Total Calls : 173, Total sessions: 358
> 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51,
> 5.76 , Total Calls : 182, Total sessions: 378
> 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53,
> 5.77 , Total Calls : 187, Total sessions: 386
> 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77,
> 5.86 , Total Calls : 190, Total sessions: 402
> 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81,
> 5.88 , Total Calls : 239, Total sessions: 496
> 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93,
> 5.93 , Total Calls : 257, Total sessions: 532
> 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09,
> 5.99 , Total Calls : 242, Total sessions: 506
> 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16,
> 6.03 , Total Calls : 233, Total sessions: 482
> 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17,
> 6.04 , Total Calls : 238, Total sessions: 492
> 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30,
> 6.10 , Total Calls : 236, Total sessions: 482
> 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46,
> 6.16 , Total Calls : 222, Total sessions: 458
> 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39,
> 6.15 , Total Calls : 221, Total sessions: 470
> 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33,
> 6.14 , Total Calls : 200, Total sessions: 422
> 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77,
> 6.30 , Total Calls : 214, Total sessions: 452
> 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93,
> 6.36 , Total Calls : 255, Total sessions: 536
> 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06,
> 6.42 , Total Calls : 252, Total sessions: 524
> 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46,
> 6.56 , Total Calls : 260, Total sessions: 536
> 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42,
> 6.56 , Total Calls : 246, Total sessions: 508
> 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52,
> 6.61 , Total Calls : 230, Total sessions: 478
> 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58,
> 6.64 , Total Calls : 194, Total sessions: 408
> 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59,
> 6.66 , Total Calls : 189, Total sessions: 396
> 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55,
> 6.66 , Total Calls : 202, Total sessions: 438
> 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49,
> 6.66 , Total Calls : 221, Total sessions: 470
> 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46,
> 6.67 , Total Calls : 220, Total sessions: 478
> 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80,
> 6.79 , Total Calls : 219, Total sessions: 480
> 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76,
> 6.79 , Total Calls : 222, Total sessions: 490
> 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69,
> 6.79 , Total Calls : 214, Total sessions: 470
> 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85,
> 6.85 , Total Calls : 210, Total sessions: 450
> 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97,
> 6.91 , Total Calls : 250, Total sessions: 524
> 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80,
> 6.87 , Total Calls : 258, Total sessions: 536
> 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64,
> 6.83 , Total Calls : 255, Total sessions: 528
> 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49,
> 6.80 , Total Calls : 246, Total sessions: 524
> 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35,
> 6.77 , Total Calls : 262, Total sessions: 564
> 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21,
> 6.74 , Total Calls : 238, Total sessions: 510
> 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06,
> 6.70 , Total Calls : 220, Total sessions: 476
> 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91,
> 6.67 , Total Calls : 228, Total sessions: 498
>
> What else could be happening.
> Did I mention that I¹m using two transcoding cards from Sangoma?
>
> Regards,
> Ricardo.-
>
>
>
>
> De: freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] En nombre de Ken Rice
> Enviado el: lunes, 22 de octubre de 2012 16:50
> Para: FreeSWITCH Users Help
> Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
>
> Load Average is the number of threads/processes waiting in the kernel¹s run
> queue at sample time... Supposedly number of cores is the how much your load
> avg can get to before your boxes is exhausted but this isnt exactly true on
> modern hardware with highly multi-threaded software like FreeSWITCH...
>
> Where did I get the 100+ number? From looking at Œtop¹ or Œw¹ on a box to see
> what the current load avg is...
>
> Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything
> in between.... When you type Œshow channels¹ from fs_cli it doesn¹t query the
> structures in RAM to show you that output, it pulls it from the sqlite db...
>
> That being said, you can test quickly to see if sqlite is hammering the disk
> at some point by shifting freeswitch/db dir to a ram drive.
> Example code for linux such as Centos or Debian assuming freeswitch is
> installed to /usr/local/freeswitch
>
> Shutdown freeswitch
> rm rf /usr/local/freeswitch/db/*
> mount t tmpfs tmpfs /usr/local/freeswitch/db
> Startup freeswitch
>
> Once you have done that stick it back in rotation...
>
> Note this will not affect anything but voicemail if you have voicemail running
> on that box and not using an external DB via mod_voicemail¹s DSN settings
> you¹ll lose the voicemail DB... (I would assume this is not the case since you
> are using this primarily as a Transcoder)
>
> K
>
>
> On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez at redvoiss.net> wrote:
> Hi Ken.
> Maybe that is the problem. I was sure that the ³load average² can¹t go more
> than #cores in your system. In my case I have 8 cores, so a ³load average² of
> 12 is high from my point of view. Where did you get the number 100 for this
> parameter?
> Anyway, what is running a sqlite in freeswitch. Does this have to do with the
> core.db file? Or with the cdr_sqlite module?.
>
> Thanks
> Ricardo.-
>
>
> De: freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] En nombre de Ken Rice
> Enviado el: lunes, 22 de octubre de 2012 16:18
> Para: FreeSWITCH Users Help
> Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
>
> Could it be just a normal sqlite running a vacuum?
>
> Sqlite does this every so often or if its calling a sync....
>
> Also freeswitch with a load avg of 12 for a short period of time isnt anything
> to really worry about. Due to the number of threads that¹s possible with FS
> (ie: 1 per call leg) its not unusual to see a box humming along and the load
> avg well above 100 in my experience... This is due to the number of threads
> waiting in the run queue when it was sampled... Now, that¹s not saying that
> this is not specifically indicative or a problem but something else to think
> about
>
> On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez at redvoiss.net> wrote:
> This is the gdb from the proccess consuming more cpu :
>
> (gdb) thread 7
> [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from
> /lib64/libpthread.so.0
> (gdb) bt
> #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0
> #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value
> optimized out>) at ./src/os_unix.c:1229
> #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229
> #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0,
> nTrunc=0) at ./src/pager.c:3770
> #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>,
> zMaster=0x0) at ./src/btree.c:6520
> #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at
> ./src/vdbeaux.c:1062
> #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at
> ./src/vdbe.c:2360
> #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at
> ./src/vdbeapi.c:236
> #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289
> #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717
> "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0)
> at ./src/legacy.c:78
> #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070,
> sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38)
> at src/switch_core_db.c:93
> #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0,
> sql=0x2aede199e717 "COMMIT", err=0x0)
> at src/switch_core_sqldb.c:445
> #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans
> (dbh=0x1a2dce0,
> sql=0x5f03bd0 "delete from calls where
> (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or
> callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and
> hostname='transcoder.red.net <http://transcoder.red.net>
> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls
> where (call"..., retries=1)
> at src/switch_core_sqldb.c:787
> #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized
> out>, obj=<value optimized out>)
> at src/switch_core_sqldb.c:1026
> #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0
> #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
>
>
> De: Ricardo Martinez [mailto:rmartinez at redvoiss.net]
> Enviado el: lunes, 22 de octubre de 2012 15:50
> Para: 'FreeSWITCH Users Help'
> Asunto: RE: [Freeswitch-users] CPU Spikes every 20 minutes
>
> AS far as i can see there is no java running.
> This is the top at the moment of the spike :
>
> top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48
> Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie
> Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st
> Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers
> Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51
> freeswitch
> 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08
> sngtc_server
> 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02
> top
> 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13
> init
> 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41
> migration/0
> 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00
> ksoftirqd/0
> 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00
> watchdog/0
> 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83
> migration/1
> 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00
> ksoftirqd/1
> 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
>
> 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14
> migration/2
> 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00
> ksoftirqd/2
> 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00
> watchdog/2
> 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07
> migration/3
> 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00
> ksoftirqd/3
> 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00
> watchdog/3
> 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06
> migration/4
> 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00
> ksoftirqd/4
> 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00
> watchdog/4
> 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06
> migration/5
> 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00
> ksoftirqd/5
> 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00
> watchdog/5
> 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07
> migration/6
>
>
> Regards,
> Ricardo.-
>
> De: freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] En nombre de Guillermo
> Ruiz Camauer
> Enviado el: lunes, 22 de octubre de 2012 15:38
> Para: FreeSWITCH Users Help
> Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
>
> Do you have anything like Java running that could be doing garbage collection?
>
>
>
> What other processes are running on the machine? Does TOP show which process
> is causing the spike?
>
>
>
> Guillermo
>
> On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez at redvoiss.net>
> wrote:
> Hi Gabe.
> I'm mean between the spikes there is nothing different as regarding to the
> calls... the cps maintains the same.. there are no scripts running or
> crontabs running, the simultaneous calls are the same also. So, for some
> reason the "load average" start to grow (every 20 or 25 minutes) without
> any clear cause.
> I have debug with gdb the moment the server is with 12.5 of "load average"
> here : http://pastebin.freeswitch.org/20112
>
> Can we detect a problem from here?
>
> Regards,
> Ricardo.
>
>
>
> -----Mensaje original-----
> De: freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] En nombre de
> Gabriel Gunderson
> Enviado el: lunes, 22 de octubre de 2012 13:31
> Para: FreeSWITCH Users Help
> Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
>
>
> On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez at redvoiss.net>
> wrote:
>> > The problem is that I'm having weird spikes of "load average" cpu
>> > every 20 minutes (aprox), without any reason, reaching near of 12.5 of
>> > load average for this machine.
>
> What have you done to verify that it happens 'without any reason'?
>
>
> Gabe
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
--
Ken
http://www.FreeSWITCH.org
http://www.ClueCon.com
http://www.OSTAG.org
irc.freenode.net #freeswitch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20121022/6c7868cb/attachment-0001.html
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list