Latest master and latest v1.4 consume more CPU for conferences
Hi guys, I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago. In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference. In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more. Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU. Any ideas as to what's causing the increased CPU usage would be greatly appreciated.
I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation. You can't use vague references like "latest" which changes every 10 min. You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace. The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions. Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup. On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi guys,
I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago.
In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference.
In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more.
Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU.
Any ideas as to what's causing the increased CPU usage would be greatly appreciated.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
Hi Anthony, thank you for responding! I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up. Here are the full versions for both the master where it happened and v1.4. Is this enough? FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit) FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit) It does not occur on this master: FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit) I figured I would try this email before bisecting git but it will likely be my next step. On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase. On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation.
You can't use vague references like "latest" which changes every 10 min. You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace. The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions. Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup.
On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi guys,
I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago.
In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference.
In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more.
Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU.
Any ideas as to what's causing the increased CPU usage would be greatly appreciated.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
It is very likely to add to the cpu to transcode everything to 12000hz On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony, thank you for responding!
I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up.
Here are the full versions for both the master where it happened and v1.4. Is this enough?
FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit) FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
It does not occur on this master: FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit)
I figured I would try this email before bisecting git but it will likely be my next step.
On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase.
On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation.
You can't use vague references like "latest" which changes every 10 min. You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace. The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions. Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup.
On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi guys,
I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago.
In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference.
In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more.
Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU.
Any ideas as to what's causing the increased CPU usage would be greatly appreciated.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change. If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened. On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It is very likely to add to the cpu to transcode everything to 12000hz
On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony, thank you for responding!
I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up.
Here are the full versions for both the master where it happened and v1.4. Is this enough?
FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit) FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
It does not occur on this master: FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit)
I figured I would try this email before bisecting git but it will likely be my next step.
On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase.
On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation.
You can't use vague references like "latest" which changes every 10 min. You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace. The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions. Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup.
On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi guys,
I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago.
In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference.
In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more.
Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU.
Any ideas as to what's causing the increased CPU usage would be greatly appreciated.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter? On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change.
If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened.
On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It is very likely to add to the cpu to transcode everything to 12000hz
On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony, thank you for responding!
I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up.
Here are the full versions for both the master where it happened and v1.4. Is this enough?
FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit) FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
It does not occur on this master: FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit)
I figured I would try this email before bisecting git but it will likely be my next step.
On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase.
On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation.
You can't use vague references like "latest" which changes every 10 min. You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace. The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions. Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup.
On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi guys,
I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago.
In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference.
In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more.
Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU.
Any ideas as to what's causing the increased CPU usage would be greatly appreciated.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
I am not sure we are on the same page with the numbers. The relative increase is about 30%-40% The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0% I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related. The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me. On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> wrote:
What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter?
On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change.
If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened.
On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It is very likely to add to the cpu to transcode everything to 12000hz
On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony, thank you for responding!
I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up.
Here are the full versions for both the master where it happened and v1.4. Is this enough?
FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit) FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
It does not occur on this master: FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit)
I figured I would try this email before bisecting git but it will likely be my next step.
On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase.
On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation.
You can't use vague references like "latest" which changes every 10 min. You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace. The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions. Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup.
On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi guys,
I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago.
In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference.
In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more.
Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU.
Any ideas as to what's causing the increased CPU usage would be greatly appreciated.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
*Brian West* brian@freeswitch.org
*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com
*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Why do you have two person conferences instead of bridged calls? Sent from my BlackBerry® PlayBook™ www.blackberry.com ------------------------------ *From:* "Oleg Stolyar" <olegstolyar@gmail.com> *To:* "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> *Sent:* 31 July, 2014 9:40 AM *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences I am not sure we are on the same page with the numbers. The relative increase is about 30%-40% The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0% I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related. The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me. On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> wrote:
What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter?
On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change.
If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened.
On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It is very likely to add to the cpu to transcode everything to 12000hz
On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony, thank you for responding!
I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up.
Here are the full versions for both the master where it happened and v1.4. Is this enough?
FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit) FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
It does not occur on this master: FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit)
I figured I would try this email before bisecting git but it will likely be my next step.
On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase.
On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation.
You can't use vague references like "latest" which changes every 10 min. You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace. The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions. Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup.
On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi guys,
I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago.
In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference.
In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more.
Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU.
Any ideas as to what's causing the increased CPU usage would be greatly appreciated.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
*Brian West* brian@freeswitch.org
*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com
*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
That's a whole different discussion. Mostly because I need the ability to easily add another person to the call. On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven <rtreleaven@bunnykick.ca> wrote:
Why do you have two person conferences instead of bridged calls?
Sent from my BlackBerry® PlayBook™ www.blackberry.com
------------------------------ *From:* "Oleg Stolyar" <olegstolyar@gmail.com> *To:* "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> *Sent:* 31 July, 2014 9:40 AM *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences
I am not sure we are on the same page with the numbers. The relative increase is about 30%-40%
The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related.
The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me.
On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> wrote:
What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter?
On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change.
If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened.
On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It is very likely to add to the cpu to transcode everything to 12000hz
On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony, thank you for responding!
I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up.
Here are the full versions for both the master where it happened and v1.4. Is this enough?
FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit) FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
It does not occur on this master: FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit)
I figured I would try this email before bisecting git but it will likely be my next step.
On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase.
On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation.
You can't use vague references like "latest" which changes every 10 min. You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace. The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions. Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup.
On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
> Hi guys, > > I have my FreeSWITCH running a bunch of conferences with two or > three users using PCMU/8000 from all the user agents. I have the > conference sample rate set to 12000 to get around a strange bug I reported > a couple of months ago. > > In my old installation with master from May 22 2014, the servers > consumed about 1.2%-1.3% CPU per two-person conference. > > In the latest master AND in the latest v1.4 it went up to about > 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH > process also jumps up and down more. > > Additionally (not sure if it's related) the latest master crashed > less than 24 hours after it started taking live traffic. I ma trying to > get the core dump and see if v1.4 also crashes but including the info here > in case it gives someone an idea of what's going on with the CPU. > > Any ideas as to what's causing the increased CPU usage would be > greatly appreciated. > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
*Brian West* brian@freeswitch.org
*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com
*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Just one extra? https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way On 31 July 2014 15:04, Oleg Stolyar <olegstolyar@gmail.com> wrote:
That's a whole different discussion. Mostly because I need the ability to easily add another person to the call.
On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven < rtreleaven@bunnykick.ca> wrote:
Why do you have two person conferences instead of bridged calls?
Sent from my BlackBerry® PlayBook™ www.blackberry.com
------------------------------ *From:* "Oleg Stolyar" <olegstolyar@gmail.com> *To:* "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> *Sent:* 31 July, 2014 9:40 AM *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences
I am not sure we are on the same page with the numbers. The relative increase is about 30%-40%
The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related.
The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me.
On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> wrote:
What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter?
On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change.
If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened.
On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It is very likely to add to the cpu to transcode everything to 12000hz
On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony, thank you for responding!
I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up.
Here are the full versions for both the master where it happened and v1.4. Is this enough?
FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit) FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
It does not occur on this master: FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit)
I figured I would try this email before bisecting git but it will likely be my next step.
On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase.
On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
> I'm not really sure how to pinpoint .8% cpu increase. You would > maybe have to git bisect? I believe you already said this in another email > then retracted it. So with all the back and forth it starts to sound like > you have misinformation. > > You can't use vague references like "latest" which changes every 10 > min. > You need to try the most current master (TODAY then recite the hash) > with a clean build and have a valid backtrace. > The things you are saying are inconsistent so I think you need to > rule out build skew from jumping around so many versions. > Your 12000 thing also makes no sense at all and does not occur for > anyone else which suggests you have environmental problem related to the > way your system is setup. > > > > > On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com > > wrote: > >> Hi guys, >> >> I have my FreeSWITCH running a bunch of conferences with two or >> three users using PCMU/8000 from all the user agents. I have the >> conference sample rate set to 12000 to get around a strange bug I reported >> a couple of months ago. >> >> In my old installation with master from May 22 2014, the servers >> consumed about 1.2%-1.3% CPU per two-person conference. >> >> In the latest master AND in the latest v1.4 it went up to about >> 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH >> process also jumps up and down more. >> >> Additionally (not sure if it's related) the latest master crashed >> less than 24 hours after it started taking live traffic. I ma trying to >> get the core dump and see if v1.4 also crashes but including the info here >> in case it gives someone an idea of what's going on with the CPU. >> >> Any ideas as to what's causing the increased CPU usage would be >> greatly appreciated. >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> > > > -- > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ > > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ > http://twitter.com/FreeSWITCH > ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ > <http://freeswitch.org/g+>* > > ClueCon Weekly Development Call > ☎ sip:888@conference.freeswitch.org ☎ +19193869900 > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
*Brian West* brian@freeswitch.org
*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com
*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Well, it gets a little complicated. I need to enable multiple simultaneous eavesdroppers and the ability to include one extra person into the conversation. Let's call it Leg C. Then Leg B should be able to drop off leaving Leg A and Leg B talking. Additionally I need the ability for Leg B and Leg C to talk to each other putting Leg A on hold. I considered using the eavesdrop tool for this which would give me the ability for the eavesdropper to talk to Leg A, Leg B or both. However, I was not sure that having Leg A drop off would work and ultimately decided that the simplicity of implementing this with conferences is worth the CPU hit. Again, the only reason I am concerned with the current additional CPU jump is because it's unexplained and I suspect it has something to do with the crash I experienced. On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre@gmail.com> wrote:
Just one extra?
https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
On 31 July 2014 15:04, Oleg Stolyar <olegstolyar@gmail.com> wrote:
That's a whole different discussion. Mostly because I need the ability to easily add another person to the call.
On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven < rtreleaven@bunnykick.ca> wrote:
Why do you have two person conferences instead of bridged calls?
Sent from my BlackBerry® PlayBook™ www.blackberry.com
------------------------------ *From:* "Oleg Stolyar" <olegstolyar@gmail.com> *To:* "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> *Sent:* 31 July, 2014 9:40 AM *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences
I am not sure we are on the same page with the numbers. The relative increase is about 30%-40%
The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related.
The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me.
On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> wrote:
What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter?
On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change.
If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened.
On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It is very likely to add to the cpu to transcode everything to 12000hz
On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
> Hi Anthony, thank you for responding! > > I apologize for the previous email - I retracted it too soon - the > CPU simply jumped down for a bit then went back up. > > Here are the full versions for both the master where it happened and > v1.4. Is this enough? > > FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit > (git af64da8 2014-07-29 13:22:47Z 64bit) > FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git > 9d1c0f7 2014-07-24 23:05:06Z 64bit) > > It does not occur on this master: > FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit > (git 9d99c34 2014-05-22 17:56:48Z 64bit) > > I figured I would try this email before bisecting git but it will > likely be my next step. > > On the subject of the 12000 rate related bug - I'd love to discuss > this more with you - perhaps I'll try to get a little bit of your time at > ClueCon to just show you the problem. It actually happens on every > platform I tested it on except Debian 7. Just in the last couple of days I > reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in > this thread in case the 12000 sample rate conference on 8000 codecs is > relevant to the CPU increase. > > > > On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < > anthony.minessale@gmail.com> wrote: > >> I'm not really sure how to pinpoint .8% cpu increase. You would >> maybe have to git bisect? I believe you already said this in another email >> then retracted it. So with all the back and forth it starts to sound like >> you have misinformation. >> >> You can't use vague references like "latest" which changes every 10 >> min. >> You need to try the most current master (TODAY then recite the >> hash) with a clean build and have a valid backtrace. >> The things you are saying are inconsistent so I think you need to >> rule out build skew from jumping around so many versions. >> Your 12000 thing also makes no sense at all and does not occur for >> anyone else which suggests you have environmental problem related to the >> way your system is setup. >> >> >> >> >> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar < >> olegstolyar@gmail.com> wrote: >> >>> Hi guys, >>> >>> I have my FreeSWITCH running a bunch of conferences with two or >>> three users using PCMU/8000 from all the user agents. I have the >>> conference sample rate set to 12000 to get around a strange bug I reported >>> a couple of months ago. >>> >>> In my old installation with master from May 22 2014, the servers >>> consumed about 1.2%-1.3% CPU per two-person conference. >>> >>> In the latest master AND in the latest v1.4 it went up to about >>> 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH >>> process also jumps up and down more. >>> >>> Additionally (not sure if it's related) the latest master crashed >>> less than 24 hours after it started taking live traffic. I ma trying to >>> get the core dump and see if v1.4 also crashes but including the info here >>> in case it gives someone an idea of what's going on with the CPU. >>> >>> Any ideas as to what's causing the increased CPU usage would be >>> greatly appreciated. >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >>> >> >> >> -- >> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >> >> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >> http://twitter.com/FreeSWITCH >> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >> <http://freeswitch.org/g+>* >> >> ClueCon Weekly Development Call >> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
*Brian West* brian@freeswitch.org
*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com
*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
There is no reason at all to think it has anything to do with the crash. For the crash, simply get a backtrace of it so we can take a look at what is going on with the crash. On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Well, it gets a little complicated. I need to enable multiple simultaneous eavesdroppers and the ability to include one extra person into the conversation. Let's call it Leg C. Then Leg B should be able to drop off leaving Leg A and Leg B talking. Additionally I need the ability for Leg B and Leg C to talk to each other putting Leg A on hold.
I considered using the eavesdrop tool for this which would give me the ability for the eavesdropper to talk to Leg A, Leg B or both. However, I was not sure that having Leg A drop off would work and ultimately decided that the simplicity of implementing this with conferences is worth the CPU hit.
Again, the only reason I am concerned with the current additional CPU jump is because it's unexplained and I suspect it has something to do with the crash I experienced.
On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre@gmail.com> wrote: Just one extra?
https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
On 31 July 2014 15:04, Oleg Stolyar <olegstolyar@gmail.com> wrote: That's a whole different discussion. Mostly because I need the ability to easily add another person to the call.
On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven <rtreleaven@bunnykick.ca> wrote: Why do you have two person conferences instead of bridged calls?
Sent from my BlackBerry® PlayBook™ www.blackberry.com
From: "Oleg Stolyar" <olegstolyar@gmail.com> To: "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> Sent: 31 July, 2014 9:40 AM Subject: Re: [Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences
I am not sure we are on the same page with the numbers. The relative increase is about 30%-40%
The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related.
The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me.
On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> wrote: What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter?
On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote: Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change.
If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened.
On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale <anthony.minessale@gmail.com> wrote: It is very likely to add to the cpu to transcode everything to 12000hz
On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote: Hi Anthony, thank you for responding!
I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up.
Here are the full versions for both the master where it happened and v1.4. Is this enough?
FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit) FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
It does not occur on this master: FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit)
I figured I would try this email before bisecting git but it will likely be my next step.
On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase.
On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale <anthony.minessale@gmail.com> wrote: I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation.
You can't use vague references like "latest" which changes every 10 min. You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace. The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions. Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup.
On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote: Hi guys,
I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago.
In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference.
In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more.
Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU.
Any ideas as to what's causing the increased CPU usage would be greatly appreciated.
___
Yes, I enabled core dump and will get the backtrace if it crashes again. I need to make preparations for the rest of my system to handle FreeSWITCH crashes better before I put the new code back in production, so it may take a while. If I find the time I will also try to bisect git to find the commit that caused this as Anthony suggested. On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <mike@jerris.com> wrote:
There is no reason at all to think it has anything to do with the crash. For the crash, simply get a backtrace of it so we can take a look at what is going on with the crash.
On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Well, it gets a little complicated. I need to enable multiple simultaneous eavesdroppers and the ability to include one extra person into the conversation. Let's call it Leg C. Then Leg B should be able to drop off leaving Leg A and Leg B talking. Additionally I need the ability for Leg B and Leg C to talk to each other putting Leg A on hold.
I considered using the eavesdrop tool for this which would give me the ability for the eavesdropper to talk to Leg A, Leg B or both. However, I was not sure that having Leg A drop off would work and ultimately decided that the simplicity of implementing this with conferences is worth the CPU hit.
Again, the only reason I am concerned with the current additional CPU jump is because it's unexplained and I suspect it has something to do with the crash I experienced.
On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre@gmail.com> wrote:
Just one extra?
https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
On 31 July 2014 15:04, Oleg Stolyar <olegstolyar@gmail.com> wrote:
That's a whole different discussion. Mostly because I need the ability to easily add another person to the call.
On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven < rtreleaven@bunnykick.ca> wrote:
Why do you have two person conferences instead of bridged calls?
Sent from my BlackBerry® PlayBook™ www.blackberry.com
------------------------------ *From:* "Oleg Stolyar" <olegstolyar@gmail.com> *To:* "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> *Sent:* 31 July, 2014 9:40 AM *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences
I am not sure we are on the same page with the numbers. The relative increase is about 30%-40%
The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related.
The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me.
On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> wrote:
What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter?
On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change.
If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened.
On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
> It is very likely to add to the cpu to transcode everything to > 12000hz > > > > On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <olegstolyar@gmail.com > > wrote: > >> Hi Anthony, thank you for responding! >> >> I apologize for the previous email - I retracted it too soon - the >> CPU simply jumped down for a bit then went back up. >> >> Here are the full versions for both the master where it happened >> and v1.4. Is this enough? >> >> FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit >> (git af64da8 2014-07-29 13:22:47Z 64bit) >> FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git >> 9d1c0f7 2014-07-24 23:05:06Z 64bit) >> >> It does not occur on this master: >> FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit >> (git 9d99c34 2014-05-22 17:56:48Z 64bit) >> >> I figured I would try this email before bisecting git but it will >> likely be my next step. >> >> On the subject of the 12000 rate related bug - I'd love to discuss >> this more with you - perhaps I'll try to get a little bit of your time at >> ClueCon to just show you the problem. It actually happens on every >> platform I tested it on except Debian 7. Just in the last couple of days I >> reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in >> this thread in case the 12000 sample rate conference on 8000 codecs is >> relevant to the CPU increase. >> >> >> >> On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < >> anthony.minessale@gmail.com> wrote: >> >>> I'm not really sure how to pinpoint .8% cpu increase. You would >>> maybe have to git bisect? I believe you already said this in another email >>> then retracted it. So with all the back and forth it starts to sound like >>> you have misinformation. >>> >>> You can't use vague references like "latest" which changes every >>> 10 min. >>> You need to try the most current master (TODAY then recite the >>> hash) with a clean build and have a valid backtrace. >>> The things you are saying are inconsistent so I think you need to >>> rule out build skew from jumping around so many versions. >>> Your 12000 thing also makes no sense at all and does not occur for >>> anyone else which suggests you have environmental problem related to the >>> way your system is setup. >>> >>> >>> >>> >>> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar < >>> olegstolyar@gmail.com> wrote: >>> >>>> Hi guys, >>>> >>>> I have my FreeSWITCH running a bunch of conferences with two or >>>> three users using PCMU/8000 from all the user agents. I have the >>>> conference sample rate set to 12000 to get around a strange bug I reported >>>> a couple of months ago. >>>> >>>> In my old installation with master from May 22 2014, the servers >>>> consumed about 1.2%-1.3% CPU per two-person conference. >>>> >>>> In the latest master AND in the latest v1.4 it went up to about >>>> 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH >>>> process also jumps up and down more. >>>> >>>> Additionally (not sure if it's related) the latest master crashed >>>> less than 24 hours after it started taking live traffic. I ma trying to >>>> get the core dump and see if v1.4 also crashes but including the info here >>>> in case it gives someone an idea of what's going on with the CPU. >>>> >>>> Any ideas as to what's causing the increased CPU usage would be >>>> greatly appreciated. >>>> >>>> ___ >>> >>>
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I actually think you need to start over. Nothing you say is consistent and I have a rule when doing debugging that when things don't add up, start over. Run latest GIT with clean build (remove all old binaries from bin lib and mod in FS install target) Start from there getting BT, evaluating CPU etc. Jumping around versions is the leading way to cause problems that cannot be explained. On Thu, Jul 31, 2014 at 12:06 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Yes, I enabled core dump and will get the backtrace if it crashes again. I need to make preparations for the rest of my system to handle FreeSWITCH crashes better before I put the new code back in production, so it may take a while.
If I find the time I will also try to bisect git to find the commit that caused this as Anthony suggested.
On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <mike@jerris.com> wrote:
There is no reason at all to think it has anything to do with the crash. For the crash, simply get a backtrace of it so we can take a look at what is going on with the crash.
On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Well, it gets a little complicated. I need to enable multiple simultaneous eavesdroppers and the ability to include one extra person into the conversation. Let's call it Leg C. Then Leg B should be able to drop off leaving Leg A and Leg B talking. Additionally I need the ability for Leg B and Leg C to talk to each other putting Leg A on hold.
I considered using the eavesdrop tool for this which would give me the ability for the eavesdropper to talk to Leg A, Leg B or both. However, I was not sure that having Leg A drop off would work and ultimately decided that the simplicity of implementing this with conferences is worth the CPU hit.
Again, the only reason I am concerned with the current additional CPU jump is because it's unexplained and I suspect it has something to do with the crash I experienced.
On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre@gmail.com> wrote:
Just one extra?
https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
On 31 July 2014 15:04, Oleg Stolyar <olegstolyar@gmail.com> wrote:
That's a whole different discussion. Mostly because I need the ability to easily add another person to the call.
On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven < rtreleaven@bunnykick.ca> wrote:
Why do you have two person conferences instead of bridged calls?
Sent from my BlackBerry® PlayBook™ www.blackberry.com
------------------------------ *From:* "Oleg Stolyar" <olegstolyar@gmail.com> *To:* "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> *Sent:* 31 July, 2014 9:40 AM *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences
I am not sure we are on the same page with the numbers. The relative increase is about 30%-40%
The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related.
The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me.
On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> wrote:
What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter?
On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
> Of course, and it did when I first made the rate change. It went > from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am > running into now is that there is another jump in CPU with no additional > config change. > > If nothing comes to mind, I will patch the May master with the fix > for WebRTC password size to fix my immediate issue and then try to find the > time to bisect git and pinpoint what happened. > > > On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < > anthony.minessale@gmail.com> wrote: > >> It is very likely to add to the cpu to transcode everything to >> 12000hz >> >> >> >> On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar < >> olegstolyar@gmail.com> wrote: >> >>> Hi Anthony, thank you for responding! >>> >>> I apologize for the previous email - I retracted it too soon - the >>> CPU simply jumped down for a bit then went back up. >>> >>> Here are the full versions for both the master where it happened >>> and v1.4. Is this enough? >>> >>> FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit >>> (git af64da8 2014-07-29 13:22:47Z 64bit) >>> FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit >>> (git 9d1c0f7 2014-07-24 23:05:06Z 64bit) >>> >>> It does not occur on this master: >>> FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit >>> (git 9d99c34 2014-05-22 17:56:48Z 64bit) >>> >>> I figured I would try this email before bisecting git but it will >>> likely be my next step. >>> >>> On the subject of the 12000 rate related bug - I'd love to discuss >>> this more with you - perhaps I'll try to get a little bit of your time at >>> ClueCon to just show you the problem. It actually happens on every >>> platform I tested it on except Debian 7. Just in the last couple of days I >>> reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in >>> this thread in case the 12000 sample rate conference on 8000 codecs is >>> relevant to the CPU increase. >>> >>> >>> >>> On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < >>> anthony.minessale@gmail.com> wrote: >>> >>>> I'm not really sure how to pinpoint .8% cpu increase. You would >>>> maybe have to git bisect? I believe you already said this in another email >>>> then retracted it. So with all the back and forth it starts to sound like >>>> you have misinformation. >>>> >>>> You can't use vague references like "latest" which changes every >>>> 10 min. >>>> You need to try the most current master (TODAY then recite the >>>> hash) with a clean build and have a valid backtrace. >>>> The things you are saying are inconsistent so I think you need to >>>> rule out build skew from jumping around so many versions. >>>> Your 12000 thing also makes no sense at all and does not occur >>>> for anyone else which suggests you have environmental problem related to >>>> the way your system is setup. >>>> >>>> >>>> >>>> >>>> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar < >>>> olegstolyar@gmail.com> wrote: >>>> >>>>> Hi guys, >>>>> >>>>> I have my FreeSWITCH running a bunch of conferences with two or >>>>> three users using PCMU/8000 from all the user agents. I have the >>>>> conference sample rate set to 12000 to get around a strange bug I reported >>>>> a couple of months ago. >>>>> >>>>> In my old installation with master from May 22 2014, the servers >>>>> consumed about 1.2%-1.3% CPU per two-person conference. >>>>> >>>>> In the latest master AND in the latest v1.4 it went up to about >>>>> 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH >>>>> process also jumps up and down more. >>>>> >>>>> Additionally (not sure if it's related) the latest master >>>>> crashed less than 24 hours after it started taking live traffic. I ma >>>>> trying to get the core dump and see if v1.4 also crashes but including the >>>>> info here in case it gives someone an idea of what's going on with the CPU. >>>>> >>>>> Any ideas as to what's causing the increased CPU usage would be >>>>> greatly appreciated. >>>>> >>>>> ___ >>>> >>>>
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
Running a clean build is a great idea! I will do that and report my findings - it may be after ClueCon though. On Thu, Jul 31, 2014 at 11:22 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I actually think you need to start over. Nothing you say is consistent and I have a rule when doing debugging that when things don't add up, start over.
Run latest GIT with clean build (remove all old binaries from bin lib and mod in FS install target)
Start from there getting BT, evaluating CPU etc. Jumping around versions is the leading way to cause problems that cannot be explained.
On Thu, Jul 31, 2014 at 12:06 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Yes, I enabled core dump and will get the backtrace if it crashes again. I need to make preparations for the rest of my system to handle FreeSWITCH crashes better before I put the new code back in production, so it may take a while.
If I find the time I will also try to bisect git to find the commit that caused this as Anthony suggested.
On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <mike@jerris.com> wrote:
There is no reason at all to think it has anything to do with the crash. For the crash, simply get a backtrace of it so we can take a look at what is going on with the crash.
On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Well, it gets a little complicated. I need to enable multiple simultaneous eavesdroppers and the ability to include one extra person into the conversation. Let's call it Leg C. Then Leg B should be able to drop off leaving Leg A and Leg B talking. Additionally I need the ability for Leg B and Leg C to talk to each other putting Leg A on hold.
I considered using the eavesdrop tool for this which would give me the ability for the eavesdropper to talk to Leg A, Leg B or both. However, I was not sure that having Leg A drop off would work and ultimately decided that the simplicity of implementing this with conferences is worth the CPU hit.
Again, the only reason I am concerned with the current additional CPU jump is because it's unexplained and I suspect it has something to do with the crash I experienced.
On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre@gmail.com> wrote:
Just one extra?
https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
On 31 July 2014 15:04, Oleg Stolyar <olegstolyar@gmail.com> wrote:
That's a whole different discussion. Mostly because I need the ability to easily add another person to the call.
On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven < rtreleaven@bunnykick.ca> wrote:
Why do you have two person conferences instead of bridged calls?
Sent from my BlackBerry® PlayBook™ www.blackberry.com
------------------------------ *From:* "Oleg Stolyar" <olegstolyar@gmail.com> *To:* "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> *Sent:* 31 July, 2014 9:40 AM *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences
I am not sure we are on the same page with the numbers. The relative increase is about 30%-40%
The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related.
The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me.
On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> wrote:
> What kind of system are you running on where 0.2 to 0.3% CPU > increase would really matter? > > > On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar@gmail.com > > wrote: > >> Of course, and it did when I first made the rate change. It went >> from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am >> running into now is that there is another jump in CPU with no additional >> config change. >> >> If nothing comes to mind, I will patch the May master with the fix >> for WebRTC password size to fix my immediate issue and then try to find the >> time to bisect git and pinpoint what happened. >> >> >> On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < >> anthony.minessale@gmail.com> wrote: >> >>> It is very likely to add to the cpu to transcode everything to >>> 12000hz >>> >>> >>> >>> On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar < >>> olegstolyar@gmail.com> wrote: >>> >>>> Hi Anthony, thank you for responding! >>>> >>>> I apologize for the previous email - I retracted it too soon - >>>> the CPU simply jumped down for a bit then went back up. >>>> >>>> Here are the full versions for both the master where it happened >>>> and v1.4. Is this enough? >>>> >>>> FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit >>>> (git af64da8 2014-07-29 13:22:47Z 64bit) >>>> FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit >>>> (git 9d1c0f7 2014-07-24 23:05:06Z 64bit) >>>> >>>> It does not occur on this master: >>>> FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit >>>> (git 9d99c34 2014-05-22 17:56:48Z 64bit) >>>> >>>> I figured I would try this email before bisecting git but it will >>>> likely be my next step. >>>> >>>> On the subject of the 12000 rate related bug - I'd love to >>>> discuss this more with you - perhaps I'll try to get a little bit of your >>>> time at ClueCon to just show you the problem. It actually happens on every >>>> platform I tested it on except Debian 7. Just in the last couple of days I >>>> reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in >>>> this thread in case the 12000 sample rate conference on 8000 codecs is >>>> relevant to the CPU increase. >>>> >>>> >>>> >>>> On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < >>>> anthony.minessale@gmail.com> wrote: >>>> >>>>> I'm not really sure how to pinpoint .8% cpu increase. You would >>>>> maybe have to git bisect? I believe you already said this in another email >>>>> then retracted it. So with all the back and forth it starts to sound like >>>>> you have misinformation. >>>>> >>>>> You can't use vague references like "latest" which changes every >>>>> 10 min. >>>>> You need to try the most current master (TODAY then recite the >>>>> hash) with a clean build and have a valid backtrace. >>>>> The things you are saying are inconsistent so I think you need >>>>> to rule out build skew from jumping around so many versions. >>>>> Your 12000 thing also makes no sense at all and does not occur >>>>> for anyone else which suggests you have environmental problem related to >>>>> the way your system is setup. >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar < >>>>> olegstolyar@gmail.com> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> I have my FreeSWITCH running a bunch of conferences with two or >>>>>> three users using PCMU/8000 from all the user agents. I have the >>>>>> conference sample rate set to 12000 to get around a strange bug I reported >>>>>> a couple of months ago. >>>>>> >>>>>> In my old installation with master from May 22 2014, the >>>>>> servers consumed about 1.2%-1.3% CPU per two-person conference. >>>>>> >>>>>> In the latest master AND in the latest v1.4 it went up to about >>>>>> 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH >>>>>> process also jumps up and down more. >>>>>> >>>>>> Additionally (not sure if it's related) the latest master >>>>>> crashed less than 24 hours after it started taking live traffic. I ma >>>>>> trying to get the core dump and see if v1.4 also crashes but including the >>>>>> info here in case it gives someone an idea of what's going on with the CPU. >>>>>> >>>>>> Any ideas as to what's causing the increased CPU usage would be >>>>>> greatly appreciated. >>>>>> >>>>>> ___ >>>>> >>>>>
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Talk about "inconsistent" (and embarrassing for me for accusing the new version) Now the older version that was rock solid for months crashed on me also. This time I have the backtrace. It's here: https://www.dropbox.com/s/y0u48n61rul0x30/20140801T200335Z-bt.txt. Please let me know if you'd like me to file a Jira and attach it to it or if you need this reproduced on the latest master before you can look into it. Really appreciate all the help and advice! On Thu, Jul 31, 2014 at 7:27 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Running a clean build is a great idea! I will do that and report my findings - it may be after ClueCon though.
On Thu, Jul 31, 2014 at 11:22 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I actually think you need to start over. Nothing you say is consistent and I have a rule when doing debugging that when things don't add up, start over.
Run latest GIT with clean build (remove all old binaries from bin lib and mod in FS install target)
Start from there getting BT, evaluating CPU etc. Jumping around versions is the leading way to cause problems that cannot be explained.
On Thu, Jul 31, 2014 at 12:06 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Yes, I enabled core dump and will get the backtrace if it crashes again. I need to make preparations for the rest of my system to handle FreeSWITCH crashes better before I put the new code back in production, so it may take a while.
If I find the time I will also try to bisect git to find the commit that caused this as Anthony suggested.
On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <mike@jerris.com> wrote:
There is no reason at all to think it has anything to do with the crash. For the crash, simply get a backtrace of it so we can take a look at what is going on with the crash.
On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Well, it gets a little complicated. I need to enable multiple simultaneous eavesdroppers and the ability to include one extra person into the conversation. Let's call it Leg C. Then Leg B should be able to drop off leaving Leg A and Leg B talking. Additionally I need the ability for Leg B and Leg C to talk to each other putting Leg A on hold.
I considered using the eavesdrop tool for this which would give me the ability for the eavesdropper to talk to Leg A, Leg B or both. However, I was not sure that having Leg A drop off would work and ultimately decided that the simplicity of implementing this with conferences is worth the CPU hit.
Again, the only reason I am concerned with the current additional CPU jump is because it's unexplained and I suspect it has something to do with the crash I experienced.
On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre@gmail.com> wrote:
Just one extra?
https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
On 31 July 2014 15:04, Oleg Stolyar <olegstolyar@gmail.com> wrote:
That's a whole different discussion. Mostly because I need the ability to easily add another person to the call.
On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven < rtreleaven@bunnykick.ca> wrote:
> Why do you have two person conferences instead of bridged calls? > > Sent from my BlackBerry® PlayBook™ > www.blackberry.com > > ------------------------------ > *From:* "Oleg Stolyar" <olegstolyar@gmail.com> > *To:* "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org > > > *Sent:* 31 July, 2014 9:40 AM > *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4 > consume more CPU for conferences > > I am not sure we are on the same page with the numbers. The > relative increase is about 30%-40% > > The 0.3% is the absolute increase per conference. CPU usage per > single two-member conference went from 1.2%-1.3% to 1.6%-2.0% > > I am using AWS m3.xlarge instances with CentOS. It's a 4-core > system, so the increase in CPU of usage of 30% would reduce the number of > calls each instance can take. The increase itself would not necessarily be > a huge deal except if it's unexplained, it may be an indicator that > something is wrong. As I said in my original email, the system running the > recent code also crashed less than 24 hours after starting to take live > calls. I wanted to call out the increase as it seems likely to me > (although by no means certain) that the two are related. > > The previous FreeSWITCH version I used have always been rock solid - > this was the first time FreeSWITCH crashed on me. > > > On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> > wrote: > >> What kind of system are you running on where 0.2 to 0.3% CPU >> increase would really matter? >> >> >> On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar < >> olegstolyar@gmail.com> wrote: >> >>> Of course, and it did when I first made the rate change. It went >>> from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am >>> running into now is that there is another jump in CPU with no additional >>> config change. >>> >>> If nothing comes to mind, I will patch the May master with the fix >>> for WebRTC password size to fix my immediate issue and then try to find the >>> time to bisect git and pinpoint what happened. >>> >>> >>> On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < >>> anthony.minessale@gmail.com> wrote: >>> >>>> It is very likely to add to the cpu to transcode everything to >>>> 12000hz >>>> >>>> >>>> >>>> On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar < >>>> olegstolyar@gmail.com> wrote: >>>> >>>>> Hi Anthony, thank you for responding! >>>>> >>>>> I apologize for the previous email - I retracted it too soon - >>>>> the CPU simply jumped down for a bit then went back up. >>>>> >>>>> Here are the full versions for both the master where it happened >>>>> and v1.4. Is this enough? >>>>> >>>>> FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit >>>>> (git af64da8 2014-07-29 13:22:47Z 64bit) >>>>> FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit >>>>> (git 9d1c0f7 2014-07-24 23:05:06Z 64bit) >>>>> >>>>> It does not occur on this master: >>>>> FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit >>>>> (git 9d99c34 2014-05-22 17:56:48Z 64bit) >>>>> >>>>> I figured I would try this email before bisecting git but it >>>>> will likely be my next step. >>>>> >>>>> On the subject of the 12000 rate related bug - I'd love to >>>>> discuss this more with you - perhaps I'll try to get a little bit of your >>>>> time at ClueCon to just show you the problem. It actually happens on every >>>>> platform I tested it on except Debian 7. Just in the last couple of days I >>>>> reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in >>>>> this thread in case the 12000 sample rate conference on 8000 codecs is >>>>> relevant to the CPU increase. >>>>> >>>>> >>>>> >>>>> On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < >>>>> anthony.minessale@gmail.com> wrote: >>>>> >>>>>> I'm not really sure how to pinpoint .8% cpu increase. You >>>>>> would maybe have to git bisect? I believe you already said this in another >>>>>> email then retracted it. So with all the back and forth it starts to sound >>>>>> like you have misinformation. >>>>>> >>>>>> You can't use vague references like "latest" which changes >>>>>> every 10 min. >>>>>> You need to try the most current master (TODAY then recite the >>>>>> hash) with a clean build and have a valid backtrace. >>>>>> The things you are saying are inconsistent so I think you need >>>>>> to rule out build skew from jumping around so many versions. >>>>>> Your 12000 thing also makes no sense at all and does not occur >>>>>> for anyone else which suggests you have environmental problem related to >>>>>> the way your system is setup. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar < >>>>>> olegstolyar@gmail.com> wrote: >>>>>> >>>>>>> Hi guys, >>>>>>> >>>>>>> I have my FreeSWITCH running a bunch of conferences with two >>>>>>> or three users using PCMU/8000 from all the user agents. I have the >>>>>>> conference sample rate set to 12000 to get around a strange bug I reported >>>>>>> a couple of months ago. >>>>>>> >>>>>>> In my old installation with master from May 22 2014, the >>>>>>> servers consumed about 1.2%-1.3% CPU per two-person conference. >>>>>>> >>>>>>> In the latest master AND in the latest v1.4 it went up to >>>>>>> about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH >>>>>>> process also jumps up and down more. >>>>>>> >>>>>>> Additionally (not sure if it's related) the latest master >>>>>>> crashed less than 24 hours after it started taking live traffic. I ma >>>>>>> trying to get the core dump and see if v1.4 also crashes but including the >>>>>>> info here in case it gives someone an idea of what's going on with the CPU. >>>>>>> >>>>>>> Any ideas as to what's causing the increased CPU usage would >>>>>>> be greatly appreciated. >>>>>>> >>>>>>> ___ >>>>>> >>>>>>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Its almost certain that you should do a fresh clean build. Once you start moving around in versions you can cause lots of problems with inconsistent binaries etc. On Fri, Aug 1, 2014 at 3:18 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Talk about "inconsistent" (and embarrassing for me for accusing the new version)
Now the older version that was rock solid for months crashed on me also. This time I have the backtrace. It's here: https://www.dropbox.com/s/y0u48n61rul0x30/20140801T200335Z-bt.txt.
Please let me know if you'd like me to file a Jira and attach it to it or if you need this reproduced on the latest master before you can look into it.
Really appreciate all the help and advice!
On Thu, Jul 31, 2014 at 7:27 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Running a clean build is a great idea! I will do that and report my findings - it may be after ClueCon though.
On Thu, Jul 31, 2014 at 11:22 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
I actually think you need to start over. Nothing you say is consistent and I have a rule when doing debugging that when things don't add up, start over.
Run latest GIT with clean build (remove all old binaries from bin lib and mod in FS install target)
Start from there getting BT, evaluating CPU etc. Jumping around versions is the leading way to cause problems that cannot be explained.
On Thu, Jul 31, 2014 at 12:06 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Yes, I enabled core dump and will get the backtrace if it crashes again. I need to make preparations for the rest of my system to handle FreeSWITCH crashes better before I put the new code back in production, so it may take a while.
If I find the time I will also try to bisect git to find the commit that caused this as Anthony suggested.
On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <mike@jerris.com> wrote:
There is no reason at all to think it has anything to do with the crash. For the crash, simply get a backtrace of it so we can take a look at what is going on with the crash.
On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Well, it gets a little complicated. I need to enable multiple simultaneous eavesdroppers and the ability to include one extra person into the conversation. Let's call it Leg C. Then Leg B should be able to drop off leaving Leg A and Leg B talking. Additionally I need the ability for Leg B and Leg C to talk to each other putting Leg A on hold.
I considered using the eavesdrop tool for this which would give me the ability for the eavesdropper to talk to Leg A, Leg B or both. However, I was not sure that having Leg A drop off would work and ultimately decided that the simplicity of implementing this with conferences is worth the CPU hit.
Again, the only reason I am concerned with the current additional CPU jump is because it's unexplained and I suspect it has something to do with the crash I experienced.
On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre@gmail.com> wrote:
Just one extra?
https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
On 31 July 2014 15:04, Oleg Stolyar <olegstolyar@gmail.com> wrote:
> That's a whole different discussion. Mostly because I need the > ability to easily add another person to the call. > > > On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven < > rtreleaven@bunnykick.ca> wrote: > >> Why do you have two person conferences instead of bridged calls? >> >> Sent from my BlackBerry® PlayBook™ >> www.blackberry.com >> >> ------------------------------ >> *From:* "Oleg Stolyar" <olegstolyar@gmail.com> >> *To:* "FreeSWITCH Users Help" < >> freeswitch-users@lists.freeswitch.org> >> *Sent:* 31 July, 2014 9:40 AM >> *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4 >> consume more CPU for conferences >> >> I am not sure we are on the same page with the numbers. The >> relative increase is about 30%-40% >> >> The 0.3% is the absolute increase per conference. CPU usage per >> single two-member conference went from 1.2%-1.3% to 1.6%-2.0% >> >> I am using AWS m3.xlarge instances with CentOS. It's a 4-core >> system, so the increase in CPU of usage of 30% would reduce the number of >> calls each instance can take. The increase itself would not necessarily be >> a huge deal except if it's unexplained, it may be an indicator that >> something is wrong. As I said in my original email, the system running the >> recent code also crashed less than 24 hours after starting to take live >> calls. I wanted to call out the increase as it seems likely to me >> (although by no means certain) that the two are related. >> >> The previous FreeSWITCH version I used have always been rock solid >> - this was the first time FreeSWITCH crashed on me. >> >> >> On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian@freeswitch.org> >> wrote: >> >>> What kind of system are you running on where 0.2 to 0.3% CPU >>> increase would really matter? >>> >>> >>> On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar < >>> olegstolyar@gmail.com> wrote: >>> >>>> Of course, and it did when I first made the rate change. It went >>>> from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am >>>> running into now is that there is another jump in CPU with no additional >>>> config change. >>>> >>>> If nothing comes to mind, I will patch the May master with the >>>> fix for WebRTC password size to fix my immediate issue and then try to find >>>> the time to bisect git and pinpoint what happened. >>>> >>>> >>>> On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale < >>>> anthony.minessale@gmail.com> wrote: >>>> >>>>> It is very likely to add to the cpu to transcode everything to >>>>> 12000hz >>>>> >>>>> >>>>> >>>>> On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar < >>>>> olegstolyar@gmail.com> wrote: >>>>> >>>>>> Hi Anthony, thank you for responding! >>>>>> >>>>>> I apologize for the previous email - I retracted it too soon - >>>>>> the CPU simply jumped down for a bit then went back up. >>>>>> >>>>>> Here are the full versions for both the master where it >>>>>> happened and v1.4. Is this enough? >>>>>> >>>>>> FreeSWITCH Version >>>>>> 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 >>>>>> 13:22:47Z 64bit) >>>>>> FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit >>>>>> (git 9d1c0f7 2014-07-24 23:05:06Z 64bit) >>>>>> >>>>>> It does not occur on this master: >>>>>> FreeSWITCH Version >>>>>> 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 >>>>>> 17:56:48Z 64bit) >>>>>> >>>>>> I figured I would try this email before bisecting git but it >>>>>> will likely be my next step. >>>>>> >>>>>> On the subject of the 12000 rate related bug - I'd love to >>>>>> discuss this more with you - perhaps I'll try to get a little bit of your >>>>>> time at ClueCon to just show you the problem. It actually happens on every >>>>>> platform I tested it on except Debian 7. Just in the last couple of days I >>>>>> reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in >>>>>> this thread in case the 12000 sample rate conference on 8000 codecs is >>>>>> relevant to the CPU increase. >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale < >>>>>> anthony.minessale@gmail.com> wrote: >>>>>> >>>>>>> I'm not really sure how to pinpoint .8% cpu increase. You >>>>>>> would maybe have to git bisect? I believe you already said this in another >>>>>>> email then retracted it. So with all the back and forth it starts to sound >>>>>>> like you have misinformation. >>>>>>> >>>>>>> You can't use vague references like "latest" which changes >>>>>>> every 10 min. >>>>>>> You need to try the most current master (TODAY then recite the >>>>>>> hash) with a clean build and have a valid backtrace. >>>>>>> The things you are saying are inconsistent so I think you need >>>>>>> to rule out build skew from jumping around so many versions. >>>>>>> Your 12000 thing also makes no sense at all and does not occur >>>>>>> for anyone else which suggests you have environmental problem related to >>>>>>> the way your system is setup. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar < >>>>>>> olegstolyar@gmail.com> wrote: >>>>>>> >>>>>>>> Hi guys, >>>>>>>> >>>>>>>> I have my FreeSWITCH running a bunch of conferences with two >>>>>>>> or three users using PCMU/8000 from all the user agents. I have the >>>>>>>> conference sample rate set to 12000 to get around a strange bug I reported >>>>>>>> a couple of months ago. >>>>>>>> >>>>>>>> In my old installation with master from May 22 2014, the >>>>>>>> servers consumed about 1.2%-1.3% CPU per two-person conference. >>>>>>>> >>>>>>>> In the latest master AND in the latest v1.4 it went up to >>>>>>>> about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH >>>>>>>> process also jumps up and down more. >>>>>>>> >>>>>>>> Additionally (not sure if it's related) the latest master >>>>>>>> crashed less than 24 hours after it started taking live traffic. I ma >>>>>>>> trying to get the core dump and see if v1.4 also crashes but including the >>>>>>>> info here in case it gives someone an idea of what's going on with the CPU. >>>>>>>> >>>>>>>> Any ideas as to what's causing the increased CPU usage would >>>>>>>> be greatly appreciated. >>>>>>>> >>>>>>>> ___ >>>>>>> >>>>>>>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
participants (6)
-
Anthony Minessale
-
Brian West
-
Michael Jerris
-
Oleg Stolyar
-
Russell Treleaven
-
Steven Ayre