Yessir. I am using the exact same ulimits below. I also increased the sessions per second to 50 and max-sessions to 5000. Also as we are moving into a high traffic period based on time of day. I've had it die repeatedly on me, the following line was the last line printed ... freeswitch: nua_stack.c:2396: nua_client_restart_request: Assertion `nua_client_is_queued(cr)' failed. this is the other last line I get if the system itself seems to be overloaded freeswitch: sofia.c:2138: sofia_handle_sip_i_state: Assertion `tech_pvt->nh != ((void *)0)' failed. I don't know if that is of any help or not. At the time I was seeing approx 80-90 calls per sec hitting the server, based on the 'status' command in the FS cli. I don't have this built with debug symbols either at the moment. I wanted to get a baseline of server with x can do y , based on how we use it. I will likely get it rebuilt and find a way to reduce some of the traffic load so I can effectively tshoot. -e It's seems fuzzy now but I think on Mon, Nov 03, 2008 at 05:19:42PM -0600 , Anthony Minessale said:

Did you increase the artificial limits in switch.conf.xml ?

try these uimits

ulimit -c unlimited ulimit -d unlimited ulimit -f unlimited ulimit -i unlimited ulimit -n 999999 ulimit -q unlimited

ulimit -u unlimited ulimit -v unlimited ulimit -x unlimited ulimit -s 244 ulimit -l unlimited

On Mon, Nov 3, 2008 at 4:46 PM, Eric Liedtke <e@musinghalfwit.org> wrote:

Greetings,

I'm working on a server in an SBC setup. It has a copper 10/100 for admin and 2 GiGe connections for private/public peering(SIP signaling and media). I've been testing with some traffic and am finding I hit a ceiling around 800-900 active calls. The system CPU ends up being about 40% which seemed really high to me. Is this simply due to the amount of networking happening with the rtp in and out the different interfaces or am I operating something in a non optimized fasion ?

It's an e1000 based card, I've tried messing around with the interrupt throttling but in the end the dynamic setting seemed to work best. Given that a chassis has done 3k simultaneous calls with about half the processing power I currently have in this server I must be running sub optimal.

Any idea's of where I might look to correct this ? kernel tuning ? network tuning ? Other than seeing a lot of irq's from the network cards I"m not sure what else might be using up the system cpu.

To squeeze out as much as possible I have the core databases housed in a tmpfs partiion, and I've also disabled the sql scoreboard (-nosql). Any other areas I should be exploring to increase the number of calls this chassis can process ?

Thanks in advance for any ideas

-Eric

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

I'm running the 1.0.1-phoenix release. No debug because our admin built the debian packages to install with so dh_strip got em. It's running on an ubuntu 8.04 server. I certainly can. I will let you know how it goes. -e It's seems fuzzy now but I think on Mon, Nov 03, 2008 at 08:57:01PM -0600 , Anthony Minessale said:

What revision are you on because the line numbers of those errors don't match up even slightly close? Did you do something to disable building with debug symbols because the current default is to build with them? What platform are you on?

Could you try a clean build of the latest revision?

On Mon, Nov 3, 2008 at 5:43 PM, Eric Liedtke <e@musinghalfwit.org> wrote:

Yessir. I am using the exact same ulimits below. I also increased the sessions per second to 50 and max-sessions to 5000. Also as we are moving into a high traffic period based on time of day. I've had it die repeatedly on me, the following line was the last line printed ...

freeswitch: nua_stack.c:2396: nua_client_restart_request: Assertion `nua_client_is_queued(cr)' failed.

this is the other last line I get if the system itself seems to be overloaded

freeswitch: sofia.c:2138: sofia_handle_sip_i_state: Assertion `tech_pvt->nh != ((void *)0)' failed.

I don't know if that is of any help or not. At the time I was seeing approx 80-90 calls per sec hitting the server, based on the 'status' command in the FS cli.

I don't have this built with debug symbols either at the moment. I wanted to get a baseline of server with x can do y , based on how we use it. I will likely get it rebuilt and find a way to reduce some of the traffic load so I can effectively tshoot.

-e

It's seems fuzzy now but I think on Mon, Nov 03, 2008 at 05:19:42PM -0600 , Anthony Minessale said: > Did you increase the artificial limits in switch.conf.xml ? > > try these uimits > > ulimit -c unlimited > ulimit -d unlimited > ulimit -f unlimited > ulimit -i unlimited > ulimit -n 999999 > ulimit -q unlimited > > ulimit -u unlimited > ulimit -v unlimited > ulimit -x unlimited > ulimit -s 244 > ulimit -l unlimited > > > > > On Mon, Nov 3, 2008 at 4:46 PM, Eric Liedtke <e@musinghalfwit.org> wrote: > > Greetings, > > I'm working on a server in an SBC setup. It has a copper 10/100 for > admin and 2 GiGe connections for private/public peering(SIP signaling > and media). I've been testing with some traffic and am finding I hit a > ceiling around 800-900 active calls. The system CPU ends up being about > 40% which seemed really high to me. Is this simply due to the amount of > networking happening with the rtp in and out the different interfaces or am > I > operating something in a non optimized fasion ? > > It's an e1000 based card, I've tried messing around with the interrupt > throttling but in the end the dynamic setting seemed to work best. Given > that a chassis has done 3k simultaneous calls with about half the > processing power I currently have in this server I must be running sub > optimal. > > Any idea's of where I might look to correct this ? kernel tuning ? > network tuning ? Other than seeing a lot of irq's from the network cards > I"m not sure what else might be using up the system cpu. > > To squeeze out as much as possible I have the core databases housed in a > tmpfs partiion, and I've also disabled the sql scoreboard (-nosql). Any > other areas I should be exploring to increase the number of calls this > chassis can process ? > > Thanks in advance for any ideas > > -Eric > > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ freeswitch-users > http://www.freeswitch.org > > > > > -- > Anthony Minessale II > > FreeSWITCH http://www.freeswitch.org/ > ClueCon http://www.cluecon.com/ > > AIM: anthm > MSN:anthony_minessale@hotmail.com > GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com > IRC: irc.freenode.net #freeswitch > > FreeSWITCH Developer Conference > sip:888@conference.freeswitch.org > iax:guest@conference.freeswitch.org/888 > googletalk:conf+888@conference.freeswitch.org > pstn:213-799-1400

> _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Yep Full-Duplex on both and these are what the offload parameters look like Offload parameters for eth2: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp segmentation offload: on udp fragmentation offload: off generic segmentation offload: off Since it's lot of little packets I left udp frag seg off because I didn't think it would be of much use. -e It's seems fuzzy now but I think on Tue, Nov 04, 2008 at 09:59:54AM +0500 , Wasim Baig said:

Please check if your net interface is full duplex (ethtool should give you this info).

We had this problem once, and spent weeks trying to identify and turned out to be a half-duplex.

Also, see if you can offload checksum to the NIC if it'll do that.

-wasim

On Tue, Nov 4, 2008 at 8:20 AM, Eric Liedtke <e@musinghalfwit.org> wrote:

I'm running the 1.0.1-phoenix release. No debug because our admin built the debian packages to install with so dh_strip got em. It's running on an ubuntu 8.04 server. I certainly can. I will let you know how it goes.

-e

It's seems fuzzy now but I think on Mon, Nov 03, 2008 at 08:57:01PM -0600 , Anthony Minessale said: > What revision are you on because the line numbers of those errors don't match > up even slightly close? > Did you do something to disable building with debug symbols because the current > default is to build with them? > What platform are you on? > > Could you try a clean build of the latest revision? > > > On Mon, Nov 3, 2008 at 5:43 PM, Eric Liedtke <e@musinghalfwit.org> wrote: > > Yessir. I am using the exact same ulimits below. I also increased the > sessions per second to 50 and max-sessions to 5000. Also as we are > moving into a high traffic period based on time of day. I've had it die > repeatedly on me, the following line was the last line printed ... > > freeswitch: nua_stack.c:2396: nua_client_restart_request: Assertion > `nua_client_is_queued(cr)' failed. > > this is the other last line I get if the system itself seems to be > overloaded > > freeswitch: sofia.c:2138: sofia_handle_sip_i_state: Assertion > `tech_pvt->nh != ((void *)0)' failed. > > I don't know if that is of any help or not. At the time I was seeing > approx 80-90 calls per sec hitting the server, based on the 'status' > command in the FS cli. > > I don't have this built with debug symbols either at the moment. I > wanted to get a baseline of server with x can do y , based on how we use > it. I will likely get it rebuilt and find a way to reduce some of the > traffic load so I can effectively tshoot. > > -e > > It's seems fuzzy now but I think on Mon, Nov 03, 2008 at 05:19:42PM -0600 , > Anthony Minessale said: > > Did you increase the artificial limits in switch.conf.xml ? > > > > try these uimits > > > > ulimit -c unlimited > > ulimit -d unlimited > > ulimit -f unlimited > > ulimit -i unlimited > > ulimit -n 999999 > > ulimit -q unlimited > > > > ulimit -u unlimited > > ulimit -v unlimited > > ulimit -x unlimited > > ulimit -s 244 > > ulimit -l unlimited > > > > > > > > > > On Mon, Nov 3, 2008 at 4:46 PM, Eric Liedtke <e@musinghalfwit.org> wrote: > > > > Greetings, > > > > I'm working on a server in an SBC setup. It has a copper 10/100 for > > admin and 2 GiGe connections for private/public peering(SIP signaling > > and media). I've been testing with some traffic and am finding I hit > a > > ceiling around 800-900 active calls. The system CPU ends up being > about > > 40% which seemed really high to me. Is this simply due to the amount > of > > networking happening with the rtp in and out the different interfaces > or am > > I > > operating something in a non optimized fasion ? > > > > It's an e1000 based card, I've tried messing around with the > interrupt > > throttling but in the end the dynamic setting seemed to work best. > Given > > that a chassis has done 3k simultaneous calls with about half the > > processing power I currently have in this server I must be running > sub > > optimal. > > > > Any idea's of where I might look to correct this ? kernel tuning ? > > network tuning ? Other than seeing a lot of irq's from the network > cards > > I"m not sure what else might be using up the system cpu. > > > > To squeeze out as much as possible I have the core databases housed > in a > > tmpfs partiion, and I've also disabled the sql scoreboard (-nosql). > Any > > other areas I should be exploring to increase the number of calls > this > > chassis can process ? > > > > Thanks in advance for any ideas > > > > -Eric > > > > _______________________________________________ > > Freeswitch-users mailing list > > Freeswitch-users@lists.freeswitch.org > > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ > freeswitch-users > > http://www.freeswitch.org > > > > > > > > > > -- > > Anthony Minessale II > > > > FreeSWITCH http://www.freeswitch.org/ > > ClueCon http://www.cluecon.com/ > > > > AIM: anthm > > MSN:anthony_minessale@hotmail.com > > GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com > > IRC: irc.freenode.net #freeswitch > > > > FreeSWITCH Developer Conference > > sip:888@conference.freeswitch.org > > iax:guest@conference.freeswitch.org/888 > > googletalk:conf+888@conference.freeswitch.org > > pstn:213-799-1400 > > > _______________________________________________ > > Freeswitch-users mailing list > > Freeswitch-users@lists.freeswitch.org > > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ freeswitch-users > > http://www.freeswitch.org > > > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ freeswitch-users > http://www.freeswitch.org > > > > > -- > Anthony Minessale II > > FreeSWITCH http://www.freeswitch.org/ > ClueCon http://www.cluecon.com/ > > AIM: anthm > MSN:anthony_minessale@hotmail.com > GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com > IRC: irc.freenode.net #freeswitch > > FreeSWITCH Developer Conference > sip:888@conference.freeswitch.org > iax:guest@conference.freeswitch.org/888 > googletalk:conf+888@conference.freeswitch.org > pstn:213-799-1400

> _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- wasim h. baig | principal consultant | convergence pk | +92 300 8508070 | as you scope creep, so shall we reap ...

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Not a problem. So far this looks to be a vast improvement. I'm already at around 800 calls, which yesteday had me down to 30% idle cpu, and today am running around 75% idle. Also my system cpu is back down under the user cpu. So some definite marked improvements. I will let you know where we get with it today. Thanks!!! -e It's seems fuzzy now but I think on Tue, Nov 04, 2008 at 08:50:32AM -0600 , Anthony Minessale said:

Thank you. Since we are in final beta stage for 1.0.2 I appreciate you testing it for us.

On Mon, Nov 3, 2008 at 9:20 PM, Eric Liedtke <e@musinghalfwit.org> wrote:

I'm running the 1.0.1-phoenix release. No debug because our admin built the debian packages to install with so dh_strip got em. It's running on an ubuntu 8.04 server. I certainly can. I will let you know how it goes.

-e

It's seems fuzzy now but I think on Mon, Nov 03, 2008 at 08:57:01PM -0600 , Anthony Minessale said: > What revision are you on because the line numbers of those errors don't match > up even slightly close? > Did you do something to disable building with debug symbols because the current > default is to build with them? > What platform are you on? > > Could you try a clean build of the latest revision? > > > On Mon, Nov 3, 2008 at 5:43 PM, Eric Liedtke <e@musinghalfwit.org> wrote: > > Yessir. I am using the exact same ulimits below. I also increased the > sessions per second to 50 and max-sessions to 5000. Also as we are > moving into a high traffic period based on time of day. I've had it die > repeatedly on me, the following line was the last line printed ... > > freeswitch: nua_stack.c:2396: nua_client_restart_request: Assertion > `nua_client_is_queued(cr)' failed. > > this is the other last line I get if the system itself seems to be > overloaded > > freeswitch: sofia.c:2138: sofia_handle_sip_i_state: Assertion > `tech_pvt->nh != ((void *)0)' failed. > > I don't know if that is of any help or not. At the time I was seeing > approx 80-90 calls per sec hitting the server, based on the 'status' > command in the FS cli. > > I don't have this built with debug symbols either at the moment. I > wanted to get a baseline of server with x can do y , based on how we use > it. I will likely get it rebuilt and find a way to reduce some of the > traffic load so I can effectively tshoot. > > -e > > It's seems fuzzy now but I think on Mon, Nov 03, 2008 at 05:19:42PM -0600 , > Anthony Minessale said: > > Did you increase the artificial limits in switch.conf.xml ? > > > > try these uimits > > > > ulimit -c unlimited > > ulimit -d unlimited > > ulimit -f unlimited > > ulimit -i unlimited > > ulimit -n 999999 > > ulimit -q unlimited > > > > ulimit -u unlimited > > ulimit -v unlimited > > ulimit -x unlimited > > ulimit -s 244 > > ulimit -l unlimited > > > > > > > > > > On Mon, Nov 3, 2008 at 4:46 PM, Eric Liedtke <e@musinghalfwit.org> wrote: > > > > Greetings, > > > > I'm working on a server in an SBC setup. It has a copper 10/100 for > > admin and 2 GiGe connections for private/public peering(SIP signaling > > and media). I've been testing with some traffic and am finding I hit > a > > ceiling around 800-900 active calls. The system CPU ends up being > about > > 40% which seemed really high to me. Is this simply due to the amount > of > > networking happening with the rtp in and out the different interfaces > or am > > I > > operating something in a non optimized fasion ? > > > > It's an e1000 based card, I've tried messing around with the > interrupt > > throttling but in the end the dynamic setting seemed to work best. > Given > > that a chassis has done 3k simultaneous calls with about half the > > processing power I currently have in this server I must be running > sub > > optimal. > > > > Any idea's of where I might look to correct this ? kernel tuning ? > > network tuning ? Other than seeing a lot of irq's from the network > cards > > I"m not sure what else might be using up the system cpu. > > > > To squeeze out as much as possible I have the core databases housed > in a > > tmpfs partiion, and I've also disabled the sql scoreboard (-nosql). > Any > > other areas I should be exploring to increase the number of calls > this > > chassis can process ? > > > > Thanks in advance for any ideas > > > > -Eric > > > > _______________________________________________ > > Freeswitch-users mailing list > > Freeswitch-users@lists.freeswitch.org > > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ > freeswitch-users > > http://www.freeswitch.org > > > > > > > > > > -- > > Anthony Minessale II > > > > FreeSWITCH http://www.freeswitch.org/ > > ClueCon http://www.cluecon.com/ > > > > AIM: anthm > > MSN:anthony_minessale@hotmail.com > > GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com > > IRC: irc.freenode.net #freeswitch > > > > FreeSWITCH Developer Conference > > sip:888@conference.freeswitch.org > > iax:guest@conference.freeswitch.org/888 > > googletalk:conf+888@conference.freeswitch.org > > pstn:213-799-1400 > > > _______________________________________________ > > Freeswitch-users mailing list > > Freeswitch-users@lists.freeswitch.org > > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ freeswitch-users > > http://www.freeswitch.org > > > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ freeswitch-users > http://www.freeswitch.org > > > > > -- > Anthony Minessale II > > FreeSWITCH http://www.freeswitch.org/ > ClueCon http://www.cluecon.com/ > > AIM: anthm > MSN:anthony_minessale@hotmail.com > GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com > IRC: irc.freenode.net #freeswitch > > FreeSWITCH Developer Conference > sip:888@conference.freeswitch.org > iax:guest@conference.freeswitch.org/888 > googletalk:conf+888@conference.freeswitch.org > pstn:213-799-1400

> _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org