FYI, I added a note on the reporting bugs page that specifically talks about this scenario: http://wiki.freeswitch.org/wiki/Reporting_Bugs#Upgrading_From_Older_Revision... -MC On Wed, Jan 28, 2009 at 6:01 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:

Also remember, Actually completely uninstall and erase /usr/local/freeswitch and the 1.0.1 source tree and freshly install the new one. If you try to upgrade on top of a release with trunk it will cause more problems for you.

On Wed, Jan 28, 2009 at 3:11 AM, Ken Rice <krice@suspicious.org> wrote:

...

Upgrade to trunk... Many many issues have been resolved since 1.0.1 was the current release

...

From: shehzad p <pmhshz@gmail.com> Reply-To: <freeswitch-users@lists.freeswitch.org> Date: Wed, 28 Jan 2009 00:54:13 -0800 (PST) To: <freeswitch-users@lists.freeswitch.org> Subject: [Freeswitch-users] Freeswitch freezes on increasing call traffic

Hi all,

Yesterday my Freeswitch server faced a problem when call traffic increased to more than 100.

When I start Freeswitch, it works fine and then after some time (approximately 15 to 20 minutes) it stops functioning (means no call is being processed, no CLI command is working and it just freezes) until I restart the freeswitch.

I am using Freeswitch 1.0.1. Debug (gdb) trace as on wiki page http://wiki.freeswitch.org/wiki/Debugging_Freeswitch#fs_debug.sh is attached http://www.nabble.com/file/p21701744/fs_debgu.txt fs_debgu.txt -- View this message in context:

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... p21701744.html Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

You might also want to set max-proceeding on the sofia profile. /b On Jan 29, 2009, at 10:06 AM, shehzad p wrote:

When I put ngrep trace, System responds with 503 Maximum Calls In Progress. (as below)

Thanks, Anthony In my previous test sps did not changed, but in recent test sps was dropped to 0 itself (as below). =============================================================== UP 0 years, 0 days, 5 hours, 1 minute, 53 seconds, 878 milliseconds, 190 microseconds 5474 session(s) since startup 75 session(s) 0/0 ============================================================= My system is 32 bit. CPU is Intel(R) Xeon(R) CPU X3220 @ 2.40GHz And RAM is 4GB Output of ulimit -a is: ulimit -a: (set after first test) core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited max nice (-e) 20 file size (blocks, -f) unlimited pending signals (-i) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 999999 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) unlimited max rt priority (-r) unlimited stack size (kbytes, -s) 244 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited =================================================== BTW using trunk on production system is safe? Warm thanks for kind responses... Anthony Minessale-2 wrote:

When you get it in that state what do you see when you execute

fsctl sps

is the sps a very low number?

Did the sps drop by itself from the value you originally set it to?

Are you using 32 bit?

if so try all of these commands in your shell before starting FS

ulimit -c unlimited ulimit -d unlimited ulimit -f unlimited ulimit -i unlimited ulimit -n 999999 ulimit -q unlimited ulimit -u unlimited ulimit -v unlimited ulimit -x unlimited ulimit -s 244 ulimit -l unlimited

DO NOT put them in a script unless you source the script with . . myscript or they will be undone instantly when the script exits

BTW, I said to try latest trunk not 1.0.2 We can only debug the development code at this point.

On Thu, Jan 29, 2009 at 10:06 AM, shehzad p <pmhshz@gmail.com> wrote:

...

Hi Anthony,

I found interesting result while testing Freeswitch, and it might be cause of freezing out of freeswitch,

I updated my system (as you told) with latest stable version Freeswitch 1.0.2 First of all I set sps to 100, Then I sends call approximately 100 per seconds, Freeswitch works fine and handles all the calls very well.

After that I send 130 calls per seconds, and magic happen now, Freeswitch handles first 100 calls only. all the preceding calls were failed (even not appeared in freeswitch console why?)

When I put ngrep trace, System responds with 503 Maximum Calls In Progress. (as below) ########################################################### # U FSFSFSFSFS -> GWGWGWGWGW SIP/2.0 503 Maximum Calls In Progress. Via: SIP/2.0/UDP GWGWGWGWGW;branch=z9hG4bK53eafabb;rport=5060. From: "99999" <sip:99999@GWGWGWGWGW>;tag=as2e10c170. To: <sip:1234567890@FSFSFSFSFS >;tag=K3jSUFrDHpmmB. Call-ID: 0feb229e58afe7be17110deb361dc234@GWGWGWGWGW. CSeq: 102 INVITE. Retry-After: 300. User-Agent: FreeSWITCH-mod_sofia/1.0.2-exported. Accept: application/sdp. Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, PRACK, MESSAGE, SUBSCRIBE, NOTIFY, REFER, UPDATE, REGISTER, INFO, PUBLISH. Supported: timer, precondition, path, replaces. Allow-Events: talk, presence, dialog, call-info, sla, include-session-description, presence.winfo, message-summary, refer. Content-Length: 0. . #####################################################################

Now another issue to note down is that, After all above happened and active calls comes to zero, I just make a single call which also fails with response 503 - Maximum Calls In Progress.

Is this intended behaviour, should I increase SPS to overcome this. or something like bug.

Please let me know what should be the resolution for this.

Thanks, msp

Anthony Minessale-2 wrote:

...

Also remember, Actually completely uninstall and erase /usr/local/freeswitch and the 1.0.1 source tree and freshly install the new one. If you try to upgrade on top of a release with trunk it will cause more problems for you.

On Wed, Jan 28, 2009 at 3:11 AM, Ken Rice <krice@suspicious.org> wrote:

...

Upgrade to trunk... Many many issues have been resolved since 1.0.1

was

...

...

the current release

...

From: shehzad p <pmhshz@gmail.com> Reply-To: <freeswitch-users@lists.freeswitch.org> Date: Wed, 28 Jan 2009 00:54:13 -0800 (PST) To: <freeswitch-users@lists.freeswitch.org> Subject: [Freeswitch-users] Freeswitch freezes on increasing call traffic

Hi all,

Yesterday my Freeswitch server faced a problem when call traffic increased to more than 100.

When I start Freeswitch, it works fine and then after some time (approximately 15 to 20 minutes) it stops functioning (means no call is being processed, no CLI command is working and it just freezes) until I restart the freeswitch.

I am using Freeswitch 1.0.1. Debug (gdb) trace as on wiki page http://wiki.freeswitch.org/wiki/Debugging_Freeswitch#fs_debug.sh is attached http://www.nabble.com/file/p21701744/fs_debgu.txt fs_debgu.txt -- View this message in context:

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

...

p21701744.html Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users

...

http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com>< MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com>

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org

<sip%3A888@conference.freeswitch.org>< sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org>

...

iax:guest@conference.freeswitch.org/888

googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

...

http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

if you are using unix you can use the supplied script scripts/freeswitch-gcore to capture a copy of the resident memory and I can have a look perhaps. Trunk is safe for production as we are in beta stage for a release of 1.0.3 at this time. On Fri, Jan 30, 2009 at 9:29 AM, shehzad p <pmhshz@gmail.com> wrote:

When freeswitch freezes, we can't connect to it to check sps status, but once we were able to connect and at that time it was showing 0/0 sps.

thanks...

shehzad p wrote:

...

Thanks, Anthony

In my previous test sps did not changed, but in recent test sps was dropped to 0 itself (as below). =============================================================== UP 0 years, 0 days, 5 hours, 1 minute, 53 seconds, 878 milliseconds, 190 microseconds 5474 session(s) since startup 75 session(s) 0/0 =============================================================

My system is 32 bit. CPU is Intel(R) Xeon(R) CPU X3220 @ 2.40GHz And RAM is 4GB

Output of ulimit -a is: ulimit -a: (set after first test) core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited max nice (-e) 20 file size (blocks, -f) unlimited pending signals (-i) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 999999 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) unlimited max rt priority (-r) unlimited stack size (kbytes, -s) 244 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ===================================================

BTW using trunk on production system is safe?

Warm thanks for kind responses...

Anthony Minessale-2 wrote:

...

When you get it in that state what do you see when you execute

fsctl sps

is the sps a very low number?

Did the sps drop by itself from the value you originally set it to?

Are you using 32 bit?

if so try all of these commands in your shell before starting FS

ulimit -c unlimited ulimit -d unlimited ulimit -f unlimited ulimit -i unlimited ulimit -n 999999 ulimit -q unlimited ulimit -u unlimited ulimit -v unlimited ulimit -x unlimited ulimit -s 244 ulimit -l unlimited

DO NOT put them in a script unless you source the script with . . myscript or they will be undone instantly when the script exits

BTW, I said to try latest trunk not 1.0.2 We can only debug the development code at this point.

On Thu, Jan 29, 2009 at 10:06 AM, shehzad p <pmhshz@gmail.com> wrote:

...

Hi Anthony,

I found interesting result while testing Freeswitch, and it might be cause of freezing out of freeswitch,

I updated my system (as you told) with latest stable version Freeswitch 1.0.2 First of all I set sps to 100, Then I sends call approximately 100 per seconds, Freeswitch works fine and handles all the calls very well.

After that I send 130 calls per seconds, and magic happen now, Freeswitch handles first 100 calls only. all the preceding calls were failed (even not appeared in freeswitch console why?)

When I put ngrep trace, System responds with 503 Maximum Calls In Progress. (as below) ########################################################### # U FSFSFSFSFS -> GWGWGWGWGW SIP/2.0 503 Maximum Calls In Progress. Via: SIP/2.0/UDP GWGWGWGWGW;branch=z9hG4bK53eafabb;rport=5060. From: "99999" <sip:99999@GWGWGWGWGW>;tag=as2e10c170. To: <sip:1234567890@FSFSFSFSFS >;tag=K3jSUFrDHpmmB. Call-ID: 0feb229e58afe7be17110deb361dc234@GWGWGWGWGW. CSeq: 102 INVITE. Retry-After: 300. User-Agent: FreeSWITCH-mod_sofia/1.0.2-exported. Accept: application/sdp. Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, PRACK, MESSAGE, SUBSCRIBE, NOTIFY, REFER, UPDATE, REGISTER, INFO, PUBLISH. Supported: timer, precondition, path, replaces. Allow-Events: talk, presence, dialog, call-info, sla, include-session-description, presence.winfo, message-summary, refer. Content-Length: 0. . #####################################################################

Now another issue to note down is that, After all above happened and active calls comes to zero, I just make a single call which also fails with response 503 - Maximum Calls In Progress.

Is this intended behaviour, should I increase SPS to overcome this. or something like bug.

Please let me know what should be the resolution for this.

Thanks, msp

Anthony Minessale-2 wrote:

...

Also remember, Actually completely uninstall and erase /usr/local/freeswitch and the 1.0.1 source tree and freshly install the new one. If you try to upgrade on top of a release with trunk it will cause

more

...

problems for you.

On Wed, Jan 28, 2009 at 3:11 AM, Ken Rice <krice@suspicious.org> wrote:

...

Upgrade to trunk... Many many issues have been resolved since 1.0.1 was the current release

> From: shehzad p <pmhshz@gmail.com> > Reply-To: <freeswitch-users@lists.freeswitch.org> > Date: Wed, 28 Jan 2009 00:54:13 -0800 (PST) > To: <freeswitch-users@lists.freeswitch.org> > Subject: [Freeswitch-users] Freeswitch freezes on increasing call traffic > > > Hi all, > > Yesterday my Freeswitch server faced a problem when call traffic increased > to more than 100. > > When I start Freeswitch, it works fine and then after some time > (approximately 15 to 20 minutes) it stops functioning (means no call is > being processed, no CLI command is working and it just freezes) until I > restart the freeswitch. > > I am using Freeswitch 1.0.1. > Debug (gdb) trace as on wiki page > http://wiki.freeswitch.org/wiki/Debugging_Freeswitch#fs_debug.shis attached > http://www.nabble.com/file/p21701744/fs_debgu.txt fs_debgu.txt > -- > View this message in context: >

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

...

...

...

...

> p21701744.html > Sent from the Freeswitch-users mailing list archive at Nabble.com. > > > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com<MSN%3Aanthony_minessale@hotmail.com> <MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> < MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com>

...

...

...

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

...

<PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com> <PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com>

...

...

...

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org<sip%3A888@conference.freeswitch.org>

<sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> < sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org>

...

...

...

iax:guest@conference.freeswitch.org/888

googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

...

<googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org> <googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org>

...

...

...

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users

...

http://www.freeswitch.org

-- View this message in context:

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com>< MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com>

...

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org>< sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org>

...

iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400

Clearly you have an issue with your javascript code. You have the Garbage collector blocking in every thread. Are you doing any endless loops in your code where you do not check session.ready() as a condition for continuing the script? any time session.ready() fails you must immediately exit. Are you using session.execute to execute long blocking operations like bridging many calls or entering a conference? You should avoid doing this as all the collective scripts on the system share a common Garbage Collector provided by the JS engine and it can lead to the exact issues you describe if the code is not properly designed. What else does you script do that are things provided by FS such as playing files and executing applications. On Sat, Jan 31, 2009 at 3:44 AM, shehzad p <pmhshz@gmail.com> wrote:

Hi Anthony,

Freeswitch 1.0.2 was crashed on last test again... BT is on http://pastebin.freeswitch.org/6979.

I tried to use scripts/freeswitch-gcore, to capture resident memory, but before the command complete the process system hanged up so only half output was captured.< http://www.nabble.com/file/p21761523/gcore-fs.txt gcore-fs.txt >

Now I have checkout from trunk and will post back if any new thing found...

Thanks, msp

Anthony Minessale-2 wrote:

...

if you are using unix you can use the supplied script

scripts/freeswitch-gcore

to capture a copy of the resident memory and I can have a look perhaps.

Trunk is safe for production as we are in beta stage for a release of 1.0.3 at this time.

On Fri, Jan 30, 2009 at 9:29 AM, shehzad p <pmhshz@gmail.com> wrote:

...

When freeswitch freezes, we can't connect to it to check sps status, but once we were able to connect and at that time it was showing 0/0

sps.

...

...

thanks...

shehzad p wrote:

...

Thanks, Anthony

In my previous test sps did not changed, but in recent test sps was dropped to 0 itself (as below). =============================================================== UP 0 years, 0 days, 5 hours, 1 minute, 53 seconds, 878 milliseconds,

190

...

microseconds 5474 session(s) since startup 75 session(s) 0/0 =============================================================

My system is 32 bit. CPU is Intel(R) Xeon(R) CPU X3220 @ 2.40GHz And RAM is 4GB

Output of ulimit -a is: ulimit -a: (set after first test) core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited max nice (-e) 20 file size (blocks, -f) unlimited pending signals (-i) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 999999 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) unlimited max rt priority (-r) unlimited stack size (kbytes, -s) 244 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ===================================================

BTW using trunk on production system is safe?

Warm thanks for kind responses...

Anthony Minessale-2 wrote:

...

When you get it in that state what do you see when you execute

fsctl sps

is the sps a very low number?

Did the sps drop by itself from the value you originally set it to?

Are you using 32 bit?

if so try all of these commands in your shell before starting FS

ulimit -c unlimited ulimit -d unlimited ulimit -f unlimited ulimit -i unlimited ulimit -n 999999 ulimit -q unlimited ulimit -u unlimited ulimit -v unlimited ulimit -x unlimited ulimit -s 244 ulimit -l unlimited

DO NOT put them in a script unless you source the script with . . myscript or they will be undone instantly when the script exits

BTW, I said to try latest trunk not 1.0.2 We can only debug the development code at this point.

On Thu, Jan 29, 2009 at 10:06 AM, shehzad p <pmhshz@gmail.com>

wrote:

...

...

...

...

Hi Anthony,

I found interesting result while testing Freeswitch, and it might be cause of freezing out of freeswitch,

I updated my system (as you told) with latest stable version

Freeswitch

...

...

1.0.2 First of all I set sps to 100, Then I sends call approximately 100 per seconds, Freeswitch works fine and handles all the calls very well.

After that I send 130 calls per seconds, and magic happen now, Freeswitch handles first 100 calls only. all the preceding calls were failed (even not appeared in freeswitch console why?)

When I put ngrep trace, System responds with 503 Maximum Calls In Progress. (as below) ########################################################### # U FSFSFSFSFS -> GWGWGWGWGW SIP/2.0 503 Maximum Calls In Progress. Via: SIP/2.0/UDP GWGWGWGWGW;branch=z9hG4bK53eafabb;rport=5060. From: "99999" <sip:99999@GWGWGWGWGW>;tag=as2e10c170. To: <sip:1234567890@FSFSFSFSFS >;tag=K3jSUFrDHpmmB. Call-ID: 0feb229e58afe7be17110deb361dc234@GWGWGWGWGW. CSeq: 102 INVITE. Retry-After: 300. User-Agent: FreeSWITCH-mod_sofia/1.0.2-exported. Accept: application/sdp. Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, PRACK, MESSAGE, SUBSCRIBE, NOTIFY, REFER, UPDATE, REGISTER, INFO, PUBLISH. Supported: timer, precondition, path, replaces. Allow-Events: talk, presence, dialog, call-info, sla, include-session-description, presence.winfo, message-summary, refer. Content-Length: 0. .

#####################################################################

...

Now another issue to note down is that, After all above happened and active calls comes to zero, I just make a single call which also fails with response 503 -

Maximum

...

Calls In Progress.

Is this intended behaviour, should I increase SPS to overcome this. or something like bug.

Please let me know what should be the resolution for this.

Thanks, msp

Anthony Minessale-2 wrote: > > Also remember, > Actually completely uninstall and erase /usr/local/freeswitch and the > 1.0.1 > source tree and freshly install the new one. > If you try to upgrade on top of a release with trunk it will cause more > problems for you. > > > On Wed, Jan 28, 2009 at 3:11 AM, Ken Rice <krice@suspicious.org> wrote: > >> Upgrade to trunk... Many many issues have been resolved since 1.0.1 was >> the >> current release >> >> >> > From: shehzad p <pmhshz@gmail.com> >> > Reply-To: <freeswitch-users@lists.freeswitch.org> >> > Date: Wed, 28 Jan 2009 00:54:13 -0800 (PST) >> > To: <freeswitch-users@lists.freeswitch.org> >> > Subject: [Freeswitch-users] Freeswitch freezes on increasing call >> traffic >> > >> > >> > Hi all, >> > >> > Yesterday my Freeswitch server faced a problem when call traffic >> increased >> > to more than 100. >> > >> > When I start Freeswitch, it works fine and then after some time >> > (approximately 15 to 20 minutes) it stops functioning (means no call >> is >> > being processed, no CLI command is working and it just freezes) until I >> > restart the freeswitch. >> > >> > I am using Freeswitch 1.0.1. >> > Debug (gdb) trace as on wiki page >> > http://wiki.freeswitch.org/wiki/Debugging_Freeswitch#fs_debug.shis >> attached >> > http://www.nabble.com/file/p21701744/fs_debgu.txt fs_debgu.txt >> > -- >> > View this message in context: >> > >>

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

...

...

...

>> > p21701744.html >> > Sent from the Freeswitch-users mailing list archive at Nabble.com. >> > >> > >> > _______________________________________________ >> > Freeswitch-users mailing list >> > Freeswitch-users@lists.freeswitch.org >> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> > >> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >> > http://www.freeswitch.org >> >> >> >> _______________________________________________ >> Freeswitch-users mailing list >> Freeswitch-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > > > -- > Anthony Minessale II > > FreeSWITCH http://www.freeswitch.org/ > ClueCon http://www.cluecon.com/ > > AIM: anthm > MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> <MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com>

...

...

...

...

<MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com>

...

...

<

...

...

MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com>

...

<MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com> <MSN%25253Aanthony_minessale@hotmail.com<MSN%2525253Aanthony_minessale@hotmail.com>

...

...

...

...

> >

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

<PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com> <PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com>

...

...

...

...

<PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com> <PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com>

...

<PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com> <PAYPAL%25253Aanthony.minessale@gmail.com<PAYPAL%2525253Aanthony.minessale@gmail.com>

...

...

...

...

> > IRC: irc.freenode.net #freeswitch > > FreeSWITCH Developer Conference >

sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> <sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org>

...

...

...

...

<sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org>

...

...

<

...

...

sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org>

...

<sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org> <sip%25253A888@conference.freeswitch.org<sip%2525253A888@conference.freeswitch.org>

...

...

...

...

> > iax:guest@conference.freeswitch.org/888 >

googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

<googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org> <googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org>

...

...

...

...

<googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org> <googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org>

...

<googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org> <googletalk%25253Aconf%25252B888@conference.freeswitch.org<googletalk%2525253Aconf%2525252B888@conference.freeswitch.org>

...

...

...

...

> > pstn:213-799-1400 > > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:

http://lists.freeswitch.org/mailman/options/freeswitch-users

...

...

...

> http://www.freeswitch.org > >

-- View this message in context:

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

...

...

...

Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com<MSN%3Aanthony_minessale@hotmail.com> <MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> < MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com>

...

...

...

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

<PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com> <PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com>

...

...

...

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org<sip%3A888@conference.freeswitch.org>

<sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> < sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org>

...

...

...

iax:guest@conference.freeswitch.org/888

googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

<googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org> <googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org>

...

...

...

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:

http://lists.freeswitch.org/mailman/options/freeswitch-users

...

...

http://www.freeswitch.org

-- View this message in context:

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com>< MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com>

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org>< sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org>

iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400

Hi anthony, There are so many crash occured, on FS 1.0.1 Server, all of them were same (means BT was same) but this last one was looking totally different so I feel that it should be posted... http://www.nabble.com/file/p21775256/bt_full_new.txt bt_full_new.txt shehzad p wrote:

Hi Anthony,

How can I handle Garbage collector related things. Is there any way to look out where js variables are causing some kind of fault related to Garbage collector?

There are some loops but they finish in hardly two or three iterations.

I didn't check session.ready() before/after fetching values from database (odbc), which might block until response from server comes back. might this cause fault as you said?

Thanks for help...

Anthony Minessale-2 wrote:

...

Clearly you have an issue with your javascript code.

You have the Garbage collector blocking in every thread.

Are you doing any endless loops in your code where you do not check session.ready() as a condition for continuing the script?

any time session.ready() fails you must immediately exit.

Are you using session.execute to execute long blocking operations like bridging many calls or entering a conference? You should avoid doing this as all the collective scripts on the system share a common Garbage Collector provided by the JS engine and it can lead to the exact issues you describe if the code is not properly designed.

What else does you script do that are things provided by FS such as playing files and executing applications.

On Sat, Jan 31, 2009 at 3:44 AM, shehzad p <pmhshz@gmail.com> wrote:

...

Hi Anthony,

Freeswitch 1.0.2 was crashed on last test again... BT is on http://pastebin.freeswitch.org/6979.

I tried to use scripts/freeswitch-gcore, to capture resident memory, but before the command complete the process system hanged up so only half output was captured.< http://www.nabble.com/file/p21761523/gcore-fs.txt gcore-fs.txt >

Now I have checkout from trunk and will post back if any new thing found...

Thanks, msp

Anthony Minessale-2 wrote:

...

if you are using unix you can use the supplied script

scripts/freeswitch-gcore

to capture a copy of the resident memory and I can have a look

perhaps.

...

Trunk is safe for production as we are in beta stage for a release of 1.0.3 at this time.

On Fri, Jan 30, 2009 at 9:29 AM, shehzad p <pmhshz@gmail.com> wrote:

...

When freeswitch freezes, we can't connect to it to check sps status, but once we were able to connect and at that time it was showing 0/0

sps.

...

...

thanks...

shehzad p wrote:

...

Thanks, Anthony

In my previous test sps did not changed, but in recent test sps was dropped to 0 itself (as below). =============================================================== UP 0 years, 0 days, 5 hours, 1 minute, 53 seconds, 878

milliseconds,

...

190

...

microseconds 5474 session(s) since startup 75 session(s) 0/0 =============================================================

My system is 32 bit. CPU is Intel(R) Xeon(R) CPU X3220 @ 2.40GHz And RAM is 4GB

Output of ulimit -a is: ulimit -a: (set after first test) core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited max nice (-e) 20 file size (blocks, -f) unlimited pending signals (-i) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 999999 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) unlimited max rt priority (-r) unlimited stack size (kbytes, -s) 244 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ===================================================

BTW using trunk on production system is safe?

Warm thanks for kind responses...

Anthony Minessale-2 wrote: > > When you get it in that state what do you see when you execute > > fsctl sps > > is the sps a very low number? > > Did the sps drop by itself from the value you originally set it to? > > Are you using 32 bit? > > if so try all of these commands in your shell before starting FS > > ulimit -c unlimited > ulimit -d unlimited > ulimit -f unlimited > ulimit -i unlimited > ulimit -n 999999 > ulimit -q unlimited > ulimit -u unlimited > ulimit -v unlimited > ulimit -x unlimited > ulimit -s 244 > ulimit -l unlimited > > > DO NOT put them in a script unless you source the script with . > . myscript or they will be undone instantly when the script exits > > BTW, I said to try latest trunk not 1.0.2 We can only debug the > development > code at this point. > > > > > > On Thu, Jan 29, 2009 at 10:06 AM, shehzad p <pmhshz@gmail.com> wrote: > >> >> Hi Anthony, >> >> I found interesting result while testing Freeswitch, and it might be >> cause >> of freezing out of freeswitch, >> >> I updated my system (as you told) with latest stable version Freeswitch >> 1.0.2 >> First of all I set sps to 100, >> Then I sends call approximately 100 per seconds, Freeswitch works fine >> and >> handles all the calls very well. >> >> After that I send 130 calls per seconds, and magic happen now, >> Freeswitch >> handles first 100 calls only. >> all the preceding calls were failed (even not appeared in freeswitch >> console >> why?) >> >> When I put ngrep trace, System responds with 503 Maximum Calls In >> Progress. >> (as below) >> ########################################################### >> # >> U FSFSFSFSFS -> GWGWGWGWGW >> SIP/2.0 503 Maximum Calls In Progress. >> Via: SIP/2.0/UDP GWGWGWGWGW;branch=z9hG4bK53eafabb;rport=5060. >> From: "99999" <sip:99999@GWGWGWGWGW>;tag=as2e10c170. >> To: <sip:1234567890@FSFSFSFSFS >;tag=K3jSUFrDHpmmB. >> Call-ID: 0feb229e58afe7be17110deb361dc234@GWGWGWGWGW. >> CSeq: 102 INVITE. >> Retry-After: 300. >> User-Agent: FreeSWITCH-mod_sofia/1.0.2-exported. >> Accept: application/sdp. >> Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, PRACK, MESSAGE, SUBSCRIBE, >> NOTIFY, >> REFER, UPDATE, REGISTER, INFO, PUBLISH. >> Supported: timer, precondition, path, replaces. >> Allow-Events: talk, presence, dialog, call-info, sla, >> include-session-description, presence.winfo, message-summary, refer. >> Content-Length: 0. >> . >> ##################################################################### >> >> >> Now another issue to note down is that, >> After all above happened and active calls comes to zero, >> I just make a single call which also fails with response 503 - Maximum >> Calls >> In Progress. >> >> >> Is this intended behaviour, should I increase SPS to overcome this. or >> something like bug. >> >> Please let me know what should be the resolution for this. >> >> Thanks, >> msp >> >> >> >> Anthony Minessale-2 wrote: >> > >> > Also remember, >> > Actually completely uninstall and erase /usr/local/freeswitch and the >> > 1.0.1 >> > source tree and freshly install the new one. >> > If you try to upgrade on top of a release with trunk it will cause >> more >> > problems for you. >> > >> > >> > On Wed, Jan 28, 2009 at 3:11 AM, Ken Rice <krice@suspicious.org> >> wrote: >> > >> >> Upgrade to trunk... Many many issues have been resolved since 1.0.1 >> was >> >> the >> >> current release >> >> >> >> >> >> > From: shehzad p <pmhshz@gmail.com> >> >> > Reply-To: <freeswitch-users@lists.freeswitch.org> >> >> > Date: Wed, 28 Jan 2009 00:54:13 -0800 (PST) >> >> > To: <freeswitch-users@lists.freeswitch.org> >> >> > Subject: [Freeswitch-users] Freeswitch freezes on increasing call >> >> traffic >> >> > >> >> > >> >> > Hi all, >> >> > >> >> > Yesterday my Freeswitch server faced a problem when call traffic >> >> increased >> >> > to more than 100. >> >> > >> >> > When I start Freeswitch, it works fine and then after some time >> >> > (approximately 15 to 20 minutes) it stops functioning (means no >> call >> >> is >> >> > being processed, no CLI command is working and it just freezes) >> until >> I >> >> > restart the freeswitch. >> >> > >> >> > I am using Freeswitch 1.0.1. >> >> > Debug (gdb) trace as on wiki page >> >> > http://wiki.freeswitch.org/wiki/Debugging_Freeswitch#fs_debug.shis >> >> attached >> >> > http://www.nabble.com/file/p21701744/fs_debgu.txt fs_debgu.txt >> >> > -- >> >> > View this message in context: >> >> > >> >> >>

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

...

>> >> > p21701744.html >> >> > Sent from the Freeswitch-users mailing list archive at Nabble.com. >> >> > >> >> > >> >> > _______________________________________________ >> >> > Freeswitch-users mailing list >> >> > Freeswitch-users@lists.freeswitch.org >> >> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> >> > >> >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> >> > http://www.freeswitch.org >> >> >> >> >> >> >> >> _______________________________________________ >> >> Freeswitch-users mailing list >> >> Freeswitch-users@lists.freeswitch.org >> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> >> http://www.freeswitch.org >> >> >> > >> > >> > >> > -- >> > Anthony Minessale II >> > >> > FreeSWITCH http://www.freeswitch.org/ >> > ClueCon http://www.cluecon.com/ >> > >> > AIM: anthm >> > MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> <MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com>

...

...

>>

<MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com>

...

...

< >>

MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com>

...

<MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com> <MSN%25253Aanthony_minessale@hotmail.com<MSN%2525253Aanthony_minessale@hotmail.com>

...

...

>> > >> > >>

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

<PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com> <PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com>

...

...

>>

<PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com> <PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com>

...

<PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com> <PAYPAL%25253Aanthony.minessale@gmail.com<PAYPAL%2525253Aanthony.minessale@gmail.com>

...

...

>> > >> > IRC: irc.freenode.net #freeswitch >> > >> > FreeSWITCH Developer Conference >> >

sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> <sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org>

...

...

>>

<sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org>

...

...

< >>

sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org>

...

<sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org> <sip%25253A888@conference.freeswitch.org<sip%2525253A888@conference.freeswitch.org>

...

...

>> > >> > iax:guest@conference.freeswitch.org/888 >> > >>

googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

<googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org> <googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org>

...

...

>>

<googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org> <googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org>

...

<googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org> <googletalk%25253Aconf%25252B888@conference.freeswitch.org<googletalk%2525253Aconf%2525252B888@conference.freeswitch.org>

...

...

>> > >> > pstn:213-799-1400 >> > >> > _______________________________________________ >> > Freeswitch-users mailing list >> > Freeswitch-users@lists.freeswitch.org >> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> > >> UNSUBSCRIBE:

http://lists.freeswitch.org/mailman/options/freeswitch-users

...

>> > http://www.freeswitch.org >> > >> > >> >> -- >> View this message in context: >>

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

...

>> Sent from the Freeswitch-users mailing list archive at Nabble.com. >> >> >> _______________________________________________ >> Freeswitch-users mailing list >> Freeswitch-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > > > -- > Anthony Minessale II > > FreeSWITCH http://www.freeswitch.org/ > ClueCon http://www.cluecon.com/ > > AIM: anthm > MSN:anthony_minessale@hotmail.com<MSN%3Aanthony_minessale@hotmail.com>

<MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <

...

MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com>

...

...

>

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

<PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com> <PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com>

...

...

> IRC: irc.freenode.net #freeswitch > > FreeSWITCH Developer Conference >

sip:888@conference.freeswitch.org<sip%3A888@conference.freeswitch.org>

<sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <

...

sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org>

...

...

> iax:guest@conference.freeswitch.org/888 >

googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

<googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org> <googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org>

...

...

> pstn:213-799-1400 > > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:

http://lists.freeswitch.org/mailman/options/freeswitch-users

...

> http://www.freeswitch.org > >

-- View this message in context:

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com>< MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com>

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org

<sip%3A888@conference.freeswitch.org>< sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org>

...

iax:guest@conference.freeswitch.org/888

googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

...

http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

Hi Brian, I am first looking to modify the script, as there are several things to modify (thanks to Anthony) then paralally I will test 1.0.3 RC1 also and any new thing come out then I will post update here... Thanks, msp Brian West-3 wrote:

I highly recommend you upgrade to 1.03RC1 or SVN Trunk. Chances are we have already fixed these issues.

/b

On Feb 1, 2009, at 7:35 AM, shehzad p wrote:

...

Hi anthony,

There are so many crash occured, on FS 1.0.1 Server, all of them were same (means BT was same) but this last one was looking totally different so I feel that it should be posted... http://www.nabble.com/file/p21775256/bt_full_new.txt bt_full_new.txt

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

Hi anthony, I Modified the whole architecture of call routing system, Now after getting required routes, script exit and, control comes back to Dialplan, and call is bridged there, And call hangup, CDR is posted to cdr.php file (using xml_cdr). So now there is no blocking statement (bridge or anything like that) in current javascript, It return back control instantly. So, setting up all above architecture... First I tested FS 1.0.1 , It get crashed two times, in interval of 3 to 5 hours and simultaneous call of about 100 to 150. BT is the same as before... http://www.nabble.com/file/p21825226/bt_new_arch.txt bt_new_arch.txt Now I am also testing 1.0.3RC1, and post it back if any found. Thanks msp Clearly you have an issue with your javascript code. You have the Garbage collector blocking in every thread. Are you doing any endless loops in your code where you do not check session.ready() as a condition for continuing the script? any time session.ready() fails you must immediately exit. Are you using session.execute to execute long blocking operations like bridging many calls or entering a conference? You should avoid doing this as all the collective scripts on the system share a common Garbage Collector provided by the JS engine and it can lead to the exact issues you describe if the code is not properly designed. What else does you script do that are things provided by FS such as playing files and executing applications. -- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

Hi anthony, In my previous post I already attached the BT for the testing of FS 1.0.1 Posting it again, please find it on link==>http://www.nabble.com/file/p21825226/bt_new_arch.txt Now I got the same result while testing FS 1.0.3RC1, And its BT is also same... BT Link: http://www.nabble.com/file/p21847332/fs_1_0_3_bt_new_arch.log fs_1_0_3_bt_new_arch.log (Note: Same in the sens the functions listed in the sequence are almost same as before...) Anthony Minessale-2 wrote:

If you still get a crash on SVN trunk please post the bt even if you think it's the same, since it won't be exactly the same, the line numbers etc will be accurate with our development code making it easier to debug.

On Wed, Feb 4, 2009 at 12:38 AM, shehzad p <pmhshz@gmail.com> wrote:

...

Hi anthony,

I Modified the whole architecture of call routing system, Now after getting required routes, script exit and, control comes back to Dialplan, and call is bridged there, And call hangup, CDR is posted to cdr.php file (using xml_cdr).

So now there is no blocking statement (bridge or anything like that) in current javascript, It return back control instantly.

So, setting up all above architecture... First I tested FS 1.0.1 , It get crashed two times, in interval of 3 to 5 hours and simultaneous call of about 100 to 150. BT is the same as before... http://www.nabble.com/file/p21825226/bt_new_arch.txt bt_new_arch.txt

Now I am also testing 1.0.3RC1, and post it back if any found.

Thanks msp

Clearly you have an issue with your javascript code.

You have the Garbage collector blocking in every thread.

Are you doing any endless loops in your code where you do not check session.ready() as a condition for continuing the script?

any time session.ready() fails you must immediately exit.

Are you using session.execute to execute long blocking operations like bridging many calls or entering a conference? You should avoid doing this as all the collective scripts on the system share a common Garbage Collector provided by the JS engine and it can lead to the exact issues you describe if the code is not properly designed.

What else does you script do that are things provided by FS such as playing files and executing applications.

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

HI Brian, Output of ulimit -a and /proc/cpuinfo is attached... http://www.nabble.com/file/p21848148/12_ulimit_and_cpuinfo.log 12_ulimit_and_cpuinfo.log BUT...................... I am running the freeswitch using below command (So ulimit set according to Anthony's previous post): =================================================================================== ulimit -d unlimited; ulimit -f unlimited; ulimit -i unlimited; ulimit -n 999999; ulimit -q unlimited ; ulimit -u unlimited; ulimit -v unlimited ; ulimit -x unlimited; ulimit -s 244; ulimit -l unlimited; freeswitch =================================================================================== Thanks msp Brian West-3 wrote:

Can you give me the output of uname -a and the contents of /proc/ cpuinfo? Not sure I asked for this info already or not.

Thanks, Brian

On Feb 5, 2009, at 2:42 AM, shehzad p wrote:

...

Hi anthony, In my previous post I already attached the BT for the testing of FS 1.0.1 Posting it again, please find it on link==>http://www.nabble.com/file/p21825226/bt_new_arch.txt

Now I got the same result while testing FS 1.0.3RC1, And its BT is also same... BT Link: http://www.nabble.com/file/p21847332/fs_1_0_3_bt_new_arch.log fs_1_0_3_bt_new_arch.log

(Note: Same in the sens the functions listed in the sequence are almost same as before...)

Anthony Minessale-2 wrote:

...

If you still get a crash on SVN trunk please post the bt even if you think it's the same, since it won't be exactly the same, the line numbers etc will be accurate with our development code making it easier to debug.

On Wed, Feb 4, 2009 at 12:38 AM, shehzad p <pmhshz@gmail.com> wrote:

...

Hi anthony,

I Modified the whole architecture of call routing system, Now after getting required routes, script exit and, control comes back to Dialplan, and call is bridged there, And call hangup, CDR is posted to cdr.php file (using xml_cdr).

So now there is no blocking statement (bridge or anything like that) in current javascript, It return back control instantly.

So, setting up all above architecture... First I tested FS 1.0.1 , It get crashed two times, in interval of 3 to 5 hours and simultaneous call of about 100 to 150. BT is the same as before... http://www.nabble.com/file/p21825226/bt_new_arch.txt bt_new_arch.txt

Now I am also testing 1.0.3RC1, and post it back if any found.

Thanks msp

Clearly you have an issue with your javascript code.

You have the Garbage collector blocking in every thread.

Are you doing any endless loops in your code where you do not check session.ready() as a condition for continuing the script?

any time session.ready() fails you must immediately exit.

Are you using session.execute to execute long blocking operations like bridging many calls or entering a conference? You should avoid doing this as all the collective scripts on the system share a common Garbage Collector provided by the JS engine and it can lead to the exact issues you describe if the code is not properly designed.

What else does you script do that are things provided by FS such as playing files and executing applications.

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com

...

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com

...

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org

...

iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org

...

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

Hi Brian, As it can be seen from the system information, there require any change in system or any suggestion... out put of uname -a is : Linux localhost.localdomain 2.6.23.1-42.fc8 #1 SMP Tue Oct 30 13:55:12 EDT 2007 i686 i686 i386 GNU/Linux Thanks, msp shehzad p wrote:

HI Brian,

Output of ulimit -a and /proc/cpuinfo is attached... http://www.nabble.com/file/p21848148/12_ulimit_and_cpuinfo.log 12_ulimit_and_cpuinfo.log

BUT...................... I am running the freeswitch using below command (So ulimit set according to Anthony's previous post): =================================================================================== ulimit -d unlimited; ulimit -f unlimited; ulimit -i unlimited; ulimit -n 999999; ulimit -q unlimited ; ulimit -u unlimited; ulimit -v unlimited ; ulimit -x unlimited; ulimit -s 244; ulimit -l unlimited; freeswitch ===================================================================================

Thanks msp

Brian West-3 wrote:

...

Can you give me the output of uname -a and the contents of /proc/ cpuinfo? Not sure I asked for this info already or not.

Thanks, Brian

On Feb 5, 2009, at 2:42 AM, shehzad p wrote:

...

Hi anthony, In my previous post I already attached the BT for the testing of FS 1.0.1 Posting it again, please find it on link==>http://www.nabble.com/file/p21825226/bt_new_arch.txt

Now I got the same result while testing FS 1.0.3RC1, And its BT is also same... BT Link: http://www.nabble.com/file/p21847332/fs_1_0_3_bt_new_arch.log fs_1_0_3_bt_new_arch.log

(Note: Same in the sens the functions listed in the sequence are almost same as before...)

Anthony Minessale-2 wrote:

...

If you still get a crash on SVN trunk please post the bt even if you think it's the same, since it won't be exactly the same, the line numbers etc will be accurate with our development code making it easier to debug.

On Wed, Feb 4, 2009 at 12:38 AM, shehzad p <pmhshz@gmail.com> wrote:

...

Hi anthony,

I Modified the whole architecture of call routing system, Now after getting required routes, script exit and, control comes back to Dialplan, and call is bridged there, And call hangup, CDR is posted to cdr.php file (using xml_cdr).

So now there is no blocking statement (bridge or anything like that) in current javascript, It return back control instantly.

So, setting up all above architecture... First I tested FS 1.0.1 , It get crashed two times, in interval of 3 to 5 hours and simultaneous call of about 100 to 150. BT is the same as before... http://www.nabble.com/file/p21825226/bt_new_arch.txt bt_new_arch.txt

Now I am also testing 1.0.3RC1, and post it back if any found.

Thanks msp

Clearly you have an issue with your javascript code.

You have the Garbage collector blocking in every thread.

Are you doing any endless loops in your code where you do not check session.ready() as a condition for continuing the script?

any time session.ready() fails you must immediately exit.

Are you using session.execute to execute long blocking operations like bridging many calls or entering a conference? You should avoid doing this as all the collective scripts on the system share a common Garbage Collector provided by the JS engine and it can lead to the exact issues you describe if the code is not properly designed.

What else does you script do that are things provided by FS such as playing files and executing applications.

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com

...

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com

...

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org

...

iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org

...

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

Hi all, I have opened JIRA for the same. http://jira.freeswitch.org/browse/FSCORE-285 One system is Fedora, and another one is Ubuntu. Although fs 1.0.1 was also get crashed in Ubuntu many times. Now 1.0.3.RC1 is loaded on Ubuntu, so this happen again with my Ubuntu I will surely post it. And I am now going to test latest trunk version and post back if any thing found... thanks, msp paul.degt wrote:

Look like you use Fedora. I had a lot of issues with using Fedora as production or load test system, in my opinion it's more like work in progress than a production ready stable linux. If you cannot buy RHEL or SLES use Centos.

shehzad p wrote:

...

Hi Brian,

As it can be seen from the system information, there require any change in system or any suggestion...

out put of uname -a is : Linux localhost.localdomain 2.6.23.1-42.fc8 #1 SMP Tue Oct 30 13:55:12 EDT 2007 i686 i686 i386 GNU/Linux

Thanks, msp

shehzad p wrote:

...

HI Brian,

Output of ulimit -a and /proc/cpuinfo is attached... http://www.nabble.com/file/p21848148/12_ulimit_and_cpuinfo.log 12_ulimit_and_cpuinfo.log

BUT...................... I am running the freeswitch using below command (So ulimit set according to Anthony's previous post): =================================================================================== ulimit -d unlimited; ulimit -f unlimited; ulimit -i unlimited; ulimit -n 999999; ulimit -q unlimited ; ulimit -u unlimited; ulimit -v unlimited ; ulimit -x unlimited; ulimit -s 244; ulimit -l unlimited; freeswitch ===================================================================================

Thanks msp

Brian West-3 wrote:

...

Can you give me the output of uname -a and the contents of /proc/ cpuinfo? Not sure I asked for this info already or not.

Thanks, Brian

On Feb 5, 2009, at 2:42 AM, shehzad p wrote:

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

Hi Anthoy, I just tested the trunk-11575, In one test FS served for 1 Hour and 10 minutes and processed about 28600 sesisons. When it get crashed there was 244 active sessions and cps was 5/200 Here is full BT attached http://www.nabble.com/file/p21765660/fs_bt_full.txt fs_bt_full.txt Thanks, msp Anthony Minessale-2 wrote:

if you are using unix you can use the supplied script

scripts/freeswitch-gcore

to capture a copy of the resident memory and I can have a look perhaps.

Trunk is safe for production as we are in beta stage for a release of 1.0.3 at this time.

On Fri, Jan 30, 2009 at 9:29 AM, shehzad p <pmhshz@gmail.com> wrote:

...

When freeswitch freezes, we can't connect to it to check sps status, but once we were able to connect and at that time it was showing 0/0 sps.

thanks...

shehzad p wrote:

...

Thanks, Anthony

In my previous test sps did not changed, but in recent test sps was dropped to 0 itself (as below). =============================================================== UP 0 years, 0 days, 5 hours, 1 minute, 53 seconds, 878 milliseconds,

190

...

microseconds 5474 session(s) since startup 75 session(s) 0/0 =============================================================

My system is 32 bit. CPU is Intel(R) Xeon(R) CPU X3220 @ 2.40GHz And RAM is 4GB

Output of ulimit -a is: ulimit -a: (set after first test) core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited max nice (-e) 20 file size (blocks, -f) unlimited pending signals (-i) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 999999 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) unlimited max rt priority (-r) unlimited stack size (kbytes, -s) 244 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ===================================================

BTW using trunk on production system is safe?

Warm thanks for kind responses...

Anthony Minessale-2 wrote:

...

When you get it in that state what do you see when you execute

fsctl sps

is the sps a very low number?

Did the sps drop by itself from the value you originally set it to?

Are you using 32 bit?

if so try all of these commands in your shell before starting FS

ulimit -c unlimited ulimit -d unlimited ulimit -f unlimited ulimit -i unlimited ulimit -n 999999 ulimit -q unlimited ulimit -u unlimited ulimit -v unlimited ulimit -x unlimited ulimit -s 244 ulimit -l unlimited

DO NOT put them in a script unless you source the script with . . myscript or they will be undone instantly when the script exits

BTW, I said to try latest trunk not 1.0.2 We can only debug the development code at this point.

On Thu, Jan 29, 2009 at 10:06 AM, shehzad p <pmhshz@gmail.com> wrote:

...

Hi Anthony,

I found interesting result while testing Freeswitch, and it might be cause of freezing out of freeswitch,

I updated my system (as you told) with latest stable version

Freeswitch

...

...

1.0.2 First of all I set sps to 100, Then I sends call approximately 100 per seconds, Freeswitch works fine and handles all the calls very well.

After that I send 130 calls per seconds, and magic happen now, Freeswitch handles first 100 calls only. all the preceding calls were failed (even not appeared in freeswitch console why?)

When I put ngrep trace, System responds with 503 Maximum Calls In Progress. (as below) ########################################################### # U FSFSFSFSFS -> GWGWGWGWGW SIP/2.0 503 Maximum Calls In Progress. Via: SIP/2.0/UDP GWGWGWGWGW;branch=z9hG4bK53eafabb;rport=5060. From: "99999" <sip:99999@GWGWGWGWGW>;tag=as2e10c170. To: <sip:1234567890@FSFSFSFSFS >;tag=K3jSUFrDHpmmB. Call-ID: 0feb229e58afe7be17110deb361dc234@GWGWGWGWGW. CSeq: 102 INVITE. Retry-After: 300. User-Agent: FreeSWITCH-mod_sofia/1.0.2-exported. Accept: application/sdp. Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, PRACK, MESSAGE, SUBSCRIBE, NOTIFY, REFER, UPDATE, REGISTER, INFO, PUBLISH. Supported: timer, precondition, path, replaces. Allow-Events: talk, presence, dialog, call-info, sla, include-session-description, presence.winfo, message-summary, refer. Content-Length: 0. . #####################################################################

Now another issue to note down is that, After all above happened and active calls comes to zero, I just make a single call which also fails with response 503 - Maximum Calls In Progress.

Is this intended behaviour, should I increase SPS to overcome this. or something like bug.

Please let me know what should be the resolution for this.

Thanks, msp

Anthony Minessale-2 wrote:

...

Also remember, Actually completely uninstall and erase /usr/local/freeswitch and

the

...

1.0.1 source tree and freshly install the new one. If you try to upgrade on top of a release with trunk it will cause more problems for you.

On Wed, Jan 28, 2009 at 3:11 AM, Ken Rice <krice@suspicious.org> wrote:

> Upgrade to trunk... Many many issues have been resolved since 1.0.1 was > the > current release > > > > From: shehzad p <pmhshz@gmail.com> > > Reply-To: <freeswitch-users@lists.freeswitch.org> > > Date: Wed, 28 Jan 2009 00:54:13 -0800 (PST) > > To: <freeswitch-users@lists.freeswitch.org> > > Subject: [Freeswitch-users] Freeswitch freezes on increasing call > traffic > > > > > > Hi all, > > > > Yesterday my Freeswitch server faced a problem when call traffic > increased > > to more than 100. > > > > When I start Freeswitch, it works fine and then after some time > > (approximately 15 to 20 minutes) it stops functioning (means no call > is > > being processed, no CLI command is working and it just freezes) until I > > restart the freeswitch. > > > > I am using Freeswitch 1.0.1. > > Debug (gdb) trace as on wiki page > > http://wiki.freeswitch.org/wiki/Debugging_Freeswitch#fs_debug.shis > attached > > http://www.nabble.com/file/p21701744/fs_debgu.txt fs_debgu.txt > > -- > > View this message in context: > > >

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

...

> > p21701744.html > > Sent from the Freeswitch-users mailing list archive at Nabble.com. > > > > > > _______________________________________________ > > Freeswitch-users mailing list > > Freeswitch-users@lists.freeswitch.org > > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > > > UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users > > http://www.freeswitch.org > > > > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm

MSN:anthony_minessale@hotmail.com<MSN%3Aanthony_minessale@hotmail.com>

<MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <

...

MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com> <MSN%253Aanthony_minessale@hotmail.com<MSN%25253Aanthony_minessale@hotmail.com>

...

...

...

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

...

<PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com> <PAYPAL%253Aanthony.minessale@gmail.com<PAYPAL%25253Aanthony.minessale@gmail.com>

...

...

...

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference

sip:888@conference.freeswitch.org<sip%3A888@conference.freeswitch.org>

<sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <

...

sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org> <sip%253A888@conference.freeswitch.org<sip%25253A888@conference.freeswitch.org>

...

...

...

iax:guest@conference.freeswitch.org/888

googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

...

<googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org> <googletalk%253Aconf%252B888@conference.freeswitch.org<googletalk%25253Aconf%25252B888@conference.freeswitch.org>

...

...

...

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users

...

http://www.freeswitch.org

-- View this message in context:

http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017...

...

Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com>< MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com>

...

GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>

...

IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org>< sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org>

...

iax:guest@conference.freeswitch.org/888

googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>

...

pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- View this message in context: http://www.nabble.com/Freeswitch-freezes-on-increasing-call-traffic-tp217017... Sent from the Freeswitch-users mailing list archive at Nabble.com.