Hello I got problems with hanging spidermonkey sessions and need some advice on how to debug them. I've made a javascript queue application that uses mod_spidermonkey_socket. It works fine for a while, but after some calls I noticed that calls didnt get transferred to agents. The reason was that earlier calls had not been terminated properly. freeswitch@test1> hupall 2009-01-15 12:15:04 [CRIT] switch_core_session.c:147 switch_core_session_hupall() Giving up with 8 sessions remaining API CALL [hupall()] output: +OK hangup all channels with cause MANAGER_REQUEST freeswitch@test1> show calls API CALL [show(calls)] output: 0 total. As you can see, 8 sessions are alive, but none of them are listed as calls. What kind of logs should I turn on to see what is happening with those sessions? Thanks, Jonas
http://wiki.freeswitch.org/wiki/Report_Issue_Checklist Please open a jira and include your script and a test case. /b On Jan 15, 2009, at 5:20 AM, Jonas Gauffin wrote:
Hello
I got problems with hanging spidermonkey sessions and need some advice on how to debug them.
I've made a javascript queue application that uses mod_spidermonkey_socket. It works fine for a while, but after some calls I noticed that calls didnt get transferred to agents. The reason was that earlier calls had not been terminated properly.
freeswitch@test1> hupall 2009-01-15 12:15:04 [CRIT] switch_core_session.c:147 switch_core_session_hupall() Giving up with 8 sessions remaining API CALL [hupall()] output: +OK hangup all channels with cause MANAGER_REQUEST
freeswitch@test1> show calls API CALL [show(calls)] output:
0 total.
As you can see, 8 sessions are alive, but none of them are listed as calls. What kind of logs should I turn on to see what is happening with those sessions?
Thanks, Jonas _________
do you have any loops in your code that might not check for session.ready() in a exit when its not true. The symptoms you posted would be consistent with held readlocks so if you got a gcore (or windows equiv) of the process you might be able to see what threads where doing what to hang on to the read lock. also are you creating sessions in the script then executing app with them, beware of this because the thread of the script is used to execute apps on a session created that way and not the session thread. On Thu, Jan 15, 2009 at 5:20 AM, Jonas Gauffin <jonas.gauffin@gmail.com>wrote:
Hello I got problems with hanging spidermonkey sessions and need some advice on how to debug them.
I've made a javascript queue application that uses mod_spidermonkey_socket. It works fine for a while, but after some calls I noticed that calls didnt get transferred to agents. The reason was that earlier calls had not been terminated properly.
freeswitch@test1> hupall 2009-01-15 12:15:04 [CRIT] switch_core_session.c:147 switch_core_session_hupall() Giving up with 8 sessions remaining API CALL [hupall()] output: +OK hangup all channels with cause MANAGER_REQUEST
freeswitch@test1> show calls API CALL [show(calls)] output:
0 total.
As you can see, 8 sessions are alive, but none of them are listed as calls. What kind of logs should I turn on to see what is happening with those sessions?
Thanks, Jonas
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
I've got a loop, but the first thing checked in each iteration is if session.ready() returns false (and in that case exit the loop). I do create sessions in the script: create, try to originate to a destination and then finally bridge together the caller and the new session. I'll try to give you more details during the day. On Fri, Jan 16, 2009 at 12:48 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
do you have any loops in your code that might not check for session.ready() in a exit when its not true.
The symptoms you posted would be consistent with held readlocks so if you got a gcore (or windows equiv) of the process you might be able to see what threads where doing what to hang on to the read lock.
also are you creating sessions in the script then executing app with them, beware of this because the thread of the script is used to execute apps on a session created that way and not the session thread.
On Thu, Jan 15, 2009 at 5:20 AM, Jonas Gauffin <jonas.gauffin@gmail.com>wrote:
Hello I got problems with hanging spidermonkey sessions and need some advice on how to debug them.
I've made a javascript queue application that uses mod_spidermonkey_socket. It works fine for a while, but after some calls I noticed that calls didnt get transferred to agents. The reason was that earlier calls had not been terminated properly.
freeswitch@test1> hupall 2009-01-15 12:15:04 [CRIT] switch_core_session.c:147 switch_core_session_hupall() Giving up with 8 sessions remaining API CALL [hupall()] output: +OK hangup all channels with cause MANAGER_REQUEST
freeswitch@test1> show calls API CALL [show(calls)] output:
0 total.
As you can see, 8 sessions are alive, but none of them are listed as calls. What kind of logs should I turn on to see what is happening with those sessions?
Thanks, Jonas
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I've found the problem. one js thread wait in socket.read (mod_spidermonkey_socket) on data. That caller have hangup, which means that the garbage collector waits on it to close. All new javascript sessions waits in JS_AWAIT_GC_DONE for the garbage collector to be done before proceeding (which means that all new javascript calls don't do anything after being launched). My server will not send anything until an agent gets free or the session hangs up (detects it through the event socket). And the event socket will not send that the session has been hangup until the socket have received anything (and the script can exit). So it's kind of deadlock between my server and the spidermonkey_socket. Is it possible to add an option to socket.read to make it abort if the session have been closed? I know that I wrote mod_spidermonkey_socket from the start, but I can't figure out how to do it. Will new sessions always wait on old ones to be garbage collected properly? For instance, what happens if a script have a lenghty post process after caller have hang up? On Fri, Jan 16, 2009 at 9:38 AM, Jonas Gauffin <jonas.gauffin@gmail.com>wrote:
I've got a loop, but the first thing checked in each iteration is if session.ready() returns false (and in that case exit the loop). I do create sessions in the script: create, try to originate to a destination and then finally bridge together the caller and the new session.
I'll try to give you more details during the day.
On Fri, Jan 16, 2009 at 12:48 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
do you have any loops in your code that might not check for session.ready() in a exit when its not true.
The symptoms you posted would be consistent with held readlocks so if you got a gcore (or windows equiv) of the process you might be able to see what threads where doing what to hang on to the read lock.
also are you creating sessions in the script then executing app with them, beware of this because the thread of the script is used to execute apps on a session created that way and not the session thread.
On Thu, Jan 15, 2009 at 5:20 AM, Jonas Gauffin <jonas.gauffin@gmail.com>wrote:
Hello I got problems with hanging spidermonkey sessions and need some advice on how to debug them.
I've made a javascript queue application that uses mod_spidermonkey_socket. It works fine for a while, but after some calls I noticed that calls didnt get transferred to agents. The reason was that earlier calls had not been terminated properly.
freeswitch@test1> hupall 2009-01-15 12:15:04 [CRIT] switch_core_session.c:147 switch_core_session_hupall() Giving up with 8 sessions remaining API CALL [hupall()] output: +OK hangup all channels with cause MANAGER_REQUEST
freeswitch@test1> show calls API CALL [show(calls)] output:
0 total.
As you can see, 8 sessions are alive, but none of them are listed as calls. What kind of logs should I turn on to see what is happening with those sessions?
Thanks, Jonas
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
All long running non js code should be wrapped in the suspend/resume gc stuff. For example: cb_state.ret = BOOLEAN_TO_JSVAL(JS_FALSE); cb_state.saveDepth = JS_SuspendRequest(cx); args.input_callback = dtmf_func; args.buf = bp; args.buflen = len; switch_ivr_sleep(jss->session, ms, sync, &args); JS_ResumeRequest(cx, cb_state.saveDepth); I think this is your issue. Can you please file a bug on jira for this issue (even better with a patch) Mike On Jan 16, 2009, at 5:54 AM, Jonas Gauffin wrote:
I've found the problem. one js thread wait in socket.read (mod_spidermonkey_socket) on data. That caller have hangup, which means that the garbage collector waits on it to close.
All new javascript sessions waits in JS_AWAIT_GC_DONE for the garbage collector to be done before proceeding (which means that all new javascript calls don't do anything after being launched).
My server will not send anything until an agent gets free or the session hangs up (detects it through the event socket). And the event socket will not send that the session has been hangup until the socket have received anything (and the script can exit). So it's kind of deadlock between my server and the spidermonkey_socket.
Is it possible to add an option to socket.read to make it abort if the session have been closed? I know that I wrote mod_spidermonkey_socket from the start, but I can't figure out how to do it.
Will new sessions always wait on old ones to be garbage collected properly? For instance, what happens if a script have a lenghty post process after caller have hang up?
On Fri, Jan 16, 2009 at 9:38 AM, Jonas Gauffin <jonas.gauffin@gmail.com
wrote: I've got a loop, but the first thing checked in each iteration is if session.ready() returns false (and in that case exit the loop).
I do create sessions in the script: create, try to originate to a destination and then finally bridge together the caller and the new session.
I'll try to give you more details during the day.
On Fri, Jan 16, 2009 at 12:48 AM, Anthony Minessale <anthony.minessale@gmail.com
wrote: do you have any loops in your code that might not check for session.ready() in a exit when its not true.
The symptoms you posted would be consistent with held readlocks so if you got a gcore (or windows equiv) of the process you might be able to see what threads where doing what to hang on to the read lock.
also are you creating sessions in the script then executing app with them, beware of this because the thread of the script is used to execute apps on a session created that way and not the session thread.
On Thu, Jan 15, 2009 at 5:20 AM, Jonas Gauffin <jonas.gauffin@gmail.com
wrote: Hello
I got problems with hanging spidermonkey sessions and need some advice on how to debug them.
I've made a javascript queue application that uses mod_spidermonkey_socket. It works fine for a while, but after some calls I noticed that calls didnt get transferred to agents. The reason was that earlier calls had not been terminated properly.
freeswitch@test1> hupall 2009-01-15 12:15:04 [CRIT] switch_core_session.c:147 switch_core_session_hupall() Giving up with 8 sessions remaining API CALL [hupall()] output: +OK hangup all channels with cause MANAGER_REQUEST
freeswitch@test1> show calls API CALL [show(calls)] output:
0 total.
As you can see, 8 sessions are alive, but none of them are listed as calls. What kind of logs should I turn on to see what is happening with those sessions?
Thanks, Jonas
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
FYI, I opened http://jira.freeswitch.org/browse/MODLANG-97 on this issue. -MC On Fri, Jan 16, 2009 at 9:03 AM, Michael Jerris <mike@jerris.com> wrote:
All long running non js code should be wrapped in the suspend/resume gc stuff. For example: cb_state.ret = BOOLEAN_TO_JSVAL(JS_FALSE); cb_state.saveDepth = JS_SuspendRequest(cx); args.input_callback = dtmf_func; args.buf = bp; args.buflen = len; switch_ivr_sleep(jss->session, ms, sync, &args); JS_ResumeRequest(cx, cb_state.saveDepth); I think this is your issue. Can you please file a bug on jira for this issue (even better with a patch) Mike
On Jan 16, 2009, at 5:54 AM, Jonas Gauffin wrote:
I've found the problem. one js thread wait in socket.read (mod_spidermonkey_socket) on data. That caller have hangup, which means that the garbage collector waits on it to close.
All new javascript sessions waits in JS_AWAIT_GC_DONE for the garbage collector to be done before proceeding (which means that all new javascript calls don't do anything after being launched). My server will not send anything until an agent gets free or the session hangs up (detects it through the event socket). And the event socket will not send that the session has been hangup until the socket have received anything (and the script can exit). So it's kind of deadlock between my server and the spidermonkey_socket. Is it possible to add an option to socket.read to make it abort if the session have been closed? I know that I wrote mod_spidermonkey_socket from the start, but I can't figure out how to do it. Will new sessions always wait on old ones to be garbage collected properly? For instance, what happens if a script have a lenghty post process after caller have hang up? On Fri, Jan 16, 2009 at 9:38 AM, Jonas Gauffin <jonas.gauffin@gmail.com> wrote:
I've got a loop, but the first thing checked in each iteration is if session.ready() returns false (and in that case exit the loop). I do create sessions in the script: create, try to originate to a destination and then finally bridge together the caller and the new session. I'll try to give you more details during the day. On Fri, Jan 16, 2009 at 12:48 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
do you have any loops in your code that might not check for session.ready() in a exit when its not true.
The symptoms you posted would be consistent with held readlocks so if you got a gcore (or windows equiv) of the process you might be able to see what threads where doing what to hang on to the read lock.
also are you creating sessions in the script then executing app with them, beware of this because the thread of the script is used to execute apps on a session created that way and not the session thread.
On Thu, Jan 15, 2009 at 5:20 AM, Jonas Gauffin <jonas.gauffin@gmail.com> wrote:
Hello I got problems with hanging spidermonkey sessions and need some advice on how to debug them. I've made a javascript queue application that uses mod_spidermonkey_socket. It works fine for a while, but after some calls I noticed that calls didnt get transferred to agents. The reason was that earlier calls had not been terminated properly. freeswitch@test1> hupall 2009-01-15 12:15:04 [CRIT] switch_core_session.c:147 switch_core_session_hupall() Giving up with 8 sessions remaining API CALL [hupall()] output: +OK hangup all channels with cause MANAGER_REQUEST
freeswitch@test1> show calls API CALL [show(calls)] output: 0 total.
As you can see, 8 sessions are alive, but none of them are listed as calls. What kind of logs should I turn on to see what is happening with those sessions? Thanks, Jonas _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
This should now be fixed in svn trunk. Please re-test this with trunk and confirm that all is working correctly now. Mike On Jan 16, 2009, at 12:03 PM, Michael Jerris wrote:
All long running non js code should be wrapped in the suspend/resume gc stuff. For example:
cb_state.ret = BOOLEAN_TO_JSVAL(JS_FALSE); cb_state.saveDepth = JS_SuspendRequest(cx); args.input_callback = dtmf_func; args.buf = bp; args.buflen = len; switch_ivr_sleep(jss->session, ms, sync, &args); JS_ResumeRequest(cx, cb_state.saveDepth);
I think this is your issue. Can you please file a bug on jira for this issue (even better with a patch)
Mike
On Jan 16, 2009, at 5:54 AM, Jonas Gauffin wrote:
I've found the problem. one js thread wait in socket.read (mod_spidermonkey_socket) on data. That caller have hangup, which means that the garbage collector waits on it to close.
All new javascript sessions waits in JS_AWAIT_GC_DONE for the garbage collector to be done before proceeding (which means that all new javascript calls don't do anything after being launched).
My server will not send anything until an agent gets free or the session hangs up (detects it through the event socket). And the event socket will not send that the session has been hangup until the socket have received anything (and the script can exit). So it's kind of deadlock between my server and the spidermonkey_socket.
Is it possible to add an option to socket.read to make it abort if the session have been closed? I know that I wrote mod_spidermonkey_socket from the start, but I can't figure out how to do it.
Will new sessions always wait on old ones to be garbage collected properly? For instance, what happens if a script have a lenghty post process after caller have hang up?
On Fri, Jan 16, 2009 at 9:38 AM, Jonas Gauffin <jonas.gauffin@gmail.com
wrote: I've got a loop, but the first thing checked in each iteration is if session.ready() returns false (and in that case exit the loop).
I do create sessions in the script: create, try to originate to a destination and then finally bridge together the caller and the new session.
I'll try to give you more details during the day.
On Fri, Jan 16, 2009 at 12:48 AM, Anthony Minessale <anthony.minessale@gmail.com
wrote: do you have any loops in your code that might not check for session.ready() in a exit when its not true.
The symptoms you posted would be consistent with held readlocks so if you got a gcore (or windows equiv) of the process you might be able to see what threads where doing what to hang on to the read lock.
also are you creating sessions in the script then executing app with them, beware of this because the thread of the script is used to execute apps on a session created that way and not the session thread.
On Thu, Jan 15, 2009 at 5:20 AM, Jonas Gauffin <jonas.gauffin@gmail.com
wrote: Hello
I got problems with hanging spidermonkey sessions and need some advice on how to debug them.
I've made a javascript queue application that uses mod_spidermonkey_socket. It works fine for a while, but after some calls I noticed that calls didnt get transferred to agents. The reason was that earlier calls had not been terminated properly.
freeswitch@test1> hupall 2009-01-15 12:15:04 [CRIT] switch_core_session.c:147 switch_core_session_hupall() Giving up with 8 sessions remaining API CALL [hupall()] output: +OK hangup all channels with cause MANAGER_REQUEST
freeswitch@test1> show calls API CALL [show(calls)] output:
0 total.
As you can see, 8 sessions are alive, but none of them are listed as calls. What kind of logs should I turn on to see what is happening with those sessions?
Thanks, Jonas
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (5)
-
Anthony Minessale
-
Brian West
-
Jonas Gauffin
-
Michael Collins
-
Michael Jerris