mod_openzap stops working after some calls
Hello, since yesterday I do a real life test with FS in a 40 sip extension environment with TDM connection via wanpipe/sangoma A104d. I detected two times the problem, that openzap stops working, while SIP calls worked. Only restarting helped. Maybe reloading of mod_openzap helps as well, but I didn't tested that, yet. I filtered FS logfile for hints of that problem and I found a growing number of this line: 2009-02-06 11:09:51 [INFO] ozmod_isdn.c:706 zap_isdn_931_34() Duplicate SETUP message(?) for Channel 1:21 ~ 1:21 in state DOWN [ignoring] Befor restart I saw such a line for *each* TDM channel and no one was able to dial out. mod_openzap handled around 40 outgoing calls until last restart . Since last restart FS runs good for 2 hours now without any duplicate SETUP in log ... Has anybody similar problems? regards Helmut
I think we have some trouble surviving issues. So when everything is ok we do fine but if something goes wrong we don't recover. We are still missing state timers in the q931. maybe you can use your new pcap thing to see what goes wrong =D On Fri, Feb 6, 2009 at 6:32 AM, Helmut Kuper <helmut.kuper@ewetel.de> wrote:
Hello,
since yesterday I do a real life test with FS in a 40 sip extension environment with TDM connection via wanpipe/sangoma A104d.
I detected two times the problem, that openzap stops working, while SIP calls worked. Only restarting helped. Maybe reloading of mod_openzap helps as well, but I didn't tested that, yet. I filtered FS logfile for hints of that problem and I found a growing number of this line:
2009-02-06 11:09:51 [INFO] ozmod_isdn.c:706 zap_isdn_931_34() Duplicate SETUP message(?) for Channel 1:21 ~ 1:21 in state DOWN [ignoring]
Befor restart I saw such a line for *each* TDM channel and no one was able to dial out. mod_openzap handled around 40 outgoing calls until last restart .
Since last restart FS runs good for 2 hours now without any duplicate SETUP in log ...
Has anybody similar problems?
regards Helmut
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Hello Anthony, :D yes that's what I'm doing ... beneath some code changes in openzap ... So I found a real timestamp in pcap is quite usefull if you have more than one call at a time ... I added that function today. It uses "libapr-1" functions. Unfortunately I introduced a dependency to libs/apr to openzap by that. If it delivers micro seconds, maybe it's better to use zap_time_now(). Have to check that tomorrow. I agree there are some problems in maintaining channel states correctly. Once a day I have to restart FS. I get "TOMANYCALLS" errors, no matching channels on RELEASE, SETUP duplicates and "oz dump 1" shows more and more channels with states other than DOWN, even, when no current calls are there. I did some timebased changes in ozmod_isdn SETUP handling and hope it helps out until state timers a available. If it works I would like to upload it to trunk, if you allow. regards helmut On 06.02.2009 17:02, Anthony Minessale wrote:
I think we have some trouble surviving issues. So when everything is ok we do fine but if something goes wrong we don't recover. We are still missing state timers in the q931.
maybe you can use your new pcap thing to see what goes wrong =D
We can not add apr dependency in openzap, we should use the native openzap calls instead. If there is anything you NEED that you don't have, please let me know and we will try to add replacement functions. Mike On Feb 9, 2009, at 1:17 PM, Helmut Kuper wrote:
Hello Anthony,
:D yes that's what I'm doing ... beneath some code changes in openzap ... So I found a real timestamp in pcap is quite usefull if you have more than one call at a time ... I added that function today. It uses "libapr-1" functions. Unfortunately I introduced a dependency to libs/apr to openzap by that. If it delivers micro seconds, maybe it's better to use zap_time_now(). Have to check that tomorrow.
I agree there are some problems in maintaining channel states correctly. Once a day I have to restart FS. I get "TOMANYCALLS" errors, no matching channels on RELEASE, SETUP duplicates and "oz dump 1" shows more and more channels with states other than DOWN, even, when no current calls are there. I did some timebased changes in ozmod_isdn SETUP handling and hope it helps out until state timers a available. If it works I would like to upload it to trunk, if you allow.
regards helmut
On 06.02.2009 17:02, Anthony Minessale wrote:
I think we have some trouble surviving issues. So when everything is ok we do fine but if something goes wrong we don't recover. We are still missing state timers in the q931.
maybe you can use your new pcap thing to see what goes wrong =D
Hi Mike, I would like to have a function which gives current time in sec, usec since unix epoch. It's only for pcap timestamp. I found a zap_time_now() somewhere in openzap maybe it helps ... regards helmut
Hi Mike, I removed the apr dependency for timestamp and use the function openzap delivers. Works good so far. For unstabilities in openzap and Q931: On my side main problem seems to be, that channels for inbound traffic sometimes not be freed during runtime. Maybe our remote TDM end (AVAYA) simply doesn't release calls as it should, maybe openzap doesn't catch all q931 messages. I added a hack, which forces channels which are in TERMINATE or beyond *AND* this state is older than e.g. 500ms for inbound SETUP to DOWN. Openzap uses those down forced channels. I have the patch in production and I saw the first aid hack serveral times "InUse" channels freeing. "oz dump 1" shows channel states which hung in TERMINATE or above are down serveral minutes later :) This is just a quite brutal hack because it assumes that the remote TDM end is able to recover channels as well. regards helmut
This is just a quite brutal hack because it assumes that the remote TDM end is able to recover channels as well.
Could you possibly modify your hack to be less brutal? For example, could it send a STATUS ENQ to the far end for the channel in question? Just curious. -MC
you should come into irc on irc.freenode.net and join #openzap Stefan Knoblich (stkn) in the channel is doing some work on implementation actual q931 timers which would solve the problem the real way. Maybe you could collaberate with him. On Wed, Feb 11, 2009 at 11:43 AM, Michael Collins <msc@freeswitch.org>wrote:
This is just a quite brutal hack because it assumes that the remote TDM end is able to recover channels as well.
Could you possibly modify your hack to be less brutal? For example, could it send a STATUS ENQ to the far end for the channel in question? Just curious. -MC
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Hi Mike, at least for incoming calls this shouldn't be too brutal, cause far end seems to know that the channel should be free otherwise it never would allocate it. By now the hack works at least for me quite good. Nobody from AVAYA side moaned about it, yet. But I have to wait one or two further days to be sure ... I guess I have to talk to stkn in irc to get an idea how long I have to use it. regards helmut On 11.02.2009 18:43, Michael Collins wrote:
This is just a quite brutal hack because it assumes that the remote TDM end is able to recover channels as well.
Could you possibly modify your hack to be less brutal? For example, could it send a STATUS ENQ to the far end for the channel in question? Just curious. -MC
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hello, just to keep you informed about this problem. As mentioned I added a hack to free allocated channels depending on last event time. I enhanced oz dump as well to display "last event time" and "InUse"-Flag. What I found is this: 1. InUse channel flag wasn't set for inbound calls. I fixed that as far as I understood the openzap code ;) and I tested the patch successfully for 7 days now... 2. In my setup (AVAYA as remote end for a E1) channels tend to hang in a state <> DOWN after terminating a call. Then I found TOMANYCALLS entries in FS log. I had to restart FS resp. openzap module. The hack I added is in production and works for 7 days now. No channels hanging anymore. Of course, the hack is not the final solution, but it seems to solve at least my problems in production until openzap has state timers. If the board wants, I can upload the hack as well. regards Helmut On 12.02.2009 14:03, Helmut Kuper wrote:
Hi Mike,
at least for incoming calls this shouldn't be too brutal, cause far end seems to know that the channel should be free otherwise it never would allocate it. By now the hack works at least for me quite good. Nobody from AVAYA side moaned about it, yet. But I have to wait one or two further days to be sure ... I guess I have to talk to stkn in irc to get an idea how long I have to use it.
regards helmut
participants (4)
-
Anthony Minessale
-
Helmut Kuper
-
Michael Collins
-
Michael Jerris