[Freeswitch-users] FS fails to restart when TLS endpoints are registered (Solaris 11.3)

Tue Feb 13 13:06:00 UTC 2018

Thanks Alexey for your help in diagnosing the problem.
I dug further into the TCP stack and found out we are having 2 separate
restart problems.

The first problem is not that the socket is failing to be released, but
apparently Sofia is trying to open the port without the SO_REUSEADDR
option, which means FS must wait for all connection to close first.
So as long as there are any TCP connections to those ports in the TIME_WAIT
state, FS will not start Sofia.
I tested with netcat and that program can bind to the port regardless of
the state of old connections, so I suspect this is a compile error.

The second problem is that the TLS and TCP registrations in
sofia_reg_internal.db cause FS to throw the assertion error “Assertion
failed: orq->orq_queue, file nta.c, line 8101”.
UDP registrations are fine.
Does FS try to continue these registrations assuming that the connection is
still up or reopen a connection to the same port?
The assert line is found in nta.c’s outgoing_create function. It seems that
the outgoing message queue is uninitialized, so the assertion error is
likely more a symptom of some other problem, instead of being the cause of
the problem.
How should I go about debugging the outgoing message queue?

Thanks in advance,
Mikko

---------- Edelleenlähetetty viesti ----------

> From: Alexey Sibyakin <alex at freeswitch.com>
> To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> Cc:
> Bcc:
> Date: Tue, 13 Feb 2018 17:23:06 +0900
> Subject: Re: [Freeswitch-users] FS fails to restart when TLS endpoints are
> registered (Solaris 11.3)
> It looks like Sofia is not releasing ip on Solaris. If so it's a bug and
> it should be reported to Sofia's bug tracker or Solaris or both. If you can
> reproduce this problem on supported OS (Debian 8 at this moment) you can
> open an issue on FreeSWITCH Jira.
>
> Consider to read this first: https://freeswitch.org/
> confluence/display/FREESWITCH/Reporting+Bugs+to+JIRA
>
> On Tue, Feb 13, 2018 at 2:30 AM, GBC Networks Oy <info at gbc.fi> wrote:
>
>> We're running FS on Solaris 11.3 and it is working perfectly, except that
>> we're unable to restart FS without deleting the TLS registrations from
>> sofia_reg_internal.db.
>> Our FS version is FreeSWITCH Version 1.9.0+git~20171010T231940Z~5940b04d4e~64bit
>> (git 5940b04 2017-10-10 23:19:40Z 64bit)
>>
>> Here are the symptoms:
>> 1.    Service stops FS with fs_cli -x "fsctl shutdown now"
>> 2.    FS stops and the service is restarted
>> 3.    FS starts up and fs_cli is working, but ” sofia status” is empty
>> The console shows:
>> freeswitch at fs> 2018-02-12 17:06:23.875991 [ERR] sofia.c:3221 Error
>> Creating SIP UA for profile: internal (sip:mod_sofia at 123.123.123.123
>> :5060;maddr=123.123.123.123;transport=udp,tcp) ATTEMPT 3 (RETRY IN 5
>> SEC)
>>
>> 2018-02-12 17:06:23.875991 [ERR] sofia.c:3231 Error Creating SIP UA for
>> profile: internal (sip:mod_sofia at 123.123.123.123
>> :5060;maddr=123.123.123.123;transport=udp,tcp)
>>
>>
>>
>> The likely causes for this are:
>>
>>
>> 1) Another application is already listening on the specified
>> address.
>>
>> 2) The IP the profile is attempting to bind to is not local to this
>> system.
>>
>> Note: the IP is correct and nothing is listening on the port.
>>
>> 4.    Next we stop FS and restart
>> This time the console shows:
>> 2018-02-12 13:33:51.132676 [INFO] switch_core_sqldb.c:1693 sofia:internal
>> Starting SQL thread.
>> 2018-02-12 13:33:51.132770 [DEBUG] sofia.c:3373 Starting thread for
>> internal
>> 2018-02-12 13:33:51.132777 [DEBUG] sofia.c:3020 Launching worker thread
>> for internal
>> 2018-02-12 13:33:51.138255 [NOTICE] sofia_reg.c:3398 Added gateway
>> '5e4ac47f-6ce4-462d-9247-565711fe3d4e' to profile 'internal'
>> 2018-02-12 13:33:51.138405 [NOTICE] sofia_reg.c:3398 Added gateway
>> 'b2563fad-0051-4067-abc3-6a431e05481e' to profile 'internal'
>> 2018-02-12 13:33:51.247731 [DEBUG] freeswitch_lua.cpp:372 DBH handle
>> 125b300 Connected.
>> 2018-02-12 13:33:51.254128 [DEBUG] freeswitch_lua.cpp:401 DBH handle
>> 125b300 released.
>> Assertion failed: orq->orq_queue, file nta.c, line 8101
>>
>> 5.    sqlite3 sofia_reg_internal.db "DELETE from sip_registrations WHERE
>> status LIKE '%TLS%'"
>> 6.    FS starts normally
>>
>>
>> Here is a registration (non-TLS) that doesn’t break sofia:
>> # sqlite3 -header sofia_reg_internal.db "SELECT * from sip_registrations"
>> call_id|sip_user|sip_host|presence_hosts|contact|status|ping
>> _status|ping_count|ping_time|force_ping|rpid|expires|ping_
>> expires|user_agent|server_user|server_host|profile_name|
>> hostname|network_ip|network_port|sip_username|sip_realm|
>> mwi_user|mwi_host|orig_server_host|orig_hostname|sub_host
>> 313531383139323735393532323132-m5mp9qlsdu27|USERNAME|fs.domain.com||"FS"
>> <sip:USERNAME at REMOTE-IP:2054>|Registered(UDP)|Reachable|3|39
>> 824|1|unknown|1518455533|1518453577|snom320/8.7.5.35|USERNAM
>> E||internal|fs|REMOTE-IP|2054|USERNAME|fs.domain.com|USERNAME|
>> fs.domain.com||fs|fs.domain.com
>>
>> Here is a registration (TLS) that breaks the restart:
>> # sqlite3 -header sofia_reg_internal.db "SELECT * from
>> sip_registrations"
>> call_id|sip_user|sip_host|presence_hosts|contact|status|ping
>> _status|ping_count|ping_time|force_ping|rpid|expires|ping_
>> expires|user_agent|server_user|server_host|profile_name|
>> hostname|network_ip|network_port|sip_username|sip_realm|
>> mwi_user|mwi_host|orig_server_host|orig_hostname|sub_host
>> 313531383139323735393532323132-m5mp9qlsdu27|USERNAME|fs.domain.com||"FS"
>> <sip:USERNAME at 192.168.6.236:2170;transport=tls;fs_nat=yes;fs
>> _path=sip%3AUSERNAME%40REMOTE-IP%3A2170%3Btransport%3Dtls>|
>> Registered(TLS-NAT)|Reachable|0||1|unknown|1518457302|1518453664|snom320/
>> 8.7.5.35|USERNAME||internal|fs|REMOTE-IP|2170|USERNAME|fs.domain.com
>> |USERNAME|fs.domain.com||fs|fs.domain.com
>>
>>
>> Best regards,
>> Mikko
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
>
> --
>
> Alex Sibyakin | Support Engineer
> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
> Email: alex at freeswitch.com
> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
> Need commercial support? Contact sales at freeswitch.com for details.
>
> _______________________________________________
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20180213/c94a5374/attachment.html>