info about sofia sip dns cache and graylisting?
Hello all, we have a problem with a 1.4-version of Freeswitch that FS goes to 100% cpu. The problem has been traced to a dns-cache-problem. The dns-cache contains srv-records, which should have 2 a-records, but they have 100's of a-records, which are mostly the same (some with a different priority). This causes FS do to a lot of compares (to sort the records) and it goes to 100% cpu in the end. The priority almost never is the original priority, so I was thinking that the cache is maybe growing because of graylisting of entries, causing the priority to increase and the ttl to be reset. The code is however very difficult to follow as it is all event-based and part of big C-structs in sofia-sip. A bug in the gdb-debugger causing static variables to be almost unreadable makes life not much easier. The graylisting policy, as well as the dns cache policy is not really documented as far as I can find, so any info about this would be great. Anyway, my current thought is that the dns-cache is maybe growing, becuase the graylisting-code sets the priority to a higher value and resets the ttl to a certain value (no idea which one, but the default is 10 minutes and the maximum a day). The srv-record-compare-function however checks also using priority and weight and returns a negative response if they are different. I would think that fs maybe then readds the original srv-records (with priority 0) when it requeries the dns. Those entries will also increase in priority and ttl when they are added, causing the cache to keep growing ... Nscd is also being used for dns-caching, which doesn't help, but I'm not sure if that is the problem as sofia_dig seems to return valid values (except maybe ttl). I would think that ttl is never touched and that the cached is cleared when the ttl expires, but I have no idea when the dns-cache is cleared in FS... I know this is maybe a lot of text, but any info on this topic (sofia dns cache) is welcome. Thanks, Michel
can you get some logs, sofia loglevel all 9, and file a JIRA and attach the logs and detailed info to the JIRA please. Also verify which rev of FreeSWITCH you're running when filing the JIRA. On Fri, May 8, 2015 at 9:34 AM, Michel Brabants <michel.brabants@gmail.com> wrote:
Hello all,
we have a problem with a 1.4-version of Freeswitch that FS goes to 100% cpu. The problem has been traced to a dns-cache-problem. The dns-cache contains srv-records, which should have 2 a-records, but they have 100's of a-records, which are mostly the same (some with a different priority). This causes FS do to a lot of compares (to sort the records) and it goes to 100% cpu in the end. The priority almost never is the original priority, so I was thinking that the cache is maybe growing because of graylisting of entries, causing the priority to increase and the ttl to be reset.
The code is however very difficult to follow as it is all event-based and part of big C-structs in sofia-sip. A bug in the gdb-debugger causing static variables to be almost unreadable makes life not much easier. The graylisting policy, as well as the dns cache policy is not really documented as far as I can find, so any info about this would be great.
Anyway, my current thought is that the dns-cache is maybe growing, becuase the graylisting-code sets the priority to a higher value and resets the ttl to a certain value (no idea which one, but the default is 10 minutes and the maximum a day). The srv-record-compare-function however checks also using priority and weight and returns a negative response if they are different. I would think that fs maybe then readds the original srv-records (with priority 0) when it requeries the dns. Those entries will also increase in priority and ttl when they are added, causing the cache to keep growing ... Nscd is also being used for dns-caching, which doesn't help, but I'm not sure if that is the problem as sofia_dig seems to return valid values (except maybe ttl).
I would think that ttl is never touched and that the cached is cleared when the ttl expires, but I have no idea when the dns-cache is cleared in FS...
I know this is maybe a lot of text, but any info on this topic (sofia dns cache) is welcome.
Thanks,
Michel
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com ClueCon 2015 Call for Speakers <https://www.cluecon.com/call-for-speakers/> | Register <https://freeswitch.com/cart.php?gid=1> TODAY! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
Hey Brian, thanks for the reply. I created a new ticket " https://freeswitch.org/jira/browse/FS-7543" with my take on the issue and how to resolve it. Michel On Fri, May 8, 2015 at 4:38 PM, Brian West <brian@freeswitch.org> wrote:
can you get some logs, sofia loglevel all 9, and file a JIRA and attach the logs and detailed info to the JIRA please.
Also verify which rev of FreeSWITCH you're running when filing the JIRA.
On Fri, May 8, 2015 at 9:34 AM, Michel Brabants <michel.brabants@gmail.com
wrote:
Hello all,
we have a problem with a 1.4-version of Freeswitch that FS goes to 100% cpu. The problem has been traced to a dns-cache-problem. The dns-cache contains srv-records, which should have 2 a-records, but they have 100's of a-records, which are mostly the same (some with a different priority). This causes FS do to a lot of compares (to sort the records) and it goes to 100% cpu in the end. The priority almost never is the original priority, so I was thinking that the cache is maybe growing because of graylisting of entries, causing the priority to increase and the ttl to be reset.
The code is however very difficult to follow as it is all event-based and part of big C-structs in sofia-sip. A bug in the gdb-debugger causing static variables to be almost unreadable makes life not much easier. The graylisting policy, as well as the dns cache policy is not really documented as far as I can find, so any info about this would be great.
Anyway, my current thought is that the dns-cache is maybe growing, becuase the graylisting-code sets the priority to a higher value and resets the ttl to a certain value (no idea which one, but the default is 10 minutes and the maximum a day). The srv-record-compare-function however checks also using priority and weight and returns a negative response if they are different. I would think that fs maybe then readds the original srv-records (with priority 0) when it requeries the dns. Those entries will also increase in priority and ttl when they are added, causing the cache to keep growing ... Nscd is also being used for dns-caching, which doesn't help, but I'm not sure if that is the problem as sofia_dig seems to return valid values (except maybe ttl).
I would think that ttl is never touched and that the cached is cleared when the ttl expires, but I have no idea when the dns-cache is cleared in FS...
I know this is maybe a lot of text, but any info on this topic (sofia dns cache) is welcome.
Thanks,
Michel
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
*Brian West* brian@freeswitch.org
*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com
ClueCon 2015 Call for Speakers <https://www.cluecon.com/call-for-speakers/> | Register <https://freeswitch.com/cart.php?gid=1> TODAY! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>
*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (2)
-
Brian West
-
Michel Brabants