Greetings, Has anyone seen an instance where freeswitch gets hung with the following stack trace? I have about 400 threads in this state and eventually FS (1.6.19) just stops responding. I am running FS against MySQL 5.6.38 on CentOS using ODBC. Guessing if this is known, it probably isn't fixed in 1.6.20... but I can hope. Any input on what a possible solution would be greatly appreciated. The MySQL server and FS are running on the same host connecting via localhost interface, so it's not a networking issue. Thread 445 (Thread 0x7fa2df6db700 (LWP 9395)): #0 0x00007fa3712da42d in __lll_lock_wait () from /usr/lib64/libpthread.so.0 #1 0x00007fa3712d5de6 in _L_lock_870 () from /usr/lib64/libpthread.so.0 #2 0x00007fa3712d5cdf in pthread_mutex_lock () from /usr/lib64/libpthread.so.0 #3 0x00007fa35742468a in sofia_glue_execute_sql_callback () from /usr/lib64/freeswitch/mod/mod_sofia.so #4 0x00007fa3573df231 in select_from_profile () from /usr/lib64/freeswitch/mod/mod_sofia.so #5 0x00007fa3573e3b17 in sofia_contact_function () from /usr/lib64/freeswitch/mod/mod_sofia.so #6 0x00007fa3730594cc in switch_api_execute () from /usr/lib64/libfreeswitch.so.1 #7 0x00007fa372fe69d6 in switch_channel_expand_variables_check () from /usr/lib64/libfreeswitch.so.1 #8 0x00007fa3552d45e4 in user_outgoing_channel () from /usr/lib64/freeswitch/mod/mod_dptools.so #9 0x00007fa37300b42a in switch_core_session_outgoing_channel () from /usr/lib64/libfreeswitch.so.1 #10 0x00007fa37308dc3b in switch_ivr_originate () from /usr/lib64/libfreeswitch.so.1 #11 0x00007fa3552db0b5 in audio_bridge_function () from /usr/lib64/freeswitch/mod/mod_dptools.so #12 0x00007fa37300ebcb in switch_core_session_exec () from /usr/lib64/libfreeswitch.so.1 #13 0x00007fa37300f179 in switch_core_session_execute_application_get_flags () from /usr/lib64/libfreeswitch.so.1 #14 0x00007fa373012a34 in switch_core_session_run () from /usr/lib64/libfreeswitch.so.1 #15 0x00007fa37300c33e in switch_core_session_thread () from /usr/lib64/libfreeswitch.so.1 #16 0x00007fa373007d83 in switch_core_session_thread_pool_worker () from /usr/lib64/libfreeswitch.so.1 #17 0x00007fa3732ce7c0 in dummy_worker () from /usr/lib64/libfreeswitch.so.1 #18 0x00007fa3712d3e25 in start_thread () from /usr/lib64/libpthread.so.0 #19 0x00007fa37092e34d in clone () from /usr/lib64/libc.so.6
look for the other thread inside sofia_glue_execute_sql_callback that is NOT blocked on pthread_mutex_lock for your answer. My guess is this is a deadlock inside the mysql drivers blocking us. Mysql drivers have a long long history of thread safety issues.
On Jun 8, 2018, at 1:04 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Greetings, Has anyone seen an instance where freeswitch gets hung with the following stack trace? I have about 400 threads in this state and eventually FS (1.6.19) just stops responding.
I am running FS against MySQL 5.6.38 on CentOS using ODBC. Guessing if this is known, it probably isn't fixed in 1.6.20... but I can hope. Any input on what a possible solution would be greatly appreciated. The MySQL server and FS are running on the same host connecting via localhost interface, so it's not a networking issue.
Thread 445 (Thread 0x7fa2df6db700 (LWP 9395)): #0 0x00007fa3712da42d in __lll_lock_wait () from /usr/lib64/libpthread.so.0 #1 0x00007fa3712d5de6 in _L_lock_870 () from /usr/lib64/libpthread.so.0 #2 0x00007fa3712d5cdf in pthread_mutex_lock () from /usr/lib64/libpthread.so.0 #3 0x00007fa35742468a in sofia_glue_execute_sql_callback () from /usr/lib64/freeswitch/mod/mod_sofia.so #4 0x00007fa3573df231 in select_from_profile () from /usr/lib64/freeswitch/mod/mod_sofia.so #5 0x00007fa3573e3b17 in sofia_contact_function () from /usr/lib64/freeswitch/mod/mod_sofia.so #6 0x00007fa3730594cc in switch_api_execute () from /usr/lib64/libfreeswitch.so.1 #7 0x00007fa372fe69d6 in switch_channel_expand_variables_check () from /usr/lib64/libfreeswitch.so.1 #8 0x00007fa3552d45e4 in user_outgoing_channel () from /usr/lib64/freeswitch/mod/mod_dptools.so #9 0x00007fa37300b42a in switch_core_session_outgoing_channel () from /usr/lib64/libfreeswitch.so.1 #10 0x00007fa37308dc3b in switch_ivr_originate () from /usr/lib64/libfreeswitch.so.1 #11 0x00007fa3552db0b5 in audio_bridge_function () from /usr/lib64/freeswitch/mod/mod_dptools.so #12 0x00007fa37300ebcb in switch_core_session_exec () from /usr/lib64/libfreeswitch.so.1 #13 0x00007fa37300f179 in switch_core_session_execute_application_get_flags () from /usr/lib64/libfreeswitch.so.1 #14 0x00007fa373012a34 in switch_core_session_run () from /usr/lib64/libfreeswitch.so.1 #15 0x00007fa37300c33e in switch_core_session_thread () from /usr/lib64/libfreeswitch.so.1 #16 0x00007fa373007d83 in switch_core_session_thread_pool_worker () from /usr/lib64/libfreeswitch.so.1 #17 0x00007fa3732ce7c0 in dummy_worker () from /usr/lib64/libfreeswitch.so.1 #18 0x00007fa3712d3e25 in start_thread () from /usr/lib64/libpthread.so.0 #19 0x00007fa37092e34d in clone () from /usr/lib64/libc.so.6
You would need to do a thread apply all bt to see what the other threads are doing. The thread you posted is stuck on a mutex waiting for something else, probably your DB in another thread. If you do get that info file it to jira because its easier to deal with backtraces there. On Fri, Jun 8, 2018 at 12:04 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Greetings, Has anyone seen an instance where freeswitch gets hung with the following stack trace? I have about 400 threads in this state and eventually FS (1.6.19) just stops responding.
I am running FS against MySQL 5.6.38 on CentOS using ODBC. Guessing if this is known, it probably isn't fixed in 1.6.20... but I can hope. Any input on what a possible solution would be greatly appreciated. The MySQL server and FS are running on the same host connecting via localhost interface, so it's not a networking issue.
Thread 445 (Thread 0x7fa2df6db700 (LWP 9395)): #0 0x00007fa3712da42d in __lll_lock_wait () from /usr/lib64/libpthread.so.0 #1 0x00007fa3712d5de6 in _L_lock_870 () from /usr/lib64/libpthread.so.0 #2 0x00007fa3712d5cdf in pthread_mutex_lock () from /usr/lib64/libpthread.so.0 #3 0x00007fa35742468a in sofia_glue_execute_sql_callback () from /usr/lib64/freeswitch/mod/mod_sofia.so #4 0x00007fa3573df231 in select_from_profile () from /usr/lib64/freeswitch/mod/mod_sofia.so #5 0x00007fa3573e3b17 in sofia_contact_function () from /usr/lib64/freeswitch/mod/mod_sofia.so #6 0x00007fa3730594cc in switch_api_execute () from /usr/lib64/libfreeswitch.so.1 #7 0x00007fa372fe69d6 in switch_channel_expand_variables_check () from /usr/lib64/libfreeswitch.so.1 #8 0x00007fa3552d45e4 in user_outgoing_channel () from /usr/lib64/freeswitch/mod/mod_dptools.so #9 0x00007fa37300b42a in switch_core_session_outgoing_channel () from /usr/lib64/libfreeswitch.so.1 #10 0x00007fa37308dc3b in switch_ivr_originate () from /usr/lib64/libfreeswitch.so.1 #11 0x00007fa3552db0b5 in audio_bridge_function () from /usr/lib64/freeswitch/mod/mod_dptools.so #12 0x00007fa37300ebcb in switch_core_session_exec () from /usr/lib64/libfreeswitch.so.1 #13 0x00007fa37300f179 in switch_core_session_execute_application_get_flags () from /usr/lib64/libfreeswitch.so.1 #14 0x00007fa373012a34 in switch_core_session_run () from /usr/lib64/libfreeswitch.so.1 #15 0x00007fa37300c33e in switch_core_session_thread () from /usr/lib64/libfreeswitch.so.1 #16 0x00007fa373007d83 in switch_core_session_thread_pool_worker () from /usr/lib64/libfreeswitch.so.1 #17 0x00007fa3732ce7c0 in dummy_worker () from /usr/lib64/libfreeswitch.so.1 #18 0x00007fa3712d3e25 in start_thread () from /usr/lib64/libpthread.so.0 #19 0x00007fa37092e34d in clone () from /usr/lib64/libc.so.6
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
-- Anthony Minessale II Founder, FreeSWITCH. http://freeswitch.com https://youtu.be/l_hOxzCt6X4 https://www.youtube.com/watch?v=oAxXgyx5jUw https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Would you guys suggest switching to Postgres on the back-end? I am guessing if there are known MySQL issues, the suggestion is probably native PGSQL? On Fri, Jun 8, 2018 at 2:48 PM Anthony Minessale < anthony.minessale@gmail.com> wrote:
You would need to do a thread apply all bt to see what the other threads are doing. The thread you posted is stuck on a mutex waiting for something else, probably your DB in another thread. If you do get that info file it to jira because its easier to deal with backtraces there.
On Fri, Jun 8, 2018 at 12:04 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Greetings, Has anyone seen an instance where freeswitch gets hung with the following stack trace? I have about 400 threads in this state and eventually FS (1.6.19) just stops responding.
I am running FS against MySQL 5.6.38 on CentOS using ODBC. Guessing if this is known, it probably isn't fixed in 1.6.20... but I can hope. Any input on what a possible solution would be greatly appreciated. The MySQL server and FS are running on the same host connecting via localhost interface, so it's not a networking issue.
Thread 445 (Thread 0x7fa2df6db700 (LWP 9395)): #0 0x00007fa3712da42d in __lll_lock_wait () from /usr/lib64/libpthread.so.0 #1 0x00007fa3712d5de6 in _L_lock_870 () from /usr/lib64/libpthread.so.0 #2 0x00007fa3712d5cdf in pthread_mutex_lock () from /usr/lib64/libpthread.so.0 #3 0x00007fa35742468a in sofia_glue_execute_sql_callback () from /usr/lib64/freeswitch/mod/mod_sofia.so #4 0x00007fa3573df231 in select_from_profile () from /usr/lib64/freeswitch/mod/mod_sofia.so #5 0x00007fa3573e3b17 in sofia_contact_function () from /usr/lib64/freeswitch/mod/mod_sofia.so #6 0x00007fa3730594cc in switch_api_execute () from /usr/lib64/libfreeswitch.so.1 #7 0x00007fa372fe69d6 in switch_channel_expand_variables_check () from /usr/lib64/libfreeswitch.so.1 #8 0x00007fa3552d45e4 in user_outgoing_channel () from /usr/lib64/freeswitch/mod/mod_dptools.so #9 0x00007fa37300b42a in switch_core_session_outgoing_channel () from /usr/lib64/libfreeswitch.so.1 #10 0x00007fa37308dc3b in switch_ivr_originate () from /usr/lib64/libfreeswitch.so.1 #11 0x00007fa3552db0b5 in audio_bridge_function () from /usr/lib64/freeswitch/mod/mod_dptools.so #12 0x00007fa37300ebcb in switch_core_session_exec () from /usr/lib64/libfreeswitch.so.1 #13 0x00007fa37300f179 in switch_core_session_execute_application_get_flags () from /usr/lib64/libfreeswitch.so.1 #14 0x00007fa373012a34 in switch_core_session_run () from /usr/lib64/libfreeswitch.so.1 #15 0x00007fa37300c33e in switch_core_session_thread () from /usr/lib64/libfreeswitch.so.1 #16 0x00007fa373007d83 in switch_core_session_thread_pool_worker () from /usr/lib64/libfreeswitch.so.1 #17 0x00007fa3732ce7c0 in dummy_worker () from /usr/lib64/libfreeswitch.so.1 #18 0x00007fa3712d3e25 in start_thread () from /usr/lib64/libpthread.so.0 #19 0x00007fa37092e34d in clone () from /usr/lib64/libc.so.6
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
-- Anthony Minessale II Founder, FreeSWITCH. http://freeswitch.com
https://youtu.be/l_hOxzCt6X4 https://www.youtube.com/watch?v=oAxXgyx5jUw https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
There are some params on confluence that show how to optimize the mysql. We don't like evangelizing one or the other technology but ya, postgres was at least better at threading when we first added db support many years ago. On Fri, Jun 8, 2018 at 4:12 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Would you guys suggest switching to Postgres on the back-end? I am guessing if there are known MySQL issues, the suggestion is probably native PGSQL?
On Fri, Jun 8, 2018 at 2:48 PM Anthony Minessale < anthony.minessale@gmail.com> wrote:
You would need to do a thread apply all bt to see what the other threads are doing. The thread you posted is stuck on a mutex waiting for something else, probably your DB in another thread. If you do get that info file it to jira because its easier to deal with backtraces there.
On Fri, Jun 8, 2018 at 12:04 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Greetings, Has anyone seen an instance where freeswitch gets hung with the following stack trace? I have about 400 threads in this state and eventually FS (1.6.19) just stops responding.
I am running FS against MySQL 5.6.38 on CentOS using ODBC. Guessing if this is known, it probably isn't fixed in 1.6.20... but I can hope. Any input on what a possible solution would be greatly appreciated. The MySQL server and FS are running on the same host connecting via localhost interface, so it's not a networking issue.
Thread 445 (Thread 0x7fa2df6db700 (LWP 9395)): #0 0x00007fa3712da42d in __lll_lock_wait () from /usr/lib64/libpthread.so.0 #1 0x00007fa3712d5de6 in _L_lock_870 () from /usr/lib64/libpthread.so.0 #2 0x00007fa3712d5cdf in pthread_mutex_lock () from /usr/lib64/libpthread.so.0 #3 0x00007fa35742468a in sofia_glue_execute_sql_callback () from /usr/lib64/freeswitch/mod/mod_sofia.so #4 0x00007fa3573df231 in select_from_profile () from /usr/lib64/freeswitch/mod/mod_sofia.so #5 0x00007fa3573e3b17 in sofia_contact_function () from /usr/lib64/freeswitch/mod/mod_sofia.so #6 0x00007fa3730594cc in switch_api_execute () from /usr/lib64/libfreeswitch.so.1 #7 0x00007fa372fe69d6 in switch_channel_expand_variables_check () from /usr/lib64/libfreeswitch.so.1 #8 0x00007fa3552d45e4 in user_outgoing_channel () from /usr/lib64/freeswitch/mod/mod_dptools.so #9 0x00007fa37300b42a in switch_core_session_outgoing_channel () from /usr/lib64/libfreeswitch.so.1 #10 0x00007fa37308dc3b in switch_ivr_originate () from /usr/lib64/libfreeswitch.so.1 #11 0x00007fa3552db0b5 in audio_bridge_function () from /usr/lib64/freeswitch/mod/mod_dptools.so #12 0x00007fa37300ebcb in switch_core_session_exec () from /usr/lib64/libfreeswitch.so.1 #13 0x00007fa37300f179 in switch_core_session_execute_application_get_flags () from /usr/lib64/libfreeswitch.so.1 #14 0x00007fa373012a34 in switch_core_session_run () from /usr/lib64/libfreeswitch.so.1 #15 0x00007fa37300c33e in switch_core_session_thread () from /usr/lib64/libfreeswitch.so.1 #16 0x00007fa373007d83 in switch_core_session_thread_pool_worker () from /usr/lib64/libfreeswitch.so.1 #17 0x00007fa3732ce7c0 in dummy_worker () from /usr/lib64/libfreeswitch.so.1 #18 0x00007fa3712d3e25 in start_thread () from /usr/lib64/libpthread.so.0 #19 0x00007fa37092e34d in clone () from /usr/lib64/libc.so.6
____________________________________________________________ _____________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
-- Anthony Minessale II Founder, FreeSWITCH. http://freeswitch.com
https://youtu.be/l_hOxzCt6X4 https://www.youtube.com/watch?v=oAxXgyx5jUw https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
-- Anthony Minessale II Founder, FreeSWITCH. http://freeswitch.com https://youtu.be/l_hOxzCt6X4 https://www.youtube.com/watch?v=oAxXgyx5jUw https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
On Sat, Jun 9, 2018, 02:12 Geoff Mina <gmina@connectfirst.com> wrote:
Would you guys suggest switching to Postgres on the back-end? I am guessing if there are known MySQL issues, the suggestion is probably native PGSQL?
Problem is with the odbc driver. So, short answer is: yes, definitely go with postgresql native support -giovanni
On Fri, Jun 8, 2018 at 2:48 PM Anthony Minessale < anthony.minessale@gmail.com> wrote:
You would need to do a thread apply all bt to see what the other threads are doing. The thread you posted is stuck on a mutex waiting for something else, probably your DB in another thread. If you do get that info file it to jira because its easier to deal with backtraces there.
On Fri, Jun 8, 2018 at 12:04 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Greetings, Has anyone seen an instance where freeswitch gets hung with the following stack trace? I have about 400 threads in this state and eventually FS (1.6.19) just stops responding.
I am running FS against MySQL 5.6.38 on CentOS using ODBC. Guessing if this is known, it probably isn't fixed in 1.6.20... but I can hope. Any input on what a possible solution would be greatly appreciated. The MySQL server and FS are running on the same host connecting via localhost interface, so it's not a networking issue.
Thread 445 (Thread 0x7fa2df6db700 (LWP 9395)): #0 0x00007fa3712da42d in __lll_lock_wait () from /usr/lib64/libpthread.so.0 #1 0x00007fa3712d5de6 in _L_lock_870 () from /usr/lib64/libpthread.so.0 #2 0x00007fa3712d5cdf in pthread_mutex_lock () from /usr/lib64/libpthread.so.0 #3 0x00007fa35742468a in sofia_glue_execute_sql_callback () from /usr/lib64/freeswitch/mod/mod_sofia.so #4 0x00007fa3573df231 in select_from_profile () from /usr/lib64/freeswitch/mod/mod_sofia.so #5 0x00007fa3573e3b17 in sofia_contact_function () from /usr/lib64/freeswitch/mod/mod_sofia.so #6 0x00007fa3730594cc in switch_api_execute () from /usr/lib64/libfreeswitch.so.1 #7 0x00007fa372fe69d6 in switch_channel_expand_variables_check () from /usr/lib64/libfreeswitch.so.1 #8 0x00007fa3552d45e4 in user_outgoing_channel () from /usr/lib64/freeswitch/mod/mod_dptools.so #9 0x00007fa37300b42a in switch_core_session_outgoing_channel () from /usr/lib64/libfreeswitch.so.1 #10 0x00007fa37308dc3b in switch_ivr_originate () from /usr/lib64/libfreeswitch.so.1 #11 0x00007fa3552db0b5 in audio_bridge_function () from /usr/lib64/freeswitch/mod/mod_dptools.so #12 0x00007fa37300ebcb in switch_core_session_exec () from /usr/lib64/libfreeswitch.so.1 #13 0x00007fa37300f179 in switch_core_session_execute_application_get_flags () from /usr/lib64/libfreeswitch.so.1 #14 0x00007fa373012a34 in switch_core_session_run () from /usr/lib64/libfreeswitch.so.1 #15 0x00007fa37300c33e in switch_core_session_thread () from /usr/lib64/libfreeswitch.so.1 #16 0x00007fa373007d83 in switch_core_session_thread_pool_worker () from /usr/lib64/libfreeswitch.so.1 #17 0x00007fa3732ce7c0 in dummy_worker () from /usr/lib64/libfreeswitch.so.1 #18 0x00007fa3712d3e25 in start_thread () from /usr/lib64/libpthread.so.0 #19 0x00007fa37092e34d in clone () from /usr/lib64/libc.so.6
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
-- Anthony Minessale II Founder, FreeSWITCH. http://freeswitch.com
https://youtu.be/l_hOxzCt6X4 https://www.youtube.com/watch?v=oAxXgyx5jUw https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Thanks. I was assuming it was the ODBC driver since MySQL itself isn't prone to deadlocks. We have switched all of our backends to PostgreSQL as of last week. So far so good. On Wed, Jun 13, 2018 at 4:31 PM Giovanni Maruzzelli <gmaruzz@gmail.com> wrote:
On Sat, Jun 9, 2018, 02:12 Geoff Mina <gmina@connectfirst.com> wrote:
Would you guys suggest switching to Postgres on the back-end? I am guessing if there are known MySQL issues, the suggestion is probably native PGSQL?
Problem is with the odbc driver.
So, short answer is: yes, definitely go with postgresql native support
-giovanni
On Fri, Jun 8, 2018 at 2:48 PM Anthony Minessale < anthony.minessale@gmail.com> wrote:
You would need to do a thread apply all bt to see what the other threads are doing. The thread you posted is stuck on a mutex waiting for something else, probably your DB in another thread. If you do get that info file it to jira because its easier to deal with backtraces there.
On Fri, Jun 8, 2018 at 12:04 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Greetings, Has anyone seen an instance where freeswitch gets hung with the following stack trace? I have about 400 threads in this state and eventually FS (1.6.19) just stops responding.
I am running FS against MySQL 5.6.38 on CentOS using ODBC. Guessing if this is known, it probably isn't fixed in 1.6.20... but I can hope. Any input on what a possible solution would be greatly appreciated. The MySQL server and FS are running on the same host connecting via localhost interface, so it's not a networking issue.
Thread 445 (Thread 0x7fa2df6db700 (LWP 9395)): #0 0x00007fa3712da42d in __lll_lock_wait () from /usr/lib64/libpthread.so.0 #1 0x00007fa3712d5de6 in _L_lock_870 () from /usr/lib64/libpthread.so.0 #2 0x00007fa3712d5cdf in pthread_mutex_lock () from /usr/lib64/libpthread.so.0 #3 0x00007fa35742468a in sofia_glue_execute_sql_callback () from /usr/lib64/freeswitch/mod/mod_sofia.so #4 0x00007fa3573df231 in select_from_profile () from /usr/lib64/freeswitch/mod/mod_sofia.so #5 0x00007fa3573e3b17 in sofia_contact_function () from /usr/lib64/freeswitch/mod/mod_sofia.so #6 0x00007fa3730594cc in switch_api_execute () from /usr/lib64/libfreeswitch.so.1 #7 0x00007fa372fe69d6 in switch_channel_expand_variables_check () from /usr/lib64/libfreeswitch.so.1 #8 0x00007fa3552d45e4 in user_outgoing_channel () from /usr/lib64/freeswitch/mod/mod_dptools.so #9 0x00007fa37300b42a in switch_core_session_outgoing_channel () from /usr/lib64/libfreeswitch.so.1 #10 0x00007fa37308dc3b in switch_ivr_originate () from /usr/lib64/libfreeswitch.so.1 #11 0x00007fa3552db0b5 in audio_bridge_function () from /usr/lib64/freeswitch/mod/mod_dptools.so #12 0x00007fa37300ebcb in switch_core_session_exec () from /usr/lib64/libfreeswitch.so.1 #13 0x00007fa37300f179 in switch_core_session_execute_application_get_flags () from /usr/lib64/libfreeswitch.so.1 #14 0x00007fa373012a34 in switch_core_session_run () from /usr/lib64/libfreeswitch.so.1 #15 0x00007fa37300c33e in switch_core_session_thread () from /usr/lib64/libfreeswitch.so.1 #16 0x00007fa373007d83 in switch_core_session_thread_pool_worker () from /usr/lib64/libfreeswitch.so.1 #17 0x00007fa3732ce7c0 in dummy_worker () from /usr/lib64/libfreeswitch.so.1 #18 0x00007fa3712d3e25 in start_thread () from /usr/lib64/libpthread.so.0 #19 0x00007fa37092e34d in clone () from /usr/lib64/libc.so.6
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
-- Anthony Minessale II Founder, FreeSWITCH. http://freeswitch.com
https://youtu.be/l_hOxzCt6X4 https://www.youtube.com/watch?v=oAxXgyx5jUw https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
the history of issues is in the client and odbc drivers, yes. Mike
On Jun 13, 2018, at 6:42 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Thanks. I was assuming it was the ODBC driver since MySQL itself isn't prone to deadlocks.
We have switched all of our backends to PostgreSQL as of last week. So far so good.
On Wed, Jun 13, 2018 at 4:31 PM Giovanni Maruzzelli <gmaruzz@gmail.com <mailto:gmaruzz@gmail.com>> wrote: On Sat, Jun 9, 2018, 02:12 Geoff Mina <gmina@connectfirst.com <mailto:gmina@connectfirst.com>> wrote: Would you guys suggest switching to Postgres on the back-end? I am guessing if there are known MySQL issues, the suggestion is probably native PGSQL?
Problem is with the odbc driver.
So, short answer is: yes, definitely go with postgresql native support
-giovanni
On Fri, Jun 8, 2018 at 2:48 PM Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> wrote: You would need to do a thread apply all bt to see what the other threads are doing. The thread you posted is stuck on a mutex waiting for something else, probably your DB in another thread. If you do get that info file it to jira because its easier to deal with backtraces there.
On Fri, Jun 8, 2018 at 12:04 PM, Geoff Mina <gmina@connectfirst.com <mailto:gmina@connectfirst.com>> wrote: Greetings, Has anyone seen an instance where freeswitch gets hung with the following stack trace? I have about 400 threads in this state and eventually FS (1.6.19) just stops responding.
I am running FS against MySQL 5.6.38 on CentOS using ODBC. Guessing if this is known, it probably isn't fixed in 1.6.20... but I can hope. Any input on what a possible solution would be greatly appreciated. The MySQL server and FS are running on the same host connecting via localhost interface, so it's not a networking issue.
Thread 445 (Thread 0x7fa2df6db700 (LWP 9395)): #0 0x00007fa3712da42d in __lll_lock_wait () from /usr/lib64/libpthread.so.0 #1 0x00007fa3712d5de6 in _L_lock_870 () from /usr/lib64/libpthread.so.0 #2 0x00007fa3712d5cdf in pthread_mutex_lock () from /usr/lib64/libpthread.so.0 #3 0x00007fa35742468a in sofia_glue_execute_sql_callback () from /usr/lib64/freeswitch/mod/mod_sofia.so #4 0x00007fa3573df231 in select_from_profile () from /usr/lib64/freeswitch/mod/mod_sofia.so #5 0x00007fa3573e3b17 in sofia_contact_function () from /usr/lib64/freeswitch/mod/mod_sofia.so #6 0x00007fa3730594cc in switch_api_execute () from /usr/lib64/libfreeswitch.so.1 #7 0x00007fa372fe69d6 in switch_channel_expand_variables_check () from /usr/lib64/libfreeswitch.so.1 #8 0x00007fa3552d45e4 in user_outgoing_channel () from /usr/lib64/freeswitch/mod/mod_dptools.so #9 0x00007fa37300b42a in switch_core_session_outgoing_channel () from /usr/lib64/libfreeswitch.so.1 #10 0x00007fa37308dc3b in switch_ivr_originate () from /usr/lib64/libfreeswitch.so.1 #11 0x00007fa3552db0b5 in audio_bridge_function () from /usr/lib64/freeswitch/mod/mod_dptools.so #12 0x00007fa37300ebcb in switch_core_session_exec () from /usr/lib64/libfreeswitch.so.1 #13 0x00007fa37300f179 in switch_core_session_execute_application_get_flags () from /usr/lib64/libfreeswitch.so.1 #14 0x00007fa373012a34 in switch_core_session_run () from /usr/lib64/libfreeswitch.so.1 #15 0x00007fa37300c33e in switch_core_session_thread () from /usr/lib64/libfreeswitch.so.1 #16 0x00007fa373007d83 in switch_core_session_thread_pool_worker () from /usr/lib64/libfreeswitch.so.1 #17 0x00007fa3732ce7c0 in dummy_worker () from /usr/lib64/libfreeswitch.so.1 #18 0x00007fa3712d3e25 in start_thread () from /usr/lib64/libpthread.so.0 #19 0x00007fa37092e34d in clone () from /usr/lib64/libc.so.6
participants (4)
-
Anthony Minessale
-
Geoff Mina
-
Giovanni Maruzzelli
-
Michael Jerris