Freeswitch HA postgresql BDR mod_sofia fails to start
Hi guys, I am running two freeswitch instances in HA cluster in OVH cloud VPS. with postgresql in the core postgresql BDR master-master replication sometimes (almost always) when I start/restart freeswitch service or mod_sofia I get errors: 2016-10-07 19:22:22.499304 [CRIT] switch_pgsql.c:360 Query (create index sr_call_id on sip_registrations (call_id)) took too long to complete or database not responding. 2016-10-07 19:22:22.599306 [CRIT] switch_pgsql.c:360 Query (insert into tasks (task_id, task_desc, task_group, task_runtime, task_sql_manager, hostname) values(1,'zrtp_cache_save','core',1475869032,0,'fusion01-2')) took too long to complete or database not responding. 2016-10-07 19:22:22.599306 [DEBUG] switch_pgsql.c:415 Query (insert into tasks (task_id, task_desc, task_group, task_runtime, task_sql_manager, hostname) values(2,'heartbeat','core',1475868132,0,'fusion01-2')) returned PGRES_FATAL_ERROR 2016-10-07 19:22:22.599306 [ERR] switch_pgsql.c:656 Error executing query: ERROR: current transaction is aborted, commands ignored until end of transaction block ..and sofia can not initialize profiles or only one profile or freeswitch fails to start at all I can see some queries on freeswitch db with state=active when I execute (select * from pg_stat_activity) even when FS is down. top - 19:39:38 up 3:07, 7 users, load average: 1.03, 1.04, 0.95 Tasks: 126 total, 2 running, 124 sleeping, 0 stopped, 0 zombie %Cpu0 : 93.0 us, 0.3 sy, 0.0 ni, 6.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 7.1 us, 0.7 sy, 0.0 ni, 92.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st Release: Debian 8.6 freeswitch 1.6.11~19~5e413fe-1~jessie+1 amd64 postgresql-bdr-9.4 9.4.9-1jessie FS config is default both VPS LAN interconnected OpenStack KVM 2 vCores 2.4 GHz 8 GB RAM 40 GB SSD Local RAID 10 any ideas ? -- Regards, Arsen.
It looks like the issue is with the replication, I've removed BDR from freeswitch database and freeswitch is flying again.. Not sure whether it's BDR or VPS performance issue or something specific related to freeswitch DB.. If someone has experience running FS with postgre in the core and BDR your advices are very welcome! :) On Fri, Oct 7, 2016 at 10:45 PM, Arsen <arsen.semionov@gmail.com> wrote:
Hi guys,
I am running two freeswitch instances in HA cluster in OVH cloud VPS. with postgresql in the core postgresql BDR master-master replication
sometimes (almost always) when I start/restart freeswitch service or mod_sofia I get errors:
2016-10-07 19:22:22.499304 [CRIT] switch_pgsql.c:360 Query (create index sr_call_id on sip_registrations (call_id)) took too long to complete or database not responding. 2016-10-07 19:22:22.599306 [CRIT] switch_pgsql.c:360 Query (insert into tasks (task_id, task_desc, task_group, task_runtime, task_sql_manager, hostname) values(1,'zrtp_cache_save','core',1475869032,0,'fusion01-2')) took too long to complete or database not responding. 2016-10-07 19:22:22.599306 [DEBUG] switch_pgsql.c:415 Query (insert into tasks (task_id, task_desc, task_group, task_runtime, task_sql_manager, hostname) values(2,'heartbeat','core',1475868132,0,'fusion01-2')) returned PGRES_FATAL_ERROR 2016-10-07 19:22:22.599306 [ERR] switch_pgsql.c:656 Error executing query: ERROR: current transaction is aborted, commands ignored until end of transaction block
..and sofia can not initialize profiles or only one profile or freeswitch fails to start at all
I can see some queries on freeswitch db with state=active when I execute (select * from pg_stat_activity) even when FS is down.
top - 19:39:38 up 3:07, 7 users, load average: 1.03, 1.04, 0.95 Tasks: 126 total, 2 running, 124 sleeping, 0 stopped, 0 zombie %Cpu0 : 93.0 us, 0.3 sy, 0.0 ni, 6.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 7.1 us, 0.7 sy, 0.0 ni, 92.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
Release: Debian 8.6 freeswitch 1.6.11~19~5e413fe-1~jessie+1 amd64 postgresql-bdr-9.4 9.4.9-1jessie FS config is default
both VPS LAN interconnected OpenStack KVM 2 vCores 2.4 GHz 8 GB RAM 40 GB SSD Local RAID 10
any ideas ?
-- Regards, Arsen.
-- Regards, Arsen.
the issue has been completely solved by setting: auto-create-schemas=false will be happy if someone will find this useful :) On Sat, Oct 8, 2016 at 5:29 PM, Arsen <arsen.semionov@gmail.com> wrote:
It looks like the issue is with the replication, I've removed BDR from freeswitch database and freeswitch is flying again.. Not sure whether it's BDR or VPS performance issue or something specific related to freeswitch DB..
If someone has experience running FS with postgre in the core and BDR your advices are very welcome! :)
On Fri, Oct 7, 2016 at 10:45 PM, Arsen <arsen.semionov@gmail.com> wrote:
Hi guys,
I am running two freeswitch instances in HA cluster in OVH cloud VPS. with postgresql in the core postgresql BDR master-master replication
sometimes (almost always) when I start/restart freeswitch service or mod_sofia I get errors:
2016-10-07 19:22:22.499304 [CRIT] switch_pgsql.c:360 Query (create index sr_call_id on sip_registrations (call_id)) took too long to complete or database not responding. 2016-10-07 19:22:22.599306 [CRIT] switch_pgsql.c:360 Query (insert into tasks (task_id, task_desc, task_group, task_runtime, task_sql_manager, hostname) values(1,'zrtp_cache_save','core',1475869032,0,'fusion01-2')) took too long to complete or database not responding. 2016-10-07 19:22:22.599306 [DEBUG] switch_pgsql.c:415 Query (insert into tasks (task_id, task_desc, task_group, task_runtime, task_sql_manager, hostname) values(2,'heartbeat','core',1475868132,0,'fusion01-2')) returned PGRES_FATAL_ERROR 2016-10-07 19:22:22.599306 [ERR] switch_pgsql.c:656 Error executing query: ERROR: current transaction is aborted, commands ignored until end of transaction block
..and sofia can not initialize profiles or only one profile or freeswitch fails to start at all
I can see some queries on freeswitch db with state=active when I execute (select * from pg_stat_activity) even when FS is down.
top - 19:39:38 up 3:07, 7 users, load average: 1.03, 1.04, 0.95 Tasks: 126 total, 2 running, 124 sleeping, 0 stopped, 0 zombie %Cpu0 : 93.0 us, 0.3 sy, 0.0 ni, 6.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 7.1 us, 0.7 sy, 0.0 ni, 92.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
Release: Debian 8.6 freeswitch 1.6.11~19~5e413fe-1~jessie+1 amd64 postgresql-bdr-9.4 9.4.9-1jessie FS config is default
both VPS LAN interconnected OpenStack KVM 2 vCores 2.4 GHz 8 GB RAM 40 GB SSD Local RAID 10
any ideas ?
-- Regards, Arsen.
-- Regards, Arsen.
-- Regards, Arsen.
participants (1)
-
Arsen