Channels and calls lost in SQLite
Hello guys, I have a problem that started to happen after I upgraded to git head last night. No configuration changes were made and this is the intriguing part. I have SQLite being mounted on tmpfs on busy Dell 1950s. After some time on the busy moments of the day, FS starts to not delete channels/calls from SQLite and leaves very little trace for me to track. I grep'ed the logs for SQL errors and found nothing.. The only part of the log that I could spot and that called my attention was this line that repeated several times during the periods where the problem occurs: switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_WARNING, "queued event at a lower priority %d/%d!\n", index, was); 11:23:13.977874 [WARNING] switch_event.c:1353 queued event at a lower priority 1/0! Which is printed from the following code: for (index = (*event)->priority; index < 3; index++) { int was = (*event)->priority; if (switch_queue_trypush(EVENT_QUEUE[index], *event) == SWITCH_STATUS_SUCCESS) { if (index != was) { switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_WARNING, "queued event at a lower priority %d/%d!\n", index, was); } goto end; }
From what I could understand on the code, seems like there are 3 event priority queues and that message occurs when queue 0 is full. How exactly that relates to the problem, I am not sure so I see myself here lost in translation. Could the core_event_handler not receive some messages therefore not execute the sql statements? If that's the case, maybe I lost some log line telling me that we were loosing events?
I hope someone can shed some light. Thank you, João Mesquita
I've seen exactly that behavior. First some warnings about events "queued at lower priority" and some time later "Event queue is full!" Stopping the sipp traffic then shows some channels stuck, show channels display channels that are no longer there. We reverted to our previous git version of FreeSWITCH and the error is gone. Moises Silva Senior Software Engineer Sangoma Technologies Inc. | NEW 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com 2010/9/1 João Mesquita <jmesquita@freeswitch.org>
Hello guys,
I have a problem that started to happen after I upgraded to git head last night. No configuration changes were made and this is the intriguing part.
I have SQLite being mounted on tmpfs on busy Dell 1950s. After some time on the busy moments of the day, FS starts to not delete channels/calls from SQLite and leaves very little trace for me to track.
I grep'ed the logs for SQL errors and found nothing.. The only part of the log that I could spot and that called my attention was this line that repeated several times during the periods where the problem occurs:
switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_WARNING, "queued event at a lower priority %d/%d!\n", index, was);
11:23:13.977874 [WARNING] switch_event.c:1353 queued event at a lower priority 1/0!
Which is printed from the following code:
for (index = (*event)->priority; index < 3; index++) { int was = (*event)->priority; if (switch_queue_trypush(EVENT_QUEUE[index], *event) == SWITCH_STATUS_SUCCESS) { if (index != was) { switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_WARNING, "queued event at a lower priority %d/%d!\n", index, was); } goto end; }
From what I could understand on the code, seems like there are 3 event priority queues and that message occurs when queue 0 is full. How exactly that relates to the problem, I am not sure so I see myself here lost in translation. Could the core_event_handler not receive some messages therefore not execute the sql statements? If that's the case, maybe I lost some log line telling me that we were loosing events?
I hope someone can shed some light.
Thank you, João Mesquita
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Can you guys narrow down the commit that pushed things over the edge? Or doesn't git have some sort of magic tool that helps you figure this kind of thing out? -MC On Wed, Sep 1, 2010 at 9:17 PM, Moises Silva <moises.silva@gmail.com> wrote:
I've seen exactly that behavior.
First some warnings about events "queued at lower priority" and some time later "Event queue is full!"
Stopping the sipp traffic then shows some channels stuck, show channels display channels that are no longer there. We reverted to our previous git version of FreeSWITCH and the error is gone.
Moises Silva Senior Software Engineer Sangoma Technologies Inc. | NEW 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
2010/9/1 João Mesquita <jmesquita@freeswitch.org>
Hello guys,
I have a problem that started to happen after I upgraded to git head last night. No configuration changes were made and this is the intriguing part.
I have SQLite being mounted on tmpfs on busy Dell 1950s. After some time on the busy moments of the day, FS starts to not delete channels/calls from SQLite and leaves very little trace for me to track.
I grep'ed the logs for SQL errors and found nothing.. The only part of the log that I could spot and that called my attention was this line that repeated several times during the periods where the problem occurs:
switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_WARNING, "queued event at a lower priority %d/%d!\n", index, was);
11:23:13.977874 [WARNING] switch_event.c:1353 queued event at a lower priority 1/0!
Which is printed from the following code:
for (index = (*event)->priority; index < 3; index++) { int was = (*event)->priority; if (switch_queue_trypush(EVENT_QUEUE[index], *event) == SWITCH_STATUS_SUCCESS) { if (index != was) { switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_WARNING, "queued event at a lower priority %d/%d!\n", index, was); } goto end; }
From what I could understand on the code, seems like there are 3 event priority queues and that message occurs when queue 0 is full. How exactly that relates to the problem, I am not sure so I see myself here lost in translation. Could the core_event_handler not receive some messages therefore not execute the sql statements? If that's the case, maybe I lost some log line telling me that we were loosing events?
I hope someone can shed some light.
Thank you, João Mesquita
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I think this may be caused by changes that were made for FSCORE-657 Commit:af100fd1563aa6e6c335c11482f1e3d0de336dd2 and Commit:362eade2dc9b429eb047607755cdd6df355326cf 8/23/2010 -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/Channels-and-calls-lost-in-SQL... Sent from the freeswitch-users mailing list archive at Nabble.com.
try latest HEAD and see if it's better. BTW, we all know this should have been on jira and not the ML right...... On Thu, Sep 2, 2010 at 8:16 AM, Jeff Lenk <jeff@jefflenk.com> wrote:
I think this may be caused by changes that were made for FSCORE-657 Commit:af100fd1563aa6e6c335c11482f1e3d0de336dd2 and Commit:362eade2dc9b429eb047607755cdd6df355326cf 8/23/2010 -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/Channels-and-calls-lost-in-SQL... Sent from the freeswitch-users mailing list archive at Nabble.com.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Anthony, let me disagree. Only confirmed bugs should go to jira, confirmation that a "feature" is a bug, should be done in ML. Otherwise jira will be filled up with user's mistakes/misconfigurations. On Thursday 02 September 2010, Anthony Minessale wrote:
try latest HEAD and see if it's better.
BTW, we all know this should have been on jira and not the ML right......
On Thu, Sep 2, 2010 at 8:16 AM, Jeff Lenk <jeff@jefflenk.com> wrote:
I think this may be caused by changes that were made for FSCORE-657 Commit:af100fd1563aa6e6c335c11482f1e3d0de336dd2 and Commit:362eade2dc9b429eb047607755cdd6df355326cf 8/23/2010 -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/Channels-and-calls-lost-in- SQLite-tp5489482p5491044.html Sent from the freeswitch-users mailing list archive at Nabble.com.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I just put it on production. So far, looks good but it's not peak time (had to wait til tonight to upgrade it). Will know tomorrow for sure and will report. Thank you Tony for the attention and the rest for diving into the problem as well. Regards, João Mesquita On Thu, Sep 2, 2010 at 2:43 PM, Sergey Okhapkin <sos@sokhapkin.dyndns.org>wrote:
Anthony, let me disagree. Only confirmed bugs should go to jira, confirmation that a "feature" is a bug, should be done in ML. Otherwise jira will be filled up with user's mistakes/misconfigurations.
On Thursday 02 September 2010, Anthony Minessale wrote:
try latest HEAD and see if it's better.
BTW, we all know this should have been on jira and not the ML right......
On Thu, Sep 2, 2010 at 8:16 AM, Jeff Lenk <jeff@jefflenk.com> wrote:
I think this may be caused by changes that were made for FSCORE-657 Commit:af100fd1563aa6e6c335c11482f1e3d0de336dd2 and Commit:362eade2dc9b429eb047607755cdd6df355326cf 8/23/2010 -- View this message in context:
http://freeswitch-users.2379917.n2.nabble.com/Channels-and-calls-lost-in-
SQLite-tp5489482p5491044.html Sent from the freeswitch-users mailing list archive at Nabble.com.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
try latest HEAD and see if it's better. BTW, we all know this should have been on jira and not the ML right...... On Thu, Sep 2, 2010 at 8:16 AM, Jeff Lenk <jeff@jefflenk.com> wrote:
I think this may be caused by changes that were made for FSCORE-657 Commit:af100fd1563aa6e6c335c11482f1e3d0de336dd2 and Commit:362eade2dc9b429eb047607755cdd6df355326cf 8/23/2010 -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/Channels-and-calls-lost-in-SQL... Sent from the freeswitch-users mailing list archive at Nabble.com.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
participants (6)
-
Anthony Minessale
-
Jeff Lenk
-
João Mesquita
-
Michael Collins
-
Moises Silva
-
Sergey Okhapkin