Hey Everyone,
<div><br></div><div>I may have a bug for Mod_Callcenter but I wanted to see if anyone else might notice a setup issue before I send this on to Jira. I recently launched a production callcenter system for our company. We are currently beta testing the system with a small subset of our callers and we have noticed an issue where every so often (IE.<b> Intermittent issue</b>) when a caller arrives in queue and the queue offers the call to the agent,<u> the offer happens twice. </u></div>
<div><br></div><div>What gets worse is that once the agent answers one of the calls the other call continues to attempt to reach the agent even while they are talking to the person they are receiving the second call for. When the call times out we get a "NO ANSWER" and then the callcenter system attempts the call again, and again, and again until the agent is done with the caller.</div>
<div><br></div><div>Even more interesting is that the agent state and member state keep changing to reflect the state of the second calling thread that keeps trying even though the caller is actually speaking to the agent.</div>
<div><br></div><div>Here is a log from the freeswitch events. This was taken using log level 5 in the cli. Lines in BLUE are cli statements I am recording when events are fired that I care about. I am using Mod_managed to listen to events and update information in a company database. Lines in RED are notes about the logs.</div>
<div><br></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><font class="Apple-style-span" size="1" color="#3366ff">2011-11-04 15:44:06.340969 [ALERT] switch_cpp.cpp:1223 PsiQueueEventConsumer - CallLogId Not Detected, Attempting to Create New :: Unavailable(2406740599) calling cc+</font></div>
</div><div><div><font class="Apple-style-span" size="1" color="#ff0000">--- A CALLER JUST JOINED THE QUEUE ---</font></div></div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:45.160968 [NOTICE] switch_channel.c:915 New Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [35e23ce8-072e-11e1-a636-2d222f47338b]</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:45.160968 [NOTICE] switch_channel.c:915 New Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [35e24e68-072e-11e1-a63a-2d222f47338b]</font></div>
</div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- NOTICE HOW 2 CHANNELS WERE INITIATED TO THE SAME EXTENSION/AGENT ---</span></div></div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- AGENT STATE: 'Receiving' ---</span></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:45.200941 [NOTICE] sofia.c:5375 Ring-Ready sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>!</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:45.481059 [NOTICE] sofia.c:5375 Ring-Ready sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>!</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:48.740993 [NOTICE] sofia.c:5983 Channel [sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>] has been answered</font></div>
</div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- THE AGENT ANSWERED ONLY ONE OF THE CALLS ---</span></div></div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- AGENT STATE: 'In A Queue Call' ---</span></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:55.000947 [NOTICE] switch_ivr_originate.c:3167 Hangup sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_CONSUME_MEDIA] [NO_ANSWER]</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:55.000947 [NOTICE] switch_ivr_originate.c:2459 Cannot create outgoing channel of type [user] cause: [NO_ANSWER]</font></div></div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:55.000947 [NOTICE] switch_core_session.c:1395 Session 605 (sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>) Ended</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:55.000947 [NOTICE] switch_core_session.c:1397 Close Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_DESTROY]</font></div>
</div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- THE SECOND CHANNEL JUST TIMED OUT AND FLAGGED THE CALL AS A "NO ANSWER" ---</span></div></div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:55.000947 [NOTICE] switch_channel.c:915 New Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [3bc0c008-072e-11e1-a642-2d222f47338b]</font></div>
</div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- AGENT STATE: 'Receiving' ---</span></div></div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:44:55.040931 [NOTICE] sofia.c:5375 Ring-Ready sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>!</font></div>
</div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- SECOND CHANNEL ATTEMPTS AGAIN TO REACH THE AGENT, THE AGENT IS ALREADY TALKING TO THE CALLER ---</span></div></div><div>
<div><font class="Apple-style-span" size="1">2011-11-04 15:45:05.000941 [NOTICE] switch_ivr_originate.c:3167 Hangup sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_CONSUME_MEDIA] [NO_ANSWER]</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:45:05.000941 [NOTICE] switch_ivr_originate.c:2459 Cannot create outgoing channel of type [user] cause: [NO_ANSWER]</font></div></div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:45:05.000941 [NOTICE] switch_core_session.c:1395 Session 606 (sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>) Ended</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:45:05.000941 [NOTICE] switch_core_session.c:1397 Close Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_DESTROY]</font></div>
</div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- NO ANSWER TIMEOUT REPEATS ---</span></div></div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:45:05.060918 [NOTICE] switch_channel.c:915 New Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [41be0a9c-072e-11e1-a647-2d222f47338b]</font></div>
</div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- AGENT STATE: 'Receiving' ---</span></div></div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- TRIES AGAIN ---</span></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:45:05.081035 [NOTICE] sofia.c:5375 Ring-Ready sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>!</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:45:15.001026 [NOTICE] switch_ivr_originate.c:3167 Hangup sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_CONSUME_MEDIA] [NO_ANSWER]</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:45:15.001026 [NOTICE] switch_ivr_originate.c:2459 Cannot create outgoing channel of type [user] cause: [NO_ANSWER]</font></div></div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:45:15.001026 [NOTICE] switch_core_session.c:1395 Session 607 (sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>) Ended</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:45:15.001026 [NOTICE] switch_core_session.c:1397 Close Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_DESTROY]</font></div>
</div><div><br></div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- LOOPS CONTINUALLY FOR 7 MINUTES ---</span></div><div>....</div><div><br></div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- TRIES AGAIN ---</span></div>
<div><div><font class="Apple-style-span" size="1">2011-11-04 15:52:15.040968 [NOTICE] switch_channel.c:915 New Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [4208943a-072f-11e1-a72a-2d222f47338b]</font></div>
</div><div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- AGENT STATE: 'Receiving' ---</span></div></div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:52:15.080896 [NOTICE] sofia.c:5375 Ring-Ready sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>!</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:52:25.000919 [NOTICE] switch_ivr_originate.c:3167 Hangup sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_CONSUME_MEDIA] [NO_ANSWER]</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:52:25.000919 [NOTICE] switch_ivr_originate.c:2459 Cannot create outgoing channel of type [user] cause: [NO_ANSWER]</font></div></div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:52:25.000919 [NOTICE] switch_core_session.c:1395 Session 653 (sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>) Ended</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:52:25.000919 [NOTICE] switch_core_session.c:1397 Close Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_DESTROY]</font></div>
</div><div><font class="Apple-style-span" size="1">2011-11-04 15:52:28.140959 [NOTICE] switch_ivr_bridge.c:658 Hangup sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_EXCHANGE_MEDIA] [NORMAL_CLEARING]</font></div>
<div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- AGENT ACTUALLY HANGS UP ON THE REAL CALL ---</span></div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- AGENT STATE: 'Waiting' ---</span></div>
<div><div><font class="Apple-style-span" size="1">2011-11-04 15:52:28.180984 [NOTICE] switch_core_session.c:1395 Session 604 (sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a>) Ended</font></div>
</div><div><div><font class="Apple-style-span" size="1">2011-11-04 15:52:28.180984 [NOTICE] switch_core_session.c:1397 Close Channel sofia/external/<a href="http://sip:4503@97.75.182.83:59868">sip:4503@97.75.182.83:59868</a> [CS_DESTROY]</font></div>
</div><div><span class="Apple-style-span" style="color: rgb(255, 0, 0); font-size: x-small; ">--- THE LOOP IS ENDED BECAUSE THE MEMBER IS NO LONGER IN THE SYSTEM ---</span></div></blockquote><div><br></div><div>While the agent state keeps changing the agent is actually in a call with the caller this entire time. Events are firing every 10 seconds informing me this agent had a "No Answer" on an inbound call when they are actually talking to the caller.</div>
<div><br></div><div>Here are some settings in case you want to know:</div><div><br></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div><font class="Apple-style-span" size="1"><b>freeswitch@internal> callcenter_config queue list</b></font></div>
</div><div><div><font class="Apple-style-span" size="1">name|strategy|moh_sound|time_base_score|tier_rules_apply|tier_rule_wait_second|tier_rule_wait_multiply_level|tier_rule_no_agent_no_wait|discard_abandoned_after|abandoned_resume_allowed|max_wait_time|max_wait_time_with_no_agent|max_wait_time_with_no_agent_time_reached|record_template</font></div>
</div><div><div><font class="Apple-style-span" size="1"><b>callcenter@sip.</b></font><span class="Apple-style-span" style="font-size: x-small; "><b><hidden></b></span><font class="Apple-style-span" size="1"><b>.com|longest-idle-agent|local_stream://moh|system|false|300|true|false|60|false|0|0|5|/usr/local/freeswitch/recordings/${strftime(%Y-%m-%d-%H-%M-%S)}.${destination_number}.${caller_id_number}.${uuid}.wav</b></font></div>
</div><div><div><font class="Apple-style-span" size="1">+OK</font></div></div><div><font class="Apple-style-span" size="1"><br></font></div><div><font class="Apple-style-span" size="1"><div><b>freeswitch@internal> callcenter_config agent list</b></div>
</font></div><div><font class="Apple-style-span" size="1"><div>name|system|uuid|type|contact|status|state|max_no_answer|wrap_up_time|reject_delay_time|busy_delay_time|no_answer_delay_time|last_bridge_start|last_bridge_end|last_offered_call|last_status_change|no_answer_count|calls_answered|talk_time|ready_time</div>
</font></div><div><font class="Apple-style-span" size="1"><div>7003@sip.<hidden>.com|single_box|6d3dabc4-0732-11e1-a7c6-2d222f47338b|callback|[call_timeout=10]user/4513@sip.<hidden>.com|Available|In a queue call|0|30|0|0|0|1320444899|1320444795|1320444896|1320443529|0|4|1129|0</div>
</font></div><div><font class="Apple-style-span" size="1"><div><b>7004@sip.<hidden>.com|single_box||callback|[call_timeout=10]user/4503@sip.<hidden>.com|Available|Waiting|0|30|0|0|0|1320445006|1320445099|1320445000|1320434600|0|24|4254|0</b></div>
</font></div><div><font class="Apple-style-span" size="1"><div>+OK</div><div><br></div></font></div></blockquote>And here are the queue settings in XML. Same as above but easier to read. <div><font class="Apple-style-span" size="1"><br>
</font></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><font class="Apple-style-span" size="1"><div><queues></div><div> <queue name="callcenter@sip.<hidden>.com"></div>
<div> <param name="strategy" value="longest-idle-agent"/></div><div> <param name="moh-sound" value="$${hold_music}"/></div><div> <param name="record-template" value="$${base_dir}/recordings/${strftime(%Y-%m-%d-%H-%M-%S)}.${destination_number}.${caller_id_number}.${uuid}.wav"/></div>
<div> <param name="time-base-score" value="system"/></div><div> <param name="max-wait-time" value="0"/></div><div> <param name="max-wait-time-with-no-agent" value="0"/></div>
<div> <param name="max-wait-time-with-no-agent-time-reached" value="5"/></div><div> <param name="tier-rules-apply" value="false"/></div><div> <param name="tier-rule-wait-second" value="300"/></div>
<div> <param name="tier-rule-wait-multiply-level" value="true"/></div><div> <param name="tier-rule-no-agent-no-wait" value="false"/></div><div> <param name="discard-abandoned-after" value="60"/></div>
<div> <param name="abandoned-resume-allowed" value="false"/></div><div> </queue></div><div> </queues></div></font></div></blockquote><div><font class="Apple-style-span" size="1"><br>
</font></div><div>So to summarize, most calls work exactly as expected, but a percentage of our total call volume is currently having this issue. We are taking around 200 calls per day at the moment but we plan to take many more in the coming month. We have another system with an IVR in front of this and it only hands callers of a particular ivr option to this server.</div>
<div><br></div><div>I should also note that we are running Dual Quad Core Intel Xeon's, 8GB DDR2, Suse 11.4 - x86_64, Mono 1.8.2. We are also testing 3 Polycom 450's and 5 X-Lite SoftPhones with our agents.</div><div>
<br></div><div><div>Any suggestions or help would be appreciated.</div><div><br></div></div><div>Thanks!</div><div><br></div><div>Ryan H</div>