[Freeswitch-users] How to implement TTS barge-in using FS ESL

xl127 x.liu at hw.ac.uk
Wed Nov 16 18:24:29 MSK 2011


Hi Christopher,

Now it works in my ESL app though I am just able to do one dialogue ( I 
need to add the event catching for furthur dialgoues).

I have a couple of questions here:

   1. In the first try, my Nuance server was able to be accessed somehow 
(FS says the MRCP is not responding in 5000ms,
     something like that), then FS says: [WARNING] rtsp_client.c:386 () 
Failed to Connect to RTSP Server 99.185.85.31:554,
     later FS says:
      [ERR] mod_unimrcp.c:1860 (TTS-6) SYNTHESIZER channel error!
      [ERR] switch_ivr_play_say.c:2439 Invalid TTS module!

    The SYNTHESIZER channel error and Invalid TTS module error are obvious.

     What I don't understand is why it went to this stange address: 
99.185.85.31:554?

   2. I specified TTS engine in play_and_detect_speech as
          "say:unimrcp:nuance5-mrcp1-1: the text to speak"
      It works though I didn't specify the TTS voice.

      How do I specify the TTS voice? In the mrcp profile (how?)? or 
something like:
          "say:unimrcp:nuance5-mrcp1-1:Serena: the text to speak" (this 
seems not right.)

   3. The barge-in works well, thanks!. Is the barge-in configurable? In 
some scenarios, we might not allow barge-in.

   4. How could I get the text which has spoken to the user when 
barge-in occurs?
      Or Could I get the time when barge-in occurs? If I know the 
barge-in time and rough totale time for the whole text
      to be spoken I can figure out the spoken text by manually checking 
the recorded audio file later, which would be painful.

   5. when I use "speak" and "detect_speech" apps in ESL, I can catch 
event: DETECTED_SPEECH and speech-type: begin-speaking
      and "detected-speech", then I do the recognition results processing.

     The new app play_and_detect_speech seems not generate these events 
any more. The way that I can think of to get the results
     is to catch event:CHANNEL_EXECUTE_COMPLETE then check if 
variable_current_application=play_and_detect_speech, then get
     the results from variable_detect_speech_result.

     Is this the proper way to get the results in ESL app? Or will 
play_and_detect_speech later on be consistent with detect_speech
     in term of ASR events?

   6. I'd like to set start-input-timers=false in the initial request 
then start the recognition timers (start-input-timers=true)
      after the TTS finishes.
      How possibly could I do this?

   Apologies for so many questions!

   Cheers,
   Xing



On 16/11/11 13:10, Christopher Rienzo wrote:
> You are missing the tts_engine.  Set the tts_engine channel variable 
> to "unimrcp:nuance5-mrcp1-1" or set the default tts profile to 
> nuance5-mrcp1-1 in the unimrcp.conf.xml file and use "say:unimrcp:text 
> to speak"
>
>
> On Wed, Nov 16, 2011 at 7:36 AM, Liu, Xingkun <X.Liu at hw.ac.uk 
> <mailto:X.Liu at hw.ac.uk>> wrote:
>
>     Great work, thanks Christopher!
>
>     I've quick tested it from the dialplan based on the example. It
>     works on my setup.
>
>     Now I am trying to put it into my Java ESL app, FS complains:
>
>      switch_ivr_play_say.c:1167 Invalid Args
>
>     A piece of codes I am using:
>
>             SendMsg msg = new SendMsg();
>             msg.addCallCommand("execute");
>             msg.addExecuteAppName("play_and_detect_speech");
>             String arg = "say:" + utt + "
>     detect:unimrcp:nuance5-mrcp1-1
>     {start-input-timers=false,no-input-timeout=5000,"
>                       + "recognition-timeout=5000}" +this.crntGrammar;
>             msg.addExecuteAppArg(arg);
>             EslMessage response = sendSyncMultiLineCommand(channel,
>     msg.getMsgLines());
>
>     Is there something I am missing?
>
>     Cheers,
>
>     Xing
>
>
>
>
>
>     -----Original Message-----
>     From: freeswitch-users-bounces at lists.freeswitch.org
>     <mailto:freeswitch-users-bounces at lists.freeswitch.org> on behalf
>     of Christopher Rienzo
>     Sent: Tue 11/15/2011 20:05
>     To: FreeSWITCH Users Help
>     Subject: Re: [Freeswitch-users] How to implement TTS barge-in
>     using FS ESL
>
>     Latest version of FreeSWITCH now has
>     http://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_play_and_detect_speech
>     Give it a try and let us know if there is anything that can be
>     improved.
>
>
>
>     On Mon, Nov 14, 2011 at 12:06 PM, xl127 <x.liu at hw.ac.uk
>     <mailto:x.liu at hw.ac.uk>> wrote:
>
>     >  The way you did and described here is similar to what I wanted.
>     > I want to set start-input-timers=false in the initial setup,
>     then start
>     > the recognition timer once the TTS finishes playing the prompt.
>     > I do not use input callback as I am using ESL event (Java ESL Client
>     > 0.9.2).
>     >
>     > I've not yet figured out how to know the prompt playing has
>     finished, how
>     > to stop the TTS and how to start the input timer.
>     > I didn't get many events when I did barge-in: only got two
>     CHANNEL_EXECUTE
>     > and CAHNNEL_EXECUTE_COMPLETE events.
>     > so didn't get the DETECTED_SPEECH at this point.
>     >
>     > In principle it shouldn't stop FS generating events when barging
>     in, so
>     > there must be somethings wrong in my setup.
>     >
>     > From Examples_directory_lua_asr_tts in the wiki, it mentions the
>     session
>     > needs to sleep for a few seconds before getting the results.
>     > I tried but it didn't help.
>     >
>     >
>     >
>     >
>     > On 14/11/11 14:48, Christopher Rienzo wrote:
>     >
>     > I'm not sure why it locks up for you.  It works for me and I
>     know of at
>     > least one other developer that got it all to work.
>     >
>     > The way I do it in my custom APP is to set
>     {start-input-timers=false} in
>     > the speech recognition request and register an input callback
>     function to
>     > deal with the recognition and DTMF events.  The input callback
>     reacts to
>     > the start of speech and recognition complete events by stopping the
>     > prompt.  At start of speech, the input timers are started and
>     silence is
>     > played until the final result arrives from the recognizer (no
>     match, match,
>     > etc).
>     >
>     > An alternative way to do this is to watch for the recognition
>     events over
>     > ESL and react to them.
>     >
>     >
>     >
>     > On Mon, Nov 14, 2011 at 9:19 AM, xl127 <x.liu at hw.ac.uk
>     <mailto:x.liu at hw.ac.uk>> wrote:
>     >
>     >>  Hi Christopher,
>     >>
>     >> That will be great! Thanks and looking forward to your app!
>     >>
>     >> By the way, do you have a quick thoughts about the reasons the
>     FS stucks
>     >> when garge-in occurs?
>     >> Otherwise if it is not straightforward to explain it,  just
>     leave it
>     >> alone until your new app comes out.
>     >>
>     >> Cheers,
>     >> Xing
>     >>
>     >>
>     >>
>     >>
>     >> On 14/11/11 13:23, Christopher Rienzo wrote:
>     >>
>     >> I'll write up a dialplan app this week to deal with this. 
>     Getting tired
>     >> of being asked the same question over and over since it's too
>     complicated
>     >> as is currently designed :)
>     >>
>     >>
>     >>
>     >>
>     >>  On Mon, Nov 14, 2011 at 8:07 AM, xl127 <x.liu at hw.ac.uk
>     <mailto:x.liu at hw.ac.uk>> wrote:
>     >>
>     >>> Hi,
>     >>>
>     >>> The first problem I need to solve is that FS stucks when I
>     speak during
>     >>> the prompt is playing.
>     >>> It looks like the recognizer is not listening any more. It
>     works fine if
>     >>> I speak after the playing prompt finishes.
>     >>>
>     >>> I tried Nuance and PocketSphinx recognizers and got same problem.
>     >>>
>     >>> Any idea about what the possible causes are?
>     >>>
>     >>> Thanks!
>     >>> Xing
>     >>>
>     >>>
>     >>> On 11/11/11 18:59, xl127 wrote:
>     >>> > Hello,
>     >>> >
>     >>> > I found this is a question that was asked before by others,
>     but I
>     >>> didn't
>     >>> > find the answer.
>     >>> >
>     >>> > Anyway, I am using FS ESL outbound mode connecting to my IVR
>     app,
>     >>>  using
>     >>> > FS's "speak"
>     >>> > and "detect_speech" to access Nuance MRCP V1 server.
>     >>> >
>     >>> > I want the user to be able to barge-in during the system's
>     speaking.
>     >>> > How could I implement it?
>     >>> >
>     >>> > I tried to specify "kill-on-barge-in=true" in the mrcp
>     config profile.
>     >>> > The barge-in doesn't work.
>     >>> >
>     >>> > With or without setting kill-on-barge-in, FS stops
>     responding to my
>     >>> > phone call and
>     >>> > eventually it hangs up my call if I speak somthing (do the
>     barge-in)
>     >>> > during the system's speaking.
>     >>> >
>     >>> > I made a turn-by-turn loop in my app, the ASR/TTS works fine
>     if I do
>     >>> not
>     >>> > do barge-in ( I wait until the TTS finishes then I start to
>     speak)
>     >>> >
>     >>> > Any advices please?
>     >>> >
>     >>> > Thanks!
>     >>> > Xing
>     >>> >
>     >>> >
>     >>> >
>     >>>
>     >>>
>     >>>
>     >>> --
>     >>> Heriot-Watt University is a Scottish charity
>     >>> registered under charity number SC000278.
>     >>>
>     >>> Heriot-Watt University is the Sunday Times
>     >>> Scottish University of the Year 2011-2012
>     >>>
>     >>>
>     >>>
>     >>>
>     _________________________________________________________________________
>     >>> Professional FreeSWITCH Consulting Services:
>     >>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     >>> http://www.freeswitchsolutions.com
>     >>>
>     >>> 
>     >>> 
>     >>>
>     >>> Official FreeSWITCH Sites
>     >>> http://www.freeswitch.org
>     >>> http://wiki.freeswitch.org
>     >>> http://www.cluecon.com
>     >>>
>     >>> FreeSWITCH-users mailing list
>     >>> FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     >>>
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     >>> http://www.freeswitch.org
>     >>>
>     >>
>     >>
>     >>
>     >>
>     _________________________________________________________________________
>     >> Professional FreeSWITCH Consulting
>     Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>     <http://www.freeswitchsolutions.com>
>
>     >>
>     >> FreeSWITCH-powered IP PBX: The CudaTel Communication
>     Server <>
>     >>
>     >> Official FreeSWITCH
>     Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
>     <http://www.cluecon.com>
>     >>
>     >> FreeSWITCH-users mailing
>     listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>     >>
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>     >>
>     >>
>     >>
>     >>  ------------------------------
>     >>
>     >> [image: Scottish University of the Year 2011-12]   *Heriot-Watt
>
>     >> University is the Sunday Times
>     >>
>     >>   Scottish University of the Year 2011-2012
>     >> *
>     >>
>     >>    Heriot-Watt University is a Scottish charity
>     >>   registered under charity number SC000278.
>     >>
>     >>
>     _________________________________________________________________________
>     >> Professional FreeSWITCH Consulting Services:
>     >> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     >> http://www.freeswitchsolutions.com
>     >>
>     >> 
>     >> 
>     >>
>     >> Official FreeSWITCH Sites
>     >> http://www.freeswitch.org
>     >> http://wiki.freeswitch.org
>     >> http://www.cluecon.com
>     >>
>     >> FreeSWITCH-users mailing list
>     >> FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     >>
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     >> http://www.freeswitch.org
>     >>
>     >>
>     >
>     >
>     >
>     _________________________________________________________________________
>     > Professional FreeSWITCH Consulting
>     Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>     <http://www.freeswitchsolutions.com>
>
>     >
>     > FreeSWITCH-powered IP PBX: The CudaTel Communication
>     Server <>
>     >
>     > Official FreeSWITCH
>     Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
>     <http://www.cluecon.com>
>     >
>     > FreeSWITCH-users mailing
>     listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>     >
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>     >
>     >
>     >
>     > ------------------------------
>     >
>     > [image: Scottish University of the Year 2011-12]   *Heriot-Watt
>
>     > University is the Sunday Times
>     >   Scottish University of the Year 2011-2012*
>
>     >
>     >    Heriot-Watt University is a Scottish charity
>     >   registered under charity number SC000278.
>     >
>     >
>     _________________________________________________________________________
>     > Professional FreeSWITCH Consulting Services:
>     > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     > http://www.freeswitchsolutions.com
>     >
>     > 
>     > 
>     >
>     > Official FreeSWITCH Sites
>     > http://www.freeswitch.org
>     > http://wiki.freeswitch.org
>     > http://www.cluecon.com
>     >
>     > FreeSWITCH-users mailing list
>     > FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     >
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     > http://www.freeswitch.org
>     >
>     >
>
>
>     ------------------------------------------------------------------------
>
>     Scottish University of the Year 2011-12 *Heriot-Watt University is
>     the Sunday Times
>       Scottish University of the Year 2011-2012*
>
>       Heriot-Watt University is a Scottish charity
>       registered under charity number SC000278.
>
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     http://www.freeswitchsolutions.com
>
>     
>     
>
>     Official FreeSWITCH Sites
>     http://www.freeswitch.org
>     http://wiki.freeswitch.org
>     http://www.cluecon.com
>
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
>
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org



-- 
Heriot-Watt University is a Scottish charity
registered under charity number SC000278.

Heriot-Watt University is the Sunday Times
Scottish University of the Year 2011-2012


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20111116/10ac67e8/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 4803 bytes
Desc: not available
Url : http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20111116/10ac67e8/attachment-0001.jpe 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hw_uni_of_year.jpg
Type: image/jpeg
Size: 4803 bytes
Desc: not available
Url : http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20111116/10ac67e8/attachment-0001.jpg 


Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list