[Freeswitch-users] How to implement TTS barge-in using FS ESL
xl127
x.liu at hw.ac.uk
Wed Nov 16 18:24:29 MSK 2011
Hi Christopher,
Now it works in my ESL app though I am just able to do one dialogue ( I
need to add the event catching for furthur dialgoues).
I have a couple of questions here:
1. In the first try, my Nuance server was able to be accessed somehow
(FS says the MRCP is not responding in 5000ms,
something like that), then FS says: [WARNING] rtsp_client.c:386 ()
Failed to Connect to RTSP Server 99.185.85.31:554,
later FS says:
[ERR] mod_unimrcp.c:1860 (TTS-6) SYNTHESIZER channel error!
[ERR] switch_ivr_play_say.c:2439 Invalid TTS module!
The SYNTHESIZER channel error and Invalid TTS module error are obvious.
What I don't understand is why it went to this stange address:
99.185.85.31:554?
2. I specified TTS engine in play_and_detect_speech as
"say:unimrcp:nuance5-mrcp1-1: the text to speak"
It works though I didn't specify the TTS voice.
How do I specify the TTS voice? In the mrcp profile (how?)? or
something like:
"say:unimrcp:nuance5-mrcp1-1:Serena: the text to speak" (this
seems not right.)
3. The barge-in works well, thanks!. Is the barge-in configurable? In
some scenarios, we might not allow barge-in.
4. How could I get the text which has spoken to the user when
barge-in occurs?
Or Could I get the time when barge-in occurs? If I know the
barge-in time and rough totale time for the whole text
to be spoken I can figure out the spoken text by manually checking
the recorded audio file later, which would be painful.
5. when I use "speak" and "detect_speech" apps in ESL, I can catch
event: DETECTED_SPEECH and speech-type: begin-speaking
and "detected-speech", then I do the recognition results processing.
The new app play_and_detect_speech seems not generate these events
any more. The way that I can think of to get the results
is to catch event:CHANNEL_EXECUTE_COMPLETE then check if
variable_current_application=play_and_detect_speech, then get
the results from variable_detect_speech_result.
Is this the proper way to get the results in ESL app? Or will
play_and_detect_speech later on be consistent with detect_speech
in term of ASR events?
6. I'd like to set start-input-timers=false in the initial request
then start the recognition timers (start-input-timers=true)
after the TTS finishes.
How possibly could I do this?
Apologies for so many questions!
Cheers,
Xing
On 16/11/11 13:10, Christopher Rienzo wrote:
> You are missing the tts_engine. Set the tts_engine channel variable
> to "unimrcp:nuance5-mrcp1-1" or set the default tts profile to
> nuance5-mrcp1-1 in the unimrcp.conf.xml file and use "say:unimrcp:text
> to speak"
>
>
> On Wed, Nov 16, 2011 at 7:36 AM, Liu, Xingkun <X.Liu at hw.ac.uk
> <mailto:X.Liu at hw.ac.uk>> wrote:
>
> Great work, thanks Christopher!
>
> I've quick tested it from the dialplan based on the example. It
> works on my setup.
>
> Now I am trying to put it into my Java ESL app, FS complains:
>
> switch_ivr_play_say.c:1167 Invalid Args
>
> A piece of codes I am using:
>
> SendMsg msg = new SendMsg();
> msg.addCallCommand("execute");
> msg.addExecuteAppName("play_and_detect_speech");
> String arg = "say:" + utt + "
> detect:unimrcp:nuance5-mrcp1-1
> {start-input-timers=false,no-input-timeout=5000,"
> + "recognition-timeout=5000}" +this.crntGrammar;
> msg.addExecuteAppArg(arg);
> EslMessage response = sendSyncMultiLineCommand(channel,
> msg.getMsgLines());
>
> Is there something I am missing?
>
> Cheers,
>
> Xing
>
>
>
>
>
> -----Original Message-----
> From: freeswitch-users-bounces at lists.freeswitch.org
> <mailto:freeswitch-users-bounces at lists.freeswitch.org> on behalf
> of Christopher Rienzo
> Sent: Tue 11/15/2011 20:05
> To: FreeSWITCH Users Help
> Subject: Re: [Freeswitch-users] How to implement TTS barge-in
> using FS ESL
>
> Latest version of FreeSWITCH now has
> http://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_play_and_detect_speech
> Give it a try and let us know if there is anything that can be
> improved.
>
>
>
> On Mon, Nov 14, 2011 at 12:06 PM, xl127 <x.liu at hw.ac.uk
> <mailto:x.liu at hw.ac.uk>> wrote:
>
> > The way you did and described here is similar to what I wanted.
> > I want to set start-input-timers=false in the initial setup,
> then start
> > the recognition timer once the TTS finishes playing the prompt.
> > I do not use input callback as I am using ESL event (Java ESL Client
> > 0.9.2).
> >
> > I've not yet figured out how to know the prompt playing has
> finished, how
> > to stop the TTS and how to start the input timer.
> > I didn't get many events when I did barge-in: only got two
> CHANNEL_EXECUTE
> > and CAHNNEL_EXECUTE_COMPLETE events.
> > so didn't get the DETECTED_SPEECH at this point.
> >
> > In principle it shouldn't stop FS generating events when barging
> in, so
> > there must be somethings wrong in my setup.
> >
> > From Examples_directory_lua_asr_tts in the wiki, it mentions the
> session
> > needs to sleep for a few seconds before getting the results.
> > I tried but it didn't help.
> >
> >
> >
> >
> > On 14/11/11 14:48, Christopher Rienzo wrote:
> >
> > I'm not sure why it locks up for you. It works for me and I
> know of at
> > least one other developer that got it all to work.
> >
> > The way I do it in my custom APP is to set
> {start-input-timers=false} in
> > the speech recognition request and register an input callback
> function to
> > deal with the recognition and DTMF events. The input callback
> reacts to
> > the start of speech and recognition complete events by stopping the
> > prompt. At start of speech, the input timers are started and
> silence is
> > played until the final result arrives from the recognizer (no
> match, match,
> > etc).
> >
> > An alternative way to do this is to watch for the recognition
> events over
> > ESL and react to them.
> >
> >
> >
> > On Mon, Nov 14, 2011 at 9:19 AM, xl127 <x.liu at hw.ac.uk
> <mailto:x.liu at hw.ac.uk>> wrote:
> >
> >> Hi Christopher,
> >>
> >> That will be great! Thanks and looking forward to your app!
> >>
> >> By the way, do you have a quick thoughts about the reasons the
> FS stucks
> >> when garge-in occurs?
> >> Otherwise if it is not straightforward to explain it, just
> leave it
> >> alone until your new app comes out.
> >>
> >> Cheers,
> >> Xing
> >>
> >>
> >>
> >>
> >> On 14/11/11 13:23, Christopher Rienzo wrote:
> >>
> >> I'll write up a dialplan app this week to deal with this.
> Getting tired
> >> of being asked the same question over and over since it's too
> complicated
> >> as is currently designed :)
> >>
> >>
> >>
> >>
> >> On Mon, Nov 14, 2011 at 8:07 AM, xl127 <x.liu at hw.ac.uk
> <mailto:x.liu at hw.ac.uk>> wrote:
> >>
> >>> Hi,
> >>>
> >>> The first problem I need to solve is that FS stucks when I
> speak during
> >>> the prompt is playing.
> >>> It looks like the recognizer is not listening any more. It
> works fine if
> >>> I speak after the playing prompt finishes.
> >>>
> >>> I tried Nuance and PocketSphinx recognizers and got same problem.
> >>>
> >>> Any idea about what the possible causes are?
> >>>
> >>> Thanks!
> >>> Xing
> >>>
> >>>
> >>> On 11/11/11 18:59, xl127 wrote:
> >>> > Hello,
> >>> >
> >>> > I found this is a question that was asked before by others,
> but I
> >>> didn't
> >>> > find the answer.
> >>> >
> >>> > Anyway, I am using FS ESL outbound mode connecting to my IVR
> app,
> >>> using
> >>> > FS's "speak"
> >>> > and "detect_speech" to access Nuance MRCP V1 server.
> >>> >
> >>> > I want the user to be able to barge-in during the system's
> speaking.
> >>> > How could I implement it?
> >>> >
> >>> > I tried to specify "kill-on-barge-in=true" in the mrcp
> config profile.
> >>> > The barge-in doesn't work.
> >>> >
> >>> > With or without setting kill-on-barge-in, FS stops
> responding to my
> >>> > phone call and
> >>> > eventually it hangs up my call if I speak somthing (do the
> barge-in)
> >>> > during the system's speaking.
> >>> >
> >>> > I made a turn-by-turn loop in my app, the ASR/TTS works fine
> if I do
> >>> not
> >>> > do barge-in ( I wait until the TTS finishes then I start to
> speak)
> >>> >
> >>> > Any advices please?
> >>> >
> >>> > Thanks!
> >>> > Xing
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Heriot-Watt University is a Scottish charity
> >>> registered under charity number SC000278.
> >>>
> >>> Heriot-Watt University is the Sunday Times
> >>> Scottish University of the Year 2011-2012
> >>>
> >>>
> >>>
> >>>
> _________________________________________________________________________
> >>> Professional FreeSWITCH Consulting Services:
> >>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> >>> http://www.freeswitchsolutions.com
> >>>
> >>>
> >>>
> >>>
> >>> Official FreeSWITCH Sites
> >>> http://www.freeswitch.org
> >>> http://wiki.freeswitch.org
> >>> http://www.cluecon.com
> >>>
> >>> FreeSWITCH-users mailing list
> >>> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>> http://www.freeswitch.org
> >>>
> >>
> >>
> >>
> >>
> _________________________________________________________________________
> >> Professional FreeSWITCH Consulting
> Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
> <http://www.freeswitchsolutions.com>
>
> >>
> >> FreeSWITCH-powered IP PBX: The CudaTel Communication
> Server <>
> >>
> >> Official FreeSWITCH
> Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
> <http://www.cluecon.com>
> >>
> >> FreeSWITCH-users mailing
> listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
> >>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
> >>
> >>
> >>
> >> ------------------------------
> >>
> >> [image: Scottish University of the Year 2011-12] *Heriot-Watt
>
> >> University is the Sunday Times
> >>
> >> Scottish University of the Year 2011-2012
> >> *
> >>
> >> Heriot-Watt University is a Scottish charity
> >> registered under charity number SC000278.
> >>
> >>
> _________________________________________________________________________
> >> Professional FreeSWITCH Consulting Services:
> >> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> >> http://www.freeswitchsolutions.com
> >>
> >>
> >>
> >>
> >> Official FreeSWITCH Sites
> >> http://www.freeswitch.org
> >> http://wiki.freeswitch.org
> >> http://www.cluecon.com
> >>
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >>
> >>
> >
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting
> Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
> <http://www.freeswitchsolutions.com>
>
> >
> > FreeSWITCH-powered IP PBX: The CudaTel Communication
> Server <>
> >
> > Official FreeSWITCH
> Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
> <http://www.cluecon.com>
> >
> > FreeSWITCH-users mailing
> listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
> >
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
> >
> >
> >
> > ------------------------------
> >
> > [image: Scottish University of the Year 2011-12] *Heriot-Watt
>
> > University is the Sunday Times
> > Scottish University of the Year 2011-2012*
>
> >
> > Heriot-Watt University is a Scottish charity
> > registered under charity number SC000278.
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > http://www.freeswitchsolutions.com
> >
> >
> >
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
>
>
> ------------------------------------------------------------------------
>
> Scottish University of the Year 2011-12 *Heriot-Watt University is
> the Sunday Times
> Scottish University of the Year 2011-2012*
>
> Heriot-Watt University is a Scottish charity
> registered under charity number SC000278.
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
--
Heriot-Watt University is a Scottish charity
registered under charity number SC000278.
Heriot-Watt University is the Sunday Times
Scottish University of the Year 2011-2012
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20111116/10ac67e8/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 4803 bytes
Desc: not available
Url : http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20111116/10ac67e8/attachment-0001.jpe
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hw_uni_of_year.jpg
Type: image/jpeg
Size: 4803 bytes
Desc: not available
Url : http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20111116/10ac67e8/attachment-0001.jpg
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list