Author: mikej Date: Fri Jun 26 15:18:39 2009 New Revision: 13987 Log: update to current unimrcp https://unimrcp.googlecode.com/svn/trunk@1014 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Fri Jun 26 18:17:10 2009 +0000 Implemented START-INPUT-TIMERS git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1014 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Fri Jun 26 16:56:38 2009 +0000 Fixed grammar file creation (the existing file must be truncated/overridden) git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1013 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Fri Jun 26 15:31:47 2009 +0000 Added GET-RESULT processing, properly set completion causes in the responses to DEFINE-GRAMMAR and RECOGNIZE requests git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1012 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Thu Jun 25 19:21:34 2009 +0000 Implemented grammar load/unload (DEFINE-GRAMMAR requests), implemented recognition complete timeout, introduced properties (to be loaded from config file) git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1011 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Wed Jun 24 14:31:37 2009 +0000 Added the NOTICE corresponding to the section 4 d of the Apache License, Version 2.0 git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1010 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Tue Jun 23 21:04:44 2009 +0000 Added identifier@pocketsphinx traces git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1009 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Tue Jun 23 21:03:02 2009 +0000 Added session identifier to engine channel (to be used for traces) git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1008 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Tue Jun 23 19:58:48 2009 +0000 Properly handled race between STOP request and RECOGNITION-COMPLETE event, other enhancements git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1007 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Tue Jun 23 19:55:57 2009 +0000 Added mpf_activity_detector_reset() git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1006 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Tue Jun 23 16:41:51 2009 +0000 Added 32kHz and 48kHz sampling rates (still no resampling support in MPF, also make sure RTP packet length <= 1500 bytes) git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1005 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Mon Jun 22 20:13:29 2009 +0000 Added flite m4 configure stuff git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1004 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Mon Jun 22 19:31:32 2009 +0000 Added pocketsphinx.vcproj and flite.vcproj project files into unimrcp.sln (disabled by default) git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1003 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Mon Jun 22 19:26:38 2009 +0000 Integrated utterance detection stuff (start-of-input, end-of-input), triggered partial and final recognition results git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1002 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: garmt.noname@gmail.com <garmt.noname@gmail.com@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Sun Jun 21 22:12:37 2009 +0000 First implementation of flite plugin - voice awb only git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1001 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Sun Jun 21 06:23:58 2009 +0000 Added framework for the development of flite plugin git-svn-id: https://unimrcp.googlecode.com/svn/trunk@1000 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Sat Jun 20 19:52:22 2009 +0000 Added missing include paths and libs required for pocketsphinx git-svn-id: https://unimrcp.googlecode.com/svn/trunk@999 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Sat Jun 20 17:32:37 2009 +0000 Started actual implementation of pocketsphinx plugin git-svn-id: https://unimrcp.googlecode.com/svn/trunk@998 f001bc3a-424a-0410-80a0-a715b8f413a8 Author: achaloyan <achaloyan@f001bc3a-424a-0410-80a0-a715b8f413a8> Date: Sat Jun 20 13:55:41 2009 +0000 Added acmacros for pocketsphinx libs (src and installed dirs are suppored), cleaned up ./configure related stuff a bit git-svn-id: https://unimrcp.googlecode.com/svn/trunk@997 f001bc3a-424a-0410-80a0-a715b8f413a8 Added: freeswitch/trunk/libs/unimrcp/NOTICE freeswitch/trunk/libs/unimrcp/build/acmacros/flite.m4 freeswitch/trunk/libs/unimrcp/build/acmacros/pocketsphinx.m4 freeswitch/trunk/libs/unimrcp/build/acmacros/sphinxbase.m4 freeswitch/trunk/libs/unimrcp/build/vsprops/flite.vsprops freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/ freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/Makefile.am freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/mrcpflite.vcproj freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/src/ freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/src/mrcp_flite.c Modified: freeswitch/trunk/libs/unimrcp/.update freeswitch/trunk/libs/unimrcp/acinclude.m4 freeswitch/trunk/libs/unimrcp/build/acmacros/sofia-sip.m4 freeswitch/trunk/libs/unimrcp/build/acmacros/swift.m4 freeswitch/trunk/libs/unimrcp/configure.ac freeswitch/trunk/libs/unimrcp/libs/mpf/include/mpf_activity_detector.h freeswitch/trunk/libs/unimrcp/libs/mpf/include/mpf_codec_descriptor.h freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_activity_detector.c freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_codec_g711.c freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_codec_linear.c freeswitch/trunk/libs/unimrcp/libs/mrcp-engine/include/mrcp_resource_engine.h freeswitch/trunk/libs/unimrcp/libs/mrcp-engine/src/mrcp_resource_engine.c freeswitch/trunk/libs/unimrcp/libs/mrcp-server/src/mrcp_server_session.c freeswitch/trunk/libs/unimrcp/plugins/Makefile.am freeswitch/trunk/libs/unimrcp/plugins/mrcp-pocketsphinx/Makefile.am freeswitch/trunk/libs/unimrcp/plugins/mrcp-pocketsphinx/src/mrcp_pocketsphinx.c freeswitch/trunk/libs/unimrcp/unimrcp.sln Modified: freeswitch/trunk/libs/unimrcp/.update ============================================================================== --- freeswitch/trunk/libs/unimrcp/.update (original) +++ freeswitch/trunk/libs/unimrcp/.update Fri Jun 26 15:18:39 2009 @@ -1 +1 @@ -Fri Jun 19 22:39:34 CDT 2009 +Fri Jun 26 15:16:40 CDT 2009 Added: freeswitch/trunk/libs/unimrcp/NOTICE ============================================================================== --- (empty file) +++ freeswitch/trunk/libs/unimrcp/NOTICE Fri Jun 26 15:18:39 2009 @@ -0,0 +1,47 @@ +The UniMRCP Project (http://www.unimrcp.org) +Copyright (C) 2008 Arsen Chaloyan +Licensed under the Apache License, Version 2.0 (the "License"). + +This product includes a number of subcomponents with +separate copyright notices and license terms. + + +Notice for APR (Apache Portable Runtime) library +--------------------------------------------------- +This product includes software developed by +The Apache Software Foundation (http://www.apache.org/). + +Portions of this software were developed at the National Center +for Supercomputing Applications (NCSA) at the University of +Illinois at Urbana-Champaign. + +This software contains code derived from the RSA Data Security +Inc. MD5 Message-Digest Algorithm. +--------------------------------------------------- + + +Notice for Sofia-SIP library +--------------------------------------------------- +Copyright (C) 2005-2006 Nokia Corporation and others (see the +in individual files for a detailed list of copyright holders). +Contact: Pekka Pessi <Pekka.Pessi@nokia.com> + +This library is free software; you can redistribute it and/or +modify it under the terms of the GNU Lesser General Public License +as published by the Free Software Foundation; either version 2.1 of +the License, or (at your option) any later version. +--------------------------------------------------- + + +Notice for G711 implementation +--------------------------------------------------- +g711.h/g711.c - A-law and u-law transcoding routines + +Written by Steve Underwood <steveu@coppice.org> +Copyright (C) 2006 Steve Underwood + +Despite my general liking of the GPL, I place this code in the +public domain for the benefit of all mankind - even the slimy +ones who might try to proprietize my work and use it to my +detriment. +--------------------------------------------------- Modified: freeswitch/trunk/libs/unimrcp/acinclude.m4 ============================================================================== --- freeswitch/trunk/libs/unimrcp/acinclude.m4 (original) +++ freeswitch/trunk/libs/unimrcp/acinclude.m4 Fri Jun 26 15:18:39 2009 @@ -4,3 +4,6 @@ m4_include([build/acmacros/find_apu.m4]) m4_include([build/acmacros/sofia-sip.m4]) m4_include([build/acmacros/swift.m4]) +m4_include([build/acmacros/sphinxbase.m4]) +m4_include([build/acmacros/pocketsphinx.m4]) +m4_include([build/acmacros/flite.m4]) Added: freeswitch/trunk/libs/unimrcp/build/acmacros/flite.m4 ============================================================================== --- (empty file) +++ freeswitch/trunk/libs/unimrcp/build/acmacros/flite.m4 Fri Jun 26 15:18:39 2009 @@ -0,0 +1,48 @@ +dnl UNIMRCP_CHECK_FLITE + +AC_DEFUN([UNIMRCP_CHECK_FLITE], +[ + AC_MSG_NOTICE([Flite library configuration]) + + AC_MSG_CHECKING([for Flite]) + AC_ARG_WITH(flite, + [ --with-flite=PATH prefix for installed Flite or + path to Flite build tree], + [flite_path=$withval], + [flite_path="/usr/local"] + ) + + found_flite="no" + + dnl TO BE DONE + flite_libdir="build/libs" + for dir in $flite_path ; do + cd $dir && flite_dir=`pwd` && cd - > /dev/null + if test -d "$dir/$flite_libdir"; then + found_flite="yes" + UNIMRCP_FLITE_INCLUDES="-I$flite_dir/include" + UNIMRCP_FLITE_LIBS="$dir/$flite_libdir/libflite_cmu_us_awb.a \ + $dir/$flite_libdir/libflite_cmu_us_kal.a \ + $dir/$flite_libdir/libflite_cmu_us_rms.a \ + $dir/$flite_libdir/libflite_cmu_us_slt.a \ + $dir/$flite_libdir/libflite_cmulex.a \ + $dir/$flite_libdir/libflite_usenglish.a \ + $dir/$flite_libdir/libflite.a" + break + fi + done + + if test x_$found_flite != x_yes; then + AC_MSG_ERROR(Cannot find Flite - looked for srcdir:$flite_srcdir in $flite_path) + else + AC_MSG_RESULT([$found_flite]) + +case "$host" in + *darwin*) + UNIMRCP_FLITE_LIBS="$UNIMRCP_FLITE_LIBS -framework CoreFoundation -framework SystemConfiguration" ;; +esac + + AC_SUBST(UNIMRCP_FLITE_INCLUDES) + AC_SUBST(UNIMRCP_FLITE_LIBS) + fi +]) Added: freeswitch/trunk/libs/unimrcp/build/acmacros/pocketsphinx.m4 ============================================================================== --- (empty file) +++ freeswitch/trunk/libs/unimrcp/build/acmacros/pocketsphinx.m4 Fri Jun 26 15:18:39 2009 @@ -0,0 +1,50 @@ +dnl UNIMRCP_CHECK_POCKETSPHINX + +AC_DEFUN([UNIMRCP_CHECK_POCKETSPHINX], +[ + AC_MSG_NOTICE([PocketSphinx library configuration]) + + AC_MSG_CHECKING([for PocketSphinx]) + AC_ARG_WITH(pocketsphinx, + [ --with-pocketsphinx=PATH prefix for installed PocketSphinx or + path to PocketSphinx build tree], + [pocketsphinx_path=$withval], + [pocketsphinx_path="/usr/local"] + ) + + found_pocketsphinx="no" + pocketsphinx_config="lib/pkgconfig/pocketsphinx.pc" + pocketsphinx_srcdir="src" + for dir in $pocketsphinx_path ; do + cd $dir && pocketsphinx_dir=`pwd` && cd - > /dev/null + if test -f "$dir/$pocketsphinx_config"; then + found_pocketsphinx="yes" + UNIMRCP_POCKETSPHINX_INCLUDES="`pkg-config --cflags $dir/$pocketsphinx_config`" + UNIMRCP_POCKETSPHINX_LIBS="`pkg-config --libs $dir/$pocketsphinx_config`" + pocketsphinx_version="`pkg-config --modversion $dir/$pocketsphinx_config`" + break + fi + if test -d "$dir/$pocketsphinx_srcdir"; then + found_pocketsphinx="yes" + UNIMRCP_POCKETSPHINX_INCLUDES="-I$pocketsphinx_dir/include" + UNIMRCP_POCKETSPHINX_LIBS="$pocketsphinx_dir/$pocketsphinx_srcdir/libpocketsphinx/libpocketsphinx.la" + pocketsphinx_version="`pkg-config --modversion $pocketsphinx_dir/pocketsphinx.pc`" + break + fi + done + + if test x_$found_pocketsphinx != x_yes; then + AC_MSG_ERROR(Cannot find PocketSphinx - looked for pocketsphinx-config:$pocketsphinx_config and srcdir:$pocketsphinx_srcdir in $pocketsphinx_path) + else + AC_MSG_RESULT([$found_pocketsphinx]) + AC_MSG_RESULT([$pocketsphinx_version]) + +case "$host" in + *darwin*) + UNIMRCP_POCKETSPHINX_LIBS="$UNIMRCP_POCKETSPHINX_LIBS -framework CoreFoundation -framework SystemConfiguration" ;; +esac + + AC_SUBST(UNIMRCP_POCKETSPHINX_INCLUDES) + AC_SUBST(UNIMRCP_POCKETSPHINX_LIBS) + fi +]) Modified: freeswitch/trunk/libs/unimrcp/build/acmacros/sofia-sip.m4 ============================================================================== --- freeswitch/trunk/libs/unimrcp/build/acmacros/sofia-sip.m4 (original) +++ freeswitch/trunk/libs/unimrcp/build/acmacros/sofia-sip.m4 Fri Jun 26 15:18:39 2009 @@ -12,6 +12,7 @@ [sofia_path="/usr/local"] ) + found_sofia="no" sofiaconfig="lib/pkgconfig/sofia-sip-ua.pc" sofiasrcdir="libsofia-sip-ua" for dir in $sofia_path ; do Added: freeswitch/trunk/libs/unimrcp/build/acmacros/sphinxbase.m4 ============================================================================== --- (empty file) +++ freeswitch/trunk/libs/unimrcp/build/acmacros/sphinxbase.m4 Fri Jun 26 15:18:39 2009 @@ -0,0 +1,50 @@ +dnl UNIMRCP_CHECK_SPHINXBASE + +AC_DEFUN([UNIMRCP_CHECK_SPHINXBASE], +[ + AC_MSG_NOTICE([SphinxBase library configuration]) + + AC_MSG_CHECKING([for SphinxBase]) + AC_ARG_WITH(sphinxbase, + [ --with-sphinxbase=PATH prefix for installed SphinxBase or + path to SphinxBase build tree], + [sphinxbase_path=$withval], + [sphinxbase_path="/usr/local"] + ) + + found_sphinxbase="no" + sphinxbase_config="lib/pkgconfig/sphinxbase.pc" + sphinxbase_srcdir="src" + for dir in $sphinxbase_path ; do + cd $dir && sphinxbase_dir=`pwd` && cd - > /dev/null + if test -f "$dir/$sphinxbase_config"; then + found_sphinxbase="yes" + UNIMRCP_SPHINXBASE_INCLUDES="`pkg-config --cflags $dir/$sphinxbase_config`" + UNIMRCP_SPHINXBASE_LIBS="`pkg-config --libs $dir/$sphinxbase_config`" + sphinxbase_version="`pkg-config --modversion $dir/$sphinxbase_config`" + break + fi + if test -d "$dir/$sphinxbase_srcdir"; then + found_sphinxbase="yes" + UNIMRCP_SPHINXBASE_INCLUDES="-I$sphinxbase_dir/include" + UNIMRCP_SPHINXBASE_LIBS="$sphinxbase_dir/$sphinxbase_srcdir/libsphinxbase/libsphinxbase.la $sphinxbase_dir/$sphinxbase_srcdir/libsphinxad/libsphinxad.la" + sphinxbase_version="`pkg-config --modversion $sphinxbase_dir/sphinxbase.pc`" + break + fi + done + + if test x_$found_sphinxbase != x_yes; then + AC_MSG_ERROR(Cannot find SphinxBase - looked for sphinxbase-config:$sphinxbase_config and srcdir:$sphinxbase_srcdir in $sphinxbase_path) + else + AC_MSG_RESULT([$found_sphinxbase]) + AC_MSG_RESULT([$sphinxbase_version]) + +case "$host" in + *darwin*) + UNIMRCP_SPHINXBASE_LIBS="$UNIMRCP_SPHINXBASE_LIBS -framework CoreFoundation -framework SystemConfiguration" ;; +esac + + AC_SUBST(UNIMRCP_SPHINXBASE_INCLUDES) + AC_SUBST(UNIMRCP_SPHINXBASE_LIBS) + fi +]) Modified: freeswitch/trunk/libs/unimrcp/build/acmacros/swift.m4 ============================================================================== --- freeswitch/trunk/libs/unimrcp/build/acmacros/swift.m4 (original) +++ freeswitch/trunk/libs/unimrcp/build/acmacros/swift.m4 Fri Jun 26 15:18:39 2009 @@ -25,6 +25,4 @@ else AC_MSG_WARN([not found - looked for $swift_path]) fi - - AM_CONDITIONAL([CEPSTRAL_PLUGIN], [test x_$found_swift = x_yes]) ]) Added: freeswitch/trunk/libs/unimrcp/build/vsprops/flite.vsprops ============================================================================== --- (empty file) +++ freeswitch/trunk/libs/unimrcp/build/vsprops/flite.vsprops Fri Jun 26 15:18:39 2009 @@ -0,0 +1,22 @@ +<?xml version="1.0" encoding="windows-1251"?> +<VisualStudioPropertySheet + ProjectType="Visual C++" + Version="8.00" + Name="flite" + InheritedPropertySheets=".\unibase.vsprops" + > + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories=""$(FliteDir)\include"" + PreprocessorDefinitions="APT_LIB_EXPORT" + /> + <Tool + Name="VCLinkerTool" + AdditionalDependencies="flite.lib" + AdditionalLibraryDirectories=""$(FliteDir)\$(ConfigurationName)"" + /> + <UserMacro + Name="FliteDir" + Value="$(LibRootDir)libs\flite" + /> +</VisualStudioPropertySheet> Modified: freeswitch/trunk/libs/unimrcp/configure.ac ============================================================================== --- freeswitch/trunk/libs/unimrcp/configure.ac (original) +++ freeswitch/trunk/libs/unimrcp/configure.ac Fri Jun 26 15:18:39 2009 @@ -16,19 +16,19 @@ AM_INIT_AUTOMAKE(foreign) -#Set default language +# Set default language AC_LANG_C AC_PROG_CC AC_PROG_CXX AC_PROG_INSTALL -#skip detection of Fortran +# Skip detection of Fortran m4_undefine([AC_PROG_F77]) m4_defun([AC_PROG_F77],[]) AC_PROG_LIBTOOL -# get version information +# Get version information get_version="build/get-version.sh" version_hdr="build/uni_version.h" plugin_version_hdr="libs/mrcp-engine/include/mrcp_resource_plugin.h" @@ -42,12 +42,13 @@ echo "UniMRCP Version: ${UNI_DOTTED_VERSION}" +# Check APR UNIMRCP_CHECK_APR UNIMRCP_CHECK_APU +# Check SOFIA UNIMRCP_CHECK_SOFIA -UNIMRCP_CHECK_SWIFT AC_SUBST(ac_aux_dir) @@ -74,7 +75,7 @@ AX_COMPILER_VENDOR - +#Enable maintainer mode AC_ARG_ENABLE(maintainer-mode, [AC_HELP_STRING([--enable-maintainer-mode ],[turn on debugging and compile time warnings])], [enable_maintainer_mode="$enableval"], @@ -87,6 +88,7 @@ fi fi +#Enable test suites AC_ARG_ENABLE(test-suites, [AC_HELP_STRING([--enable-test-suites ],[build test suites])], [enable_test_suites="$enableval"], @@ -94,6 +96,10 @@ AM_CONDITIONAL([TEST_SUITES],[test "${enable_test_suites}" != "no"]) + +### Plugins ### + +#Enable demo synthesizer plugin AC_ARG_ENABLE(demosynth-plugin, [AC_HELP_STRING([--disable-demosynth-plugin ],[exclude demo synthesizer plugin from build])], [enable_demosynth_plugin="$enableval"], @@ -101,6 +107,8 @@ AM_CONDITIONAL([DEMOSYNTH_PLUGIN],[test "${enable_demosynth_plugin}" = "yes"]) + +#Enable demo recognizer plugin AC_ARG_ENABLE(demorecog-plugin, [AC_HELP_STRING([--disable-demorecog-plugin ],[exclude demo recognizer plugin from build])], [enable_demorecog_plugin="$enableval"], @@ -108,6 +116,51 @@ AM_CONDITIONAL([DEMORECOG_PLUGIN],[test "${enable_demorecog_plugin}" = "yes"]) + +#Enable Cepstral Swift plugin +AC_ARG_ENABLE(cepstral-plugin, + [AC_HELP_STRING([--disable-cepstral-plugin ],[exclude cepstral plugin from build])], + [enable_cepstral_plugin="$enableval"], + [enable_cepstral_plugin="yes"]) + +if test "${enable_cepstral_plugin}" != "no"; then + UNIMRCP_CHECK_SWIFT +fi + +AM_CONDITIONAL([CEPSTRAL_PLUGIN],[test "${enable_cepstral_plugin}" = "yes" &&\ + test "${found_swift}" = "yes"]) + + +#Enable PocketSphinx plugin +AC_ARG_ENABLE(pocketsphinx-plugin, + [AC_HELP_STRING([--enable-pocketsphinx-plugin ],[enable pocketsphinx plugin])], + [enable_pocketsphinx_plugin="$enableval"], + [enable_pocketsphinx_plugin="no"]) + +if test "${enable_pocketsphinx_plugin}" != "no"; then + UNIMRCP_CHECK_SPHINXBASE + UNIMRCP_CHECK_POCKETSPHINX +fi + +AM_CONDITIONAL([POCKETSPHINX_PLUGIN],[test "${enable_pocketsphinx_plugin}" = "yes" &&\ + test "${found_pocketsphinx}" = "yes" &&\ + test "${found_sphinxbase}" = "yes"]) + + +#Enable Flite plugin +AC_ARG_ENABLE(flite-plugin, + [AC_HELP_STRING([--enable-flite-plugin ],[enable flite plugin])], + [enable_flite_plugin="$enableval"], + [enable_flite_plugin="no"]) + +if test "${enable_flite_plugin}" != "no"; then + UNIMRCP_CHECK_FLITE +fi + +AM_CONDITIONAL([FLITE_PLUGIN],[test "${enable_flite_plugin}" = "yes" &&\ + test "${found_flite}" = "yes"]) + + AM_CONDITIONAL(ISMAC, [test `uname -s` = Darwin]) AC_CONFIG_FILES([ @@ -127,6 +180,8 @@ modules/mrcp-unirtsp/Makefile plugins/Makefile plugins/mrcp-cepstral/Makefile + plugins/mrcp-pocketsphinx/Makefile + plugins/mrcp-flite/Makefile plugins/demo-synth/Makefile plugins/demo-recog/Makefile platforms/Makefile @@ -148,3 +203,16 @@ ]) AC_OUTPUT + +AC_MSG_NOTICE +AC_MSG_NOTICE([Report:]) +AC_MSG_NOTICE([ UniMRCP: $UNI_DOTTED_VERSION]) +AC_MSG_NOTICE([ APR: $apr_version]) +AC_MSG_NOTICE([ APR-Util: $apu_version]) +AC_MSG_NOTICE([ Sofia-SIP: $sofia_version]) +AC_MSG_NOTICE([Plugins:]) +AC_MSG_NOTICE([ Demo Synthesizer: $enable_demosynth_plugin]) +AC_MSG_NOTICE([ Demo Recognizer: $enable_demorecog_plugin]) +AC_MSG_NOTICE([ Cepstral: $enable_cepstral_plugin]) +AC_MSG_NOTICE([ PocketSphinx: $enable_pocketsphinx_plugin]) +AC_MSG_NOTICE([ Flite: $enable_flite_plugin]) Modified: freeswitch/trunk/libs/unimrcp/libs/mpf/include/mpf_activity_detector.h ============================================================================== --- freeswitch/trunk/libs/unimrcp/libs/mpf/include/mpf_activity_detector.h (original) +++ freeswitch/trunk/libs/unimrcp/libs/mpf/include/mpf_activity_detector.h Fri Jun 26 15:18:39 2009 @@ -42,6 +42,9 @@ /** Create activity detector */ MPF_DECLARE(mpf_activity_detector_t*) mpf_activity_detector_create(apr_pool_t *pool); +/** Reset activity detector */ +MPF_DECLARE(void) mpf_activity_detector_reset(mpf_activity_detector_t *detector); + /** Set threshold of voice activity (silence) level */ MPF_DECLARE(void) mpf_activity_detector_level_set(mpf_activity_detector_t *detector, apr_size_t level_threshold); Modified: freeswitch/trunk/libs/unimrcp/libs/mpf/include/mpf_codec_descriptor.h ============================================================================== --- freeswitch/trunk/libs/unimrcp/libs/mpf/include/mpf_codec_descriptor.h (original) +++ freeswitch/trunk/libs/unimrcp/libs/mpf/include/mpf_codec_descriptor.h Fri Jun 26 15:18:39 2009 @@ -39,7 +39,9 @@ typedef enum { MPF_SAMPLE_RATE_NONE = 0x00, MPF_SAMPLE_RATE_8000 = 0x01, - MPF_SAMPLE_RATE_16000 = 0x02 + MPF_SAMPLE_RATE_16000 = 0x02, + MPF_SAMPLE_RATE_32000 = 0x04, + MPF_SAMPLE_RATE_48000 = 0x08 } mpf_sample_rates_e; /** Codec descriptor declaration */ Modified: freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_activity_detector.c ============================================================================== --- freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_activity_detector.c (original) +++ freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_activity_detector.c Fri Jun 26 15:18:39 2009 @@ -53,6 +53,13 @@ return detector; } +/** Reset activity detector */ +MPF_DECLARE(void) mpf_activity_detector_reset(mpf_activity_detector_t *detector) +{ + detector->duration = 0; + detector->state = DETECTOR_STATE_INACTIVITY; +} + /** Set threshold of voice activity (silence) level */ MPF_DECLARE(void) mpf_activity_detector_level_set(mpf_activity_detector_t *detector, apr_size_t level_threshold) { Modified: freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_codec_g711.c ============================================================================== --- freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_codec_g711.c (original) +++ freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_codec_g711.c Fri Jun 26 15:18:39 2009 @@ -140,15 +140,17 @@ }; static const mpf_codec_attribs_t g711u_attribs = { - {G711u_CODEC_NAME, G711u_CODEC_NAME_LENGTH}, /* codec name */ - 8, /* bits per sample */ - MPF_SAMPLE_RATE_8000 | MPF_SAMPLE_RATE_16000 /* sampling rates */ + {G711u_CODEC_NAME, G711u_CODEC_NAME_LENGTH}, /* codec name */ + 8, /* bits per sample */ + MPF_SAMPLE_RATE_8000 | MPF_SAMPLE_RATE_16000 | + MPF_SAMPLE_RATE_32000 | MPF_SAMPLE_RATE_48000 /* supported sampling rates */ }; static const mpf_codec_attribs_t g711a_attribs = { - {G711a_CODEC_NAME, G711a_CODEC_NAME_LENGTH}, /* codec name */ - 8, /* bits per sample */ - MPF_SAMPLE_RATE_8000 | MPF_SAMPLE_RATE_16000 /* sampling rates */ + {G711a_CODEC_NAME, G711a_CODEC_NAME_LENGTH}, /* codec name */ + 8, /* bits per sample */ + MPF_SAMPLE_RATE_8000 | MPF_SAMPLE_RATE_16000 | + MPF_SAMPLE_RATE_32000 | MPF_SAMPLE_RATE_48000 /* supported sampling rates */ }; mpf_codec_t* mpf_codec_g711u_create(apr_pool_t *pool) Modified: freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_codec_linear.c ============================================================================== --- freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_codec_linear.c (original) +++ freeswitch/trunk/libs/unimrcp/libs/mpf/src/mpf_codec_linear.c Fri Jun 26 15:18:39 2009 @@ -84,15 +84,17 @@ }; static const mpf_codec_attribs_t lpcm_attribs = { - {LPCM_CODEC_NAME, LPCM_CODEC_NAME_LENGTH}, /* codec name */ - 16, /* bits per sample */ - MPF_SAMPLE_RATE_8000 | MPF_SAMPLE_RATE_16000 /* sampling rates */ + {LPCM_CODEC_NAME, LPCM_CODEC_NAME_LENGTH}, /* codec name */ + 16, /* bits per sample */ + MPF_SAMPLE_RATE_8000 | MPF_SAMPLE_RATE_16000 | + MPF_SAMPLE_RATE_32000 | MPF_SAMPLE_RATE_48000 /* supported sampling rates */ }; static const mpf_codec_attribs_t l16_attribs = { - {L16_CODEC_NAME, L16_CODEC_NAME_LENGTH}, /* codec name */ - 16, /* bits per sample */ - MPF_SAMPLE_RATE_8000 | MPF_SAMPLE_RATE_16000 /* sampling rates */ + {L16_CODEC_NAME, L16_CODEC_NAME_LENGTH}, /* codec name */ + 16, /* bits per sample */ + MPF_SAMPLE_RATE_8000 | MPF_SAMPLE_RATE_16000 | + MPF_SAMPLE_RATE_32000 | MPF_SAMPLE_RATE_48000 /* supported sampling rates */ }; mpf_codec_descriptor_t* mpf_codec_lpcm_descriptor_create(apr_uint16_t sampling_rate, apr_byte_t channel_count, apr_pool_t *pool) Modified: freeswitch/trunk/libs/unimrcp/libs/mrcp-engine/include/mrcp_resource_engine.h ============================================================================== --- freeswitch/trunk/libs/unimrcp/libs/mrcp-engine/include/mrcp_resource_engine.h (original) +++ freeswitch/trunk/libs/unimrcp/libs/mrcp-engine/include/mrcp_resource_engine.h Fri Jun 26 15:18:39 2009 @@ -74,6 +74,8 @@ mpf_termination_t *termination; /** Back pointer to resource engine */ mrcp_resource_engine_t *engine; + /** Unique identifier (useful for traces) */ + apt_str_t id; /** Pool to allocate memory from */ apr_pool_t *pool; }; Modified: freeswitch/trunk/libs/unimrcp/libs/mrcp-engine/src/mrcp_resource_engine.c ============================================================================== --- freeswitch/trunk/libs/unimrcp/libs/mrcp-engine/src/mrcp_resource_engine.c (original) +++ freeswitch/trunk/libs/unimrcp/libs/mrcp-engine/src/mrcp_resource_engine.c Fri Jun 26 15:18:39 2009 @@ -51,6 +51,7 @@ channel->termination = termination; channel->engine = engine; channel->pool = pool; + apt_string_reset(&channel->id); return channel; } Modified: freeswitch/trunk/libs/unimrcp/libs/mrcp-server/src/mrcp_server_session.c ============================================================================== --- freeswitch/trunk/libs/unimrcp/libs/mrcp-server/src/mrcp_server_session.c (original) +++ freeswitch/trunk/libs/unimrcp/libs/mrcp-server/src/mrcp_server_session.c Fri Jun 26 15:18:39 2009 @@ -206,6 +206,7 @@ engine_channel = mrcp_server_engine_channel_create(session,resource_name); if(engine_channel) { + engine_channel->id = session->base.id; engine_channel->event_obj = channel; engine_channel->event_vtable = &engine_channel_vtable; channel->engine_channel = engine_channel; Modified: freeswitch/trunk/libs/unimrcp/plugins/Makefile.am ============================================================================== --- freeswitch/trunk/libs/unimrcp/plugins/Makefile.am (original) +++ freeswitch/trunk/libs/unimrcp/plugins/Makefile.am Fri Jun 26 15:18:39 2009 @@ -13,3 +13,11 @@ if CEPSTRAL_PLUGIN SUBDIRS += mrcp-cepstral endif + +if POCKETSPHINX_PLUGIN +SUBDIRS += mrcp-pocketsphinx +endif + +if FLITE_PLUGIN +SUBDIRS += mrcp-flite +endif Added: freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/Makefile.am ============================================================================== --- (empty file) +++ freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/Makefile.am Fri Jun 26 15:18:39 2009 @@ -0,0 +1,19 @@ +MAINTAINERCLEANFILES = Makefile.in + +INCLUDES = -Iinclude \ + -I$(top_srcdir)/libs/mrcp-engine/include \ + -I$(top_srcdir)/libs/mrcp/include \ + -I$(top_srcdir)/libs/mrcp/message/include \ + -I$(top_srcdir)/libs/mrcp/control/include \ + -I$(top_srcdir)/libs/mrcp/resources/include \ + -I$(top_srcdir)/libs/mpf/include \ + -I$(top_srcdir)/libs/apr-toolkit/include \ + $(UNIMRCP_APR_INCLUDES) \ + $(UNIMRCP_APU_INCLUDES) \ + $(UNIMRCP_FLITE_INCLUDES) + +plugin_LTLIBRARIES = mrcpflite.la + +mrcpflite_la_SOURCES = src/mrcp_flite.c +mrcpflite_la_LDFLAGS = -module $(PLUGIN_LT_VERSION) +mrcpflite_la_LIBADD = $(UNIMRCP_FLITE_LIBS) -lm Added: freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/mrcpflite.vcproj ============================================================================== --- (empty file) +++ freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/mrcpflite.vcproj Fri Jun 26 15:18:39 2009 @@ -0,0 +1,165 @@ +<?xml version="1.0" encoding="Windows-1252"?> +<VisualStudioProject + ProjectType="Visual C++" + Version="8.00" + Name="mrcpflite" + ProjectGUID="{56F6FB96-2BC7-4CAE-A8BF-6A0FAEC90556}" + RootNamespace="mrcpflite" + Keyword="Win32Proj" + > + <Platforms> + <Platform + Name="Win32" + /> + </Platforms> + <ToolFiles> + </ToolFiles> + <Configurations> + <Configuration + Name="Debug|Win32" + ConfigurationType="2" + InheritedPropertySheets="$(ProjectDir)..\..\build\vsprops\unidebug.vsprops;$(ProjectDir)..\..\build\vsprops\flite.vsprops;$(ProjectDir)..\..\build\vsprops\unimrcpplugin.vsprops" + CharacterSet="1" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="include" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCWebDeploymentTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + <Configuration + Name="Release|Win32" + ConfigurationType="2" + InheritedPropertySheets="$(ProjectDir)..\..\build\vsprops\unirelease.vsprops;$(ProjectDir)..\..\build\vsprops\flite.vsprops;$(ProjectDir)..\..\build\vsprops\unimrcpplugin.vsprops" + CharacterSet="1" + WholeProgramOptimization="1" + > + <Tool + Name="VCPreBuildEventTool" + /> + <Tool + Name="VCCustomBuildTool" + /> + <Tool + Name="VCXMLDataGeneratorTool" + /> + <Tool + Name="VCWebServiceProxyGeneratorTool" + /> + <Tool + Name="VCMIDLTool" + /> + <Tool + Name="VCCLCompilerTool" + AdditionalIncludeDirectories="include" + /> + <Tool + Name="VCManagedResourceCompilerTool" + /> + <Tool + Name="VCResourceCompilerTool" + /> + <Tool + Name="VCPreLinkEventTool" + /> + <Tool + Name="VCLinkerTool" + /> + <Tool + Name="VCALinkTool" + /> + <Tool + Name="VCManifestTool" + /> + <Tool + Name="VCXDCMakeTool" + /> + <Tool + Name="VCBscMakeTool" + /> + <Tool + Name="VCFxCopTool" + /> + <Tool + Name="VCAppVerifierTool" + /> + <Tool + Name="VCWebDeploymentTool" + /> + <Tool + Name="VCPostBuildEventTool" + /> + </Configuration> + </Configurations> + <References> + </References> + <Files> + <Filter + Name="include" + Filter="h;hpp;hxx;hm;inl;inc;xsd" + UniqueIdentifier="{93995380-89BD-4b04-88EB-625FBE52EBFB}" + > + </Filter> + <Filter + Name="src" + Filter="cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx" + > + <File + RelativePath=".\src\mrcp_flite.c" + > + </File> + </Filter> + </Files> + <Globals> + </Globals> +</VisualStudioProject> Added: freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/src/mrcp_flite.c ============================================================================== --- (empty file) +++ freeswitch/trunk/libs/unimrcp/plugins/mrcp-flite/src/mrcp_flite.c Fri Jun 26 15:18:39 2009 @@ -0,0 +1,749 @@ +/* + * Copyright 2008 Arsen Chaloyan + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * Some mandatory rules for plugin implementation. + * 1. Each plugin MUST contain the following function as an entry point of the plugin + * MRCP_PLUGIN_DECLARE(mrcp_resource_engine_t*) mrcp_plugin_create(apr_pool_t *pool) + * 2. One and only one response MUST be sent back to the received request. + * 3. Methods (callbacks) of the MRCP engine channel MUST not block. + * (asynch response can be sent from the context of other thread) + * 4. Methods (callbacks) of the MPF engine stream MUST not block. + */ + +#include "mrcp_resource_engine.h" +#include "mrcp_synth_resource.h" +#include "mrcp_synth_header.h" +#include "mrcp_generic_header.h" +#include "mrcp_message.h" +#include "mpf_buffer.h" +#include "apr_time.h" +#include "apt_consumer_task.h" +#include "apt_log.h" +#include "flite.h" + +typedef struct flite_synth_engine_t flite_synth_engine_t; +typedef struct flite_synth_channel_t flite_synth_channel_t; +typedef struct flite_synth_msg_t flite_synth_msg_t; + +/** Declaration of synthesizer engine methods */ +static apt_bool_t flite_synth_engine_destroy(mrcp_resource_engine_t *engine); +static apt_bool_t flite_synth_engine_open(mrcp_resource_engine_t *engine); +static apt_bool_t flite_synth_engine_close(mrcp_resource_engine_t *engine); +static mrcp_engine_channel_t* flite_synth_engine_channel_create(mrcp_resource_engine_t *engine, apr_pool_t *pool); + +static const struct mrcp_engine_method_vtable_t engine_vtable = { + flite_synth_engine_destroy, + flite_synth_engine_open, + flite_synth_engine_close, + flite_synth_engine_channel_create +}; + +/** Declaration of synthesizer channel methods */ +static apt_bool_t flite_synth_channel_destroy(mrcp_engine_channel_t *channel); +static apt_bool_t flite_synth_channel_open(mrcp_engine_channel_t *channel); +static apt_bool_t flite_synth_channel_close(mrcp_engine_channel_t *channel); +static apt_bool_t flite_synth_channel_request_process(mrcp_engine_channel_t *channel, mrcp_message_t *request); + +static const struct mrcp_engine_channel_method_vtable_t channel_vtable = { + flite_synth_channel_destroy, + flite_synth_channel_open, + flite_synth_channel_close, + flite_synth_channel_request_process +}; + +/** Declaration of synthesizer audio stream methods */ +static apt_bool_t flite_synth_stream_destroy(mpf_audio_stream_t *stream); +static apt_bool_t flite_synth_stream_open(mpf_audio_stream_t *stream); +static apt_bool_t flite_synth_stream_close(mpf_audio_stream_t *stream); +static apt_bool_t flite_synth_stream_read(mpf_audio_stream_t *stream, mpf_frame_t *frame); + +static const mpf_audio_stream_vtable_t audio_stream_vtable = { + flite_synth_stream_destroy, + flite_synth_stream_open, + flite_synth_stream_close, + flite_synth_stream_read, + NULL, + NULL, + NULL +}; + +/** Declaration of flite synthesizer engine */ +struct flite_synth_engine_t { + apt_consumer_task_t *task; + int iChannels; + apr_thread_mutex_t *guard; +}; + +// flite stuff +APT_BEGIN_EXTERN_C + +cst_voice *register_cmu_us_awb(void); +void unregister_cmu_us_awb(cst_voice * v); + +cst_voice *register_cmu_us_kal(void); +void unregister_cmu_us_kal(cst_voice * v); + +cst_voice *register_cmu_us_rms(void); +void unregister_cmu_us_rms(cst_voice * v); + +cst_voice *register_cmu_us_slt(void); +void unregister_cmu_us_slt(cst_voice * v); + +APT_END_EXTERN_C + +static struct { + cst_voice *awb; + cst_voice *kal; + cst_voice *rms; + cst_voice *slt; +} voices; + +/** Declaration of flite synthesizer channel */ +struct flite_synth_channel_t { + flite_synth_engine_t *flite_engine; // Back pointer to engine + mrcp_engine_channel_t *channel; // Engine channel base + mrcp_message_t *speak_request; // Active (in-progress) speak request + mrcp_message_t *stop_response; // Pending stop response + apt_bool_t paused; // Is paused + mpf_buffer_t *audio_buffer; // Audio buffer + int iId; // Synth channel simultaneous reference count + cst_voice *voice; + cst_wave *wave; + apr_pool_t *pool; + apt_consumer_task_t *task; +}; + +typedef enum flite_synth_msg_type_e { + flite_synth_MSG_OPEN_CHANNEL, + flite_synth_MSG_CLOSE_CHANNEL, + flite_synth_MSG_REQUEST_PROCESS +} flite_synth_msg_type_e; + +/** Declaration of flite synthesizer task message */ +struct flite_synth_msg_t { + flite_synth_msg_type_e type; + mrcp_engine_channel_t *channel; + mrcp_message_t *request; +}; + +/* mutex: may be flite library is not thread safe*/ +static apr_thread_mutex_t *flite_mutex; + +struct flite_speak_msg_t { + flite_synth_channel_t *channel; + mrcp_message_t *request; +}; + +typedef struct flite_speak_msg_t flite_speak_msg_t; + +// all calls to the Flite API functions (after initialization) +// will be carried out using a separate task +static apt_bool_t flite_synth_msg_signal(flite_synth_msg_type_e type, mrcp_engine_channel_t *channel, mrcp_message_t *request); +static apt_bool_t flite_synth_msg_process(apt_task_t *task, apt_task_msg_t *msg); +static apt_bool_t flite_synth_channel_open_t(mrcp_engine_channel_t * channel); +static apt_bool_t flite_synth_channel_close_t(mrcp_engine_channel_t * channel); + +// and we have a special task for the actual synthesis - +// the task is created when a mrcp speak message is received +static apt_bool_t flite_speak_msg_process(apt_task_t *task, apt_task_msg_t *msg); + +/** Declare this macro to use log routine of the server where the plugin is loaded from */ +MRCP_PLUGIN_LOGGER_IMPLEMENT + +/** Create flite synthesizer engine */ +MRCP_PLUGIN_DECLARE(mrcp_resource_engine_t*) mrcp_plugin_create(apr_pool_t *pool) +{ + /* create flite engine */ + flite_synth_engine_t *flite_engine = (flite_synth_engine_t *) apr_palloc(pool,sizeof(flite_synth_engine_t)); + apt_task_msg_pool_t *msg_pool; + apt_task_vtable_t *task_vtable = 0; + + flite_engine->iChannels = 0; + + /* create task/thread to run flite engine in the context of this task */ + msg_pool = apt_task_msg_pool_create_dynamic(sizeof(flite_synth_msg_t),pool); + flite_engine->task = apt_consumer_task_create(flite_engine,msg_pool,pool); + if (!flite_engine->task) + { + apt_log(APT_LOG_MARK, APT_PRIO_ERROR, "MRCP_PLUGIN_DECLARE cannot create task"); + return NULL; + } + task_vtable = apt_consumer_task_vtable_get(flite_engine->task); + if (task_vtable) + { + task_vtable->process_msg = flite_synth_msg_process; + } + else + { + apt_log(APT_LOG_MARK, APT_PRIO_ERROR, "MRCP_PLUGIN_DECLARE cannot use task vtable"); + return NULL; + } + + /* create flite mutex */ + if (apr_thread_mutex_create(&flite_mutex,APR_THREAD_MUTEX_DEFAULT,pool) != APR_SUCCESS) + { + apt_log(APT_LOG_MARK, APT_PRIO_ERROR,"Failed to create flite mutex"); + return NULL; + } + + /* create channel mutex */ + if (apr_thread_mutex_create(&flite_engine->guard,APR_THREAD_MUTEX_DEFAULT,pool) != APR_SUCCESS) + { + apt_log(APT_LOG_MARK, APT_PRIO_ERROR, "Failed to create channel guard"); + return NULL; + } + + /* create resource engine base */ + return mrcp_resource_engine_create( + MRCP_SYNTHESIZER_RESOURCE, /* MRCP resource identifier */ + flite_engine, /* object to associate */ + &engine_vtable, /* virtual methods table of resource engine */ + pool); /* pool to allocate memory from */ +} + +/** Destroy synthesizer engine */ +static apt_bool_t flite_synth_engine_destroy(mrcp_resource_engine_t *engine) +{ + flite_synth_engine_t *flite_engine = (flite_synth_engine_t *) engine->obj; + + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_engine_destroy"); + + if(flite_engine->task) { + apt_task_t *task = apt_consumer_task_base_get(flite_engine->task); + apt_task_destroy(task); + flite_engine->task = NULL; + } + + apr_thread_mutex_destroy(flite_engine->guard); + return TRUE; +} + +/** Open synthesizer engine */ +static apt_bool_t flite_synth_engine_open(mrcp_resource_engine_t *engine) +{ + flite_synth_engine_t *flite_engine = (flite_synth_engine_t *) engine->obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_engine_open"); + + flite_init(); + voices.awb = register_cmu_us_awb(); + voices.kal = register_cmu_us_kal(); + voices.rms = register_cmu_us_rms(); + voices.slt = register_cmu_us_slt(); + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "fliteInitialize success"); + + if (flite_engine->task) { + apt_task_t *task = apt_consumer_task_base_get(flite_engine->task); + apt_task_start(task); + } + return TRUE; +} + +/** Close synthesizer engine */ +static apt_bool_t flite_synth_engine_close(mrcp_resource_engine_t *engine) +{ + flite_synth_engine_t *flite_engine = (flite_synth_engine_t *) engine->obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_engine_close"); + + if (flite_engine->task) + { + apt_task_t *task = apt_consumer_task_base_get(flite_engine->task); + apt_task_terminate(task,TRUE); + } + + unregister_cmu_us_awb(voices.awb); + unregister_cmu_us_kal(voices.kal); + unregister_cmu_us_rms(voices.rms); + unregister_cmu_us_slt(voices.slt); + + return TRUE; +} + +/** Create flite synthesizer channel derived from engine channel base */ +static mrcp_engine_channel_t* flite_synth_engine_channel_create(mrcp_resource_engine_t *engine, apr_pool_t *pool) +{ + /* create flite synth channel */ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) apr_palloc(pool,sizeof(flite_synth_channel_t)); + mpf_codec_descriptor_t *codec_descriptor = NULL; + + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_engine_channel_create"); + +// codec_descriptor = (mpf_codec_descriptor_t *) apr_palloc(pool,sizeof(mpf_codec_descriptor_t)); +// mpf_codec_descriptor_init(codec_descriptor); +// codec_descriptor->channel_count = 1; +// codec_descriptor->payload_type = 96; +// apt_string_set(&codec_descriptor->name,"L16"); +// codec_descriptor->sampling_rate = 16000; +// + synth_channel->flite_engine = (flite_synth_engine_t *) engine->obj; + synth_channel->speak_request = NULL; // no active speak request in progress + synth_channel->stop_response = NULL; + synth_channel->paused = FALSE; + synth_channel->pool = pool; + synth_channel->wave = NULL; + synth_channel->voice = NULL; + synth_channel->iId = 0; + + /* create engine channel base */ + synth_channel->channel = mrcp_engine_source_channel_create( + engine, /* resource engine */ + &channel_vtable, /* virtual methods table of engine channel */ + &audio_stream_vtable, /* virtual methods table of audio stream */ + synth_channel, /* object to associate */ + codec_descriptor, /* codec descriptor might be NULL by default */ + pool); /* pool to allocate memory from */ + + if (!synth_channel->channel) + { + apt_log(APT_LOG_MARK, APT_PRIO_WARNING, "flite_synth_engine_channel_create failed"); + return NULL; + } + + synth_channel->audio_buffer = mpf_buffer_create(pool); + apr_thread_mutex_lock(synth_channel->flite_engine->guard); + synth_channel->iId = ++synth_channel->flite_engine->iChannels; + apr_thread_mutex_unlock(synth_channel->flite_engine->guard); + + apt_log(APT_LOG_MARK, APT_PRIO_DEBUG, "flite_synth_engine_channel_create created channel %d", synth_channel->iId); + + return synth_channel->channel; +} + +/** Destroy engine channel */ +static apt_bool_t flite_synth_channel_destroy(mrcp_engine_channel_t *channel) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_destroy - channel %d", synth_channel->iId); + if(synth_channel->task) + { + apt_task_t *task = apt_consumer_task_base_get(synth_channel->task); + if (!apt_task_destroy(task)) + { + apt_log(APT_LOG_MARK, APT_PRIO_WARNING, "Speak task destroy failed - channel %d", synth_channel->iId); + } + else + { + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "Speak task destroyed - channel %d", synth_channel->iId); + } + } + synth_channel->task = NULL; + + apr_thread_mutex_lock(synth_channel->flite_engine->guard); + synth_channel->flite_engine->iChannels--; + apr_thread_mutex_unlock(synth_channel->flite_engine->guard); + + return TRUE; +} + +/** Open engine channel (asynchronous response MUST be sent)*/ +static apt_bool_t flite_synth_channel_open(mrcp_engine_channel_t *channel) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_open - channel %d", synth_channel->iId); + return flite_synth_msg_signal(flite_synth_MSG_OPEN_CHANNEL,channel,NULL); +} + +/** Close engine channel (asynchronous response MUST be sent)*/ +static apt_bool_t flite_synth_channel_close(mrcp_engine_channel_t *channel) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_close - channel %d", synth_channel->iId); + return flite_synth_msg_signal(flite_synth_MSG_CLOSE_CHANNEL,channel,NULL); +} + +/** Process MRCP channel request (asynchronous response MUST be sent)*/ +static apt_bool_t flite_synth_channel_request_process(mrcp_engine_channel_t *channel, mrcp_message_t *request) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_request_process - channel %d", synth_channel->iId); + return flite_synth_msg_signal(flite_synth_MSG_REQUEST_PROCESS,channel,request); +} + +/** Open engine channel */ +static apt_bool_t flite_synth_channel_open_t(mrcp_engine_channel_t *channel) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_open_t - channel %d", synth_channel->iId); + + synth_channel->voice = voices.awb; + return TRUE; +} + +static apt_bool_t flite_synth_channel_close_t(mrcp_engine_channel_t *channel) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_close_t - channel %d", synth_channel->iId); + + if (synth_channel->task) + { + apt_task_t *task = apt_consumer_task_base_get(synth_channel->task); + if (!apt_task_terminate(task,TRUE)) + { + apt_log(APT_LOG_MARK, APT_PRIO_WARNING, "Speak task terminate failed - channel %d", synth_channel->iId); + } + else + { + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "Speak task terminated - channel %d", synth_channel->iId); + apt_task_destroy(task); + synth_channel->task = 0; + } + } + if (synth_channel->wave) + { + delete_wave(synth_channel->wave); + synth_channel->wave = 0; + } + else + { + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "No wave buffer to delete - channel %d", synth_channel->iId); + } + return TRUE; +} + + +/** Process SPEAK request */ +static apt_bool_t flite_synth_channel_speak(mrcp_engine_channel_t *channel, mrcp_message_t *request, mrcp_message_t *response) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_speak - channel %d", synth_channel->iId); + + if (!synth_channel->speak_request) + { + apt_task_msg_pool_t *msg_pool = apt_task_msg_pool_create_dynamic( sizeof(flite_speak_msg_t),synth_channel->pool); + apt_task_vtable_t *task_vtable = 0; + apt_task_t * task = 0; + apt_task_msg_t *msg = 0; + flite_speak_msg_t *flite_msg = 0; + + /* create task/thread to run flite so this function is not blocking */ + apt_log(APT_LOG_MARK,APT_PRIO_NOTICE,"Create flite speak task - channel: %d", synth_channel->iId); + synth_channel->task = apt_consumer_task_create(synth_channel, msg_pool, synth_channel->pool); + if (!synth_channel->task) + { + apt_log(APT_LOG_MARK,APT_PRIO_ERROR, "flite_synth_channel_speak failed to create flite speak task - channel:%d", synth_channel->iId); + return FALSE; + } + + task_vtable = apt_consumer_task_vtable_get(synth_channel->task); + if (!task_vtable) + { + apt_log(APT_LOG_MARK,APT_PRIO_ERROR, "flite_synth_channel_speak cannot use flite speak task vtable - channel:%d", synth_channel->iId); + return FALSE; + } + + task_vtable->process_msg = flite_speak_msg_process; + synth_channel->speak_request = request; + + apt_log(APT_LOG_MARK,APT_PRIO_INFO, "Start task - channel %d", synth_channel->iId); + task = apt_consumer_task_base_get(synth_channel->task); + if (apt_task_start(task) == FALSE) + { + apt_log(APT_LOG_MARK,APT_PRIO_ERROR, "flite_synth_channel_speak failed to start task - channel: %d", synth_channel->iId); + apt_task_destroy(task); + return FALSE; + } + + msg = apt_task_msg_acquire(msg_pool); + msg->type = TASK_MSG_USER; + flite_msg = (flite_speak_msg_t*) msg->data; + flite_msg->channel = synth_channel; + flite_msg->request = request; + apt_log(APT_LOG_MARK,APT_PRIO_INFO, "Send signal to start speech synthesis - channel:%d", synth_channel->iId); + if (apt_task_msg_signal(task,msg)) + { + response->start_line.request_state = MRCP_REQUEST_STATE_INPROGRESS; + mrcp_engine_channel_message_send(channel,response); + } + else + { + apt_log(APT_LOG_MARK,APT_PRIO_INFO, "Failed to send signal to start speech synthesis - channel:%d", synth_channel->iId); + apt_task_destroy(task); + return FALSE; + } + } + return TRUE; +} + +static apt_bool_t flite_speak_msg_process(apt_task_t *task, apt_task_msg_t *msg) +{ + flite_speak_msg_t *flite_msg = (flite_speak_msg_t*)msg->data; + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) flite_msg->channel; + apr_time_t start = 0; + apr_time_t elapsed = 0; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "< flite_speak_msg_process speak - channel %d", synth_channel->iId); + + // probably not necessary !? + mpf_buffer_restart(synth_channel->audio_buffer); + + // just sequential stuff + start = apr_time_now(); // in microsec + if (synth_channel->speak_request->body.length) + { + cst_wave * wave = synth_channel->wave; + wave = flite_text_to_wave(synth_channel->speak_request->body.buf, synth_channel->voice); + } + + elapsed = (apr_time_now() - start)/1000; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "TTS (chan %d) took %ld millisec", synth_channel->iId, elapsed); + + if (!synth_channel->stop_response) + { + // this will notify the callback that feeds the client that synthesis is complete + mpf_buffer_event_write(synth_channel->audio_buffer, MEDIA_FRAME_TYPE_EVENT); + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "> flite_speak_msg_process speak - end of TTS - %d", synth_channel->iId); + } + else + { + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "> flite_speak_msg_process speak - channel %d", synth_channel->iId); + } + return TRUE; +} + +/** Process STOP request */ +static apt_bool_t flite_synth_channel_stop(mrcp_engine_channel_t *channel, mrcp_message_t *request, mrcp_message_t *response) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_stop - channel %d", synth_channel->iId); + /* store the request, make sure there is no more activity and only then send the response */ + + /* TODO this should probably be mutexed */ + synth_channel->stop_response = response; + return TRUE; +} + +/** Process PAUSE request */ +static apt_bool_t flite_synth_channel_pause(mrcp_engine_channel_t *channel, mrcp_message_t *request, mrcp_message_t *response) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_pause - channel %d", synth_channel->iId); + + synth_channel->paused = TRUE; + /* send asynchronous response */ + mrcp_engine_channel_message_send(channel,response); + return TRUE; +} + +/** Process RESUME request */ +static apt_bool_t flite_synth_channel_resume(mrcp_engine_channel_t *channel, mrcp_message_t *request, mrcp_message_t *response) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_resume - channel %d", synth_channel->iId); + + synth_channel->paused = FALSE; + /* send asynchronous response */ + mrcp_engine_channel_message_send(channel,response); + return TRUE; +} + +/** Dispatch MRCP request */ +static apt_bool_t flite_synth_channel_request_dispatch(mrcp_engine_channel_t *channel, mrcp_message_t *request) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_bool_t processed = FALSE; + mrcp_message_t *response = mrcp_response_create(request,request->pool); + + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_channel_request_dispatch - channel %d", synth_channel->iId); + + switch(request->start_line.method_id) { + case SYNTHESIZER_SET_PARAMS: + // TODO set voices + // if (!strcasecmp(voice_name, "awb")) { + // synth_channel->voice = voices.awb; + // } else if (!strcasecmp(voice_name, "kal")) { + // synth_channel->voice = voices.awb; + // } else if (!strcasecmp(voice_name, "rms")) { + // synth_channel->voice = voices.awb; + // } else if (!strcasecmp(voice_name, "slt")) { + // synth_channel->voice = voices.awb; + // } else { + // apt_log(APT_LOG_MARK, APT_PRIO_INFO, "Valid voice names are awb, kal, rms or slt"); + // } + break; + case SYNTHESIZER_GET_PARAMS: + break; + case SYNTHESIZER_SPEAK: + processed = flite_synth_channel_speak(channel,request,response); + break; + case SYNTHESIZER_STOP: + processed = flite_synth_channel_stop(channel,request,response); + break; + case SYNTHESIZER_PAUSE: + processed = flite_synth_channel_pause(channel,request,response); + break; + case SYNTHESIZER_RESUME: + processed = flite_synth_channel_resume(channel,request,response); + break; + case SYNTHESIZER_BARGE_IN_OCCURRED: + processed = flite_synth_channel_stop(channel,request,response); + break; + case SYNTHESIZER_CONTROL: + break; + case SYNTHESIZER_DEFINE_LEXICON: + break; + default: + break; + } + if(processed == FALSE) { + /* send asynchronous response for not handled request */ + mrcp_engine_channel_message_send(channel,response); + } + return TRUE; +} + +/** Callback is called from MPF engine context to destroy any additional data associated with audio stream */ +static apt_bool_t flite_synth_stream_destroy(mpf_audio_stream_t *stream) +{ + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_stream_destroy"); + return TRUE; +} + +/** Callback is called from MPF engine context to perform any action before open */ +static apt_bool_t flite_synth_stream_open(mpf_audio_stream_t *stream) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) stream->obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_stream_open - channel %d", synth_channel->iId); + return TRUE; +} + +/** Callback is called from MPF engine context to perform any action after close */ +static apt_bool_t flite_synth_stream_close(mpf_audio_stream_t *stream) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) stream->obj; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_stream_close - channel %d", synth_channel->iId); + return TRUE; +} + +/** Raise SPEAK-COMPLETE event */ +static apt_bool_t flite_synth_speak_complete_raise(flite_synth_channel_t *synth_channel) +{ + mrcp_message_t *message = 0; + mrcp_synth_header_t * synth_header = 0; + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_speak_complete_raise - channel %d", synth_channel->iId); + + if (!synth_channel->speak_request) { + return FALSE; + } + + message = mrcp_event_create( + synth_channel->speak_request, + SYNTHESIZER_SPEAK_COMPLETE, + synth_channel->speak_request->pool); + if (!message) { + return FALSE; + } + + /* get/allocate synthesizer header */ + synth_header = (mrcp_synth_header_t *) mrcp_resource_header_prepare(message); + if (synth_header) { + /* set completion cause */ + synth_header->completion_cause = SYNTHESIZER_COMPLETION_CAUSE_NORMAL; + mrcp_resource_header_property_add(message,SYNTHESIZER_HEADER_COMPLETION_CAUSE); + } + /* set request state */ + message->start_line.request_state = MRCP_REQUEST_STATE_COMPLETE; + + synth_channel->speak_request = NULL; + /* send asynch event */ + return mrcp_engine_channel_message_send(synth_channel->channel,message); +} + +/** Callback is called from MPF engine context to read/get new frame */ +static apt_bool_t flite_synth_stream_read(mpf_audio_stream_t *stream, mpf_frame_t *frame) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) stream->obj; + cst_wave * wave = synth_channel->wave; + if (synth_channel->stop_response && synth_channel->speak_request) + { + /* send asynchronous response to STOP request */ + mrcp_engine_channel_message_send(synth_channel->channel, synth_channel->stop_response); + synth_channel->speak_request = NULL; + synth_channel->paused = FALSE; + return TRUE; + } + + if (wave && cst_wave_num_samples(wave)) + { + mpf_buffer_audio_write(synth_channel->audio_buffer, cst_wave_samples(wave), cst_wave_num_samples(wave) * 2); + } + + /* check if there is active SPEAK request and it isn't in paused state */ + if (synth_channel->speak_request && synth_channel->paused == FALSE) + { + /* normal processing */ + mpf_buffer_frame_read(synth_channel->audio_buffer,frame); +// apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_stream_read - channel %d - size %d", synth_channel->iId, mpf_buffer_get_size(synth_channel->audio_buffer)); + + if((frame->type & MEDIA_FRAME_TYPE_EVENT) == MEDIA_FRAME_TYPE_EVENT) + { + flite_synth_speak_complete_raise(synth_channel); + } + } + return TRUE; +} + +static apt_bool_t flite_synth_msg_signal(flite_synth_msg_type_e type, mrcp_engine_channel_t *channel, mrcp_message_t *request) +{ + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) channel->method_obj; + apt_bool_t status = FALSE; + flite_synth_engine_t *flite_engine = synth_channel->flite_engine; + apt_task_t *task = apt_consumer_task_base_get(flite_engine->task); + apt_task_msg_t *msg = apt_task_msg_get(task); + + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_msg_signal - channel %d", synth_channel->iId); + + if (msg) { + flite_synth_msg_t *flite_msg; + msg->type = TASK_MSG_USER; + flite_msg = (flite_synth_msg_t*) msg->data; + + flite_msg->type = type; + flite_msg->channel = channel; + flite_msg->request = request; + status = apt_task_msg_signal(task,msg); + } + else + { + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_msg_signal - task msg not found %d", synth_channel->iId); + } + return status; +} + +static apt_bool_t flite_synth_msg_process(apt_task_t *task, apt_task_msg_t *msg) +{ + flite_synth_msg_t *flite_msg = (flite_synth_msg_t*)msg->data; + flite_synth_channel_t *synth_channel = (flite_synth_channel_t *) flite_msg->channel->method_obj; + + apt_log(APT_LOG_MARK, APT_PRIO_INFO, "flite_synth_msg_process - channel %d", synth_channel->iId); + + switch(flite_msg->type) { + case flite_synth_MSG_OPEN_CHANNEL: + /* open channel and send asynch response */ + flite_synth_channel_open_t(flite_msg->channel); + mrcp_engine_channel_open_respond(flite_msg->channel,TRUE); + break; + case flite_synth_MSG_CLOSE_CHANNEL: + /* close channel, make sure there is no activity and send asynch response */ + flite_synth_channel_close_t(flite_msg->channel); + mrcp_engine_channel_close_respond(flite_msg->channel); + break; + case flite_synth_MSG_REQUEST_PROCESS: + flite_synth_channel_request_dispatch(flite_msg->channel,flite_msg->request); + break; + default: + break; + } + return TRUE; +} Modified: freeswitch/trunk/libs/unimrcp/plugins/mrcp-pocketsphinx/Makefile.am ============================================================================== --- freeswitch/trunk/libs/unimrcp/plugins/mrcp-pocketsphinx/Makefile.am (original) +++ freeswitch/trunk/libs/unimrcp/plugins/mrcp-pocketsphinx/Makefile.am Fri Jun 26 15:18:39 2009 @@ -8,9 +8,13 @@ -I$(top_srcdir)/libs/mrcp/resources/include \ -I$(top_srcdir)/libs/mpf/include \ -I$(top_srcdir)/libs/apr-toolkit/include \ - $(UNIMRCP_APR_INCLUDES) $(UNIMRCP_APU_INCLUDES) + $(UNIMRCP_APR_INCLUDES) \ + $(UNIMRCP_APU_INCLUDES) \ + $(UNIMRCP_SPHINXBASE_INCLUDES) \ + $(UNIMRCP_POCKETSPHINX_INCLUDES) plugin_LTLIBRARIES = mrcppocketsphinx.la -mrcppocketsphinx_la_SOURCES = src/mrcp_pocketsphinx.c.c +mrcppocketsphinx_la_SOURCES = src/mrcp_pocketsphinx.c mrcppocketsphinx_la_LDFLAGS = -module $(PLUGIN_LT_VERSION) +mrcppocketsphinx_la_LIBADD = $(UNIMRCP_POCKETSPHINX_LIBS) $(UNIMRCP_SPHINXBASE_LIBS) -lm Modified: freeswitch/trunk/libs/unimrcp/plugins/mrcp-pocketsphinx/src/mrcp_pocketsphinx.c ============================================================================== --- freeswitch/trunk/libs/unimrcp/plugins/mrcp-pocketsphinx/src/mrcp_pocketsphinx.c (original) +++ freeswitch/trunk/libs/unimrcp/plugins/mrcp-pocketsphinx/src/mrcp_pocketsphinx.c Fri Jun 26 15:18:39 2009 @@ -24,6 +24,11 @@ * 4. Methods (callbacks) of the MPF engine stream MUST not block. */ +#include <pocketsphinx.h> +#include <apr_thread_cond.h> +#include <apr_thread_proc.h> +#include <apr_tables.h> +#include <apr_file_io.h> #include "mrcp_resource_engine.h" #include "mrcp_recog_resource.h" #include "mrcp_recog_header.h" @@ -33,24 +38,27 @@ #include "apt_log.h" +#define RECOGNIZER_SIDRES(recognizer) (recognizer)->channel->id.buf, "pocketsphinx" + typedef struct pocketsphinx_engine_t pocketsphinx_engine_t; typedef struct pocketsphinx_recognizer_t pocketsphinx_recognizer_t; +typedef struct pocketsphinx_properties_t pocketsphinx_properties_t; -/** Declaration of recognizer engine methods */ +/** Methods of recognition engine */ static apt_bool_t pocketsphinx_engine_destroy(mrcp_resource_engine_t *engine); static apt_bool_t pocketsphinx_engine_open(mrcp_resource_engine_t *engine); static apt_bool_t pocketsphinx_engine_close(mrcp_resource_engine_t *engine); -static mrcp_engine_channel_t* pocketsphinx_recognizer_create(mrcp_resource_engine_t *engine, apr_pool_t *pool); +static mrcp_engine_channel_t* pocketsphinx_engine_recognizer_create(mrcp_resource_engine_t *engine, apr_pool_t *pool); static const struct mrcp_engine_method_vtable_t engine_vtable = { pocketsphinx_engine_destroy, pocketsphinx_engine_open, pocketsphinx_engine_close, - pocketsphinx_recognizer_create + pocketsphinx_engine_recognizer_create }; -/** Declaration of recognizer channel methods */ +/** Methods of recognition channel (recognizer) */ static apt_bool_t pocketsphinx_recognizer_destroy(mrcp_engine_channel_t *channel); static apt_bool_t pocketsphinx_recognizer_open(mrcp_engine_channel_t *channel); static apt_bool_t pocketsphinx_recognizer_close(mrcp_engine_channel_t *channel); @@ -63,43 +71,94 @@ pocketsphinx_recognizer_request_process }; -/** Declaration of recognizer audio stream methods */ -static apt_bool_t pocketsphinx_stream_destroy(mpf_audio_stream_t *stream); -static apt_bool_t pocketsphinx_stream_open(mpf_audio_stream_t *stream); -static apt_bool_t pocketsphinx_stream_close(mpf_audio_stream_t *stream); +/** Methods of audio stream to recognize */ static apt_bool_t pocketsphinx_stream_write(mpf_audio_stream_t *stream, const mpf_frame_t *frame); static const mpf_audio_stream_vtable_t audio_stream_vtable = { - pocketsphinx_stream_destroy, - NULL, - NULL, - NULL, - pocketsphinx_stream_open, - pocketsphinx_stream_close, + NULL, /* destroy */ + NULL, /* open_rx */ + NULL, /* close_rx */ + NULL, /* read_frame */ + NULL, /* open_tx */ + NULL, /* close_tx */ pocketsphinx_stream_write }; -/** Declaration of pocketsphinx engine */ +/** Pocketsphinx engine (engine is an aggregation of recognizers) */ struct pocketsphinx_engine_t { mrcp_resource_engine_t *base; }; -/** Declaration of pocketsphinx recognizer */ +/** Pocketsphinx properties */ +struct pocketsphinx_properties_t { + const char *dictionary; + const char *model_8k; + const char *model_16k; + apr_size_t noinput_timeout; + apr_size_t recognition_timeout; + apr_size_t partial_result_timeout; +}; + +/** Pocketsphinx channel (recognizer) */ struct pocketsphinx_recognizer_t { /** Back pointer to engine */ - pocketsphinx_engine_t *engine; + pocketsphinx_engine_t *engine; /** Engine channel base */ - mrcp_engine_channel_t *channel; + mrcp_engine_channel_t *channel; + + /** Actual recognizer object */ + ps_decoder_t *decoder; + /** Configuration */ + cmd_ln_t *config; + /** Properties (to be loaded from config file) */ + pocketsphinx_properties_t properties; + /** Is input timer started */ + apt_bool_t is_input_timer_on; + /** Noinput timeout */ + apr_size_t noinput_timeout; + /** Recognition timeout */ + apr_size_t recognition_timeout; + /** Timeout elapsed since the last partial result checking */ + apr_size_t partial_result_timeout; + /** Last (partially) recognized result */ + const char *last_result; + /** Active grammar identifier (content-id) */ + const char *grammar_id; + /** Table of defined grammars (key=content-id, value=grammar-file-path) */ + apr_table_t *grammar_table; + + /** Voice activity detector */ + mpf_activity_detector_t *detector; + + /** Thread to run recognition in */ + apr_thread_t *thread; + /** Conditional wait object */ + apr_thread_cond_t *wait_object; + /** Mutex of the wait object */ + apr_thread_mutex_t *mutex; + + /** Pending request from client stack to recognizer */ + mrcp_message_t *request; + /** Pending event from mpf layer to recognizer */ + mrcp_message_t *complete_event; + /** In-progress RECOGNIZE request */ + mrcp_message_t *inprogress_recog; + /** Pending STOP response */ + mrcp_message_t *stop_response; + /** Is recognition channel being closed */ + apt_bool_t close_requested; }; /** Declare this macro to use log routine of the server, plugin is loaded from */ MRCP_PLUGIN_LOGGER_IMPLEMENT +static void* APR_THREAD_FUNC pocketsphinx_recognizer_run(apr_thread_t *thread, void *data); /** Create pocketsphinx engine (engine is an aggregation of recognizers) */ MRCP_PLUGIN_DECLARE(mrcp_resource_engine_t*) mrcp_plugin_create(apr_pool_t *pool) { pocketsphinx_engine_t *engine = apr_palloc(pool,sizeof(pocketsphinx_engine_t)); + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Create PocketSphinx Engine"); /* create resource engine base */ engine->base = mrcp_resource_engine_create( @@ -110,27 +169,49 @@ return engine->base; } +/** Destroy pocketsphinx engine */ static apt_bool_t pocketsphinx_engine_destroy(mrcp_resource_engine_t *engine) { return TRUE; } +/** Open pocketsphinx engine */ static apt_bool_t pocketsphinx_engine_open(mrcp_resource_engine_t *engine) { return TRUE; } +/** Close pocketsphinx engine */ static apt_bool_t pocketsphinx_engine_close(mrcp_resource_engine_t *engine) { return TRUE; } /** Create pocketsphinx recognizer */ -static mrcp_engine_channel_t* pocketsphinx_recognizer_create(mrcp_resource_engine_t *engine, apr_pool_t *pool) +static mrcp_engine_channel_t* pocketsphinx_engine_recognizer_create(mrcp_resource_engine_t *engine, apr_pool_t *pool) { mrcp_engine_channel_t *channel; pocketsphinx_recognizer_t *recognizer = apr_palloc(pool,sizeof(pocketsphinx_recognizer_t)); - +// recognizer->engine = engine; + recognizer->decoder = NULL; + recognizer->config = NULL; + recognizer->is_input_timer_on = FALSE; + recognizer->noinput_timeout = 0; + recognizer->recognition_timeout = 0; + recognizer->partial_result_timeout = 0; + recognizer->last_result = NULL; + recognizer->detector = NULL; + recognizer->thread = NULL; + recognizer->wait_object = NULL; + recognizer->mutex = NULL; + recognizer->request = NULL; + recognizer->complete_event = NULL; + recognizer->inprogress_recog = FALSE; + recognizer->stop_response = NULL; + recognizer->close_requested = FALSE; + recognizer->grammar_id = NULL; + recognizer->grammar_table = apr_table_make(pool,1); + /* create engine channel base */ channel = mrcp_engine_sink_channel_create( engine, /* resource engine */ @@ -144,55 +225,452 @@ return channel; } -/** Destroy engine channel */ +/** Destroy pocketsphinx recognizer */ static apt_bool_t pocketsphinx_recognizer_destroy(mrcp_engine_channel_t *channel) { return TRUE; } -/** Open engine channel (asynchronous response MUST be sent)*/ +/** Open pocketsphinx recognizer (asynchronous response MUST be sent) */ static apt_bool_t pocketsphinx_recognizer_open(mrcp_engine_channel_t *channel) { - return mrcp_engine_channel_open_respond(channel,TRUE); + apr_status_t rv; + pocketsphinx_recognizer_t *recognizer = channel->method_obj; + + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Open Channel "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + + apr_thread_mutex_create(&recognizer->mutex,APR_THREAD_MUTEX_DEFAULT,channel->pool); + apr_thread_cond_create(&recognizer->wait_object,channel->pool); + + /* Launch a thread to run recognition in */ + rv = apr_thread_create(&recognizer->thread,NULL,pocketsphinx_recognizer_run,recognizer,channel->pool); + if(rv != APR_SUCCESS) { + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Failed to Launch Thread "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + apr_thread_mutex_destroy(recognizer->mutex); + recognizer->mutex = NULL; + apr_thread_cond_destroy(recognizer->wait_object); + recognizer->wait_object = NULL; + return mrcp_engine_channel_open_respond(channel,FALSE); + } + + return TRUE; } -/** Close engine channel (asynchronous response MUST be sent)*/ +/** Close pocketsphinx recognizer (asynchronous response MUST be sent)*/ static apt_bool_t pocketsphinx_recognizer_close(mrcp_engine_channel_t *channel) { + pocketsphinx_recognizer_t *recognizer = channel->method_obj; + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Close Channel "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + if(recognizer->thread) { + apr_status_t rv; + + /* Signal recognition thread to terminate */ + apr_thread_mutex_lock(recognizer->mutex); + recognizer->close_requested = TRUE; + apr_thread_cond_signal(recognizer->wait_object); + apr_thread_mutex_unlock(recognizer->mutex); + + apr_thread_join(&rv,recognizer->thread); + recognizer->thread = NULL; + + apr_thread_mutex_destroy(recognizer->mutex); + recognizer->mutex = NULL; + apr_thread_cond_destroy(recognizer->wait_object); + recognizer->wait_object = NULL; + } + return mrcp_engine_channel_close_respond(channel); } +/** Process MRCP request (asynchronous response MUST be sent)*/ +static apt_bool_t pocketsphinx_recognizer_request_process(mrcp_engine_channel_t *channel, mrcp_message_t *request) +{ + pocketsphinx_recognizer_t *recognizer = channel->method_obj; + + /* Store request and signal recognition thread to process the request */ + apr_thread_mutex_lock(recognizer->mutex); + recognizer->request = request; + apr_thread_cond_signal(recognizer->wait_object); + apr_thread_mutex_unlock(recognizer->mutex); + return TRUE; +} + + + +/** Load pocketsphinx properties [RECOG] */ +static apt_bool_t pocketsphinx_properties_load(pocketsphinx_recognizer_t *recognizer) +{ + mrcp_engine_channel_t *channel = recognizer->channel; + const apt_dir_layout_t *dir_layout = channel->engine->dir_layout; + pocketsphinx_properties_t *properties = &recognizer->properties; + + properties->dictionary = apt_datadir_filepath_get(dir_layout,"pocketsphinx/default.dic",channel->pool); + properties->model_8k = apt_datadir_filepath_get(dir_layout,"pocketsphinx/communicator",channel->pool); + properties->model_16k = apt_datadir_filepath_get(dir_layout,"pocketsphinx/wsj1",channel->pool); + + properties->noinput_timeout = 5000; + properties->recognition_timeout = 15000; + properties->partial_result_timeout = 100; + + return TRUE; +} + +/** Initialize pocketsphinx decoder [RECOG] */ +static apt_bool_t pocketsphinx_decoder_init(pocketsphinx_recognizer_t *recognizer, const char *grammar) +{ + apt_log(APT_LOG_MARK,APT_PRIO_DEBUG,"Init Config "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + recognizer->config = cmd_ln_init(recognizer->config, ps_args(), FALSE, + "-samprate", "8000", + "-hmm", recognizer->properties.model_8k, + "-jsgf", grammar, + "-dict", recognizer->properties.dictionary, + "-frate", "50", + "-silprob", "0.005", + NULL); + if(!recognizer->config) { + apt_log(APT_LOG_MARK,APT_PRIO_WARNING,"Failed to Init Config "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + return FALSE; + } + + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Init Decoder "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + if(recognizer->decoder) { + if(ps_reinit(recognizer->decoder,recognizer->config) < 0) { + apt_log(APT_LOG_MARK,APT_PRIO_WARNING,"Failed to Reinit Decoder "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + return FALSE; + } + } + else { + recognizer->decoder = ps_init(recognizer->config); + if(!recognizer->decoder) { + apt_log(APT_LOG_MARK,APT_PRIO_WARNING,"Failed to Init Decoder "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + return FALSE; + } + } + + if(!recognizer->detector) { + recognizer->detector = mpf_activity_detector_create(recognizer->channel->pool); + mpf_activity_detector_level_set(recognizer->detector,50); + } + return TRUE; +} + +/** Build pocketsphinx recognized result [RECOG] */ +static apt_bool_t pocketsphinx_result_build(pocketsphinx_recognizer_t *recognizer, mrcp_message_t *message) +{ + apt_str_t *body = &message->body; + if(!recognizer->last_result || !recognizer->grammar_id) { + return FALSE; + } + + body->buf = apr_psprintf(message->pool, + "<?xml version=\"1.0\"?>\n" + "<result grammar=\"%s\">\n" + " <interpretation grammar=\"%s\" confidence=\"%d\">\n" + " <input mode=\"speech\">%s</input>\n" + " </interpretation>\n" + "</result>\n", + recognizer->grammar_id, + recognizer->grammar_id, + 99, + recognizer->last_result, + recognizer->last_result); + if(body->buf) { + mrcp_generic_header_t *generic_header; + generic_header = mrcp_generic_header_prepare(message); + if(generic_header) { + /* set content type */ + apt_string_assign(&generic_header->content_type,"application/x-nlsml",message->pool); + mrcp_generic_header_property_add(message,GENERIC_HEADER_CONTENT_TYPE); + } + + body->length = strlen(body->buf); + } + return TRUE; +} + +/** Clear pocketsphinx grammars [RECOG] */ +static apt_bool_t pocketsphinx_grammars_clear(pocketsphinx_recognizer_t *recognizer) +{ + const apr_array_header_t *tarr = apr_table_elts(recognizer->grammar_table); + const apr_table_entry_t *telts = (const apr_table_entry_t*)tarr->elts; + int i; + for(i = 0; i < tarr->nelts; i++) { + const char *grammar_file_path = telts[i].val; + if(grammar_file_path) { + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Remove Grammar File [%s] "APT_SIDRES_FMT, + grammar_file_path,RECOGNIZER_SIDRES(recognizer)); + apr_file_remove(grammar_file_path,recognizer->channel->pool); + } + } + apr_table_clear(recognizer->grammar_table); + return TRUE; +} + +/** Process DEFINE-GRAMMAR request [RECOG] */ +static apt_bool_t pocketsphinx_define_grammar(pocketsphinx_recognizer_t *recognizer, mrcp_message_t *request, mrcp_message_t *response) +{ + const char *content_type = NULL; + const char *content_id = NULL; + apt_str_t *grammar = NULL; + + mrcp_engine_channel_t *channel = recognizer->channel; + + /* get recognizer header */ + mrcp_generic_header_t *generic_header = mrcp_generic_header_get(request); + mrcp_recog_header_t *recog_header = mrcp_resource_header_prepare(response); + if(!generic_header || !recog_header) { + return FALSE; + } + + recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_SUCCESS; + mrcp_resource_header_property_add(response,RECOGNIZER_HEADER_COMPLETION_CAUSE); + + /* content-id must be specified */ + if(mrcp_generic_header_property_check(request,GENERIC_HEADER_CONTENT_ID) == TRUE) { + content_id = generic_header->content_id.buf; + } + if(!content_id) { + apt_log(APT_LOG_MARK,APT_PRIO_WARNING,"Missing Content-Id "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + response->start_line.status_code = MRCP_STATUS_CODE_MISSING_PARAM; + recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_GRAM_LOAD_FAILURE; + return FALSE; + } + + if(mrcp_generic_header_property_check(request,GENERIC_HEADER_CONTENT_LENGTH) == TRUE) { + grammar = &request->body; + } + + if(grammar) { + /* load grammar */ + const apt_dir_layout_t *dir_layout = channel->engine->dir_layout; + const char *grammar_file_path = NULL; + const char *grammar_file_name = NULL; + apr_file_t *fd = NULL; + apr_status_t rv; + apr_size_t size; + + /* content-type must be specified */ + if(mrcp_generic_header_property_check(request,GENERIC_HEADER_CONTENT_TYPE) == TRUE) { + content_type = generic_header->content_type.buf; + } + if(!content_type) { + apt_log(APT_LOG_MARK,APT_PRIO_WARNING,"Missing Content-Type "APT_SIDRES_FMT,RECOGNIZER_SIDRES(recognizer)); + response->start_line.status_code = MRCP_STATUS_CODE_MISSING_PARAM; + recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_GRAM_LOAD_FAILURE; + return FALSE; + } + + /* only JSGF grammar is supported */ + if(strstr(content_type,"jsgf") == NULL) { + apt_log(APT_LOG_MARK,APT_PRIO_WARNING,"Not Supported Content-Type [%s] "APT_SIDRES_FMT, + content_type,RECOGNIZER_SIDRES(recognizer)); + response->start_line.status_code = MRCP_STATUS_CODE_UNSUPPORTED_PARAM_VALUE; + recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_GRAM_LOAD_FAILURE; + return FALSE; + } + + grammar_file_name = apr_psprintf(channel->pool,"pocketsphinx/%s-%s.gram",channel->id.buf,content_id); + grammar_file_path = apt_datadir_filepath_get(dir_layout,grammar_file_name,channel->pool); + + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Create Grammar File [%s] "APT_SIDRES_FMT, + grammar_file_path,RECOGNIZER_SIDRES(recognizer)); + rv = apr_file_open(&fd,grammar_file_path,APR_CREATE|APR_TRUNCATE|APR_WRITE|APR_BINARY, + APR_OS_DEFAULT,channel->pool); + if(rv != APR_SUCCESS) { + apt_log(APT_LOG_MARK,APT_PRIO_WARNING,"Cannot Open Grammar File to Write [%s] "APT_SIDRES_FMT, + grammar_file_path,RECOGNIZER_SIDRES(recognizer)); + response->start_line.status_code = MRCP_STATUS_CODE_METHOD_FAILED; + recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_GRAM_LOAD_FAILURE; + return FALSE; + } + + size = grammar->length; + apr_file_write(fd,grammar->buf,&size); + apr_file_close(fd); + + /* init pocketsphinx decoder */ + if(pocketsphinx_decoder_init(recognizer,grammar_file_path) == TRUE) { + recognizer->grammar_id = content_id; + apr_table_setn(recognizer->grammar_table,content_id,grammar_file_path); + } + else { + response->start_line.status_code = MRCP_STATUS_CODE_METHOD_FAILED; + recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_GRAM_LOAD_FAILURE; + apr_file_remove(grammar_file_path,channel->pool); + return FALSE; + } + } + else { + /* unload grammar */ + const char *grammar_file_path = apr_table_get(recognizer->grammar_table,content_id); + if(grammar_file_path) { + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Remove Grammar File [%s] "APT_SIDRES_FMT, + grammar_file_path,RECOGNIZER_SIDRES(recognizer)); + apr_file_remove(grammar_file_path,channel->pool); + apr_table_unset(recognizer->grammar_table,content_id); + } + } -/** Process RECOGNIZE request */ + /* send asynchronous response */ + mrcp_engine_channel_message_send(channel,response); + return TRUE; +} + +/** Process RECOGNIZE request [RECOG] */ static apt_bool_t pocketsphinx_recognize(pocketsphinx_recognizer_t *recognizer, mrcp_message_t *request, mrcp_message_t *response) { + mrcp_recog_header_t *request_recog_header; + mrcp_recog_header_t *response_recog_header = mrcp_resource_header_prepare(response); + if(!response_recog_header) { + return FALSE; + } + + response_recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_SUCCESS; + mrcp_resource_header_property_add(response,RECOGNIZER_HEADER_COMPLETION_CAUSE); + + if(!recognizer->decoder || ps_start_utt(recognizer->decoder, NULL) < 0) { + response->start_line.status_code = MRCP_STATUS_CODE_METHOD_FAILED; + response_recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_ERROR; + return FALSE; + } + + recognizer->is_input_timer_on = TRUE; + /* get recognizer header */ + request_recog_header = mrcp_resource_header_get(request); + if(request_recog_header) { + if(mrcp_resource_header_property_check(request,RECOGNIZER_HEADER_START_INPUT_TIMERS) == TRUE) { + recognizer->is_input_timer_on = request_recog_header->start_input_timers; + } + } + + response->start_line.request_state = MRCP_REQUEST_STATE_INPROGRESS; + /* send asynchronous response */ + mrcp_engine_channel_message_send(recognizer->channel,response); + + + /* reset */ + mpf_activity_detector_reset(recognizer->detector); + recognizer->noinput_timeout = 0; + recognizer->recognition_timeout = 0; + recognizer->partial_result_timeout = 0; + recognizer->last_result = NULL; + recognizer->complete_event = NULL; + + recognizer->inprogress_recog = request; return TRUE; } -/** Process STOP request */ +/** Process GET-RESULTS request [RECOG] */ +static apt_bool_t pocketsphinx_get_result(pocketsphinx_recognizer_t *recognizer, mrcp_message_t *request, mrcp_message_t *response) +{ + if(pocketsphinx_result_build(recognizer,response) != TRUE) { + response->start_line.status_code = MRCP_STATUS_CODE_METHOD_FAILED; + } + + /* send asynchronous response */ + mrcp_engine_channel_message_send(recognizer->channel,response); + return TRUE; +} + +/** Process START-INPUT-TIMERS request [RECOG] */ +static apt_bool_t pocketsphinx_start_input_timers(pocketsphinx_recognizer_t *recognizer, mrcp_message_t *request, mrcp_message_t *response) +{ + recognizer->is_input_timer_on = TRUE; + + /* send asynchronous response */ + mrcp_engine_channel_message_send(recognizer->channel,response); + return TRUE; +} + + +/** Process STOP request [RECOG] */ static apt_bool_t pocketsphinx_stop(pocketsphinx_recognizer_t *recognizer, mrcp_message_t *request, mrcp_message_t *response) { + if(recognizer->inprogress_recog) { + /* store pending STOP response for further processing */ + recognizer->stop_response = response; + return TRUE; + } + + /* send asynchronous response */ + mrcp_engine_channel_message_send(recognizer->channel,response); return TRUE; } -static apt_bool_t pocketsphinx_recognizer_request_process(mrcp_engine_channel_t *channel, mrcp_message_t *request) +/** Process RECOGNITION-COMPLETE event [RECOG] */ +static apt_bool_t pocketsphinx_recognition_complete(pocketsphinx_recognizer_t *recognizer, mrcp_message_t *complete_event) +{ + mrcp_recog_header_t *recog_header; + if(!recognizer->inprogress_recog) { + /* false event */ + return FALSE; + } + + recognizer->inprogress_recog = NULL; + ps_end_utt(recognizer->decoder); + + if(recognizer->stop_response) { + /* recognition has been stopped, send STOP response instead */ + mrcp_message_t *response = recognizer->stop_response; + recognizer->stop_response = NULL; + if(recognizer->close_requested == FALSE) { + mrcp_engine_channel_message_send(recognizer->channel,response); + } + return TRUE; + } + + recog_header = mrcp_resource_header_get(complete_event); + if(recog_header->completion_cause == RECOGNIZER_COMPLETION_CAUSE_SUCCESS || + recog_header->completion_cause == RECOGNIZER_COMPLETION_CAUSE_RECOGNITION_TIMEOUT) { + int32 score; + char const *hyp; + char const *uttid; + + hyp = ps_get_hyp(recognizer->decoder, &score, &uttid); + if(hyp && strlen(hyp) > 0) { + int32 prob; + recognizer->last_result = apr_pstrdup(recognizer->channel->pool,hyp); + prob = ps_get_prob(recognizer->decoder, &uttid); + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Get Recognition Final Result [%s] Prob [%d] Score [%d] "APT_SIDRES_FMT, + hyp,prob,score,RECOGNIZER_SIDRES(recognizer)); + if(pocketsphinx_result_build(recognizer,complete_event) != TRUE) { + recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_ERROR; + } + } + else { + recog_header->completion_cause = RECOGNIZER_COMPLETION_CAUSE_NO_MATCH; + } + } + + /* send asynchronous event */ + mrcp_engine_channel_message_send(recognizer->channel,complete_event); + return TRUE; +} + +/** Dispatch MRCP request [RECOG] */ +static apt_bool_t pocketsphinx_request_dispatch(pocketsphinx_recognizer_t *recognizer, mrcp_message_t *request) { - pocketsphinx_recognizer_t *recognizer = channel->method_obj; apt_bool_t processed = FALSE; mrcp_message_t *response = mrcp_response_create(request,request->pool); + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Dispatch Request %s "APT_SIDRES_FMT, + request->start_line.method_name.buf, + RECOGNIZER_SIDRES(recognizer)); switch(request->start_line.method_id) { case RECOGNIZER_SET_PARAMS: break; case RECOGNIZER_GET_PARAMS: break; case RECOGNIZER_DEFINE_GRAMMAR: + processed = pocketsphinx_define_grammar(recognizer,request,response); break; case RECOGNIZER_RECOGNIZE: processed = pocketsphinx_recognize(recognizer,request,response); break; case RECOGNIZER_GET_RESULT: + processed = pocketsphinx_get_result(recognizer,request,response); break; case RECOGNIZER_START_INPUT_TIMERS: + processed = pocketsphinx_start_input_timers(recognizer,request,response); break; case RECOGNIZER_STOP: processed = pocketsphinx_stop(recognizer,request,response); @@ -201,32 +679,200 @@ break; } if(processed == FALSE) { - /* send asynchronous response for not handled request */ - mrcp_engine_channel_message_send(channel,response); + /* send asynchronous response for non handled request */ + mrcp_engine_channel_message_send(recognizer->channel,response); } return TRUE; } +/** Recognition thread [RECOG] */ +static void* APR_THREAD_FUNC pocketsphinx_recognizer_run(apr_thread_t *thread, void *data) +{ + pocketsphinx_recognizer_t *recognizer = data; + apt_bool_t status; + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Run Recognition Thread "APT_SIDRES_FMT, RECOGNIZER_SIDRES(recognizer)); + status = pocketsphinx_properties_load(recognizer); + /** Send response to channel_open request */ + mrcp_engine_channel_open_respond(recognizer->channel,status); + + do { + /** Wait for MRCP requests */ + apt_log(APT_LOG_MARK,APT_PRIO_DEBUG,"Wait for incoming messages "APT_SIDRES_FMT, RECOGNIZER_SIDRES(recognizer)); + apr_thread_mutex_lock(recognizer->mutex); + apr_thread_cond_wait(recognizer->wait_object,recognizer->mutex); + apr_thread_mutex_unlock(recognizer->mutex); + + if(recognizer->request) { + /* store request message and further dispatch it */ + mrcp_message_t *request = recognizer->request; + recognizer->request = NULL; + pocketsphinx_request_dispatch(recognizer,request); + } + if(recognizer->complete_event) { + /* end of input detected, get recognition result and raise recognition complete event */ + pocketsphinx_recognition_complete(recognizer,recognizer->complete_event); + } + } + while(recognizer->close_requested == FALSE); -static apt_bool_t pocketsphinx_stream_destroy(mpf_audio_stream_t *stream) -{ - return TRUE; -} + /* check if recognition is still active */ + if(recognizer->inprogress_recog) { + apr_thread_mutex_lock(recognizer->mutex); + recognizer->stop_response = recognizer->inprogress_recog; + apr_thread_cond_wait(recognizer->wait_object,recognizer->mutex); + apr_thread_mutex_unlock(recognizer->mutex); + if(recognizer->complete_event) { + pocketsphinx_recognition_complete(recognizer,recognizer->complete_event); + } + } -static apt_bool_t pocketsphinx_stream_open(mpf_audio_stream_t *stream) -{ - return TRUE; + /** Clear all the defined grammars */ + pocketsphinx_grammars_clear(recognizer); + + if(recognizer->decoder) { + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Free Decoder "APT_SIDRES_FMT, RECOGNIZER_SIDRES(recognizer)); + /** Free pocketsphinx decoder */ + ps_free(recognizer->decoder); + recognizer->decoder = NULL; + } + + /** Exit thread */ + apr_thread_exit(thread,APR_SUCCESS); + return NULL; } -static apt_bool_t pocketsphinx_stream_close(mpf_audio_stream_t *stream) + + +/* Start of input (utterance) [MPF] */ +static apt_bool_t pocketsphinx_start_of_input(pocketsphinx_recognizer_t *recognizer) { + /* create START-OF-INPUT event */ + mrcp_message_t *message = mrcp_event_create( + recognizer->inprogress_recog, + RECOGNIZER_START_OF_INPUT, + recognizer->inprogress_recog->pool); + if(!message) { + return FALSE; + } + + /* set request state */ + message->start_line.request_state = MRCP_REQUEST_STATE_INPROGRESS; + /* send asynch event */ + return mrcp_engine_channel_message_send(recognizer->channel,message); +} + +/* End of input (utterance) [MPF] */ +static apt_bool_t pocketsphinx_end_of_input(pocketsphinx_recognizer_t *recognizer, mrcp_recog_completion_cause_e cause) +{ + mrcp_recog_header_t *recog_header; + /* create RECOGNITION-COMPLETE event */ + mrcp_message_t *message = mrcp_event_create( + recognizer->inprogress_recog, + RECOGNIZER_RECOGNITION_COMPLETE, + recognizer->inprogress_recog->pool); + if(!message) { + return FALSE; + } + + /* get/allocate recognizer header */ + recog_header = mrcp_resource_header_prepare(message); + if(recog_header) { + /* set completion cause */ + recog_header->completion_cause = cause; + mrcp_resource_header_property_add(message,RECOGNIZER_HEADER_COMPLETION_CAUSE); + } + /* set request state */ + message->start_line.request_state = MRCP_REQUEST_STATE_COMPLETE; + + /* signal recognition thread first */ + apr_thread_mutex_lock(recognizer->mutex); + recognizer->complete_event = message; + apr_thread_cond_signal(recognizer->wait_object); + apr_thread_mutex_unlock(recognizer->mutex); return TRUE; } +/* Process audio frame [MPF] */ static apt_bool_t pocketsphinx_stream_write(mpf_audio_stream_t *stream, const mpf_frame_t *frame) { + pocketsphinx_recognizer_t *recognizer = stream->obj; + + /* check whether recognition has been started and not completed yet */ + if(recognizer->inprogress_recog && !recognizer->complete_event) { + mpf_detector_event_e det_event; + + /* first check if STOP has been requested */ + if(recognizer->stop_response) { + /* recognition has been stopped -> acknowledge with complete-event */ + pocketsphinx_end_of_input(recognizer,RECOGNIZER_COMPLETION_CAUSE_SUCCESS); + return TRUE; + } + + if(ps_process_raw( + recognizer->decoder, + (const int16 *)frame->codec_frame.buffer, + frame->codec_frame.size / sizeof(int16), + FALSE, + FALSE) < 0) { + + apt_log(APT_LOG_MARK,APT_PRIO_WARNING,"Failed to Process Raw Data "APT_SIDRES_FMT, + RECOGNIZER_SIDRES(recognizer)); + } + + recognizer->partial_result_timeout += CODEC_FRAME_TIME_BASE; + if(recognizer->partial_result_timeout == recognizer->properties.partial_result_timeout) { + int32 score; + char const *hyp; + char const *uttid; + + recognizer->partial_result_timeout = 0; + hyp = ps_get_hyp(recognizer->decoder, &score, &uttid); + if(hyp && strlen(hyp) > 0) { + if(recognizer->last_result == NULL || 0 != strcmp(recognizer->last_result, hyp)) { + recognizer->last_result = apr_pstrdup(recognizer->channel->pool,hyp); + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Get Recognition Partial Result [%s] Score [%d] "APT_SIDRES_FMT, + hyp,score,RECOGNIZER_SIDRES(recognizer)); + } + } + } + + if(recognizer->is_input_timer_on) { + recognizer->noinput_timeout += CODEC_FRAME_TIME_BASE; + if(recognizer->noinput_timeout == recognizer->properties.noinput_timeout) { + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Noinput Timeout Elapsed "APT_SIDRES_FMT, + RECOGNIZER_SIDRES(recognizer)); + pocketsphinx_end_of_input(recognizer,RECOGNIZER_COMPLETION_CAUSE_NO_INPUT_TIMEOUT); + return TRUE; + } + } + + recognizer->recognition_timeout += CODEC_FRAME_TIME_BASE; + if(recognizer->recognition_timeout == recognizer->properties.recognition_timeout) { + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Recognition Timeout Elapsed "APT_SIDRES_FMT, + RECOGNIZER_SIDRES(recognizer)); + pocketsphinx_end_of_input(recognizer,RECOGNIZER_COMPLETION_CAUSE_RECOGNITION_TIMEOUT); + return TRUE; + } + + det_event = mpf_activity_detector_process(recognizer->detector,frame); + switch(det_event) { + case MPF_DETECTOR_EVENT_ACTIVITY: + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Detected Voice Activity "APT_SIDRES_FMT, + RECOGNIZER_SIDRES(recognizer)); + pocketsphinx_start_of_input(recognizer); + break; + case MPF_DETECTOR_EVENT_INACTIVITY: + apt_log(APT_LOG_MARK,APT_PRIO_INFO,"Detected Voice Inactivity "APT_SIDRES_FMT, + RECOGNIZER_SIDRES(recognizer)); + pocketsphinx_end_of_input(recognizer,RECOGNIZER_COMPLETION_CAUSE_SUCCESS); + break; + default: + break; + } + } + return TRUE; } Modified: freeswitch/trunk/libs/unimrcp/unimrcp.sln ============================================================================== --- freeswitch/trunk/libs/unimrcp/unimrcp.sln (original) +++ freeswitch/trunk/libs/unimrcp/unimrcp.sln Fri Jun 26 15:18:39 2009 @@ -27,27 +27,27 @@ EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mrcpsignaling", "libs\mrcp-signaling\mrcpsignaling.vcproj", "{12A49562-BAB9-43A3-A21D-15B60BBB4C31}" ProjectSection(ProjectDependencies) = postProject - {B5A00BFA-6083-4FAE-A097-71642D6473B5} = {B5A00BFA-6083-4FAE-A097-71642D6473B5} {1C320193-46A6-4B34-9C56-8AB584FC1B56} = {1C320193-46A6-4B34-9C56-8AB584FC1B56} + {B5A00BFA-6083-4FAE-A097-71642D6473B5} = {B5A00BFA-6083-4FAE-A097-71642D6473B5} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mrcpserver", "libs\mrcp-server\mrcpserver.vcproj", "{18B1F35A-10F8-4287-9B37-2D10501B0B38}" ProjectSection(ProjectDependencies) = postProject - {12A49562-BAB9-43A3-A21D-15B60BBB4C31} = {12A49562-BAB9-43A3-A21D-15B60BBB4C31} {A9EDAC04-6A5F-4BA7-BC0D-CCE7B255B6EA} = {A9EDAC04-6A5F-4BA7-BC0D-CCE7B255B6EA} + {12A49562-BAB9-43A3-A21D-15B60BBB4C31} = {12A49562-BAB9-43A3-A21D-15B60BBB4C31} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "libunimrcpserver", "platforms\libunimrcp-server\libunimrcpserver.vcproj", "{C98AF157-352E-4737-BD30-A24E2647F5AE}" ProjectSection(ProjectDependencies) = postProject - {DEB01ACB-D65F-4A62-AED9-58C1054499E9} = {DEB01ACB-D65F-4A62-AED9-58C1054499E9} - {18B1F35A-10F8-4287-9B37-2D10501B0B38} = {18B1F35A-10F8-4287-9B37-2D10501B0B38} {746F3632-5BB2-4570-9453-31D6D58A7D8E} = {746F3632-5BB2-4570-9453-31D6D58A7D8E} + {18B1F35A-10F8-4287-9B37-2D10501B0B38} = {18B1F35A-10F8-4287-9B37-2D10501B0B38} + {DEB01ACB-D65F-4A62-AED9-58C1054499E9} = {DEB01ACB-D65F-4A62-AED9-58C1054499E9} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mrcpsofiasip", "modules\mrcp-sofiasip\mrcpsofiasip.vcproj", "{746F3632-5BB2-4570-9453-31D6D58A7D8E}" ProjectSection(ProjectDependencies) = postProject - {12A49562-BAB9-43A3-A21D-15B60BBB4C31} = {12A49562-BAB9-43A3-A21D-15B60BBB4C31} {A9EDAC04-6A5F-4BA7-BC0D-CCE7B255B6EA} = {A9EDAC04-6A5F-4BA7-BC0D-CCE7B255B6EA} + {12A49562-BAB9-43A3-A21D-15B60BBB4C31} = {12A49562-BAB9-43A3-A21D-15B60BBB4C31} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mrcpclient", "libs\mrcp-client\mrcpclient.vcproj", "{72782932-37CC-46AE-8C7F-9A7B1A6EE108}" @@ -57,9 +57,9 @@ EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "libunimrcpclient", "platforms\libunimrcp-client\libunimrcpclient.vcproj", "{EE157390-1E85-416C-946E-620E32C9AD33}" ProjectSection(ProjectDependencies) = postProject - {DEB01ACB-D65F-4A62-AED9-58C1054499E9} = {DEB01ACB-D65F-4A62-AED9-58C1054499E9} - {746F3632-5BB2-4570-9453-31D6D58A7D8E} = {746F3632-5BB2-4570-9453-31D6D58A7D8E} {72782932-37CC-46AE-8C7F-9A7B1A6EE108} = {72782932-37CC-46AE-8C7F-9A7B1A6EE108} + {746F3632-5BB2-4570-9453-31D6D58A7D8E} = {746F3632-5BB2-4570-9453-31D6D58A7D8E} + {DEB01ACB-D65F-4A62-AED9-58C1054499E9} = {DEB01ACB-D65F-4A62-AED9-58C1054499E9} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "unimrcpclient", "platforms\unimrcp-client\unimrcpclient.vcproj", "{57FAF32E-49FD-491F-895D-132D0D5EFE0A}" @@ -74,8 +74,8 @@ EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mrcpengine", "libs\mrcp-engine\mrcpengine.vcproj", "{843425BE-9A9A-44F4-A4E3-4B57D6ABD53C}" ProjectSection(ProjectDependencies) = postProject - {B5A00BFA-6083-4FAE-A097-71642D6473B5} = {B5A00BFA-6083-4FAE-A097-71642D6473B5} {1C320193-46A6-4B34-9C56-8AB584FC1B56} = {1C320193-46A6-4B34-9C56-8AB584FC1B56} + {B5A00BFA-6083-4FAE-A097-71642D6473B5} = {B5A00BFA-6083-4FAE-A097-71642D6473B5} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "demosynth", "plugins\demo-synth\demosynth.vcproj", "{92BFA534-C419-4EB2-AAA3-510653F38F08}" @@ -127,8 +127,8 @@ EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mrcpunirtsp", "modules\mrcp-unirtsp\mrcpunirtsp.vcproj", "{DEB01ACB-D65F-4A62-AED9-58C1054499E9}" ProjectSection(ProjectDependencies) = postProject - {12A49562-BAB9-43A3-A21D-15B60BBB4C31} = {12A49562-BAB9-43A3-A21D-15B60BBB4C31} {504B3154-7A4F-459D-9877-B951021C3F1F} = {504B3154-7A4F-459D-9877-B951021C3F1F} + {12A49562-BAB9-43A3-A21D-15B60BBB4C31} = {12A49562-BAB9-43A3-A21D-15B60BBB4C31} EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "prepare", "build\tools\prepare.vcproj", "{01D63BF5-7798-4746-852A-4B45229BB735}" @@ -138,6 +138,16 @@ {13DEECA0-BDD4-4744-A1A2-8EB0A44DF3D2} = {13DEECA0-BDD4-4744-A1A2-8EB0A44DF3D2} EndProjectSection EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mrcppocketsphinx", "plugins\mrcp-pocketsphinx\mrcppocketsphinx.vcproj", "{3C614AE8-B611-4D43-A9AF-1CAA440A9F69}" + ProjectSection(ProjectDependencies) = postProject + {843425BE-9A9A-44F4-A4E3-4B57D6ABD53C} = {843425BE-9A9A-44F4-A4E3-4B57D6ABD53C} + EndProjectSection +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mrcpflite", "plugins\mrcp-flite\mrcpflite.vcproj", "{56F6FB96-2BC7-4CAE-A8BF-6A0FAEC90556}" + ProjectSection(ProjectDependencies) = postProject + {843425BE-9A9A-44F4-A4E3-4B57D6ABD53C} = {843425BE-9A9A-44F4-A4E3-4B57D6ABD53C} + EndProjectSection +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Win32 = Debug|Win32 @@ -238,6 +248,10 @@ {01D63BF5-7798-4746-852A-4B45229BB735}.Release|Win32.ActiveCfg = Release|Win32 {4714EF49-BFD5-4B22-95F7-95A07F1EAC25}.Debug|Win32.ActiveCfg = Debug|Win32 {4714EF49-BFD5-4B22-95F7-95A07F1EAC25}.Release|Win32.ActiveCfg = Release|Win32 + {3C614AE8-B611-4D43-A9AF-1CAA440A9F69}.Debug|Win32.ActiveCfg = Debug|Win32 + {3C614AE8-B611-4D43-A9AF-1CAA440A9F69}.Release|Win32.ActiveCfg = Release|Win32 + {56F6FB96-2BC7-4CAE-A8BF-6A0FAEC90556}.Debug|Win32.ActiveCfg = Debug|Win32 + {56F6FB96-2BC7-4CAE-A8BF-6A0FAEC90556}.Release|Win32.ActiveCfg = Release|Win32 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE @@ -261,6 +275,8 @@ {92BFA534-C419-4EB2-AAA3-510653F38F08} = {09BABD45-8F30-4F99-B8B8-8DD78F6804DB} {B495B6D9-AF84-479D-B30A-313C16EF8BFD} = {09BABD45-8F30-4F99-B8B8-8DD78F6804DB} {729EF28E-38C9-40DE-A138-87785F021411} = {09BABD45-8F30-4F99-B8B8-8DD78F6804DB} + {3C614AE8-B611-4D43-A9AF-1CAA440A9F69} = {09BABD45-8F30-4F99-B8B8-8DD78F6804DB} + {56F6FB96-2BC7-4CAE-A8BF-6A0FAEC90556} = {09BABD45-8F30-4F99-B8B8-8DD78F6804DB} {79EF9F1D-E211-4ED1-91D2-FC935AB3A872} = {AC4356E8-48A1-4D2D-AFB1-11CF30B974CD} {429C907B-97D1-4B2D-9B0E-A14A5BFDAD15} = {AC4356E8-48A1-4D2D-AFB1-11CF30B974CD} {DCF01B1C-5268-44F3-9130-D647FABFB663} = {AC4356E8-48A1-4D2D-AFB1-11CF30B974CD}