To have avmd running on a call leg, let's call that channel, i.e. RTP sent and/or recvd to/from a given RTP endpoint you need to start avmd on that channel: https://freeswitch.org/confluence/display/FREESWITCH/mod_avmd#mod_avmd-APPinterface
You can select direction for that channel on which avmd should be running by using inbound_channel/outbound_channel settings: https://freeswitch.org/confluence/display/FREESWITCH/mod_avmd#mod_avmd-set
for example:
<
action application="avmd_start" data="inbound_channel=0,outbound_channel=1" />will start avmd on RTP sent and received to a given endpoint.
all the best,