[Freeswitch-dev] Video synchronization

[Paulo Rogério Panhoto]

    Hi,

    This is my first post to this list and I hope this is the right one
to ask. For the last couple of weeks, I've been working on a mod to play
MP4 Video files (recording will be the next step). For simplicity, I
chose to use the same kind of MP4 files as the Asterisk app: H26x/MPEG
Video track + PCMU audio track. I used as base the mod_fsv code and
added code based on the mpeg4ip library. I've actually got a video to
work but I'm having some problem to synchronize the audio and video. I
have an idea of how the algorithm should be but I don't know how to use
the switch_core_timer to implement it.

the current loop is much like mod_fsv:

    while (switch_channel_ready(channel))
    {
        vid_frame.packetlen = vid_frame.buflen;
        write_frame.datalen = write_frame.buflen;
        bool vOk = vc.getVideoPacket(vid_frame.packet, vid_frame.packetlen);
        bool aOk = vc.getAudioPacket(write_frame.data, write_frame.datalen);
       
        if (!vOk && !aOk)
            break;

        if (vOk)
        {
            switch_rtp_hdr_t *hdr = reinterpret_cast<switch_rtp_hdr_t
*>(vid_frame.packet);

            // Adjusts timestamp to standard 90KHz clock.
            ts = ntohl(hdr->ts) * 90000 / vc.videoTrack().track.clock;
            hdr->ts = htonl(ts);
            if (pt)
                hdr->pt = pt;

            if (switch_channel_test_flag(channel, CF_VIDEO))
            {
                switch_byte_t *data = (switch_byte_t *) vid_frame.packet;

                vid_frame.data = data + 12;
                vid_frame.datalen = vid_frame.packetlen - 12;
                switch_core_session_write_video_frame(session,
&vid_frame, SWITCH_IO_FLAG_NONE, 0);
            }
            if (ts && last && last != ts) {
                switch_cond_next();
            }
            last = ts;
        }

        if(aOk)
        {
            if (write_frame.datalen > (int) write_frame.buflen)
                write_frame.datalen = write_frame.buflen;

            switch_core_session_write_frame(session, &write_frame,
SWITCH_IO_FLAG_NONE, 0);

        }
        switch_core_timer_next(&timer);
    }

I think it should be like this (pseudocode)

nextAudio = nextVideo = ts = 0

while(switch_channel_ready())
{
    read_video_frame();
    read_audio_frame();
 
    nextVideo = ntohl(hdr->ts) * 90000 / video.clock;

    waitTime = min(nextAudio, nextVideo) - ts;
    if(waitTime > 0)
     {
        wait(waitTime);
        ts += waitTime;
    }

    if(ts >= nextAudio)
    {
       send_audio_frame();
       nextAudio += 20 * 90/8; // use the same clock source
    }
    if(ts >= nextVideo)
    {
        send_video_frame();
    }
}

    Any suggestion is much appreciated.

    Regards,

    Paulo R. Panhoto