[Freeswitch-dev] Video synchronization

[Anthony Minessale]

we need to get nonblocking reads to the rtp working so the

  read_video_frame();
   read_audio_frame();

don't block when there is no data.


2010/7/13 Paulo Rogério Panhoto <paulo at voicetechnology.com.br>

>    Hi,
>
>    This is my first post to this list and I hope this is the right one
> to ask. For the last couple of weeks, I've been working on a mod to play
> MP4 Video files (recording will be the next step). For simplicity, I
> chose to use the same kind of MP4 files as the Asterisk app: H26x/MPEG
> Video track + PCMU audio track. I used as base the mod_fsv code and
> added code based on the mpeg4ip library. I've actually got a video to
> work but I'm having some problem to synchronize the audio and video. I
> have an idea of how the algorithm should be but I don't know how to use
> the switch_core_timer to implement it.
>
> the current loop is much like mod_fsv:
>
>    while (switch_channel_ready(channel))
>    {
>        vid_frame.packetlen = vid_frame.buflen;
>        write_frame.datalen = write_frame.buflen;
>        bool vOk = vc.getVideoPacket(vid_frame.packet, vid_frame.packetlen);
>        bool aOk = vc.getAudioPacket(write_frame.data, write_frame.datalen);
>
>        if (!vOk && !aOk)
>            break;
>
>        if (vOk)
>        {
>            switch_rtp_hdr_t *hdr = reinterpret_cast<switch_rtp_hdr_t
> *>(vid_frame.packet);
>
>            // Adjusts timestamp to standard 90KHz clock.
>            ts = ntohl(hdr->ts) * 90000 / vc.videoTrack().track.clock;
>            hdr->ts = htonl(ts);
>            if (pt)
>                hdr->pt = pt;
>
>            if (switch_channel_test_flag(channel, CF_VIDEO))
>            {
>                switch_byte_t *data = (switch_byte_t *) vid_frame.packet;
>
>                vid_frame.data = data + 12;
>                vid_frame.datalen = vid_frame.packetlen - 12;
>                switch_core_session_write_video_frame(session,
> &vid_frame, SWITCH_IO_FLAG_NONE, 0);
>            }
>            if (ts && last && last != ts) {
>                switch_cond_next();
>            }
>            last = ts;
>        }
>
>        if(aOk)
>        {
>            if (write_frame.datalen > (int) write_frame.buflen)
>                write_frame.datalen = write_frame.buflen;
>
>            switch_core_session_write_frame(session, &write_frame,
> SWITCH_IO_FLAG_NONE, 0);
>
>        }
>        switch_core_timer_next(&timer);
>    }
>
> I think it should be like this (pseudocode)
>
> nextAudio = nextVideo = ts = 0
>
> while(switch_channel_ready())
> {
>    read_video_frame();
>    read_audio_frame();
>
>    nextVideo = ntohl(hdr->ts) * 90000 / video.clock;
>
>    waitTime = min(nextAudio, nextVideo) - ts;
>    if(waitTime > 0)
>     {
>        wait(waitTime);
>        ts += waitTime;
>    }
>
>    if(ts >= nextAudio)
>    {
>       send_audio_frame();
>       nextAudio += 20 * 90/8; // use the same clock source
>    }
>    if(ts >= nextVideo)
>    {
>        send_video_frame();
>    }
> }
>
>    Any suggestion is much appreciated.
>
>    Regards,
>
>    Paulo R. Panhoto
>
> _______________________________________________
> FreeSWITCH-dev mailing list
> FreeSWITCH-dev at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev
> http://www.freeswitch.org
>



-- 
Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm
MSN:anthony_minessale at hotmail.com <MSN%3Aanthony_minessale at hotmail.com>
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com<PAYPAL%3Aanthony.minessale at gmail.com>
IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org <sip%3A888 at conference.freeswitch.org>
googletalk:conf+888 at conference.freeswitch.org<googletalk%3Aconf%2B888 at conference.freeswitch.org>
pstn:+19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-dev/attachments/20100713/570c68f2/attachment.html