[Freeswitch-dev] Video synchronization
Hi, This is my first post to this list and I hope this is the right one to ask. For the last couple of weeks, I've been working on a mod to play MP4 Video files (recording will be the next step). For simplicity, I chose to use the same kind of MP4 files as the Asterisk app: H26x/MPEG Video track + PCMU audio track. I used as base the mod_fsv code and added code based on the mpeg4ip library. I've actually got a video to work but I'm having some problem to synchronize the audio and video. I have an idea of how the algorithm should be but I don't know how to use the switch_core_timer to implement it. the current loop is much like mod_fsv: while (switch_channel_ready(channel)) { vid_frame.packetlen = vid_frame.buflen; write_frame.datalen = write_frame.buflen; bool vOk = vc.getVideoPacket(vid_frame.packet, vid_frame.packetlen); bool aOk = vc.getAudioPacket(write_frame.data, write_frame.datalen); if (!vOk && !aOk) break; if (vOk) { switch_rtp_hdr_t *hdr = reinterpret_cast<switch_rtp_hdr_t *>(vid_frame.packet); // Adjusts timestamp to standard 90KHz clock. ts = ntohl(hdr->ts) * 90000 / vc.videoTrack().track.clock; hdr->ts = htonl(ts); if (pt) hdr->pt = pt; if (switch_channel_test_flag(channel, CF_VIDEO)) { switch_byte_t *data = (switch_byte_t *) vid_frame.packet; vid_frame.data = data + 12; vid_frame.datalen = vid_frame.packetlen - 12; switch_core_session_write_video_frame(session, &vid_frame, SWITCH_IO_FLAG_NONE, 0); } if (ts && last && last != ts) { switch_cond_next(); } last = ts; } if(aOk) { if (write_frame.datalen > (int) write_frame.buflen) write_frame.datalen = write_frame.buflen; switch_core_session_write_frame(session, &write_frame, SWITCH_IO_FLAG_NONE, 0); } switch_core_timer_next(&timer); } I think it should be like this (pseudocode) nextAudio = nextVideo = ts = 0 while(switch_channel_ready()) { read_video_frame(); read_audio_frame(); nextVideo = ntohl(hdr->ts) * 90000 / video.clock; waitTime = min(nextAudio, nextVideo) - ts; if(waitTime > 0) { wait(waitTime); ts += waitTime; } if(ts >= nextAudio) { send_audio_frame(); nextAudio += 20 * 90/8; // use the same clock source } if(ts >= nextVideo) { send_video_frame(); } } Any suggestion is much appreciated. Regards, Paulo R. Panhoto
we need to get nonblocking reads to the rtp working so the read_video_frame(); read_audio_frame(); don't block when there is no data. 2010/7/13 Paulo Rogério Panhoto <paulo@voicetechnology.com.br>
Hi,
This is my first post to this list and I hope this is the right one to ask. For the last couple of weeks, I've been working on a mod to play MP4 Video files (recording will be the next step). For simplicity, I chose to use the same kind of MP4 files as the Asterisk app: H26x/MPEG Video track + PCMU audio track. I used as base the mod_fsv code and added code based on the mpeg4ip library. I've actually got a video to work but I'm having some problem to synchronize the audio and video. I have an idea of how the algorithm should be but I don't know how to use the switch_core_timer to implement it.
the current loop is much like mod_fsv:
while (switch_channel_ready(channel)) { vid_frame.packetlen = vid_frame.buflen; write_frame.datalen = write_frame.buflen; bool vOk = vc.getVideoPacket(vid_frame.packet, vid_frame.packetlen); bool aOk = vc.getAudioPacket(write_frame.data, write_frame.datalen);
if (!vOk && !aOk) break;
if (vOk) { switch_rtp_hdr_t *hdr = reinterpret_cast<switch_rtp_hdr_t *>(vid_frame.packet);
// Adjusts timestamp to standard 90KHz clock. ts = ntohl(hdr->ts) * 90000 / vc.videoTrack().track.clock; hdr->ts = htonl(ts); if (pt) hdr->pt = pt;
if (switch_channel_test_flag(channel, CF_VIDEO)) { switch_byte_t *data = (switch_byte_t *) vid_frame.packet;
vid_frame.data = data + 12; vid_frame.datalen = vid_frame.packetlen - 12; switch_core_session_write_video_frame(session, &vid_frame, SWITCH_IO_FLAG_NONE, 0); } if (ts && last && last != ts) { switch_cond_next(); } last = ts; }
if(aOk) { if (write_frame.datalen > (int) write_frame.buflen) write_frame.datalen = write_frame.buflen;
switch_core_session_write_frame(session, &write_frame, SWITCH_IO_FLAG_NONE, 0);
} switch_core_timer_next(&timer); }
I think it should be like this (pseudocode)
nextAudio = nextVideo = ts = 0
while(switch_channel_ready()) { read_video_frame(); read_audio_frame();
nextVideo = ntohl(hdr->ts) * 90000 / video.clock;
waitTime = min(nextAudio, nextVideo) - ts; if(waitTime > 0) { wait(waitTime); ts += waitTime; }
if(ts >= nextAudio) { send_audio_frame(); nextAudio += 20 * 90/8; // use the same clock source } if(ts >= nextVideo) { send_video_frame(); } }
Any suggestion is much appreciated.
Regards,
Paulo R. Panhoto
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
I've found a way to synchronize the A/V streams with usleep() but I don't think that's the best way to do this. Is there a switch_core.... function to perform a similar task (and maybe allow other tasks to run)? Here's the current version of the playback loop (that takes into consideration multiple RTP packets per video frame) : u_int64_t videoNext = 0, audioNext = 0; u_int64_t ts = 0; bool videoSent = true, audioSent = true; while (switch_channel_ready(channel)) { bool vOk, aOk; if(videoSent) { vid_frame.packetlen = vid_frame.buflen; vOk = vc.getVideoPacket(vid_frame.packet, vid_frame.packetlen); if (vOk) { switch_rtp_hdr_t *hdr = reinterpret_cast<switch_rtp_hdr_t *>(vid_frame.packet); videoNext = vc.videoTrack().track.get90KTimestamp(ntohl(hdr->ts)); hdr->ts = htonl(videoNext); if (pt) hdr->pt = pt; } videoSent = false; } if(audioSent) { write_frame.datalen = write_frame.buflen; aOk = vc.getAudioPacket(write_frame.data, write_frame.datalen); audioSent = false; } if (!vOk && !aOk) break; int64_t wait = min(audioNext, videoNext) - ts; if(wait > 0) { /* wait the time for the next A/V frame */ usleep(wait * 1000 / 90); //switch_core_timer_next(&timer); ts += wait; } if (switch_channel_test_flag(channel, CF_VIDEO) && ts >= videoNext) { switch_byte_t *data = (switch_byte_t *) vid_frame.packet; vid_frame.data = data + 12; vid_frame.datalen = vid_frame.packetlen - 12; switch_core_session_write_video_frame(session, &vid_frame, SWITCH_IO_FLAG_NONE, 0); videoSent = true; } if(aOk && ts >= audioNext) { if (write_frame.datalen > (int) write_frame.buflen) write_frame.datalen = write_frame.buflen; switch_core_session_write_frame(session, &write_frame, SWITCH_IO_FLAG_NONE, 0); audioSent = true; audioNext += 90000 / 50; // 20ms } } Btw, thanks for the tip. I'll keep the separate thread to store video for the record function. Regards, Paulo On 13/07/10 15:39, Anthony Minessale wrote:
we need to get nonblocking reads to the rtp working so the
read_video_frame(); read_audio_frame();
don't block when there is no data.
without threads for both it would require doing non blocking reads on the rtp streams. We have some provisions in the code for being able to do this but I am not sure it's been proven to work. We can always try to dust it off. I think its done by passing the io_flag SWITCH_IO_FLAG_NOBLOCK to the read_frame call. Another idea would be to open the audio and video tracks separately each in it's own thread and play them independently since it's 2 different IP streams. 2010/7/13 Paulo Rogério Panhoto <paulo@voicetechnology.com.br>
I've found a way to synchronize the A/V streams with usleep() but I don't think that's the best way to do this. Is there a switch_core.... function to perform a similar task (and maybe allow other tasks to run)?
Here's the current version of the playback loop (that takes into consideration multiple RTP packets per video frame) :
u_int64_t videoNext = 0, audioNext = 0; u_int64_t ts = 0;
bool videoSent = true, audioSent = true;
while (switch_channel_ready(channel)) { bool vOk, aOk; if(videoSent)
{ vid_frame.packetlen = vid_frame.buflen; vOk = vc.getVideoPacket(vid_frame.packet, vid_frame.packetlen); if (vOk) { switch_rtp_hdr_t *hdr = reinterpret_cast<switch_rtp_hdr_t *>(vid_frame.packet);
videoNext = vc.videoTrack().track.get90KTimestamp(ntohl(hdr->ts)); hdr->ts = htonl(videoNext);
if (pt) hdr->pt = pt; } videoSent = false; } if(audioSent)
{ write_frame.datalen = write_frame.buflen; aOk = vc.getAudioPacket(write_frame.data, write_frame.datalen); audioSent = false; }
if (!vOk && !aOk) break;
int64_t wait = min(audioNext, videoNext) - ts; if(wait > 0) { /* wait the time for the next A/V frame */ usleep(wait * 1000 / 90); //switch_core_timer_next(&timer); ts += wait; }
if (switch_channel_test_flag(channel, CF_VIDEO) && ts >= videoNext)
{ switch_byte_t *data = (switch_byte_t *) vid_frame.packet;
vid_frame.data = data + 12; vid_frame.datalen = vid_frame.packetlen - 12; switch_core_session_write_video_frame(session, &vid_frame, SWITCH_IO_FLAG_NONE, 0); videoSent = true; }
if(aOk && ts >= audioNext)
{ if (write_frame.datalen > (int) write_frame.buflen) write_frame.datalen = write_frame.buflen;
switch_core_session_write_frame(session, &write_frame, SWITCH_IO_FLAG_NONE, 0); audioSent = true; audioNext += 90000 / 50; // 20ms } }
Btw, thanks for the tip. I'll keep the separate thread to store video for the record function.
Regards,
Paulo
On 13/07/10 15:39, Anthony Minessale wrote:
we need to get nonblocking reads to the rtp working so the
read_video_frame(); read_audio_frame();
don't block when there is no data.
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
Perhaps threading is the best to do. Does FS handle calls in separate threads? 2010/7/13 Anthony Minessale <anthony.minessale@gmail.com>
without threads for both it would require doing non blocking reads on the rtp streams. We have some provisions in the code for being able to do this but I am not sure it's been proven to work. We can always try to dust it off.
I think its done by passing the io_flag SWITCH_IO_FLAG_NOBLOCK to the read_frame call.
Another idea would be to open the audio and video tracks separately each in it's own thread and play them independently since it's 2 different IP streams.
2010/7/13 Paulo Rogério Panhoto <paulo@voicetechnology.com.br>
I've found a way to synchronize the A/V streams with usleep() but I don't think that's the best way to do this. Is there a switch_core.... function to perform a similar task (and maybe allow other tasks to run)?
Here's the current version of the playback loop (that takes into consideration multiple RTP packets per video frame) :
u_int64_t videoNext = 0, audioNext = 0; u_int64_t ts = 0;
bool videoSent = true, audioSent = true;
while (switch_channel_ready(channel)) { bool vOk, aOk; if(videoSent)
{ vid_frame.packetlen = vid_frame.buflen; vOk = vc.getVideoPacket(vid_frame.packet, vid_frame.packetlen); if (vOk) { switch_rtp_hdr_t *hdr = reinterpret_cast<switch_rtp_hdr_t *>(vid_frame.packet);
videoNext = vc.videoTrack().track.get90KTimestamp(ntohl(hdr->ts)); hdr->ts = htonl(videoNext);
if (pt) hdr->pt = pt; } videoSent = false; } if(audioSent)
{ write_frame.datalen = write_frame.buflen; aOk = vc.getAudioPacket(write_frame.data, write_frame.datalen); audioSent = false; }
if (!vOk && !aOk) break;
int64_t wait = min(audioNext, videoNext) - ts; if(wait > 0) { /* wait the time for the next A/V frame */ usleep(wait * 1000 / 90); //switch_core_timer_next(&timer); ts += wait; }
if (switch_channel_test_flag(channel, CF_VIDEO) && ts >= videoNext)
{ switch_byte_t *data = (switch_byte_t *) vid_frame.packet;
vid_frame.data = data + 12; vid_frame.datalen = vid_frame.packetlen - 12; switch_core_session_write_video_frame(session, &vid_frame, SWITCH_IO_FLAG_NONE, 0); videoSent = true; }
if(aOk && ts >= audioNext)
{ if (write_frame.datalen > (int) write_frame.buflen) write_frame.datalen = write_frame.buflen;
switch_core_session_write_frame(session, &write_frame, SWITCH_IO_FLAG_NONE, 0); audioSent = true; audioNext += 90000 / 50; // 20ms } }
Btw, thanks for the tip. I'll keep the separate thread to store video for the record function.
Regards,
Paulo
On 13/07/10 15:39, Anthony Minessale wrote:
we need to get nonblocking reads to the rtp working so the
read_video_frame(); read_audio_frame();
don't block when there is no data.
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
each channel runs in a separate thread. so in playback you could use the current thread for audio and launch another for the video. Since they are 2 different rtp streams you could read and write from them at the same time. 2010/7/13 Paulo Rogério Panhoto <paulo@voicetechnology.com.br>
Perhaps threading is the best to do. Does FS handle calls in separate threads?
2010/7/13 Anthony Minessale <anthony.minessale@gmail.com>
without threads for both it would require doing non blocking reads on the
rtp streams. We have some provisions in the code for being able to do this but I am not sure it's been proven to work. We can always try to dust it off.
I think its done by passing the io_flag SWITCH_IO_FLAG_NOBLOCK to the read_frame call.
Another idea would be to open the audio and video tracks separately each in it's own thread and play them independently since it's 2 different IP streams.
2010/7/13 Paulo Rogério Panhoto <paulo@voicetechnology.com.br>
I've found a way to synchronize the A/V streams with usleep() but I don't think that's the best way to do this. Is there a switch_core.... function to perform a similar task (and maybe allow other tasks to run)?
Here's the current version of the playback loop (that takes into consideration multiple RTP packets per video frame) :
u_int64_t videoNext = 0, audioNext = 0; u_int64_t ts = 0;
bool videoSent = true, audioSent = true;
while (switch_channel_ready(channel)) { bool vOk, aOk; if(videoSent)
{ vid_frame.packetlen = vid_frame.buflen; vOk = vc.getVideoPacket(vid_frame.packet, vid_frame.packetlen); if (vOk) { switch_rtp_hdr_t *hdr = reinterpret_cast<switch_rtp_hdr_t *>(vid_frame.packet);
videoNext = vc.videoTrack().track.get90KTimestamp(ntohl(hdr->ts)); hdr->ts = htonl(videoNext);
if (pt) hdr->pt = pt; } videoSent = false; } if(audioSent)
{ write_frame.datalen = write_frame.buflen; aOk = vc.getAudioPacket(write_frame.data, write_frame.datalen); audioSent = false; }
if (!vOk && !aOk) break;
int64_t wait = min(audioNext, videoNext) - ts; if(wait > 0) { /* wait the time for the next A/V frame */ usleep(wait * 1000 / 90); //switch_core_timer_next(&timer); ts += wait; }
if (switch_channel_test_flag(channel, CF_VIDEO) && ts >= videoNext)
{ switch_byte_t *data = (switch_byte_t *) vid_frame.packet;
vid_frame.data = data + 12; vid_frame.datalen = vid_frame.packetlen - 12; switch_core_session_write_video_frame(session, &vid_frame, SWITCH_IO_FLAG_NONE, 0); videoSent = true; }
if(aOk && ts >= audioNext)
{ if (write_frame.datalen > (int) write_frame.buflen) write_frame.datalen = write_frame.buflen;
switch_core_session_write_frame(session, &write_frame, SWITCH_IO_FLAG_NONE, 0); audioSent = true; audioNext += 90000 / 50; // 20ms } }
Btw, thanks for the tip. I'll keep the separate thread to store video for the record function.
Regards,
Paulo
On 13/07/10 15:39, Anthony Minessale wrote:
we need to get nonblocking reads to the rtp working so the
read_video_frame(); read_audio_frame();
don't block when there is no data.
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
participants (2)
-
Anthony Minessale
-
Paulo Rogério Panhoto