RTP packet loss on outgoing streams
Hello, we need to monitor the voice quality on our server. To test this, we setup a test environment with 2 freeswitches, originating calls from freeswitch 1 and then answer them on freeswitch 2 and play an announcement. To analyze the traffic, we us tshark with the following command: tshark -q -f 'udp' -o rtp.heuristic_rtp:TRUE -z rtp,strems -w test-pcap The result is, that we see packet loss around 20% on outgoing RTP streams, when we have more then 20 calls. The load average shown with the top command is always < 0.3 with a 4 core CPU and CPU load is less then 10%. ========================= RTP Streams ======================== Src IP addr Port Dest IP addr Port SSRC Payload Pkts Lost Max Delta(ms) Max Jitter(ms) Mean Jitter(ms) Problems? 192.168.1.134 27590 192.168.1.135 22576 0xCD7A189D ITU-T G.722 800 2 (0.2%) 59.93 0.11 0.05 X 192.168.1.135 22576 192.168.1.134 27590 0x5D4E50CD ITU-T G.722 599 202 (25.2%) 160.24 0.16 0.05 X 192.168.1.134 28172 192.168.1.135 29704 0xD53EC9AD ITU-T G.722 798 2 (0.2%) 59.99 0.07 0.04 X 192.168.1.135 29704 192.168.1.134 28172 0x5D50942D ITU-T G.722 713 86 (10.8%) 199.98 0.08 0.04 X 192.168.1.135 25794 192.168.1.134 16460 0x5D51E43D ITU-T G.722 633 165 (20.7%) 180.25 0.75 0.07 X Instead of tshark, we also tried to capture SIP & RTP with tcpdump with the following command: tcpdump -i any -T rtp -vvvvv -s 0 -w test.pcap If we analyze this file with wireshark (TELEPHONY->RTP->Show all Streams), it shows almost always 0% for packet loss. Only if we increase the load to 50 calls we can see a similar scenario where inbound calls still have < 1% packet loss but outgoing RTP streams have up to 20 % packet loss. We also repeated the same test with freeswitch 1.4.15 with much more powerful hardware with almost the same results. The question is what do we see here. Does tshark show us wrong result? Can we more trust on the results captured with tcpdump or is there finally an issue in freeswitch when sending outgoing packets? Or is the method to capture on the server in general the wrong approach? Thanks for any comments and hints Markus
Where are you capturing this data? ON the system itself? or some other point in the network? On Fri, Apr 17, 2015 at 10:01 AM, mbo <mbodbg@gmx.net> wrote:
Hello,
we need to monitor the voice quality on our server. To test this, we setup a test environment with 2 freeswitches, originating calls from freeswitch 1 and then answer them on freeswitch 2 and play an announcement. To analyze the traffic, we us tshark with the following command:
tshark -q -f 'udp' -o rtp.heuristic_rtp:TRUE -z rtp,strems -w test-pcap
The result is, that we see packet loss around 20% on outgoing RTP streams, when we have more then 20 calls. The load average shown with the top command is always < 0.3 with a 4 core CPU and CPU load is less then 10%.
========================= RTP Streams ======================== Src IP addr Port Dest IP addr Port SSRC Payload Pkts Lost Max Delta(ms) Max Jitter(ms) Mean Jitter(ms) Problems? 192.168.1.134 27590 192.168.1.135 22576 0xCD7A189D ITU-T G.722 800 2 (0.2%) 59.93 0.11 0.05 X 192.168.1.135 22576 192.168.1.134 27590 0x5D4E50CD ITU-T G.722 599 202 (25.2%) 160.24 0.16 0.05 X 192.168.1.134 28172 192.168.1.135 29704 0xD53EC9AD ITU-T G.722 798 2 (0.2%) 59.99 0.07 0.04 X 192.168.1.135 29704 192.168.1.134 28172 0x5D50942D ITU-T G.722 713 86 (10.8%) 199.98 0.08 0.04 X 192.168.1.135 25794 192.168.1.134 16460 0x5D51E43D ITU-T G.722 633 165 (20.7%) 180.25 0.75 0.07 X
Instead of tshark, we also tried to capture SIP & RTP with tcpdump with the following command:
tcpdump -i any -T rtp -vvvvv -s 0 -w test.pcap
If we analyze this file with wireshark (TELEPHONY->RTP->Show all Streams), it shows almost always 0% for packet loss. Only if we increase the load to 50 calls we can see a similar scenario where inbound calls still have < 1% packet loss but outgoing RTP streams have up to 20 % packet loss. We also repeated the same test with freeswitch 1.4.15 with much more powerful hardware with almost the same results.
The question is what do we see here. Does tshark show us wrong result? Can we more trust on the results captured with tcpdump or is there finally an issue in freeswitch when sending outgoing packets? Or is the method to capture on the server in general the wrong approach?
Thanks for any comments and hints
Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com ClueCon 2015 Call for Speakers <https://www.cluecon.com/call-for-speakers/> | Register <https://freeswitch.com/cart.php?gid=1> TODAY! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
On the system itself. Am 17.04.2015 um 17:47 schrieb Brian West <brian@freeswitch.org>:
Where are you capturing this data? ON the system itself? or some other point in the network?
On Fri, Apr 17, 2015 at 10:01 AM, mbo <mbodbg@gmx.net> wrote: Hello,
we need to monitor the voice quality on our server. To test this, we setup a test environment with 2 freeswitches, originating calls from freeswitch 1 and then answer them on freeswitch 2 and play an announcement. To analyze the traffic, we us tshark with the following command:
tshark -q -f 'udp' -o rtp.heuristic_rtp:TRUE -z rtp,strems -w test-pcap
The result is, that we see packet loss around 20% on outgoing RTP streams, when we have more then 20 calls. The load average shown with the top command is always < 0.3 with a 4 core CPU and CPU load is less then 10%.
========================= RTP Streams ======================== Src IP addr Port Dest IP addr Port SSRC Payload Pkts Lost Max Delta(ms) Max Jitter(ms) Mean Jitter(ms) Problems? 192.168.1.134 27590 192.168.1.135 22576 0xCD7A189D ITU-T G.722 800 2 (0.2%) 59.93 0.11 0.05 X 192.168.1.135 22576 192.168.1.134 27590 0x5D4E50CD ITU-T G.722 599 202 (25.2%) 160.24 0.16 0.05 X 192.168.1.134 28172 192.168.1.135 29704 0xD53EC9AD ITU-T G.722 798 2 (0.2%) 59.99 0.07 0.04 X 192.168.1.135 29704 192.168.1.134 28172 0x5D50942D ITU-T G.722 713 86 (10.8%) 199.98 0.08 0.04 X 192.168.1.135 25794 192.168.1.134 16460 0x5D51E43D ITU-T G.722 633 165 (20.7%) 180.25 0.75 0.07 X
Instead of tshark, we also tried to capture SIP & RTP with tcpdump with the following command:
tcpdump -i any -T rtp -vvvvv -s 0 -w test.pcap
If we analyze this file with wireshark (TELEPHONY->RTP->Show all Streams), it shows almost always 0% for packet loss. Only if we increase the load to 50 calls we can see a similar scenario where inbound calls still have < 1% packet loss but outgoing RTP streams have up to 20 % packet loss. We also repeated the same test with freeswitch 1.4.15 with much more powerful hardware with almost the same results.
The question is what do we see here. Does tshark show us wrong result? Can we more trust on the results captured with tcpdump or is there finally an issue in freeswitch when sending outgoing packets? Or is the method to capture on the server in general the wrong approach?
Thanks for any comments and hints
Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Brian West brian@freeswitch.org
Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com http://www.freeswitchcookbook.com
ClueCon 2015 Call for Speakers | Register TODAY! | Reddit: /r/freeswitch
T:+19184209001 | F:+19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
mbo mbodbg@gmx.net
What kind of network cards does this system have? On Fri, Apr 17, 2015 at 2:06 PM, mbo <mbodbg@gmx.net> wrote:
On the system itself.
Am 17.04.2015 um 17:47 schrieb Brian West <brian@freeswitch.org>:
Where are you capturing this data? ON the system itself? or some other point in the network?
On Fri, Apr 17, 2015 at 10:01 AM, mbo <mbodbg@gmx.net> wrote:
Hello,
we need to monitor the voice quality on our server. To test this, we setup a test environment with 2 freeswitches, originating calls from freeswitch 1 and then answer them on freeswitch 2 and play an announcement. To analyze the traffic, we us tshark with the following command:
tshark -q -f 'udp' -o rtp.heuristic_rtp:TRUE -z rtp,strems -w test-pcap
The result is, that we see packet loss around 20% on outgoing RTP streams, when we have more then 20 calls. The load average shown with the top command is always < 0.3 with a 4 core CPU and CPU load is less then 10%.
========================= RTP Streams ======================== Src IP addr Port Dest IP addr Port SSRC Payload Pkts Lost Max Delta(ms) Max Jitter(ms) Mean Jitter(ms) Problems? 192.168.1.134 27590 192.168.1.135 22576 0xCD7A189D ITU-T G.722 800 2 (0.2%) 59.93 0.11 0.05 X 192.168.1.135 22576 192.168.1.134 27590 0x5D4E50CD ITU-T G.722 599 202 (25.2%) 160.24 0.16 0.05 X 192.168.1.134 28172 192.168.1.135 29704 0xD53EC9AD ITU-T G.722 798 2 (0.2%) 59.99 0.07 0.04 X 192.168.1.135 29704 192.168.1.134 28172 0x5D50942D ITU-T G.722 713 86 (10.8%) 199.98 0.08 0.04 X 192.168.1.135 25794 192.168.1.134 16460 0x5D51E43D ITU-T G.722 633 165 (20.7%) 180.25 0.75 0.07 X
Instead of tshark, we also tried to capture SIP & RTP with tcpdump with the following command:
tcpdump -i any -T rtp -vvvvv -s 0 -w test.pcap
If we analyze this file with wireshark (TELEPHONY->RTP->Show all Streams), it shows almost always 0% for packet loss. Only if we increase the load to 50 calls we can see a similar scenario where inbound calls still have < 1% packet loss but outgoing RTP streams have up to 20 % packet loss. We also repeated the same test with freeswitch 1.4.15 with much more powerful hardware with almost the same results.
The question is what do we see here. Does tshark show us wrong result? Can we more trust on the results captured with tcpdump or is there finally an issue in freeswitch when sending outgoing packets? Or is the method to capture on the server in general the wrong approach?
Thanks for any comments and hints
Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
*Brian West* brian@freeswitch.org
*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com
ClueCon 2015 Call for Speakers <https://www.cluecon.com/call-for-speakers/> | Register <https://freeswitch.com/cart.php?gid=1> TODAY! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>
*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
mbo mbodbg@gmx.net
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com ClueCon 2015 Call for Speakers <https://www.cluecon.com/call-for-speakers/> | Register <https://freeswitch.com/cart.php?gid=1> TODAY! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
you can try playing with the buffer size (-B option), but in general, tshark is quite CPU-intensive when it tries to analyze the UDP packet stream in real time. Also I'm not sure if it's multithreaded -- I guess it's not, so it tries to analyze all the streams in a single thread in real time. I made this set of scripts to run on a low-power CPU (Atom and such): https://github.com/voxserv/voip_qos_probe What I do is run tcpdump during the call, store the capture, and then analyze the output with tshark in low priority. Also I noticed that on a Xen VM on an Intel CoreDuo CPU, you need to increase the tcpdump's buffer size with -B option. With default settings, sometimes it's not even able to capture a single SIP session without dropping packets. On Fri, Apr 17, 2015 at 5:01 PM, mbo <mbodbg@gmx.net> wrote:
Hello,
we need to monitor the voice quality on our server. To test this, we setup a test environment with 2 freeswitches, originating calls from freeswitch 1 and then answer them on freeswitch 2 and play an announcement. To analyze the traffic, we us tshark with the following command:
tshark -q -f 'udp' -o rtp.heuristic_rtp:TRUE -z rtp,strems -w test-pcap
The result is, that we see packet loss around 20% on outgoing RTP streams, when we have more then 20 calls. The load average shown with the top command is always < 0.3 with a 4 core CPU and CPU load is less then 10%.
========================= RTP Streams ======================== Src IP addr Port Dest IP addr Port SSRC Payload Pkts Lost Max Delta(ms) Max Jitter(ms) Mean Jitter(ms) Problems? 192.168.1.134 27590 192.168.1.135 22576 0xCD7A189D ITU-T G.722 800 2 (0.2%) 59.93 0.11 0.05 X 192.168.1.135 22576 192.168.1.134 27590 0x5D4E50CD ITU-T G.722 599 202 (25.2%) 160.24 0.16 0.05 X 192.168.1.134 28172 192.168.1.135 29704 0xD53EC9AD ITU-T G.722 798 2 (0.2%) 59.99 0.07 0.04 X 192.168.1.135 29704 192.168.1.134 28172 0x5D50942D ITU-T G.722 713 86 (10.8%) 199.98 0.08 0.04 X 192.168.1.135 25794 192.168.1.134 16460 0x5D51E43D ITU-T G.722 633 165 (20.7%) 180.25 0.75 0.07 X
Instead of tshark, we also tried to capture SIP & RTP with tcpdump with the following command:
tcpdump -i any -T rtp -vvvvv -s 0 -w test.pcap
If we analyze this file with wireshark (TELEPHONY->RTP->Show all Streams), it shows almost always 0% for packet loss. Only if we increase the load to 50 calls we can see a similar scenario where inbound calls still have < 1% packet loss but outgoing RTP streams have up to 20 % packet loss. We also repeated the same test with freeswitch 1.4.15 with much more powerful hardware with almost the same results.
The question is what do we see here. Does tshark show us wrong result? Can we more trust on the results captured with tcpdump or is there finally an issue in freeswitch when sending outgoing packets? Or is the method to capture on the server in general the wrong approach?
Thanks for any comments and hints
Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
What i'm getting at is you should never have dropped packets on the host itself like that, If you do then you probably have WinNic's aka RealTek garbage. On Fri, Apr 17, 2015 at 3:47 PM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
you can try playing with the buffer size (-B option), but in general, tshark is quite CPU-intensive when it tries to analyze the UDP packet stream in real time. Also I'm not sure if it's multithreaded -- I guess it's not, so it tries to analyze all the streams in a single thread in real time.
I made this set of scripts to run on a low-power CPU (Atom and such): https://github.com/voxserv/voip_qos_probe What I do is run tcpdump during the call, store the capture, and then analyze the output with tshark in low priority.
Also I noticed that on a Xen VM on an Intel CoreDuo CPU, you need to increase the tcpdump's buffer size with -B option. With default settings, sometimes it's not even able to capture a single SIP session without dropping packets.
On Fri, Apr 17, 2015 at 5:01 PM, mbo <mbodbg@gmx.net> wrote:
Hello,
we need to monitor the voice quality on our server. To test this, we setup a test environment with 2 freeswitches, originating calls from freeswitch 1 and then answer them on freeswitch 2 and play an announcement. To analyze the traffic, we us tshark with the following command:
tshark -q -f 'udp' -o rtp.heuristic_rtp:TRUE -z rtp,strems -w test-pcap
The result is, that we see packet loss around 20% on outgoing RTP streams, when we have more then 20 calls. The load average shown with the top command is always < 0.3 with a 4 core CPU and CPU load is less then 10%.
========================= RTP Streams ======================== Src IP addr Port Dest IP addr Port SSRC Payload Pkts Lost Max Delta(ms) Max Jitter(ms) Mean Jitter(ms) Problems? 192.168.1.134 27590 192.168.1.135 22576 0xCD7A189D ITU-T G.722 800 2 (0.2%) 59.93 0.11 0.05 X 192.168.1.135 22576 192.168.1.134 27590 0x5D4E50CD ITU-T G.722 599 202 (25.2%) 160.24 0.16 0.05 X 192.168.1.134 28172 192.168.1.135 29704 0xD53EC9AD ITU-T G.722 798 2 (0.2%) 59.99 0.07 0.04 X 192.168.1.135 29704 192.168.1.134 28172 0x5D50942D ITU-T G.722 713 86 (10.8%) 199.98 0.08 0.04 X 192.168.1.135 25794 192.168.1.134 16460 0x5D51E43D ITU-T G.722 633 165 (20.7%) 180.25 0.75 0.07 X
Instead of tshark, we also tried to capture SIP & RTP with tcpdump with the following command:
tcpdump -i any -T rtp -vvvvv -s 0 -w test.pcap
If we analyze this file with wireshark (TELEPHONY->RTP->Show all Streams), it shows almost always 0% for packet loss. Only if we increase the load to 50 calls we can see a similar scenario where inbound calls still have < 1% packet loss but outgoing RTP streams have up to 20 % packet loss. We also repeated the same test with freeswitch 1.4.15 with much more powerful hardware with almost the same results.
The question is what do we see here. Does tshark show us wrong result? Can we more trust on the results captured with tcpdump or is there finally an issue in freeswitch when sending outgoing packets? Or is the method to capture on the server in general the wrong approach?
Thanks for any comments and hints
Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com ClueCon 2015 Call for Speakers <https://www.cluecon.com/call-for-speakers/> | Register <https://freeswitch.com/cart.php?gid=1> TODAY! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
it's not dropped packets, it's the packets that didn't make it to tshark's buffer, because it was too slow and the buffer was too short. On Fri, Apr 17, 2015 at 10:49 PM, Brian West <brian@freeswitch.org> wrote:
What i'm getting at is you should never have dropped packets on the host itself like that, If you do then you probably have WinNic's aka RealTek garbage.
On Fri, Apr 17, 2015 at 3:47 PM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
you can try playing with the buffer size (-B option), but in general, tshark is quite CPU-intensive when it tries to analyze the UDP packet stream in real time. Also I'm not sure if it's multithreaded -- I guess it's not, so it tries to analyze all the streams in a single thread in real time.
I made this set of scripts to run on a low-power CPU (Atom and such): https://github.com/voxserv/voip_qos_probe What I do is run tcpdump during the call, store the capture, and then analyze the output with tshark in low priority.
Also I noticed that on a Xen VM on an Intel CoreDuo CPU, you need to increase the tcpdump's buffer size with -B option. With default settings, sometimes it's not even able to capture a single SIP session without dropping packets.
On Fri, Apr 17, 2015 at 5:01 PM, mbo <mbodbg@gmx.net> wrote:
Hello,
we need to monitor the voice quality on our server. To test this, we setup a test environment with 2 freeswitches, originating calls from freeswitch 1 and then answer them on freeswitch 2 and play an announcement. To analyze the traffic, we us tshark with the following command:
tshark -q -f 'udp' -o rtp.heuristic_rtp:TRUE -z rtp,strems -w test-pcap
The result is, that we see packet loss around 20% on outgoing RTP streams, when we have more then 20 calls. The load average shown with the top command is always < 0.3 with a 4 core CPU and CPU load is less then 10%.
========================= RTP Streams ======================== Src IP addr Port Dest IP addr Port SSRC Payload Pkts Lost Max Delta(ms) Max Jitter(ms) Mean Jitter(ms) Problems? 192.168.1.134 27590 192.168.1.135 22576 0xCD7A189D ITU-T G.722 800 2 (0.2%) 59.93 0.11 0.05 X 192.168.1.135 22576 192.168.1.134 27590 0x5D4E50CD ITU-T G.722 599 202 (25.2%) 160.24 0.16 0.05 X 192.168.1.134 28172 192.168.1.135 29704 0xD53EC9AD ITU-T G.722 798 2 (0.2%) 59.99 0.07 0.04 X 192.168.1.135 29704 192.168.1.134 28172 0x5D50942D ITU-T G.722 713 86 (10.8%) 199.98 0.08 0.04 X 192.168.1.135 25794 192.168.1.134 16460 0x5D51E43D ITU-T G.722 633 165 (20.7%) 180.25 0.75 0.07 X
Instead of tshark, we also tried to capture SIP & RTP with tcpdump with the following command:
tcpdump -i any -T rtp -vvvvv -s 0 -w test.pcap
If we analyze this file with wireshark (TELEPHONY->RTP->Show all Streams), it shows almost always 0% for packet loss. Only if we increase the load to 50 calls we can see a similar scenario where inbound calls still have < 1% packet loss but outgoing RTP streams have up to 20 % packet loss. We also repeated the same test with freeswitch 1.4.15 with much more powerful hardware with almost the same results.
The question is what do we see here. Does tshark show us wrong result? Can we more trust on the results captured with tcpdump or is there finally an issue in freeswitch when sending outgoing packets? Or is the method to capture on the server in general the wrong approach?
Thanks for any comments and hints
Markus
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
*Brian West* brian@freeswitch.org
*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com
ClueCon 2015 Call for Speakers <https://www.cluecon.com/call-for-speakers/> | Register <https://freeswitch.com/cart.php?gid=1> TODAY! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>
*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I did some more test and when I use the -B 32768 option with tcpdump I can trace up to 150 calls without packets dropped by the kernel. Finally that means as soon as you have more than 15 parallel calls on the server it’s very important to set this parameter to not get wrong results. Many thanks for your support Markus Am 17.04.2015 um 23:13 schrieb Stanislav Sinyagin <ssinyagin@gmail.com>:
it's not dropped packets, it's the packets that didn't make it to tshark's buffer, because it was too slow and the buffer was too short.
On Fri, Apr 17, 2015 at 10:49 PM, Brian West <brian@freeswitch.org> wrote: What i'm getting at is you should never have dropped packets on the host itself like that, If you do then you probably have WinNic's aka RealTek garbage.
On Fri, Apr 17, 2015 at 3:47 PM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote: you can try playing with the buffer size (-B option), but in general, tshark is quite CPU-intensive when it tries to analyze the UDP packet stream in real time. Also I'm not sure if it's multithreaded -- I guess it's not, so it tries to analyze all the streams in a single thread in real time.
I made this set of scripts to run on a low-power CPU (Atom and such): https://github.com/voxserv/voip_qos_probe What I do is run tcpdump during the call, store the capture, and then analyze the output with tshark in low priority.
Also I noticed that on a Xen VM on an Intel CoreDuo CPU, you need to increase the tcpdump's buffer size with -B option. With default settings, sometimes it's not even able to capture a single SIP session without dropping packets.
On Fri, Apr 17, 2015 at 5:01 PM, mbo <mbodbg@gmx.net> wrote:
Hello,
we need to monitor the voice quality on our server. To test this, we setup a test environment with 2 freeswitches, originating calls from freeswitch 1 and then answer them on freeswitch 2 and play an announcement. To analyze the traffic, we us tshark with the following command:
tshark -q -f 'udp' -o rtp.heuristic_rtp:TRUE -z rtp,strems -w test-pcap
The result is, that we see packet loss around 20% on outgoing RTP streams, when we have more then 20 calls. The load average shown with the top command is always < 0.3 with a 4 core CPU and CPU load is less then 10%.
========================= RTP Streams ======================== Src IP addr Port Dest IP addr Port SSRC Payload Pkts Lost Max Delta(ms) Max Jitter(ms) Mean Jitter(ms) Problems? 192.168.1.134 27590 192.168.1.135 22576 0xCD7A189D ITU-T G.722 800 2 (0.2%) 59.93 0.11 0.05 X 192.168.1.135 22576 192.168.1.134 27590 0x5D4E50CD ITU-T G.722 599 202 (25.2%) 160.24 0.16 0.05 X 192.168.1.134 28172 192.168.1.135 29704 0xD53EC9AD ITU-T G.722 798 2 (0.2%) 59.99 0.07 0.04 X 192.168.1.135 29704 192.168.1.134 28172 0x5D50942D ITU-T G.722 713 86 (10.8%) 199.98 0.08 0.04 X 192.168.1.135 25794 192.168.1.134 16460 0x5D51E43D ITU-T G.722 633 165 (20.7%) 180.25 0.75 0.07 X
Instead of tshark, we also tried to capture SIP & RTP with tcpdump with the following command:
tcpdump -i any -T rtp -vvvvv -s 0 -w test.pcap
If we analyze this file with wireshark (TELEPHONY->RTP->Show all Streams), it shows almost always 0% for packet loss. Only if we increase the load to 50 calls we can see a similar scenario where inbound calls still have < 1% packet loss but outgoing RTP streams have up to 20 % packet loss. We also repeated the same test with freeswitch 1.4.15 with much more powerful hardware with almost the same results.
The question is what do we see here. Does tshark show us wrong result? Can we more trust on the results captured with tcpdump or is there finally an issue in freeswitch when sending outgoing packets? Or is the method to capture on the server in general the wrong approach?
Thanks for any comments and hints
Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Brian West brian@freeswitch.org
Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com http://www.freeswitchcookbook.com
ClueCon 2015 Call for Speakers | Register TODAY! | Reddit: /r/freeswitch
T:+19184209001 | F:+19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (3)
-
Brian West
-
mbo
-
Stanislav Sinyagin