[Freeswitch-users] Jitsi + Freeswitch (and general questions about SFU)

Thu Aug 4 23:26:00 MSD 2016

(Anthony, response to you is below Michael's response to consolidate
messages). Apologies in advance for the long reply - but love the
discussion.

Hey Michael,

It's not so much the number of clients that is problematic but rather the
difficulty of capacity planning when operations become increasingly more
expensive. It presses towards building a smaller number of very large FS
instances, over a smaller number of medium sized ones that you could get
away with for audio + network traffic. The system I'm building is
multi-tenant, and there's application code written around distributing
conferences across FS hosts. But say an FS host has 80% CPU, it's hard to
know in advance if a conference I'm about to place there will turn into a
10 person video chat or a 2 person audio chat. With an MCU approach,
there's a very real possibility that if 10 people join I exhaust CPU and
run into issues. Whereas with an SFU, there's much less risk of any one
conference pushing over that edge, because that 20% can still handle a
considerable number of clients. It just makes the capacity planning and
distribution process easier when a typical video conference consumes a very
small number of CPU resources.

As far as the second comment - I agree. Certain devices may be limited in
their processing power. But mobile devices are *also* limited in screen
real state, so there may not be an MCU layout that works well for a
combination of both mobile and desktop devices. With an SFU approach, you
could choose to receive all streams on the mobile device but only decode 2
or 3 of them (at the application's discretion), and lay them out in a way
that is optimal for that device. It wastes network bandwidth, but preserves
experience. You could do similar with a canvas per caller, but then you're
even further increasing the cost on the server.

In any case I'm not intending to argue that SFU is a better approach
generally (or argue at all for that matter), just trying to better
understand the required additions to FS to make it happen. With the
addition of mod_verto and full native WebRTC support, having an SFU
capability would allow FS to cater to the FaceTime/Hangouts/Skype types of
applications with highly customized (and hopefully optimized) experiences.

Anthony,

I agree that what I'm referring to would be little more than a replication
of what Jitsi does, but the "value add" would be having it baked directly
into mod_conference with no need to interface with another product. I'm
finding it hard to rationalize how I would get Jitsi Video Bridge working
with Freeswitch in a way that doesn't cause more problems than it solves -
and FS caters to far more use cases independently than Jitsi can. To
clarify, I'm not asking anyone here to contribute the code - rather just
looking for any pointers on what it might take. If SFU were to be added as
an exclusive *OR* with the current canvas muxing, is this a PR that the FS
would accept? Or is it a requirement that it be able to operate in a hybrid
MCU + SFU mode? I'm still not quite sure how useful that would be, though -
except in the cases of better supporting mobile (for which there are SFU
workarounds), or older devices that don't do multi-stream (seems unlikely
that these use cases would want SFU in the first place though).

Sorry again if I'm annoying anyone! I'm really just looking for the
dialog/hoping to learn more.

Best,
Colin

On Thu, Aug 4, 2016 at 2:59 PM Michael Jerris <mike at jerris.com> wrote:

> how many clients are you talking about?  You may want to look closely,
> particularly at mobile clients, as they wont gracefully handle multiple
> streams, they don't have the processor to handle that much video decode,
> and often don't have the bandwidth either.
>
> On Thursday, August 4, 2016, Colin Morelli <colin.morelli at gmail.com>
> wrote:
>
>> Anthony,
>>
>> Thanks for the quick reply.
>>
>> Could you elaborate more on the need for FS to support a hybrid mux/multi
>> stream model (for a single conference?) I'd love to understand more about
>> this so I can wrap my head around the efforts involved. I wonder if there's
>> a phase 1 that could be reached where you'd be forced to choose between
>> SFU/MCU modes? Understanding that it might sacrifice compatibility with
>> clients that can't accept multiple video streams, or the particular codecs
>> that are used on a stream, I would think this allows FS to *at least* play
>> the role that Jitsi would play today - and allow for future enhancements.
>>
>> I think MCU makes a lot of sense for many deployments - and is very
>> sensible to support. The two concerns I personally have with MCU are a) the
>> obvious CPU constraints imposed on the hosts, but more concerning is 2) the
>> UX constraints it imposes on the resulting interfaces. #2 becomes
>> increasingly relevant as WebRTC/mobile/desktop clients become more and more
>> prevalent.
>>
>> Best,
>> Colin
>>
>> On Thu, Aug 4, 2016 at 2:18 PM Anthony Minessale <
>> anthony.minessale at gmail.com> wrote:
>>
>>> It will be a massive project to support SFU.  It would be a hybrid of
>>> the 2 not an either/or.  Its all been considered but there is less to gain
>>> from SFU so its not a priority.
>>> Unlike Jitsi, FS would have to actually implement the media side of the
>>> equation and dual support muxing and multi streaming where Jitsi enjoys
>>> only needing to provide signaling between browsers.
>>> FreeSWITCH is better suited to an MCU and we made a specific decision to
>>> focus on that but there might be a slow march towards these other things as
>>> time progresses and interest does or doesn't happen.
>>>
>>>
>>> On Thu, Aug 4, 2016 at 1:04 PM, Colin Morelli <colin.morelli at gmail.com>
>>> wrote:
>>>
>>>> For what it's worth I also understand FS would need to be able to push
>>>> multiple video streams to each connected client, but given that FS can
>>>> already handle audio + video streams, I assumed multi stream handling
>>>> exists (though it may be limited to just 1 audio + 1 video - I haven't dug
>>>> into this at all).
>>>>
>>>> On Thu, Aug 4, 2016 at 1:52 PM Colin Morelli <colin.morelli at gmail.com>
>>>> wrote:
>>>>
>>>>> Hey there,
>>>>>
>>>>> Has anyone had success pairing up Freeswitch + Jitsi? Specifically I'm
>>>>> interested in the SFU capabilities of Jitsi to replace just the video
>>>>> conferencing portions of Freeswitch. I'm doing a lot of other things in FS
>>>>> that will continue to be important on their own (DTMF, voicemail, SIP +
>>>>> PSTN integration, etc), but the SFU support would be an awesome addition. I
>>>>> saw that there's a SIP <-> Jitsi bridge which looks like it might be
>>>>> promising - but I feel like the complexity would fairly quickly become
>>>>> overwhelming.
>>>>>
>>>>> Alternatively, I wonder how much work it would be to add
>>>>> video-mode="sfu" to mod_conference in FS. It's something that (in the
>>>>> future, not quite yet), I might be interested in taking a stab at. Has
>>>>> anyone come up with a list of things that would need to happen for SFU
>>>>> support? I might be a bit naive here, but it seems FS has most of the
>>>>> building blocks in place already, and mostly it would come down a) having a
>>>>> video stream selection process, and b) generating appropriate SDPs when
>>>>> participants are added/removed. I'm sure I must be missing something,
>>>>> though.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> Best,
>>>>> Colin
>>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>>
>>> --
>>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>>
>>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>>> http://twitter.com/FreeSWITCH
>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>> <http://freeswitch.org/g+>*
>>>
>>> ClueCon Weekly Development Call
>>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>>
>>> https://www.youtube.com/watch?v=9XXgW34t40s
>>> https://www.youtube.com/watch?v=NLaDpGQuZDA
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20160804/569ac85f/attachment-0001.html 