Hi everyone, this is my first post. I'd like to know if there is any plan for an HA solution for FreeSWITCH. I know the 'heartbeat' trick, but is there or will there be any way to mirror the configuration changes and even call state between two boxes in a sort of clustered arrangement? Thanks, Tom
Tom Warren wrote:
I'd like to know if there is any plan for an HA solution for FreeSWITCH. I know the 'heartbeat' trick, but is there or will there be any way to mirror the configuration changes and even call state between two boxes in a sort of clustered arrangement?
I don't think there's anything out of the box. I would imagine, much as you can do with web servers, some form of R-R DNS exposing some proxies that connect to 2+ FreeSWITCH/other boxen would essentially do the trick. The tricky part would be getting any new machines to announce themselves to the proxies and getting the proxies to remove machines now out of service. I'll be looking into that later... James
Hi James, I'd like to avoid front-ending the FS boxes and instead have them load-share amongst themselves or sit in a hot/standby arrangement while updating the other box(es) with call state and configuration changes. They would be connected directly via a closed tertiary network so that the heartbeat could be set to a very low value, and call state data would not traverse the same path as the call traffic. Am I just dreaming? Could a standby box take over call in progress when the primary fails? Thanks, Tom On Tue, Aug 26, 2008 at 9:25 PM, James Green <james.green@stealthnet.net> wrote:
Tom Warren wrote:
I'd like to know if there is any plan for an HA solution for FreeSWITCH. I know the 'heartbeat' trick, but is there or will there be any way to mirror the configuration changes and even call state between two boxes in a sort of clustered arrangement?
I don't think there's anything out of the box. I would imagine, much as you can do with web servers, some form of R-R DNS exposing some proxies that connect to 2+ FreeSWITCH/other boxen would essentially do the trick.
The tricky part would be getting any new machines to announce themselves to the proxies and getting the proxies to remove machines now out of service. I'll be looking into that later...
James
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Tom, Configuration: that should be easy using XML CURL to load what would otherwise be in the conf/ XML files. I would strongly recommend looking at this. Call state I wouldn't like to comment on, I have not looked into it. I would imagine it's unlikely to succeed based on the uuid of each call only being known to the handler FreeSWITCH instance. I could have my assumptions wrong of course. I take it incoming calls would be asked to try each of several IPs in turn until a connection is successful? Else you would need to "notice" one instance was no longer active and ask whomever was routing calls to it to divert. I still prefer the idea of 2+ proxy boxes. It would be simpler to add more as load increases, and far simpler to add additional IVRs (FS) into a backend pool. Heartbeat solutions have potential issues such as single point of failure and split brain. James Tom Warren wrote:
Hi James,
I'd like to avoid front-ending the FS boxes and instead have them load-share amongst themselves or sit in a hot/standby arrangement while updating the other box(es) with call state and configuration changes. They would be connected directly via a closed tertiary network so that the heartbeat could be set to a very low value, and call state data would not traverse the same path as the call traffic. Am I just dreaming? Could a standby box take over call in progress when the primary fails?
Hi, Regarding HA, I don't suppose you'll be able to go without any front-end, beacuse of IP-connectivity reasons, if you plan to work in active-standby mode . Active box needs to have an unique IP, otherwise each client will have to have 2 (or more) IP addresses configured. As solution you can use a switch with ip routing based on heartbeat (which can be a icmp ping or http get or others). The idea is you configure virtual ip on each of your boxes, and FS runs on this IP. Boxes are only connected to your IP network through the switch. the switch directs all traffic to only one of the boxes, checking the heartbeat all the time. If one machine failes, trafic starts being routed to the other box. With this solution you don't have to do any tricks to have a single IP address for your clients. there are of course other solutions, this one gives you the shortest switch time though. As for the transferring call states between the machines, much of call data is distibuted into modules. Now if we wanted to keep consistent states between two FS instances, we'd have to come up with some interfaces to firstly serialize the states and then 'revoke' them on second machine. No that is of course doable, but I'd require enhacing internal FS interfaces (especially those to endpoint modules) so that a call controls in not-initial states could be created. That is a huge amount of work, I don't suppose it's worth it. br Ł 2008/8/27 James Green <james.green@stealthnet.net>:
Tom,
Configuration: that should be easy using XML CURL to load what would otherwise be in the conf/ XML files. I would strongly recommend looking at this.
Call state I wouldn't like to comment on, I have not looked into it. I would imagine it's unlikely to succeed based on the uuid of each call only being known to the handler FreeSWITCH instance. I could have my assumptions wrong of course.
I take it incoming calls would be asked to try each of several IPs in turn until a connection is successful? Else you would need to "notice" one instance was no longer active and ask whomever was routing calls to it to divert.
I still prefer the idea of 2+ proxy boxes. It would be simpler to add more as load increases, and far simpler to add additional IVRs (FS) into a backend pool. Heartbeat solutions have potential issues such as single point of failure and split brain.
James
Tom Warren wrote:
Hi James,
I'd like to avoid front-ending the FS boxes and instead have them load-share amongst themselves or sit in a hot/standby arrangement while updating the other box(es) with call state and configuration changes. They would be connected directly via a closed tertiary network so that the heartbeat could be set to a very low value, and call state data would not traverse the same path as the call traffic. Am I just dreaming? Could a standby box take over call in progress when the primary fails?
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Łukasz Zwierko wrote:
Hi, Regarding HA, I don't suppose you'll be able to go without any front-end, beacuse of IP-connectivity reasons, if you plan to work in active-standby mode . Active box needs to have an unique IP, otherwise each client will have to have 2 (or more) IP addresses configured. As solution you can use a switch with ip routing based on heartbeat (which can be a icmp ping or http get or others). The idea is you configure virtual ip on each of your boxes, and FS runs on this IP. Boxes are only connected to your IP network through the switch. the switch directs all traffic to only one of the boxes, checking the heartbeat all the time. If one machine failes, trafic starts being routed to the other box.
And introduces a single point of failure which defeats the objective. Having TWO IP addresses to connect to allows software to simply try again on the other address. It also reduces the burden of administration as your sysadmin can simply remove a dodgy machine our of the list. Of course, the added bonus of a proxy is that it needs few configuration updates and usually less software to be installed; can also act as a firewall too...
With this solution you don't have to do any tricks to have a single IP address for your clients. there are of course other solutions, this one gives you the shortest switch time though.
OSPF.
As for the transferring call states between the machines, much of call data is distibuted into modules. Now if we wanted to keep consistent states between two FS instances, we'd have to come up with some interfaces to firstly serialize the states and then 'revoke' them on second machine. No that is of course doable, but I'd require enhacing internal FS interfaces (especially those to endpoint modules) so that a call controls in not-initial states could be created. That is a huge amount of work, I don't suppose it's worth it.
Indeed, particularly when working with inconsistent states following split brain unlocking. James
W dniu 27 sierpnia 2008 13:48 użytkownik James Green <james.green@stealthnet.net> napisał:
Łukasz Zwierko wrote:
Hi, Regarding HA, I don't suppose you'll be able to go without any front-end, beacuse of IP-connectivity reasons, if you plan to work in active-standby mode . Active box needs to have an unique IP, otherwise each client will have to have 2 (or more) IP addresses configured. As solution you can use a switch with ip routing based on heartbeat (which can be a icmp ping or http get or others). The idea is you configure virtual ip on each of your boxes, and FS runs on this IP. Boxes are only connected to your IP network through the switch. the switch directs all traffic to only one of the boxes, checking the heartbeat all the time. If one machine failes, trafic starts being routed to the other box.
And introduces a single point of failure which defeats the objective.
Having TWO IP addresses to connect to allows software to simply try again on the other address. It also reduces the burden of administration as your sysadmin can simply remove a dodgy machine our of the list. Of course, the added bonus of a proxy is that it needs few configuration updates and usually less software to be installed; can also act as a firewall too...
With this solution you don't have to do any tricks to have a single IP address for your clients. there are of course other solutions, this one gives you the shortest switch time though.
OSPF.
OSPF what? What I am talking about was to have TWO FS machines runnig in active- standby mode, without having this configuration displayed to users. Simplest solution for that is having the same virtual IP on both boxes, and directing the traffic to only one of them. OSPF can help with that, still it's not all that is to it. you're going to need heartbeat info, not just link state. I admit the 2 boxes-on one switch is not the best idea, still some means of failure detections, triggering routing changes are vital.
As for the transferring call states between the machines, much of call data is distibuted into modules. Now if we wanted to keep consistent states between two FS instances, we'd have to come up with some interfaces to firstly serialize the states and then 'revoke' them on second machine. No that is of course doable, but I'd require enhacing internal FS interfaces (especially those to endpoint modules) so that a call controls in not-initial states could be created. That is a huge amount of work, I don't suppose it's worth it.
Indeed, particularly when working with inconsistent states following split brain unlocking.
James
In the solution that i've seen, only session's in stable state remained active after active-standby switch. Ł
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Łukasz Zwierko wrote:
OSPF what? What I am talking about was to have TWO FS machines runnig in active- standby mode, without having this configuration displayed to users. Simplest solution for that is having the same virtual IP on both boxes, and directing the traffic to only one of them. OSPF can help with that, still it's not all that is to it. you're going to need heartbeat info, not just link state. I admit the 2 boxes-on one switch is not the best idea, still some means of failure detections, triggering routing changes are vital.
A software monitor on each host could detect faults and ifdown the interface, removing that host from the available cluster. That said, there is no provision to "activate" the new active host in this scenario for the purposes of catching any state information. The new active host would simply start seeing traffic on the previously [near] idle interface. Less than ideal, again.
In the solution that i've seen, only session's in stable state remained active after active-standby switch.
You have managed to get to this point? Be interested in any details. It might be nice to have a mechanism to serialise and database module states, however it may prove fragile. James
W dniu 27 sierpnia 2008 14:46 użytkownik James Green <james.green@stealthnet.net> napisał:
Łukasz Zwierko wrote:
OSPF what? What I am talking about was to have TWO FS machines runnig in active- standby mode, without having this configuration displayed to users. Simplest solution for that is having the same virtual IP on both boxes, and directing the traffic to only one of them. OSPF can help with that, still it's not all that is to it. you're going to need heartbeat info, not just link state. I admit the 2 boxes-on one switch is not the best idea, still some means of failure detections, triggering routing changes are vital.
A software monitor on each host could detect faults and ifdown the interface, removing that host from the available cluster.
External monitoring would be better, system might hang up, interface could remain up.
That said, there is no provision to "activate" the new active host in this scenario for the purposes of catching any state information. The new active host would simply start seeing traffic on the previously [near] idle interface. Less than ideal, again.
Right you are. In my opinion the state would have to be updated at all time. It can not be assumed that the host will have time/means to transfer anything to new-active host when the crash begins.
In the solution that i've seen, only session's in stable state remained active after active-standby switch.
You have managed to get to this point? Be interested in any details.
I've seen a solution similiar to the one we talking about. Two hosts were connected directly, and the active, updated the secondary all the time, with sessions of active users (it was not voip solutions, it was for GPRS traffic inspection). It was a proprietary solution - Bytemobile MSSP. It a switchover occured (for any reasons), unstable sessions got lost, active sessions remained active. The IP sharing worked a bit differently than I described, as the machines exchanged heartbeat, were not controlled by external monitoring.
It might be nice to have a mechanism to serialize and database module states, however it may prove fragile.
It's hard but doable. More problem would be with restoring call states in standby FS (including voice transcoded connections) . And what about calls being served by external scripts? That's a real challenge! Ł
James
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Łukasz Zwierko wrote:
External monitoring would be better, system might hang up, interface could remain up.
Wherein lay an additional problem. Should the system be under severe load, the "system" may appear to hang, indeed become unreachable temporarily. Do you remove the host or check for a flapping state? I think it best to say that architecturally there are lots of considerations!
In my opinion the state would have to be updated at all time. It can not be assumed that the host will have time/means to transfer anything to new-active host when the crash begins.
Indeed, but I would prefer to know that the software developers had designed their modules / core with syncronisation in mind rather than as an afterthought.
I've seen a solution similiar to the one we talking about. Two hosts were connected directly, and the active, updated the secondary all the time, with sessions of active users (it was not voip solutions, it was for GPRS traffic inspection). It was a proprietary solution - Bytemobile MSSP. It a switchover occured (for any reasons), unstable sessions got lost, active sessions remained active. The IP sharing worked a bit differently than I described, as the machines exchanged heartbeat, were not controlled by external monitoring.
There comes a time where the degree of reliability needs to be weighed against commercial requirements. Unstable sessions being transferred in my opinion is in the "not worth it" category unless you have commercial reasons to spend development time on testing it. Banks have cash (well, they did...) open source tends to rely on goodwill :-)
It's hard but doable. More problem would be with restoring call states in standby FS (including voice transcoded connections) . And what about calls being served by external scripts? That's a real challenge!
Walk before you can run! I do wonder if there is scope for thought given FS can act as a proxy itself... James
Wow, this has turned into quite a debate. I think HA could be the killer app that catapults FS into the mainstream. Even if not full implemented today, laying the groundwork to allow for call state synchronization with a standby box would be a great leap ahead of any other platform I'm aware of. Although not fully analogous, if you want to see an example of state mirroring look at Cisco SNAT. This synchronizes NAT entries to a standby router so that IP conversations are not reset when switchover occurs. Imagine running just two boxes and having a fully redundant and mirrored configuration where when the primary box fails, the caller hears only a momentary 'hiccup' then conversation continues on.... Tom 2008/8/27 James Green <james.green@stealthnet.net>:
Łukasz Zwierko wrote:
External monitoring would be better, system might hang up, interface could remain up.
Wherein lay an additional problem. Should the system be under severe load, the "system" may appear to hang, indeed become unreachable temporarily. Do you remove the host or check for a flapping state? I think it best to say that architecturally there are lots of considerations!
In my opinion the state would have to be updated at all time. It can not be assumed that the host will have time/means to transfer anything to new-active host when the crash begins.
Indeed, but I would prefer to know that the software developers had designed their modules / core with syncronisation in mind rather than as an afterthought.
I've seen a solution similiar to the one we talking about. Two hosts were connected directly, and the active, updated the secondary all the time, with sessions of active users (it was not voip solutions, it was for GPRS traffic inspection). It was a proprietary solution - Bytemobile MSSP. It a switchover occured (for any reasons), unstable sessions got lost, active sessions remained active. The IP sharing worked a bit differently than I described, as the machines exchanged heartbeat, were not controlled by external monitoring.
There comes a time where the degree of reliability needs to be weighed against commercial requirements. Unstable sessions being transferred in my opinion is in the "not worth it" category unless you have commercial reasons to spend development time on testing it. Banks have cash (well, they did...) open source tends to rely on goodwill :-)
It's hard but doable. More problem would be with restoring call states in standby FS (including voice transcoded connections) . And what about calls being served by external scripts? That's a real challenge!
Walk before you can run! I do wonder if there is scope for thought given FS can act as a proxy itself...
James
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Wow, this has turned into quite a debate.
I think HA could be the killer app that catapults FS into the mainstream. Even if not full implemented today, laying the groundwork to allow for call state synchronization with a standby box would be a great leap ahead of any other platform I'm aware of.
Although not fully analogous, if you want to see an example of state mirroring look at Cisco SNAT. This synchronizes NAT entries to a standby router so that IP conversations are not reset when switchover occurs.
Imagine running just two boxes and having a fully redundant and mirrored configuration where when the primary box fails, the caller hears only a momentary 'hiccup' then conversation continues on....
Tom
AFAIK all commercial B2BUA/Proxy/SBCs all have the ability to share/exchange session/call states in real-time such that where implemented as active/passive configuration, current states are viewable on the passive member machine, although only few have the ability to influence and/or Mod/Add/Chg these sessions from the passive member. These states are usually maintained in a DB which in some cases is RAMDISK resident. In our proprietary solution, we have a web-based GUI to manage the 'cluster' where a change made in any one member (we allow 4 total members, where 1 is a master and 3 backups (Geographically remote, if needed)) propagates the changes in the DBs of all members including that of the active/master. Hence real-time control and modifications are possible. We do this over a proprietary but very simple TCP based protocol where this information is exchanged beteween the members. Other solutions out there e.g. Nextone uses the technique as was mentioned earlier in one of the posts, where a Virtual IP is configured for every interface as a virtual/logical interface which is what is configured in all UAs/Clients to connect to. A dedicated network port for a heartbeat type mechanism tells the other members if they're 'still there'. If not, rapid failover occurs where theVirtual IP is transferred to the passive member via a gratuitous ARP to the upstream switch (for people worried about the switch being a point of failure, we have configured ours with EtherChannel or Trunking so as to allow tables to be shared by both switches) upon which all requests are now sent to the new Master. Mind you, all of this happens within msecs and although it's never happened in a Live scenario, during testing, none of the calls were dropped when the switch/failover occurred other than a brief period of silence (enough for someone to say 'hello?') and boom the call's back on. I know nothing about the architecture of FS or I'd be able to make more qualified comments but clearly it's not an impossible task or 'a first' but seeing the limited number of developers working on FS, it might be a little while before it happens here, but I have no clue what the current state is even. I'm sure with people like Anthony's calibre around, it might not be that big a deal :) Cheers, \RR
Service Provider Self healing ha-cluster can and are being manage with the following tools: 1- NGINX or POUND or PERLBAL or HAProxy for LBS 2- WackaMole/Spread ..... For DNS / FOS 3- PUPPET / RPATH...... Rapid System Build 4- GroundWork/Hyperic for Network Management/System Management/Apps 5- XMPP ... no need for explanation 6- Kayako .... eTickets -E http://gpro.ws
participants (5)
-
EdPimentl
-
James Green
-
RR
-
Tom Warren
-
Łukasz Zwierko