That is why I keep mentioning the XML hooks. This is exactly what it's for. When you have an xml hook every single request to the xml document can be hooked into and your custom callback function is called with every search param allowing you to dynamically find the right xml and return it on-the-fly. This way you don't need to manage a giant root of XML for user data.
Try looking at it from further back. You can even make an xml hook module that uses a database directly and builds binary xml structures and returns them without them ever having to exist as text. With combinations of the C interfaces like dialplan app, dialplan module, xml_hooks and the like you can scale to any size.
Anyway, I do understand the point of the xml patch, I am very concerned about the stability of the xml parser with that patch you made applied. The entire design of FreeSWITCH is based around allowing you to extend it through modules and hooks. If you can find a hook that is missing that is a much more compelling case than modifying the core for multi-home configurations which is not a primary purpose of the software.
I don't know if you even care about my advice but I assume since you came here asking for it that you want me to continue to help you understand the right direction to take.
On Fri, Oct 26, 2012 at 12:23 AM, Yiftach Golan <yiftah@choochee.com> wrote:
Yes it is kind specific to our needs, which is very fast loading of specific XML in big environments We have some experience with FreeSWITCH modules, we wrote the Voicemail module from scratch after the existing one did not fit our need
I think on the FreeSWITCH clustering we were really struggling and this is were the scaling became very challenging for us Do you have any good document or reference on good practice to cluster FreeSWITCH, specifically cluster of FreeSWITCHes that can hold tenth of thousands of users in a single cluster?
OK I loaded the code to : http://jira.freeswitch.org/browse/FS-4766
I have to say that this version is much less tested than the one we had for FreeSWITCH1.0.7 but hopefully we can still gain from this improvement
I have a different question on the same issue : Even with this fix it is very hard to put huge XMLs, the memory consumption is going crazy Did you think about adding MySQL support ? I thought on adding something like :
<action application="bridge" data="MYSQL(SELECT bridge_data FROM users WHERE destination_number = ${destination_number})"/> This will allow us to still use the flexibility of the XMLs and the scalability of a database
Do you know if someone did already?
It may be a bit problematic since I will need 3 hooks : 1. switch_xml_parse_fd : Create the initial pre allocated buffer
2. preprocess : indexing the file names (adding a note before and after each file )
3. switch_xml_parse_file : use my reload instead the regular one for creating the big xml and then I just let the function continue as usual through the "__switch_xml_open_root" function I am reloading the file to the memory and then let the regular "switch_xml.c" do its work, so using the existing functions in "switch_xml.c" will not do the work
However, we do not have to add it I just thought it would be good improvement for FreeSWITCH as it was very helpful for us
I merged my code to the HEAD here are the results (I yet checked in the code) : # time /opt/freeswitch/bin/fs_cli -x "reloadxml"
+OK [Success] real 0m24.424s user 0m0.000s sys 0m0.020s
# time /opt/freeswitch/bin/fs_cli -x "reloadxml /repository/generated_configs/tenant_dialplan/156792923823_tenant_dialplan.xml" +OK [Success] real 0m0.315s user 0m0.000s sys 0m0.010s
Looks like there is major improvement in the loading time How does it work, I just need to check in my code to the git? with my subscribe user? Thanks, Yiftach.
Yes I agree, I can solve it this way for me, if anyone else in the community is looking for a very fast way of loading a specific file from the "reloadxml" command I can update it to the HEAD
The man xml registry is already reloaded using the method described. In fact the xml root has reference counting and everyone who was on the old one will be able to hold on to that root until they release it even when a new one has been loaded into place.
I think if you pursue the custom C module with the XML hook, that you can put your same logic into a module and not bother the core as any access to the core xml will be redirected to your module callback where you can do whatever you want and not disturb the core of FS unless you have some kind of functionality that is useful to everyone in general.
You may be able to develop it as well but it will not solve our problem of pushing configuration fast We have a front end that pushes the configuration changes and it needs to be fast, so even if we change it in the background it will be slow
On Tue, Oct 23, 2012 at 2:03 PM, Puskás Zsolt <errotan@elder.hu> wrote:
I'm not an expert ( correct me if I'm wrong ) but I suppose the
xml config memory section is locked while the files are proccessed
so other modules are waiting for it. Instead of loading just one
file you could continue to load all the files into somekind of a
new temporary config section without locking the main. Once the
files are loaded and processed the temporary and main section
could be swapped or the temporary is used and the other is deleted
from memory. No idea how many time this could take to implement.
2012-10-23 02:56 keltezéssel, Yiftach Golan írta:
Hi FreeSWITCH community,
My name is Yiftah and I am working for ChooChee which is a hosted
PBX solution company
We have been using FreeSWITCH for almost 2 years and we added a
lot of code changes to the core and to the different modules
Lately we have made a decision to get more close to the community
so that we can enjoy the bug fixes and the new feature from the
community and we can contribute our knowledge and bug fixes
I would please like to start with a very basic question :
We are using dialplan xmls in order to load the configuration, we
have a lot of users on each freeswitch (+4k), it took a very long
time for the configuration to load (around 5 minutes) during this
time no new calls could be served by FreeSWITCH
In order to solve this problem we introduce reload xml that
accepts file names and has the ability to load just one file name
into the memory (we achieved 10ms loading time instead of 5
minutes)
The problem is that we had to introduce a lot of code in order to
solve this problem, has anyone else encounter this problem? we do
not want to use mod_xml_curl as we want to take care bringing the
file to the machine
Another approach that we thought is that a database might be much
better fit in here, because every xml update includes the entire
tenant (business) and has the potential in harming other users in
the same tenant, also DB has much higher capacities then XMLs, did
anyone think of enabling MySQL as a directory of parameters
instead of xml?
Thanks,
Yiftach.