[Freeswitch-users] ReloadXML takes very long time

Fri Oct 26 04:04:16 MSD 2012

OK I loaded the code to :
http://jira.freeswitch.org/browse/FS-4766
I have to say that this version is much less tested than the one we had for
FreeSWITCH1.0.7 but hopefully we can still gain from this improvement

I have a different question on the same issue :
Even with this fix it is very hard to put huge XMLs, the memory consumption
is going crazy
Did you think about adding MySQL support ?
I thought on adding something like :
<action application="bridge" data="MYSQL(SELECT bridge_data FROM users
WHERE destination_number = ${destination_number})"/>
This will allow us to still use the flexibility of the XMLs and the
scalability of a database
Do you know if someone did already?

Thanks,
Yiftach.

On Thu, Oct 25, 2012 at 3:19 PM, Anthony Minessale <
anthony.minessale at gmail.com> wrote:

> Can you git diff it and post it to JIRA so we can have some people look at
> it and test?
>
>
> On Thu, Oct 25, 2012 at 5:11 PM, Yiftach Golan <yiftah at choochee.com>wrote:
>
>> It may be a bit problematic since I will need 3 hooks :
>> 1. *switch_xml_parse_fd :* Create the initial pre allocated buffer
>> 2. *preprocess :* indexing the file names (adding a note before and
>> after each file )
>> 3. *switch_xml_parse_file :*  use my reload instead the regular one for
>> creating the big xml and then I just let the function continue as usual
>> through the "__switch_xml_open_root" function
>> I am reloading the file to the memory and then let the regular
>> "switch_xml.c" do its work, so using the existing functions in
>> "switch_xml.c" will not do the work
>> However, we do not have to add it I just thought it would be good
>> improvement for FreeSWITCH as it was very helpful for us
>>
>> On Thu, Oct 25, 2012 at 2:58 PM, Anthony Minessale <
>> anthony.minessale at gmail.com> wrote:
>>
>>> I thought you were going to move it to be a module?
>>>
>>> On Thu, Oct 25, 2012 at 4:50 PM, Yiftach Golan <yiftah at choochee.com>wrote:
>>>
>>>> I merged my code to the HEAD here are the results (I yet checked in the
>>>> code) :
>>>> # time /opt/freeswitch/bin/fs_cli -x "reloadxml"
>>>> +OK [Success]
>>>> real    *0m24.424s*
>>>> user    0m0.000s
>>>> sys    0m0.020s
>>>>
>>>> # time /opt/freeswitch/bin/fs_cli -x "reloadxml
>>>> /repository/generated_configs/tenant_dialplan/156792923823_tenant_dialplan.xml"
>>>> +OK [Success]
>>>> real    *0m0.315s*
>>>> user    0m0.000s
>>>> sys    0m0.010s
>>>>
>>>> Looks like there is major improvement in the loading time
>>>> How does it work, I just need to check in my code to the git? with my
>>>> subscribe user?
>>>> Thanks,
>>>> Yiftach.
>>>>
>>>>
>>>>
>>>> On Tue, Oct 23, 2012 at 5:42 PM, Yiftach Golan <yiftah at choochee.com>wrote:
>>>>
>>>>> OK I will do so
>>>>>
>>>>> On Tue, Oct 23, 2012 at 5:41 PM, Anthony Minessale <
>>>>> anthony.minessale at gmail.com> wrote:
>>>>>
>>>>>> Also, if you design it cleanly and generic enough you can check the
>>>>>> module itself into HEAD and maintain it yourself and others can choose to
>>>>>> use it.
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 23, 2012 at 5:54 PM, Yiftach Golan <yiftah at choochee.com>wrote:
>>>>>>
>>>>>>> Yes I agree, I can solve it this way for me, if anyone else in the
>>>>>>> community is looking for a very fast way of loading a specific file from
>>>>>>> the "reloadxml" command I can update it to the HEAD
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 23, 2012 at 3:49 PM, Anthony Minessale <
>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>
>>>>>>>> The man xml registry is already reloaded using the method
>>>>>>>> described.  In fact the xml root has reference counting and everyone who
>>>>>>>> was on the old one will be able to hold on to that root until they release
>>>>>>>> it even when a new one has been loaded into place.
>>>>>>>>
>>>>>>>> I think if you pursue the custom C module with the XML hook, that
>>>>>>>> you can put your same logic into a module and not bother the core as any
>>>>>>>> access to the core xml will be redirected to your module callback where you
>>>>>>>> can do whatever you want and not disturb the core of FS unless you have
>>>>>>>> some kind of functionality that is useful to everyone in general.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 23, 2012 at 4:32 PM, Yiftach Golan <yiftah at choochee.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> You may be able to develop it as well but it will not solve our
>>>>>>>>> problem of pushing configuration fast
>>>>>>>>> We have a front end that pushes the configuration changes and it
>>>>>>>>> needs to be fast, so even if we change it in the background it will be slow
>>>>>>>>>
>>>>>>>>> On Tue, Oct 23, 2012 at 2:03 PM, Puskás Zsolt <errotan at elder.hu>wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm not an expert ( correct me if I'm wrong ) but I suppose the
>>>>>>>>>> xml config memory section is locked while the files are proccessed so other
>>>>>>>>>> modules are waiting for it. Instead of loading just one file you could
>>>>>>>>>> continue to load all the files into somekind of a new temporary config
>>>>>>>>>> section without locking the main. Once the files are loaded and processed
>>>>>>>>>> the temporary and main section could be swapped or the temporary is used
>>>>>>>>>> and the other is deleted from memory. No idea how many time this could take
>>>>>>>>>> to implement.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2012-10-23 02:56 keltezéssel, Yiftach Golan írta:
>>>>>>>>>>
>>>>>>>>>> Hi FreeSWITCH community,
>>>>>>>>>> My name is Yiftah and I am working for ChooChee which is a hosted
>>>>>>>>>> PBX solution company
>>>>>>>>>> We have been using FreeSWITCH for almost 2 years and we added a
>>>>>>>>>> lot of code changes to the core and to the different modules
>>>>>>>>>> Lately we have made a decision to get more close to the community
>>>>>>>>>> so that we can enjoy the bug fixes and the new feature from the community
>>>>>>>>>> and we can contribute our knowledge and bug fixes
>>>>>>>>>> I would please like to start with a very basic question :
>>>>>>>>>> We are using dialplan xmls in order to load the configuration, we
>>>>>>>>>> have a lot of users on each freeswitch (+4k), it took a very long time for
>>>>>>>>>> the configuration to load (around 5 minutes) during this time no new calls
>>>>>>>>>> could be served by FreeSWITCH
>>>>>>>>>> In order to solve this problem we introduce reload xml that
>>>>>>>>>> accepts file names and has the ability to load just one file name into the
>>>>>>>>>> memory (we achieved 10ms loading time instead of 5 minutes)
>>>>>>>>>> The problem is that we had to introduce a lot of code in order to
>>>>>>>>>> solve this problem, has anyone else encounter this problem? we do not want
>>>>>>>>>> to use mod_xml_curl as we want to take care bringing the file to the
>>>>>>>>>> machine
>>>>>>>>>> Another approach that we thought is that a database might be much
>>>>>>>>>> better fit in here, because every xml update includes the entire tenant
>>>>>>>>>> (business) and has the potential in harming other users in the same tenant,
>>>>>>>>>> also DB has much higher capacities then XMLs, did anyone think of enabling
>>>>>>>>>> MySQL as a directory of parameters instead of xml?
>>>>>>>>>> Thanks,
>>>>>>>>>> Yiftach.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Anthony Minessale II
>>>>>>>>
>>>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>>>> ClueCon http://www.cluecon.com/
>>>>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>>>>
>>>>>>>> AIM: anthm
>>>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>>> IRC: irc.freenode.net #freeswitch
>>>>>>>>
>>>>>>>> FreeSWITCH Developer Conference
>>>>>>>> sip:888 at conference.freeswitch.org
>>>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>>> pstn:+19193869900
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> 
>>>>>>>> 
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://wiki.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> 
>>>>>>> 
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://wiki.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Anthony Minessale II
>>>>>>
>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>> ClueCon http://www.cluecon.com/
>>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>>
>>>>>> AIM: anthm
>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>> IRC: irc.freenode.net #freeswitch
>>>>>>
>>>>>> FreeSWITCH Developer Conference
>>>>>> sip:888 at conference.freeswitch.org
>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>> pstn:+19193869900
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> 
>>>>>> 
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Anthony Minessale II
>>>
>>> FreeSWITCH http://www.freeswitch.org/
>>> ClueCon http://www.cluecon.com/
>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>
>>> AIM: anthm
>>> MSN:anthony_minessale at hotmail.com
>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>> IRC: irc.freenode.net #freeswitch
>>>
>>> FreeSWITCH Developer Conference
>>> sip:888 at conference.freeswitch.org
>>> googletalk:conf+888 at conference.freeswitch.org
>>> pstn:+19193869900
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> IRC: irc.freenode.net #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> googletalk:conf+888 at conference.freeswitch.org
> pstn:+19193869900
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20121025/743c3ca5/attachment-0001.html 