[Freeswitch-users] ReloadXML takes very long time

Fri Oct 26 06:05:01 MSD 2012

I appreciate your patch but it looks very specific to your needs.

I think you should stop and join the community and learn the philosophy
behind how things work and learn how to make a module.
You are breaking holes in the wall to get through because you are not
seeing the door.

I really think you should understand how the xml hooks or dialplan modules
work and how to actually scale things in FreeSWITCH.

Join IRC and call the weekly call and maybe study the wiki.

On Thu, Oct 25, 2012 at 7:04 PM, Yiftach Golan <yiftah at choochee.com> wrote:

> OK I loaded the code to :
> http://jira.freeswitch.org/browse/FS-4766
> I have to say that this version is much less tested than the one we had
> for FreeSWITCH1.0.7 but hopefully we can still gain from this improvement
>
> I have a different question on the same issue :
> Even with this fix it is very hard to put huge XMLs, the memory
> consumption is going crazy
> Did you think about adding MySQL support ?
> I thought on adding something like :
> <action application="bridge" data="MYSQL(SELECT bridge_data FROM users
> WHERE destination_number = ${destination_number})"/>
> This will allow us to still use the flexibility of the XMLs and the
> scalability of a database
> Do you know if someone did already?
>
> Thanks,
> Yiftach.
>
>
>
> On Thu, Oct 25, 2012 at 3:19 PM, Anthony Minessale <
> anthony.minessale at gmail.com> wrote:
>
>> Can you git diff it and post it to JIRA so we can have some people look
>> at it and test?
>>
>>
>> On Thu, Oct 25, 2012 at 5:11 PM, Yiftach Golan <yiftah at choochee.com>wrote:
>>
>>> It may be a bit problematic since I will need 3 hooks :
>>> 1. *switch_xml_parse_fd :* Create the initial pre allocated buffer
>>> 2. *preprocess :* indexing the file names (adding a note before and
>>> after each file )
>>> 3. *switch_xml_parse_file :*  use my reload instead the regular one for
>>> creating the big xml and then I just let the function continue as usual
>>> through the "__switch_xml_open_root" function
>>> I am reloading the file to the memory and then let the regular
>>> "switch_xml.c" do its work, so using the existing functions in
>>> "switch_xml.c" will not do the work
>>> However, we do not have to add it I just thought it would be good
>>> improvement for FreeSWITCH as it was very helpful for us
>>>
>>> On Thu, Oct 25, 2012 at 2:58 PM, Anthony Minessale <
>>> anthony.minessale at gmail.com> wrote:
>>>
>>>> I thought you were going to move it to be a module?
>>>>
>>>> On Thu, Oct 25, 2012 at 4:50 PM, Yiftach Golan <yiftah at choochee.com>wrote:
>>>>
>>>>> I merged my code to the HEAD here are the results (I yet checked in
>>>>> the code) :
>>>>> # time /opt/freeswitch/bin/fs_cli -x "reloadxml"
>>>>> +OK [Success]
>>>>> real    *0m24.424s*
>>>>> user    0m0.000s
>>>>> sys    0m0.020s
>>>>>
>>>>> # time /opt/freeswitch/bin/fs_cli -x "reloadxml
>>>>> /repository/generated_configs/tenant_dialplan/156792923823_tenant_dialplan.xml"
>>>>> +OK [Success]
>>>>> real    *0m0.315s*
>>>>> user    0m0.000s
>>>>> sys    0m0.010s
>>>>>
>>>>> Looks like there is major improvement in the loading time
>>>>> How does it work, I just need to check in my code to the git? with my
>>>>> subscribe user?
>>>>> Thanks,
>>>>> Yiftach.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 23, 2012 at 5:42 PM, Yiftach Golan <yiftah at choochee.com>wrote:
>>>>>
>>>>>> OK I will do so
>>>>>>
>>>>>> On Tue, Oct 23, 2012 at 5:41 PM, Anthony Minessale <
>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>
>>>>>>> Also, if you design it cleanly and generic enough you can check the
>>>>>>> module itself into HEAD and maintain it yourself and others can choose to
>>>>>>> use it.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 23, 2012 at 5:54 PM, Yiftach Golan <yiftah at choochee.com>wrote:
>>>>>>>
>>>>>>>> Yes I agree, I can solve it this way for me, if anyone else in the
>>>>>>>> community is looking for a very fast way of loading a specific file from
>>>>>>>> the "reloadxml" command I can update it to the HEAD
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 23, 2012 at 3:49 PM, Anthony Minessale <
>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> The man xml registry is already reloaded using the method
>>>>>>>>> described.  In fact the xml root has reference counting and everyone who
>>>>>>>>> was on the old one will be able to hold on to that root until they release
>>>>>>>>> it even when a new one has been loaded into place.
>>>>>>>>>
>>>>>>>>> I think if you pursue the custom C module with the XML hook, that
>>>>>>>>> you can put your same logic into a module and not bother the core as any
>>>>>>>>> access to the core xml will be redirected to your module callback where you
>>>>>>>>> can do whatever you want and not disturb the core of FS unless you have
>>>>>>>>> some kind of functionality that is useful to everyone in general.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Oct 23, 2012 at 4:32 PM, Yiftach Golan <
>>>>>>>>> yiftah at choochee.com> wrote:
>>>>>>>>>
>>>>>>>>>> You may be able to develop it as well but it will not solve our
>>>>>>>>>> problem of pushing configuration fast
>>>>>>>>>> We have a front end that pushes the configuration changes and it
>>>>>>>>>> needs to be fast, so even if we change it in the background it will be slow
>>>>>>>>>>
>>>>>>>>>> On Tue, Oct 23, 2012 at 2:03 PM, Puskás Zsolt <errotan at elder.hu>wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm not an expert ( correct me if I'm wrong ) but I suppose the
>>>>>>>>>>> xml config memory section is locked while the files are proccessed so other
>>>>>>>>>>> modules are waiting for it. Instead of loading just one file you could
>>>>>>>>>>> continue to load all the files into somekind of a new temporary config
>>>>>>>>>>> section without locking the main. Once the files are loaded and processed
>>>>>>>>>>> the temporary and main section could be swapped or the temporary is used
>>>>>>>>>>> and the other is deleted from memory. No idea how many time this could take
>>>>>>>>>>> to implement.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2012-10-23 02:56 keltezéssel, Yiftach Golan írta:
>>>>>>>>>>>
>>>>>>>>>>> Hi FreeSWITCH community,
>>>>>>>>>>> My name is Yiftah and I am working for ChooChee which is a
>>>>>>>>>>> hosted PBX solution company
>>>>>>>>>>> We have been using FreeSWITCH for almost 2 years and we added a
>>>>>>>>>>> lot of code changes to the core and to the different modules
>>>>>>>>>>> Lately we have made a decision to get more close to the
>>>>>>>>>>> community so that we can enjoy the bug fixes and the new feature from the
>>>>>>>>>>> community and we can contribute our knowledge and bug fixes
>>>>>>>>>>> I would please like to start with a very basic question :
>>>>>>>>>>> We are using dialplan xmls in order to load the configuration,
>>>>>>>>>>> we have a lot of users on each freeswitch (+4k), it took a very long time
>>>>>>>>>>> for the configuration to load (around 5 minutes) during this time no new
>>>>>>>>>>> calls could be served by FreeSWITCH
>>>>>>>>>>> In order to solve this problem we introduce reload xml that
>>>>>>>>>>> accepts file names and has the ability to load just one file name into the
>>>>>>>>>>> memory (we achieved 10ms loading time instead of 5 minutes)
>>>>>>>>>>> The problem is that we had to introduce a lot of code in order
>>>>>>>>>>> to solve this problem, has anyone else encounter this problem? we do not
>>>>>>>>>>> want to use mod_xml_curl as we want to take care bringing the file to the
>>>>>>>>>>> machine
>>>>>>>>>>> Another approach that we thought is that a database might be
>>>>>>>>>>> much better fit in here, because every xml update includes the entire
>>>>>>>>>>> tenant (business) and has the potential in harming other users in the same
>>>>>>>>>>> tenant, also DB has much higher capacities then XMLs, did anyone think of
>>>>>>>>>>> enabling MySQL as a directory of parameters instead of xml?
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Yiftach.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>> Professional FreeSWITCH Consulting Services:consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server
>>>>>>>>>>>
>>>>>>>>>>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>
>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Anthony Minessale II
>>>>>>>>>
>>>>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>>>>> ClueCon http://www.cluecon.com/
>>>>>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>>>>>
>>>>>>>>> AIM: anthm
>>>>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>>>> IRC: irc.freenode.net #freeswitch
>>>>>>>>>
>>>>>>>>> FreeSWITCH Developer Conference
>>>>>>>>> sip:888 at conference.freeswitch.org
>>>>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>>>> pstn:+19193869900
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> 
>>>>>>>> 
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://wiki.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Anthony Minessale II
>>>>>>>
>>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>>> ClueCon http://www.cluecon.com/
>>>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>>>
>>>>>>> AIM: anthm
>>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>> IRC: irc.freenode.net #freeswitch
>>>>>>>
>>>>>>> FreeSWITCH Developer Conference
>>>>>>> sip:888 at conference.freeswitch.org
>>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>> pstn:+19193869900
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> 
>>>>>>> 
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://wiki.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Anthony Minessale II
>>>>
>>>> FreeSWITCH http://www.freeswitch.org/
>>>> ClueCon http://www.cluecon.com/
>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>
>>>> AIM: anthm
>>>> MSN:anthony_minessale at hotmail.com
>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>> IRC: irc.freenode.net #freeswitch
>>>>
>>>> FreeSWITCH Developer Conference
>>>> sip:888 at conference.freeswitch.org
>>>> googletalk:conf+888 at conference.freeswitch.org
>>>> pstn:+19193869900
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>>
>> --
>> Anthony Minessale II
>>
>> FreeSWITCH http://www.freeswitch.org/
>> ClueCon http://www.cluecon.com/
>> Twitter: http://twitter.com/FreeSWITCH_wire
>>
>> AIM: anthm
>> MSN:anthony_minessale at hotmail.com
>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>> IRC: irc.freenode.net #freeswitch
>>
>> FreeSWITCH Developer Conference
>> sip:888 at conference.freeswitch.org
>> googletalk:conf+888 at conference.freeswitch.org
>> pstn:+19193869900
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>

-- 
Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
googletalk:conf+888 at conference.freeswitch.org
pstn:+19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20121025/30fb01b0/attachment-0001.html 