Don't worry, I don't think you are undermining anything. It's just a discussion thread.<br>If Yossi gets mod_cdr back up to speed we will happily accept it.<br><br>BTW, *hint*<br>You use the perl script to turn the current batch of xml files into a single file or array of sql stmts.<br>
then you pass that on to a single insert / update transaction.<br><br>Your realtime insert into the db idea will suffer the same slow effect of<br>single transaction per insert which is your bottleneck. But it will scale<br>
rather far before it becomes a problem. The bigger issue is the chicken or the <br>egg on what to do when the db is down. File system is the safest place to put the data<br>cos if it goes, so does you box and there is little chance of dispare apart from filling it up and<br>
you have the ability to reprocess the files again if the db dies. Single file per record solves<br>file locking problems and the xml data is so big anyway you would want to seggregate it.<br><br>Like I said I don't go on giving out hints all the time so that's the last from me for now. ;)<br>
<br><br><br><div class="gmail_quote">On Thu, Oct 30, 2008 at 3:27 PM, <a href="mailto:regs@kinetix.gr">regs@kinetix.gr</a> <span dir="ltr"><<a href="mailto:regs@kinetix.gr">regs@kinetix.gr</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Don't get me wrong. I am not trying to undermine the excellent work that<br>
has already been done on the event handling<br>
modules of FS.<br>
<br>
I myself am a big fun of modular constructs in order to achieve<br>
something complicated (e.g. the UNIX way).<br>
<br>
But there are some situations that we will always need some extra<br>
features in order to accomplish our goal.<br>
<br>
As I wrote before, my current system is using the cdr-push (from the<br>
gateway side) method to gather my cdrs.<br>
Many billing systems and many terminating providers are satisfied with<br>
that model alone. I am not. In the end of each hour<br>
I am gathering all the cdrs and checking them one by one against my<br>
database in order to verify that ALL of my cdrs were<br>
handled by my radius servers.<br>
<br>
In my future system (where a more batch-like mode is preferable) I am<br>
"forced" to use a cdr-pull method. My billing system will<br>
be responisble for "pulling" the cdrs from the gateways and then process<br>
them. So if we have a look at the event handling weaponry of FS the<br>
following modules cannot meet me needs :<br>
<br>
mod_event_multicast<br>
mod_event socket<br>
mod_radius_cdr<br>
mod_xmpp_event<br>
<br>
because they all rely on a mechanism where the gateway is pushing my<br>
cdrs to my billing system (and you need a checking mechanism to verify<br>
that all the cdrs were handled).<br>
<br>
So now, lets have a look at the alternatives :<br>
<br>
mod_xml_cdr : this was my first choice as it had all the info needed<br>
(even more). But the cdr per file approach proved out to be<br>
inadequate in terms of performance. I wrote a few lines of code (in<br>
perl) that did a listing of the directory, parsed the cdrs using<br>
XML::Simple (without even doing anything meaningful, e.g. checks,<br>
inserts to a database etc.) and I was not able to make my system<br>
parse more than a 30 cdrs/sec (reference 1000 USD average system, not<br>
drawing any conclusions, don't get me wrong).<br>
Also while the parser was running I had a high cpu utilization (as<br>
expected) that was competing with my FS service. Wouldn't that make<br>
a bad impact of my freeswitch performance?<br>
<br>
so I went to a different module<br>
<br>
mod_cdr_csv : the text .csv format with the plethora of attributes was<br>
perfect for my application but it would be a daunting task<br>
for me to make a script tha purges the Master csv while it was being<br>
used by FS. I googled the issue and found about the HUP method<br>
and the rotate directive in the config. So far so good. But it lacked<br>
some functionality.<br>
<br>
All I was trying to state in my previous messages was that it would be<br>
nice if the rotation was being initiated by FS, and maybe have different<br>
behavior depending on another directive or something. Let me explain<br>
what I mean.<br>
<br>
There are some people who don't care about what cdrs exist in a given<br>
rotated log file, as long as a) no cdrs get lost in the process of rotation<br>
, b) no duplicates exist. I can see that the HUP method satisfied this need.<br>
<br>
There is another group of people that DO care about what cdrs are in a<br>
rotated log file. E.g. : The file with the name<br>
Master.csv.2008-01-01-09-00-00<br>
would only contain cdrs that were terminated from "2008-01-01 09:00:00"<br>
to "2008-01-01 09:59:59" if an hourly setup was desirable or from<br>
"2008-01-01 09:00:00" to "2008-01-01 09:04:59" if a five-minutes setup<br>
was desirable. That need is not covered by the current HUP method<br>
because some cdrs might "escape" from the next to the previous file due<br>
to the fact that there is a delay between cron executing the HUP and FS<br>
doing<br>
the file log rotation.<br>
<br>
In simple words : I am getting my job done with FS they way it already<br>
is (no question about that). BUT, I (or someone with different needs)<br>
could do<br>
his/her job better if some minor features were present.<br>
<br>
I am not saying that you should should embrace modules that meet the<br>
average needs yet sacrifice flexibility. I am merely suggesting<br>
extending the flexibility<br>
of the already existing ones. Put some more lego tiles in your box set :)<br>
<div><div></div><div class="Wj3C7c"><br>
<br>
Michael Collins wrote:<br>
><br>
> /me sends Anthony's post to the printer to be laminated and framed… J<br>
><br>
> ------------------------------------------------------------------------<br>
><br>
> *From:* <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a><br>
> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a>] *On Behalf Of<br>
> *Anthony Minessale<br>
> *Sent:* Thursday, October 30, 2008 6:10 AM<br>
> *To:* <a href="mailto:freeswitch-users@lists.freeswitch.org">freeswitch-users@lists.freeswitch.org</a><br>
> *Subject:* Re: [Freeswitch-users] mod_cdr revival (or new module maybe)<br>
><br>
> Actually,<br>
><br>
> The goal is to not limit the functionality by over thinking how things<br>
> will be used but to provide building blocks to make much<br>
> more possible. Our analogy for this is that if you take the common<br>
> lego reference, If you have something cool built out of legos but they<br>
> are super glued together it limits what you *could* have done had then<br>
> not been.<br>
><br>
> The system that David described is indeed ideal. I have mentioned it<br>
> more than once and its no coincidence that FreeSWITCH plays into that<br>
> model to a tee. I do not mention it very often because I think<br>
> understanding that concept alone is valuable advice that I'd prefer<br>
> not to waste on deaf ears so I recommend you heed his suggestion.<br>
><br>
> Logging direct to DB is not a sin or anything but it's just not very<br>
> reliable. We have so many ways to process call records as it is.<br>
> you can use the cdr_csv if you want legacy support from asterisk that<br>
> you can eventually tweak with templatable output, you can even make a<br>
> template that generates the log as ready to roll sql statements you<br>
> can just cat into mysql.<br>
><br>
> We have event_socket + channel_hangup event which has all the same<br>
> info that the cdr_csv uses to generate the file.<br>
> We have the XML cdr that is a 3 dimensional view of the call with an<br>
> account of nearly everything that happend including<br>
> each application executed, branching records when the call is<br>
> transferred complete with a timestamp for each occurance.<br>
><br>
> --"However it would be great if freeswitch had the options for extra<br>
> functionality (auto log rotation, db cdrs etc)<br>
> so that it meets the peculiarities of every different project."<br>
><br>
> A single cron entry that does kill -HUP `cat freeswitch.pid` and you<br>
> have automatic rotation in unix.<br>
> In windows there are several cron equivs and instead of kill you can<br>
> use an xml-rpc script to call the FSAPI call "fsctl send_sighup"<br>
> also you can schedule it to happen automaticly with the FSAPI<br>
><br>
> sched_api @300 mygroup fsctl send_sighup<br>
><br>
> A script connected to event socket subscribed to channel_hangup can<br>
> act as your DB cdr engine and has the bonus that it could<br>
> connect to N boxes at once as well as be able to tell if the system<br>
> was in trouble by listening for system heartbeat events and or socket<br>
> disconnect.<br>
><br>
> Moral of the story, there is more already there than first appears.<br>
> I've always hated the M$ paper clip so I strive not to bring him back.<br>
> "It looks like you are trying to run a VoIP termination company! Shall<br>
> I assume how you want your call details?"<br>
><br>
><br>
> On Thu, Oct 30, 2008 at 7:43 AM, <a href="mailto:regs@kinetix.gr">regs@kinetix.gr</a><br>
</div></div><div><div></div><div class="Wj3C7c">> <mailto:<a href="mailto:regs@kinetix.gr">regs@kinetix.gr</a>> <<a href="mailto:regs@kinetix.gr">regs@kinetix.gr</a> <mailto:<a href="mailto:regs@kinetix.gr">regs@kinetix.gr</a>>> wrote:<br>
><br>
> I'll try some tests with various combinations first and then decide<br>
> what's best.<br>
><br>
> I have to say that I was astonished to find out that freeswitch had so<br>
> many event handlers from the very beginning.<br>
> However it would be great if freeswitch had the options for extra<br>
> functionality (auto log rotation, db cdrs etc)<br>
> so that it meets the peculiarities of every different project. That<br>
> would make a big difference compared to<br>
> other softswitch solutions where the lack of such features prohibits<br>
> people from using them (especially in<br>
> the carrier grade level). User feedback (and wish lists) is the key<br>
> for the success of an open source (or not)<br>
> project...<br>
><br>
> Thank you all for your replies. You' ve been very helpful.<br>
><br>
><br>
><br>
> David Knell wrote:<br>
><br>
> The sleep's done each time the directory's empty, not each time a<br>
> file's written. File open and close<br>
> are trivial (it's probably still cached), and the contents are going<br>
> to have to be parsed wherever you<br>
> process it.<br>
><br>
> We've used exactly this to process deliver CDRs on boxes handling in<br>
> excess of 500K mins/day<br>
> without issue. And, looking at one now, the CDR processor's used about<br>
> 4% of the CPU time<br>
> of FreeSWITCH, and about half that of the MySQL database which it<br>
> writes the records to, also<br>
> on the local machine, from which they're simply copied to the main CDR<br>
> processor.<br>
><br>
> It performance simply isn't worth worrying about.<br>
><br>
> --Dave<br>
><br>
> "sleep for a couple of seconds"<br>
><br>
> But then you could only insert 1800 cdrs per hour...<br>
> If I was to insert 36000 cdrs per hour this means that I have to<br>
> open<br>
> parse<br>
> close<br>
> 10 files per second. Imagine the I/O penalty just for opening - closing the file.<br>
> (the persing is the same for both situations)<br>
><br>
><br>
><br>
><br>
><br>
> David Knell wrote:<br>
><br>
</div></div><div><div></div><div class="Wj3C7c">> <a href="mailto:regs@kinetix.gr">regs@kinetix.gr</a> <mailto:<a href="mailto:regs@kinetix.gr">regs@kinetix.gr</a>> wrote:<br>
><br>
>> Yes, the xml files give you tons of info... but isn't it a little<br>
>> insufficient - performance wise -<br>
>> to open and close so many files in such a little time. In a PBX<br>
>> environment that wouldn't be an<br>
>> issue but if we get to the small-voip-carrier level (some thousand cdrs<br>
>> per hour)<br>
>> that could slow things down considerably, wouldn't it?<br>
>><br>
>><br>
> Not that you'd notice. We run XML CDR to database scripting on each box<br>
> that we use<br>
> for switching, and it's a pretty trivial task compared with switching<br>
> all that media. Doing it<br>
> this way is:-<br>
> (a) distributed - one process per box scales nicely;<br>
> (b) robust - script down, DB down, no problem: files just queue up;<br>
> (c) simple - the script logic is trivial:<br>
> - while 1<br>
> - for each file in the XML CDR directory<br>
> - open it<br>
> - parse it (XML::Simple for us)<br>
> - insert it in to the DB<br>
> - delete it<br>
> - sleep for a couple of seconds<br>
> Two error cases: can't parse or can't find data which should be there:<br>
> move the file in to<br>
> another directory to be examined by real eyes; DB insert fails: break<br>
> out of inner loop and<br>
> it'll be retried after a short pause.<br>
><br>
> --Dave<br>
><br>
><br>
><br>
><br>
><br>
> ------------------------------------------------------------------------<br>
><br>
><br>
><br>
> _______________________________________________<br>
> Freeswitch-users mailing list<br>
</div></div>> <a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a> <mailto:<a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a>><br>
<div class="Ih2E3d">> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
><br>
><br>
> --<br>
> David Knell, Director, 3C Limited<br>
> T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031<br>
> <a href="http://www.3c.co.uk" target="_blank">http://www.3c.co.uk</a><br>
><br>
><br>
> ------------------------------------------------------------------------<br>
><br>
><br>
><br>
> _______________________________________________<br>
> Freeswitch-users mailing list<br>
</div>> <a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a> <mailto:<a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a>><br>
<div class="Ih2E3d">> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Freeswitch-users mailing list<br>
> <a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a><br>
</div>> <mailto:<a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a>><br>
<div class="Ih2E3d">> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Anthony Minessale II<br>
><br>
> FreeSWITCH <a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org/</a><br>
> ClueCon <a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com/</a><br>
><br>
> AIM: anthm<br>
> <a href="mailto:MSN%3Aanthony_minessale@hotmail.com">MSN:anthony_minessale@hotmail.com</a><br>
</div>> <mailto:<a href="mailto:MSN%253Aanthony_minessale@hotmail.com">MSN%3Aanthony_minessale@hotmail.com</a>><br>
<div class="Ih2E3d">> GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com">PAYPAL:anthony.minessale@gmail.com</a><br>
</div>> <mailto:<a href="mailto:PAYPAL%253Aanthony.minessale@gmail.com">PAYPAL%3Aanthony.minessale@gmail.com</a>><br>
> IRC: <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> <<a href="http://irc.freenode.net" target="_blank">http://irc.freenode.net</a>> #freeswitch<br>
<div class="Ih2E3d">><br>
> FreeSWITCH Developer Conference<br>
> <a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a><br>
</div>> <mailto:<a href="mailto:sip%253A888@conference.freeswitch.org">sip%3A888@conference.freeswitch.org</a>><br>
<div class="Ih2E3d">> <a href="http://iax:guest@conference.freeswitch.org/888" target="_blank">iax:guest@conference.freeswitch.org/888</a><br>
</div>> <<a href="http://iax:guest@conference.freeswitch.org/888" target="_blank">http://iax:guest@conference.freeswitch.org/888</a>><br>
<div class="Ih2E3d">> <a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org">googletalk:conf+888@conference.freeswitch.org</a><br>
</div>> <mailto:<a href="mailto:googletalk%253Aconf%252B888@conference.freeswitch.org">googletalk%3Aconf%2B888@conference.freeswitch.org</a>><br>
> pstn:213-799-1400<br>
><br>
> ------------------------------------------------------------------------<br>
<div><div></div><div class="Wj3C7c">><br>
> _______________________________________________<br>
> Freeswitch-users mailing list<br>
> <a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
<br>
<br>
_______________________________________________<br>
Freeswitch-users mailing list<br>
<a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Anthony Minessale II<br><br>FreeSWITCH <a href="http://www.freeswitch.org/">http://www.freeswitch.org/</a><br>ClueCon <a href="http://www.cluecon.com/">http://www.cluecon.com/</a><br>
<br>AIM: anthm<br><a href="mailto:MSN%3Aanthony_minessale@hotmail.com">MSN:anthony_minessale@hotmail.com</a><br>GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net">irc.freenode.net</a> #freeswitch<br><br>FreeSWITCH Developer Conference<br><a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a><br><a href="http://iax:guest@conference.freeswitch.org/888">iax:guest@conference.freeswitch.org/888</a><br>
<a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org">googletalk:conf+888@conference.freeswitch.org</a><br>pstn:213-799-1400<br>