[Freeswitch-users] mod_cdr revival (or new module maybe)

Anthony Minessale anthony.minessale at gmail.com
Thu Oct 30 16:53:23 PDT 2008


Don't worry, I don't think you are undermining anything.  It's just a
discussion thread.
If Yossi gets mod_cdr back up to speed we will happily accept it.

BTW, *hint*
You use the perl script to turn the current batch of xml files into a single
file or array of sql stmts.
then you pass that on to a single insert / update transaction.

Your realtime insert into the db idea will suffer the same slow effect of
single transaction per insert which is your bottleneck. But it will scale
rather far before it becomes a problem.  The bigger issue is the chicken or
the
egg on what to do when the db is down.  File system is the safest place to
put the data
cos if it goes, so does you box and there is little chance of dispare apart
from filling it up and
you have the ability to reprocess the files again if the db dies.  Single
file per record solves
file locking problems and the xml data is so big anyway you would want to
seggregate it.

Like I said I don't go on giving out hints all the time so that's the last
from me for now. ;)



On Thu, Oct 30, 2008 at 3:27 PM, regs at kinetix.gr <regs at kinetix.gr> wrote:

> Don't get me wrong. I am not trying to undermine the excellent work that
> has already been done on the event handling
> modules of FS.
>
> I myself am a big fun of modular constructs in order to achieve
> something complicated (e.g. the UNIX way).
>
> But there are some situations that we will always need some extra
> features in order to accomplish our goal.
>
> As I wrote before, my current system is using the cdr-push (from the
> gateway side) method to gather my cdrs.
> Many billing systems and many terminating providers are satisfied with
> that model alone. I am not. In the end of each hour
> I am gathering all the cdrs and checking them one by one against my
> database in order to verify that ALL of my cdrs were
> handled by my radius servers.
>
> In my future system (where a more batch-like mode is preferable) I am
> "forced" to use a cdr-pull method. My billing system will
> be responisble for "pulling" the cdrs from the gateways and then process
> them. So if we have a look at the event handling weaponry of FS the
> following modules cannot meet me needs :
>
> mod_event_multicast
> mod_event socket
> mod_radius_cdr
> mod_xmpp_event
>
> because they all rely on a mechanism where the gateway is pushing my
> cdrs to my billing system (and you need a checking mechanism to verify
> that all the cdrs were handled).
>
> So now, lets have a look at the alternatives :
>
> mod_xml_cdr : this was my first choice as it had all the info needed
> (even more). But the cdr per file approach proved out to be
> inadequate in terms of performance. I wrote a few lines of code (in
> perl) that did a listing of the directory, parsed the cdrs using
> XML::Simple (without even doing anything meaningful, e.g. checks,
> inserts to a database etc.) and I was not able to make my system
> parse more than a 30 cdrs/sec (reference 1000 USD average system, not
> drawing any conclusions, don't get me wrong).
> Also while the parser was running I had a high cpu utilization (as
> expected) that was competing with my FS service. Wouldn't that make
> a bad impact of my freeswitch performance?
>
> so I went to a different module
>
> mod_cdr_csv : the text .csv format with the plethora of attributes was
> perfect for my application but it would be a daunting task
> for me to make a script tha purges the Master csv while it was being
> used by FS. I googled the issue and found about the HUP method
> and the rotate directive in the config. So far so good. But it lacked
> some functionality.
>
> All I was trying to state in my previous messages was that it would be
> nice if the rotation was being initiated by FS, and maybe have different
> behavior depending on another directive or something. Let me explain
> what I mean.
>
> There are some people who don't care about what cdrs exist in a given
> rotated log file, as long as a) no cdrs get lost in the process of rotation
> , b) no duplicates exist. I can see that the HUP method satisfied this
> need.
>
> There is another group of people that DO care about what cdrs are in a
> rotated log file. E.g. : The file with the name
> Master.csv.2008-01-01-09-00-00
> would only contain cdrs that were terminated from "2008-01-01 09:00:00"
> to "2008-01-01 09:59:59" if an hourly setup was desirable or from
> "2008-01-01 09:00:00" to "2008-01-01 09:04:59" if a five-minutes setup
> was desirable. That need is not covered by the current HUP method
> because some cdrs might "escape" from the next to the previous file due
> to the fact that there is a delay between cron executing the HUP and FS
> doing
> the file log rotation.
>
> In simple words : I am getting my job done with FS they way it already
> is (no question about that). BUT, I (or someone with different needs)
> could do
> his/her job better if some minor features were present.
>
> I am not saying that you should should embrace modules that meet the
> average needs yet sacrifice flexibility. I am merely suggesting
> extending the flexibility
> of the already existing ones. Put some more lego tiles in your box set :)
>
>
> Michael Collins wrote:
> >
> > /me sends Anthony's post to the printer to be laminated and framed… J
> >
> > ------------------------------------------------------------------------
> >
> > *From:* freeswitch-users-bounces at lists.freeswitch.org
> > [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of
> > *Anthony Minessale
> > *Sent:* Thursday, October 30, 2008 6:10 AM
> > *To:* freeswitch-users at lists.freeswitch.org
> > *Subject:* Re: [Freeswitch-users] mod_cdr revival (or new module maybe)
> >
> > Actually,
> >
> > The goal is to not limit the functionality by over thinking how things
> > will be used but to provide building blocks to make much
> > more possible. Our analogy for this is that if you take the common
> > lego reference, If you have something cool built out of legos but they
> > are super glued together it limits what you *could* have done had then
> > not been.
> >
> > The system that David described is indeed ideal. I have mentioned it
> > more than once and its no coincidence that FreeSWITCH plays into that
> > model to a tee. I do not mention it very often because I think
> > understanding that concept alone is valuable advice that I'd prefer
> > not to waste on deaf ears so I recommend you heed his suggestion.
> >
> > Logging direct to DB is not a sin or anything but it's just not very
> > reliable. We have so many ways to process call records as it is.
> > you can use the cdr_csv if you want legacy support from asterisk that
> > you can eventually tweak with templatable output, you can even make a
> > template that generates the log as ready to roll sql statements you
> > can just cat into mysql.
> >
> > We have event_socket + channel_hangup event which has all the same
> > info that the cdr_csv uses to generate the file.
> > We have the XML cdr that is a 3 dimensional view of the call with an
> > account of nearly everything that happend including
> > each application executed, branching records when the call is
> > transferred complete with a timestamp for each occurance.
> >
> > --"However it would be great if freeswitch had the options for extra
> > functionality (auto log rotation, db cdrs etc)
> > so that it meets the peculiarities of every different project."
> >
> > A single cron entry that does kill -HUP `cat freeswitch.pid` and you
> > have automatic rotation in unix.
> > In windows there are several cron equivs and instead of kill you can
> > use an xml-rpc script to call the FSAPI call "fsctl send_sighup"
> > also you can schedule it to happen automaticly with the FSAPI
> >
> > sched_api @300 mygroup fsctl send_sighup
> >
> > A script connected to event socket subscribed to channel_hangup can
> > act as your DB cdr engine and has the bonus that it could
> > connect to N boxes at once as well as be able to tell if the system
> > was in trouble by listening for system heartbeat events and or socket
> > disconnect.
> >
> > Moral of the story, there is more already there than first appears.
> > I've always hated the M$ paper clip so I strive not to bring him back.
> > "It looks like you are trying to run a VoIP termination company! Shall
> > I assume how you want your call details?"
> >
> >
> > On Thu, Oct 30, 2008 at 7:43 AM, regs at kinetix.gr
> > <mailto:regs at kinetix.gr> <regs at kinetix.gr <mailto:regs at kinetix.gr>>
> wrote:
> >
> > I'll try some tests with various combinations first and then decide
> > what's best.
> >
> > I have to say that I was astonished to find out that freeswitch had so
> > many event handlers from the very beginning.
> > However it would be great if freeswitch had the options for extra
> > functionality (auto log rotation, db cdrs etc)
> > so that it meets the peculiarities of every different project. That
> > would make a big difference compared to
> > other softswitch solutions where the lack of such features prohibits
> > people from using them (especially in
> > the carrier grade level). User feedback (and wish lists) is the key
> > for the success of an open source (or not)
> > project...
> >
> > Thank you all for your replies. You' ve been very helpful.
> >
> >
> >
> > David Knell wrote:
> >
> > The sleep's done each time the directory's empty, not each time a
> > file's written. File open and close
> > are trivial (it's probably still cached), and the contents are going
> > to have to be parsed wherever you
> > process it.
> >
> > We've used exactly this to process deliver CDRs on boxes handling in
> > excess of 500K mins/day
> > without issue. And, looking at one now, the CDR processor's used about
> > 4% of the CPU time
> > of FreeSWITCH, and about half that of the MySQL database which it
> > writes the records to, also
> > on the local machine, from which they're simply copied to the main CDR
> > processor.
> >
> > It performance simply isn't worth worrying about.
> >
> > --Dave
> >
> > "sleep for a couple of seconds"
> >
> > But then you could only insert 1800 cdrs per hour...
> > If I was to insert 36000 cdrs per hour this means that I have to
> > open
> > parse
> > close
> > 10 files per second. Imagine the I/O penalty just for opening - closing
> the file.
> > (the persing is the same for both situations)
> >
> >
> >
> >
> >
> > David Knell wrote:
> >
> > regs at kinetix.gr <mailto:regs at kinetix.gr> wrote:
> >
> >> Yes, the xml files give you tons of info... but isn't it a little
> >> insufficient - performance wise -
> >> to open and close so many files in such a little time. In a PBX
> >> environment that wouldn't be an
> >> issue but if we get to the small-voip-carrier level (some thousand cdrs
> >> per hour)
> >> that could slow things down considerably, wouldn't it?
> >>
> >>
> > Not that you'd notice.  We run XML CDR to database scripting on each box
> > that we use
> > for switching, and it's a pretty trivial task compared with switching
> > all that media.  Doing it
> > this way is:-
> > (a) distributed - one process per box scales nicely;
> > (b) robust - script down, DB down, no problem: files just queue up;
> > (c) simple - the script logic is trivial:
> > - while 1
> >   - for each file in the XML CDR directory
> >     - open it
> >     - parse it (XML::Simple for us)
> >     - insert it in to the DB
> >     - delete it
> >   - sleep for a couple of seconds
> > Two error cases: can't parse or can't find data which should be there:
> > move the file in to
> > another directory to be examined by real eyes; DB insert fails: break
> > out of inner loop and
> > it'll be retried after a short pause.
> >
> > --Dave
> >
> >
> >
> >
> >
> > ------------------------------------------------------------------------
> >
> >
> >
> > _______________________________________________
> > Freeswitch-users mailing list
> > Freeswitch-users at lists.freeswitch.org <mailto:
> Freeswitch-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> >
> >
> > --
> > David Knell, Director, 3C Limited
> > T: 020 8114 8901  F: 020 3002 7257  M: 001 415 630 3031
> > http://www.3c.co.uk
> >
> >
> > ------------------------------------------------------------------------
> >
> >
> >
> > _______________________________________________
> > Freeswitch-users mailing list
> > Freeswitch-users at lists.freeswitch.org <mailto:
> Freeswitch-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> > _______________________________________________
> > Freeswitch-users mailing list
> > Freeswitch-users at lists.freeswitch.org
> > <mailto:Freeswitch-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> >
> > --
> > Anthony Minessale II
> >
> > FreeSWITCH http://www.freeswitch.org/
> > ClueCon http://www.cluecon.com/
> >
> > AIM: anthm
> > MSN:anthony_minessale at hotmail.com <MSN%3Aanthony_minessale at hotmail.com>
> > <mailto:MSN%3Aanthony_minessale at hotmail.com<MSN%253Aanthony_minessale at hotmail.com>
> >
> > GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com<PAYPAL%3Aanthony.minessale at gmail.com>
> > <mailto:PAYPAL%3Aanthony.minessale at gmail.com<PAYPAL%253Aanthony.minessale at gmail.com>
> >
> > IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
> >
> > FreeSWITCH Developer Conference
> > sip:888 at conference.freeswitch.org <sip%3A888 at conference.freeswitch.org>
> > <mailto:sip%3A888 at conference.freeswitch.org<sip%253A888 at conference.freeswitch.org>
> >
> > iax:guest at conference.freeswitch.org/888
> > <http://iax:guest@conference.freeswitch.org/888>
> > googletalk:conf+888 at conference.freeswitch.org<googletalk%3Aconf%2B888 at conference.freeswitch.org>
> > <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org<googletalk%253Aconf%252B888 at conference.freeswitch.org>
> >
> > pstn:213-799-1400
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Freeswitch-users mailing list
> > Freeswitch-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>



-- 
Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/

AIM: anthm
MSN:anthony_minessale at hotmail.com <MSN%3Aanthony_minessale at hotmail.com>
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com<PAYPAL%3Aanthony.minessale at gmail.com>
IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org <sip%3A888 at conference.freeswitch.org>
iax:guest at conference.freeswitch.org/888
googletalk:conf+888 at conference.freeswitch.org<googletalk%3Aconf%2B888 at conference.freeswitch.org>
pstn:213-799-1400
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20081030/dd9588d5/attachment-0002.html 


More information about the FreeSWITCH-users mailing list