[Freeswitch-users] mod_cdr revival (or new module maybe)
regs at kinetix.gr
regs at kinetix.gr
Thu Oct 30 13:27:28 PDT 2008
Don't get me wrong. I am not trying to undermine the excellent work that
has already been done on the event handling
modules of FS.
I myself am a big fun of modular constructs in order to achieve
something complicated (e.g. the UNIX way).
But there are some situations that we will always need some extra
features in order to accomplish our goal.
As I wrote before, my current system is using the cdr-push (from the
gateway side) method to gather my cdrs.
Many billing systems and many terminating providers are satisfied with
that model alone. I am not. In the end of each hour
I am gathering all the cdrs and checking them one by one against my
database in order to verify that ALL of my cdrs were
handled by my radius servers.
In my future system (where a more batch-like mode is preferable) I am
"forced" to use a cdr-pull method. My billing system will
be responisble for "pulling" the cdrs from the gateways and then process
them. So if we have a look at the event handling weaponry of FS the
following modules cannot meet me needs :
mod_event_multicast
mod_event socket
mod_radius_cdr
mod_xmpp_event
because they all rely on a mechanism where the gateway is pushing my
cdrs to my billing system (and you need a checking mechanism to verify
that all the cdrs were handled).
So now, lets have a look at the alternatives :
mod_xml_cdr : this was my first choice as it had all the info needed
(even more). But the cdr per file approach proved out to be
inadequate in terms of performance. I wrote a few lines of code (in
perl) that did a listing of the directory, parsed the cdrs using
XML::Simple (without even doing anything meaningful, e.g. checks,
inserts to a database etc.) and I was not able to make my system
parse more than a 30 cdrs/sec (reference 1000 USD average system, not
drawing any conclusions, don't get me wrong).
Also while the parser was running I had a high cpu utilization (as
expected) that was competing with my FS service. Wouldn't that make
a bad impact of my freeswitch performance?
so I went to a different module
mod_cdr_csv : the text .csv format with the plethora of attributes was
perfect for my application but it would be a daunting task
for me to make a script tha purges the Master csv while it was being
used by FS. I googled the issue and found about the HUP method
and the rotate directive in the config. So far so good. But it lacked
some functionality.
All I was trying to state in my previous messages was that it would be
nice if the rotation was being initiated by FS, and maybe have different
behavior depending on another directive or something. Let me explain
what I mean.
There are some people who don't care about what cdrs exist in a given
rotated log file, as long as a) no cdrs get lost in the process of rotation
, b) no duplicates exist. I can see that the HUP method satisfied this need.
There is another group of people that DO care about what cdrs are in a
rotated log file. E.g. : The file with the name
Master.csv.2008-01-01-09-00-00
would only contain cdrs that were terminated from "2008-01-01 09:00:00"
to "2008-01-01 09:59:59" if an hourly setup was desirable or from
"2008-01-01 09:00:00" to "2008-01-01 09:04:59" if a five-minutes setup
was desirable. That need is not covered by the current HUP method
because some cdrs might "escape" from the next to the previous file due
to the fact that there is a delay between cron executing the HUP and FS
doing
the file log rotation.
In simple words : I am getting my job done with FS they way it already
is (no question about that). BUT, I (or someone with different needs)
could do
his/her job better if some minor features were present.
I am not saying that you should should embrace modules that meet the
average needs yet sacrifice flexibility. I am merely suggesting
extending the flexibility
of the already existing ones. Put some more lego tiles in your box set :)
Michael Collins wrote:
>
> /me sends Anthony’s post to the printer to be laminated and framed… J
>
> ------------------------------------------------------------------------
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of
> *Anthony Minessale
> *Sent:* Thursday, October 30, 2008 6:10 AM
> *To:* freeswitch-users at lists.freeswitch.org
> *Subject:* Re: [Freeswitch-users] mod_cdr revival (or new module maybe)
>
> Actually,
>
> The goal is to not limit the functionality by over thinking how things
> will be used but to provide building blocks to make much
> more possible. Our analogy for this is that if you take the common
> lego reference, If you have something cool built out of legos but they
> are super glued together it limits what you *could* have done had then
> not been.
>
> The system that David described is indeed ideal. I have mentioned it
> more than once and its no coincidence that FreeSWITCH plays into that
> model to a tee. I do not mention it very often because I think
> understanding that concept alone is valuable advice that I'd prefer
> not to waste on deaf ears so I recommend you heed his suggestion.
>
> Logging direct to DB is not a sin or anything but it's just not very
> reliable. We have so many ways to process call records as it is.
> you can use the cdr_csv if you want legacy support from asterisk that
> you can eventually tweak with templatable output, you can even make a
> template that generates the log as ready to roll sql statements you
> can just cat into mysql.
>
> We have event_socket + channel_hangup event which has all the same
> info that the cdr_csv uses to generate the file.
> We have the XML cdr that is a 3 dimensional view of the call with an
> account of nearly everything that happend including
> each application executed, branching records when the call is
> transferred complete with a timestamp for each occurance.
>
> --"However it would be great if freeswitch had the options for extra
> functionality (auto log rotation, db cdrs etc)
> so that it meets the peculiarities of every different project."
>
> A single cron entry that does kill -HUP `cat freeswitch.pid` and you
> have automatic rotation in unix.
> In windows there are several cron equivs and instead of kill you can
> use an xml-rpc script to call the FSAPI call "fsctl send_sighup"
> also you can schedule it to happen automaticly with the FSAPI
>
> sched_api @300 mygroup fsctl send_sighup
>
> A script connected to event socket subscribed to channel_hangup can
> act as your DB cdr engine and has the bonus that it could
> connect to N boxes at once as well as be able to tell if the system
> was in trouble by listening for system heartbeat events and or socket
> disconnect.
>
> Moral of the story, there is more already there than first appears.
> I've always hated the M$ paper clip so I strive not to bring him back.
> "It looks like you are trying to run a VoIP termination company! Shall
> I assume how you want your call details?"
>
>
> On Thu, Oct 30, 2008 at 7:43 AM, regs at kinetix.gr
> <mailto:regs at kinetix.gr> <regs at kinetix.gr <mailto:regs at kinetix.gr>> wrote:
>
> I'll try some tests with various combinations first and then decide
> what's best.
>
> I have to say that I was astonished to find out that freeswitch had so
> many event handlers from the very beginning.
> However it would be great if freeswitch had the options for extra
> functionality (auto log rotation, db cdrs etc)
> so that it meets the peculiarities of every different project. That
> would make a big difference compared to
> other softswitch solutions where the lack of such features prohibits
> people from using them (especially in
> the carrier grade level). User feedback (and wish lists) is the key
> for the success of an open source (or not)
> project...
>
> Thank you all for your replies. You' ve been very helpful.
>
>
>
> David Knell wrote:
>
> The sleep's done each time the directory's empty, not each time a
> file's written. File open and close
> are trivial (it's probably still cached), and the contents are going
> to have to be parsed wherever you
> process it.
>
> We've used exactly this to process deliver CDRs on boxes handling in
> excess of 500K mins/day
> without issue. And, looking at one now, the CDR processor's used about
> 4% of the CPU time
> of FreeSWITCH, and about half that of the MySQL database which it
> writes the records to, also
> on the local machine, from which they're simply copied to the main CDR
> processor.
>
> It performance simply isn't worth worrying about.
>
> --Dave
>
> "sleep for a couple of seconds"
>
> But then you could only insert 1800 cdrs per hour...
> If I was to insert 36000 cdrs per hour this means that I have to
> open
> parse
> close
> 10 files per second. Imagine the I/O penalty just for opening - closing the file.
> (the persing is the same for both situations)
>
>
>
>
>
> David Knell wrote:
>
> regs at kinetix.gr <mailto:regs at kinetix.gr> wrote:
>
>> Yes, the xml files give you tons of info... but isn't it a little
>> insufficient - performance wise -
>> to open and close so many files in such a little time. In a PBX
>> environment that wouldn't be an
>> issue but if we get to the small-voip-carrier level (some thousand cdrs
>> per hour)
>> that could slow things down considerably, wouldn't it?
>>
>>
> Not that you'd notice. We run XML CDR to database scripting on each box
> that we use
> for switching, and it's a pretty trivial task compared with switching
> all that media. Doing it
> this way is:-
> (a) distributed - one process per box scales nicely;
> (b) robust - script down, DB down, no problem: files just queue up;
> (c) simple - the script logic is trivial:
> - while 1
> - for each file in the XML CDR directory
> - open it
> - parse it (XML::Simple for us)
> - insert it in to the DB
> - delete it
> - sleep for a couple of seconds
> Two error cases: can't parse or can't find data which should be there:
> move the file in to
> another directory to be examined by real eyes; DB insert fails: break
> out of inner loop and
> it'll be retried after a short pause.
>
> --Dave
>
>
>
>
>
> ------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org <mailto:Freeswitch-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
>
> --
> David Knell, Director, 3C Limited
> T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031
> http://www.3c.co.uk
>
>
> ------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org <mailto:Freeswitch-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> <mailto:Freeswitch-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> <mailto:sip%3A888 at conference.freeswitch.org>
> iax:guest at conference.freeswitch.org/888
> <http://iax:guest@conference.freeswitch.org/888>
> googletalk:conf+888 at conference.freeswitch.org
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:213-799-1400
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
More information about the FreeSWITCH-users
mailing list