mod_cdr revival (or new module maybe)
Hi, I saw in the wiki that the mod_cdr module is now unsupported. There is also a note about a revival of the module. I would like to ask the following : What is the current state of the revival process? (should we expect something in the near future?) Will it have the same functionality as before (DB support for instance)? Are there any plans for a brand new database specific event handler module? It would be great if there was one so that developers (especially those who develop billing applications) would not have to create their own hacks (cron scripts etc.) Thank you for your time, -- ------------------------------------------- Apostolos Pantsiopoulos Kinetix Tele.com Support Center
Unsure at this time. There has been some work on mod_cdr_odbc. We generally advise against direct to db cdr methods without a very robust backup method for when the db is down. On Oct 29, 2008, at 9:57 AM, "regs@kinetix.gr" <regs@kinetix.gr> wrote:
Hi,
I saw in the wiki that the mod_cdr module is now unsupported. There is also a note about a revival of the module. I would like to ask the following :
What is the current state of the revival process? (should we expect something in the near future?)
Will it have the same functionality as before (DB support for instance)?
Are there any plans for a brand new database specific event handler module?
It would be great if there was one so that developers (especially those who develop billing applications) would not have to create their own hacks (cron scripts etc.)
Thank you for your time,
-- ------------------------------------------- Apostolos Pantsiopoulos Kinetix Tele.com Support Center
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Yes, I agree. But one could use the two methods combined (csv or xml + db) for redundancy. Is there any consideration regarding automatic log rotation (e.g. hourly, or user specified) without the need of a HUP? Now, that could make things a lot easier for the development of an external csv to db aggregation script because the script would read from a closed (not used by freeswitch at the time) CDRs file. And the developer could be sure that the cdrs contained in that file would have a hangup timestamp that could be described by the filename (e.g. 20080101_010000.csv). Michael Jerris wrote:
Unsure at this time. There has been some work on mod_cdr_odbc. We generally advise against direct to db cdr methods without a very robust backup method for when the db is down.
On Oct 29, 2008, at 9:57 AM, "regs@kinetix.gr" <regs@kinetix.gr> wrote:
Hi,
I saw in the wiki that the mod_cdr module is now unsupported. There is also a note about a revival of the module. I would like to ask the following :
What is the current state of the revival process? (should we expect something in the near future?)
Will it have the same functionality as before (DB support for instance)?
Are there any plans for a brand new database specific event handler module?
It would be great if there was one so that developers (especially those who develop billing applications) would not have to create their own hacks (cron scripts etc.)
Thank you for your time,
-- ------------------------------------------- Apostolos Pantsiopoulos Kinetix Tele.com Support Center
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Yes, I agree. But one could use the two methods combined (csv or xml + db) for redundancy.
Is there any consideration regarding automatic log rotation (e.g. hourly, or user specified) without the need of a HUP? Now, that could make things a lot easier for the development of an external csv to db aggregation script because the script would read from a closed (not used by freeswitch at the time) CDRs file. And the developer could be sure that the cdrs contained in that file would have a hangup timestamp that could be described by the filename (e.g. 20080101_010000.csv).
For the record, I've been dumping all my XML CDRs into a particular directory and letting a script pick them up and process them. I think this is the best of both worlds: you get individual files with tons of info on each call and you can have a process that picks up those files and inserts them into the db. If the db is down then the CDRs aren't lost - they just accumulate in the directory until you get the db/script thing working again. Just my $.02 -MC
In regards to auto log rotation - YES YES ANTHM just completed that item for me, where by you can set the time in minutes i believe it was. I have not tested it yet, hope to this week. Shawn Michael Collins wrote:
Yes, I agree. But one could use the two methods combined (csv or xml + db) for redundancy.
Is there any consideration regarding automatic log rotation (e.g. hourly, or user specified) without the need of a HUP? Now, that could make things a lot easier
for
the development of an external csv to db aggregation script because the script would read from a closed (not used by freeswitch at the time) CDRs file. And the developer could be sure that the cdrs contained in that file would have a hangup timestamp that could be described by the filename (e.g. 20080101_010000.csv).
For the record, I've been dumping all my XML CDRs into a particular directory and letting a script pick them up and process them. I think this is the best of both worlds: you get individual files with tons of info on each call and you can have a process that picks up those files and inserts them into the db. If the db is down then the CDRs aren't lost - they just accumulate in the directory until you get the db/script thing working again.
Just my $.02
-MC
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
That's very good news. :) Shawn Lewis wrote:
In regards to auto log rotation - YES YES
ANTHM just completed that item for me, where by you can set the time in minutes i believe it was.
I have not tested it yet, hope to this week.
Shawn
Michael Collins wrote:
Yes, I agree. But one could use the two methods combined (csv or xml + db) for redundancy.
Is there any consideration regarding automatic log rotation (e.g. hourly, or user specified) without the need of a HUP? Now, that could make things a lot easier
for
the development of an external csv to db aggregation script because the script would read from a closed (not used by freeswitch at the time) CDRs file. And the developer could be sure that the cdrs contained in that file would have a hangup timestamp that could be described by the filename (e.g. 20080101_010000.csv).
For the record, I've been dumping all my XML CDRs into a particular directory and letting a script pick them up and process them. I think this is the best of both worlds: you get individual files with tons of info on each call and you can have a process that picks up those files and inserts them into the db. If the db is down then the CDRs aren't lost - they just accumulate in the directory until you get the db/script thing working again.
Just my $.02
-MC
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it? Thanks again for your prompt replies, Michael Collins wrote:
Yes, I agree. But one could use the two methods combined (csv or xml + db) for redundancy.
Is there any consideration regarding automatic log rotation (e.g. hourly, or user specified) without the need of a HUP? Now, that could make things a lot easier
for
the development of an external csv to db aggregation script because the script would read from a closed (not used by freeswitch at the time) CDRs file. And the developer could be sure that the cdrs contained in that file would have a hangup timestamp that could be described by the filename (e.g. 20080101_010000.csv).
For the record, I've been dumping all my XML CDRs into a particular directory and letting a script pick them up and process them. I think this is the best of both worlds: you get individual files with tons of info on each call and you can have a process that picks up those files and inserts them into the db. If the db is down then the CDRs aren't lost - they just accumulate in the directory until you get the db/script thing working again.
Just my $.02
-MC
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it?
Thanks again for your prompt replies,
At that level of activity then I would assume you'd want a more robust solution which obviously would involve a server handling the CDRs separately. That's where XML is a real winner: it can POST CDRs to a web server and the webserver can handle all the pre-processing and db fun stuff. And if the connection to the webserver failed, the CDRs would be put on disk so that they aren't lost forever. Also, the webserver could cache the CDRs to its disk (or whatever storage) if the db itself went down but the webserver stayed up. Just a thought, anyway. It may be extra layers but it's also extra control. -MC
Michael Collins wrote:
Yes, I agree. But one could use the two methods combined (csv or xml + db) for redundancy.
Is there any consideration regarding automatic log rotation (e.g. hourly, or user specified) without the need of a HUP? Now, that could make things a lot easier
for
the development of an external csv to db aggregation script because the script would read from a closed (not used by freeswitch at the time) CDRs file. And the developer could be sure that the cdrs contained in that file would have a hangup timestamp that could be described by the filename (e.g. 20080101_010000.csv).
For the record, I've been dumping all my XML CDRs into a particular directory and letting a script pick them up and process them. I think this is the best of both worlds: you get individual files with tons of info on each call and you can have a process that picks up those files and inserts them into the db. If the db is down then the CDRs aren't lost - they just accumulate in the directory until you get the db/script thing working again.
Just my $.02
-MC
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
Good point. I have got this kind of behavior (cdrs push model) in my current system (using radius servers). The only drawback of this method is that if you want to be absolutely sure that all the cdrs were handled by the web server (or radius server) you have to check at certain intervals every cdr one by one (and handle those left unhandled for various reasons (network, excessive web server load etc.)) But for my next project I am somewhat forced to use a "cdrs-pull" method where a process will "pull" cdrs from the server at its own pace making this extra check unnecessary. I will wait for an automatic log rotation as Shawn Lewis wrote. I think that will do the job. Michael Collins wrote:
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand
cdrs
per hour) that could slow things down considerably, wouldn't it?
Thanks again for your prompt replies,
At that level of activity then I would assume you'd want a more robust solution which obviously would involve a server handling the CDRs separately. That's where XML is a real winner: it can POST CDRs to a web server and the webserver can handle all the pre-processing and db fun stuff. And if the connection to the webserver failed, the CDRs would be put on disk so that they aren't lost forever. Also, the webserver could cache the CDRs to its disk (or whatever storage) if the db itself went down but the webserver stayed up.
Just a thought, anyway. It may be extra layers but it's also extra control.
-MC
Michael Collins wrote:
Yes, I agree. But one could use the two methods combined (csv or
xml +
db) for redundancy.
Is there any consideration regarding automatic log rotation (e.g. hourly, or user specified) without the need of a HUP? Now, that could make things a lot easier
for
the development of an external csv to db aggregation script because the script would
read
from a closed (not used by freeswitch at the time) CDRs file. And the developer could be sure that the
cdrs
contained in that file would have a hangup timestamp that could be described by the filename
(e.g.
20080101_010000.csv).
For the record, I've been dumping all my XML CDRs into a particular directory and letting a script pick them up and process them. I
think
this is the best of both worlds: you get individual files with tons
of
info on each call and you can have a process that picks up those
files
and inserts them into the db. If the db is down then the CDRs aren't lost - they just accumulate in the directory until you get the
db/script
thing working again.
Just my $.02
-MC
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
regs@kinetix.gr wrote:
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it?
Not that you'd notice. We run XML CDR to database scripting on each box that we use for switching, and it's a pretty trivial task compared with switching all that media. Doing it this way is:- (a) distributed - one process per box scales nicely; (b) robust - script down, DB down, no problem: files just queue up; (c) simple - the script logic is trivial: - while 1 - for each file in the XML CDR directory - open it - parse it (XML::Simple for us) - insert it in to the DB - delete it - sleep for a couple of seconds Two error cases: can't parse or can't find data which should be there: move the file in to another directory to be examined by real eyes; DB insert fails: break out of inner loop and it'll be retried after a short pause. --Dave -- David Knell, Director, 3C Limited T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031 http://www.3c.co.uk
"sleep for a couple of seconds" But then you could only insert 1800 cdrs per hour... If I was to insert 36000 cdrs per hour this means that I have to open parse close 10 files per second. Imagine the I/O penalty just for opening - closing the file. (the persing is the same for both situations) David Knell wrote:
regs@kinetix.gr wrote:
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it?
Not that you'd notice. We run XML CDR to database scripting on each box that we use for switching, and it's a pretty trivial task compared with switching all that media. Doing it this way is:- (a) distributed - one process per box scales nicely; (b) robust - script down, DB down, no problem: files just queue up; (c) simple - the script logic is trivial: - while 1 - for each file in the XML CDR directory - open it - parse it (XML::Simple for us) - insert it in to the DB - delete it - sleep for a couple of seconds Two error cases: can't parse or can't find data which should be there: move the file in to another directory to be examined by real eyes; DB insert fails: break out of inner loop and it'll be retried after a short pause.
--Dave
The sleep's done each time the directory's empty, not each time a file's written. File open and close are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you process it. We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also on the local machine, from which they're simply copied to the main CDR processor. It performance simply isn't worth worrying about. --Dave
"sleep for a couple of seconds"
But then you could only insert 1800 cdrs per hour... If I was to insert 36000 cdrs per hour this means that I have to open parse close 10 files per second. Imagine the I/O penalty just for opening - closing the file. (the persing is the same for both situations)
David Knell wrote:
regs@kinetix.gr wrote:
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it?
Not that you'd notice. We run XML CDR to database scripting on each box that we use for switching, and it's a pretty trivial task compared with switching all that media. Doing it this way is:- (a) distributed - one process per box scales nicely; (b) robust - script down, DB down, no problem: files just queue up; (c) simple - the script logic is trivial: - while 1 - for each file in the XML CDR directory - open it - parse it (XML::Simple for us) - insert it in to the DB - delete it - sleep for a couple of seconds Two error cases: can't parse or can't find data which should be there: move the file in to another directory to be examined by real eyes; DB insert fails: break out of inner loop and it'll be retried after a short pause.
--Dave
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- David Knell, Director, 3C Limited T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031 http://www.3c.co.uk
I'll try some tests with various combinations first and then decide what's best. I have to say that I was astonished to find out that freeswitch had so many event handlers from the very beginning. However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc) so that it meets the peculiarities of every different project. That would make a big difference compared to other softswitch solutions where the lack of such features prohibits people from using them (especially in the carrier grade level). User feedback (and wish lists) is the key for the success of an open source (or not) project... Thank you all for your replies. You' ve been very helpful. David Knell wrote:
The sleep's done each time the directory's empty, not each time a file's written. File open and close are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you process it.
We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also on the local machine, from which they're simply copied to the main CDR processor.
It performance simply isn't worth worrying about.
--Dave
"sleep for a couple of seconds"
But then you could only insert 1800 cdrs per hour... If I was to insert 36000 cdrs per hour this means that I have to open parse close 10 files per second. Imagine the I/O penalty just for opening - closing the file. (the persing is the same for both situations)
David Knell wrote:
regs@kinetix.gr wrote:
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it?
Not that you'd notice. We run XML CDR to database scripting on each box that we use for switching, and it's a pretty trivial task compared with switching all that media. Doing it this way is:- (a) distributed - one process per box scales nicely; (b) robust - script down, DB down, no problem: files just queue up; (c) simple - the script logic is trivial: - while 1 - for each file in the XML CDR directory - open it - parse it (XML::Simple for us) - insert it in to the DB - delete it - sleep for a couple of seconds Two error cases: can't parse or can't find data which should be there: move the file in to another directory to be examined by real eyes; DB insert fails: break out of inner loop and it'll be retried after a short pause.
--Dave
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- David Knell, Director, 3C Limited T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031 http://www.3c.co.uk ------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Actually, The goal is to not limit the functionality by over thinking how things will be used but to provide building blocks to make much more possible. Our analogy for this is that if you take the common lego reference, If you have something cool built out of legos but they are super glued together it limits what you *could* have done had then not been. The system that David described is indeed ideal. I have mentioned it more than once and its no coincidence that FreeSWITCH plays into that model to a tee. I do not mention it very often because I think understanding that concept alone is valuable advice that I'd prefer not to waste on deaf ears so I recommend you heed his suggestion. Logging direct to DB is not a sin or anything but it's just not very reliable. We have so many ways to process call records as it is. you can use the cdr_csv if you want legacy support from asterisk that you can eventually tweak with templatable output, you can even make a template that generates the log as ready to roll sql statements you can just cat into mysql. We have event_socket + channel_hangup event which has all the same info that the cdr_csv uses to generate the file. We have the XML cdr that is a 3 dimensional view of the call with an account of nearly everything that happend including each application executed, branching records when the call is transferred complete with a timestamp for each occurance. --"However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc) so that it meets the peculiarities of every different project." A single cron entry that does kill -HUP `cat freeswitch.pid` and you have automatic rotation in unix. In windows there are several cron equivs and instead of kill you can use an xml-rpc script to call the FSAPI call "fsctl send_sighup" also you can schedule it to happen automaticly with the FSAPI sched_api @300 mygroup fsctl send_sighup A script connected to event socket subscribed to channel_hangup can act as your DB cdr engine and has the bonus that it could connect to N boxes at once as well as be able to tell if the system was in trouble by listening for system heartbeat events and or socket disconnect. Moral of the story, there is more already there than first appears. I've always hated the M$ paper clip so I strive not to bring him back. "It looks like you are trying to run a VoIP termination company! Shall I assume how you want your call details?" On Thu, Oct 30, 2008 at 7:43 AM, regs@kinetix.gr <regs@kinetix.gr> wrote:
I'll try some tests with various combinations first and then decide what's best.
I have to say that I was astonished to find out that freeswitch had so many event handlers from the very beginning. However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc) so that it meets the peculiarities of every different project. That would make a big difference compared to other softswitch solutions where the lack of such features prohibits people from using them (especially in the carrier grade level). User feedback (and wish lists) is the key for the success of an open source (or not) project...
Thank you all for your replies. You' ve been very helpful.
David Knell wrote:
The sleep's done each time the directory's empty, not each time a file's written. File open and close are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you process it.
We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also on the local machine, from which they're simply copied to the main CDR processor.
It performance simply isn't worth worrying about.
--Dave
"sleep for a couple of seconds"
But then you could only insert 1800 cdrs per hour... If I was to insert 36000 cdrs per hour this means that I have to open parse close 10 files per second. Imagine the I/O penalty just for opening - closing the file. (the persing is the same for both situations)
David Knell wrote:
regs@kinetix.gr wrote:
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it?
Not that you'd notice. We run XML CDR to database scripting on each box that we use for switching, and it's a pretty trivial task compared with switching all that media. Doing it this way is:- (a) distributed - one process per box scales nicely; (b) robust - script down, DB down, no problem: files just queue up; (c) simple - the script logic is trivial: - while 1 - for each file in the XML CDR directory - open it - parse it (XML::Simple for us) - insert it in to the DB - delete it - sleep for a couple of seconds Two error cases: can't parse or can't find data which should be there: move the file in to another directory to be examined by real eyes; DB insert fails: break out of inner loop and it'll be retried after a short pause.
--Dave
------------------------------
_______________________________________________ Freeswitch-users mailing listFreeswitch-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...
-- David Knell, Director, 3C Limited T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031http://www.3c.co.uk
------------------------------
_______________________________________________ Freeswitch-users mailing listFreeswitch-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
/me sends Anthony's post to the printer to be laminated and framed... :-) ________________________________ From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Thursday, October 30, 2008 6:10 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_cdr revival (or new module maybe) Actually, The goal is to not limit the functionality by over thinking how things will be used but to provide building blocks to make much more possible. Our analogy for this is that if you take the common lego reference, If you have something cool built out of legos but they are super glued together it limits what you *could* have done had then not been. The system that David described is indeed ideal. I have mentioned it more than once and its no coincidence that FreeSWITCH plays into that model to a tee. I do not mention it very often because I think understanding that concept alone is valuable advice that I'd prefer not to waste on deaf ears so I recommend you heed his suggestion. Logging direct to DB is not a sin or anything but it's just not very reliable. We have so many ways to process call records as it is. you can use the cdr_csv if you want legacy support from asterisk that you can eventually tweak with templatable output, you can even make a template that generates the log as ready to roll sql statements you can just cat into mysql. We have event_socket + channel_hangup event which has all the same info that the cdr_csv uses to generate the file. We have the XML cdr that is a 3 dimensional view of the call with an account of nearly everything that happend including each application executed, branching records when the call is transferred complete with a timestamp for each occurance. --"However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc) so that it meets the peculiarities of every different project." A single cron entry that does kill -HUP `cat freeswitch.pid` and you have automatic rotation in unix. In windows there are several cron equivs and instead of kill you can use an xml-rpc script to call the FSAPI call "fsctl send_sighup" also you can schedule it to happen automaticly with the FSAPI sched_api @300 mygroup fsctl send_sighup A script connected to event socket subscribed to channel_hangup can act as your DB cdr engine and has the bonus that it could connect to N boxes at once as well as be able to tell if the system was in trouble by listening for system heartbeat events and or socket disconnect. Moral of the story, there is more already there than first appears. I've always hated the M$ paper clip so I strive not to bring him back. "It looks like you are trying to run a VoIP termination company! Shall I assume how you want your call details?" On Thu, Oct 30, 2008 at 7:43 AM, regs@kinetix.gr <regs@kinetix.gr> wrote: I'll try some tests with various combinations first and then decide what's best. I have to say that I was astonished to find out that freeswitch had so many event handlers from the very beginning. However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc) so that it meets the peculiarities of every different project. That would make a big difference compared to other softswitch solutions where the lack of such features prohibits people from using them (especially in the carrier grade level). User feedback (and wish lists) is the key for the success of an open source (or not) project... Thank you all for your replies. You' ve been very helpful. David Knell wrote: The sleep's done each time the directory's empty, not each time a file's written. File open and close are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you process it. We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also on the local machine, from which they're simply copied to the main CDR processor. It performance simply isn't worth worrying about. --Dave "sleep for a couple of seconds" But then you could only insert 1800 cdrs per hour... If I was to insert 36000 cdrs per hour this means that I have to open parse close 10 files per second. Imagine the I/O penalty just for opening - closing the file. (the persing is the same for both situations) David Knell wrote: regs@kinetix.gr wrote: Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it? Not that you'd notice. We run XML CDR to database scripting on each box that we use for switching, and it's a pretty trivial task compared with switching all that media. Doing it this way is:- (a) distributed - one process per box scales nicely; (b) robust - script down, DB down, no problem: files just queue up; (c) simple - the script logic is trivial: - while 1 - for each file in the XML CDR directory - open it - parse it (XML::Simple for us) - insert it in to the DB - delete it - sleep for a couple of seconds Two error cases: can't parse or can't find data which should be there: move the file in to another directory to be examined by real eyes; DB insert fails: break out of inner loop and it'll be retried after a short pause. --Dave ________________________________ _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- David Knell, Director, 3C Limited T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031 http://www.3c.co.uk ________________________________ _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Don't get me wrong. I am not trying to undermine the excellent work that has already been done on the event handling modules of FS. I myself am a big fun of modular constructs in order to achieve something complicated (e.g. the UNIX way). But there are some situations that we will always need some extra features in order to accomplish our goal. As I wrote before, my current system is using the cdr-push (from the gateway side) method to gather my cdrs. Many billing systems and many terminating providers are satisfied with that model alone. I am not. In the end of each hour I am gathering all the cdrs and checking them one by one against my database in order to verify that ALL of my cdrs were handled by my radius servers. In my future system (where a more batch-like mode is preferable) I am "forced" to use a cdr-pull method. My billing system will be responisble for "pulling" the cdrs from the gateways and then process them. So if we have a look at the event handling weaponry of FS the following modules cannot meet me needs : mod_event_multicast mod_event socket mod_radius_cdr mod_xmpp_event because they all rely on a mechanism where the gateway is pushing my cdrs to my billing system (and you need a checking mechanism to verify that all the cdrs were handled). So now, lets have a look at the alternatives : mod_xml_cdr : this was my first choice as it had all the info needed (even more). But the cdr per file approach proved out to be inadequate in terms of performance. I wrote a few lines of code (in perl) that did a listing of the directory, parsed the cdrs using XML::Simple (without even doing anything meaningful, e.g. checks, inserts to a database etc.) and I was not able to make my system parse more than a 30 cdrs/sec (reference 1000 USD average system, not drawing any conclusions, don't get me wrong). Also while the parser was running I had a high cpu utilization (as expected) that was competing with my FS service. Wouldn't that make a bad impact of my freeswitch performance? so I went to a different module mod_cdr_csv : the text .csv format with the plethora of attributes was perfect for my application but it would be a daunting task for me to make a script tha purges the Master csv while it was being used by FS. I googled the issue and found about the HUP method and the rotate directive in the config. So far so good. But it lacked some functionality. All I was trying to state in my previous messages was that it would be nice if the rotation was being initiated by FS, and maybe have different behavior depending on another directive or something. Let me explain what I mean. There are some people who don't care about what cdrs exist in a given rotated log file, as long as a) no cdrs get lost in the process of rotation , b) no duplicates exist. I can see that the HUP method satisfied this need. There is another group of people that DO care about what cdrs are in a rotated log file. E.g. : The file with the name Master.csv.2008-01-01-09-00-00 would only contain cdrs that were terminated from "2008-01-01 09:00:00" to "2008-01-01 09:59:59" if an hourly setup was desirable or from "2008-01-01 09:00:00" to "2008-01-01 09:04:59" if a five-minutes setup was desirable. That need is not covered by the current HUP method because some cdrs might "escape" from the next to the previous file due to the fact that there is a delay between cron executing the HUP and FS doing the file log rotation. In simple words : I am getting my job done with FS they way it already is (no question about that). BUT, I (or someone with different needs) could do his/her job better if some minor features were present. I am not saying that you should should embrace modules that meet the average needs yet sacrifice flexibility. I am merely suggesting extending the flexibility of the already existing ones. Put some more lego tiles in your box set :) Michael Collins wrote:
/me sends Anthony’s post to the printer to be laminated and framed… J
------------------------------------------------------------------------
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Anthony Minessale *Sent:* Thursday, October 30, 2008 6:10 AM *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] mod_cdr revival (or new module maybe)
Actually,
The goal is to not limit the functionality by over thinking how things will be used but to provide building blocks to make much more possible. Our analogy for this is that if you take the common lego reference, If you have something cool built out of legos but they are super glued together it limits what you *could* have done had then not been.
The system that David described is indeed ideal. I have mentioned it more than once and its no coincidence that FreeSWITCH plays into that model to a tee. I do not mention it very often because I think understanding that concept alone is valuable advice that I'd prefer not to waste on deaf ears so I recommend you heed his suggestion.
Logging direct to DB is not a sin or anything but it's just not very reliable. We have so many ways to process call records as it is. you can use the cdr_csv if you want legacy support from asterisk that you can eventually tweak with templatable output, you can even make a template that generates the log as ready to roll sql statements you can just cat into mysql.
We have event_socket + channel_hangup event which has all the same info that the cdr_csv uses to generate the file. We have the XML cdr that is a 3 dimensional view of the call with an account of nearly everything that happend including each application executed, branching records when the call is transferred complete with a timestamp for each occurance.
--"However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc) so that it meets the peculiarities of every different project."
A single cron entry that does kill -HUP `cat freeswitch.pid` and you have automatic rotation in unix. In windows there are several cron equivs and instead of kill you can use an xml-rpc script to call the FSAPI call "fsctl send_sighup" also you can schedule it to happen automaticly with the FSAPI
sched_api @300 mygroup fsctl send_sighup
A script connected to event socket subscribed to channel_hangup can act as your DB cdr engine and has the bonus that it could connect to N boxes at once as well as be able to tell if the system was in trouble by listening for system heartbeat events and or socket disconnect.
Moral of the story, there is more already there than first appears. I've always hated the M$ paper clip so I strive not to bring him back. "It looks like you are trying to run a VoIP termination company! Shall I assume how you want your call details?"
On Thu, Oct 30, 2008 at 7:43 AM, regs@kinetix.gr <mailto:regs@kinetix.gr> <regs@kinetix.gr <mailto:regs@kinetix.gr>> wrote:
I'll try some tests with various combinations first and then decide what's best.
I have to say that I was astonished to find out that freeswitch had so many event handlers from the very beginning. However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc) so that it meets the peculiarities of every different project. That would make a big difference compared to other softswitch solutions where the lack of such features prohibits people from using them (especially in the carrier grade level). User feedback (and wish lists) is the key for the success of an open source (or not) project...
Thank you all for your replies. You' ve been very helpful.
David Knell wrote:
The sleep's done each time the directory's empty, not each time a file's written. File open and close are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you process it.
We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also on the local machine, from which they're simply copied to the main CDR processor.
It performance simply isn't worth worrying about.
--Dave
"sleep for a couple of seconds"
But then you could only insert 1800 cdrs per hour... If I was to insert 36000 cdrs per hour this means that I have to open parse close 10 files per second. Imagine the I/O penalty just for opening - closing the file. (the persing is the same for both situations)
David Knell wrote:
regs@kinetix.gr <mailto:regs@kinetix.gr> wrote:
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it?
Not that you'd notice. We run XML CDR to database scripting on each box that we use for switching, and it's a pretty trivial task compared with switching all that media. Doing it this way is:- (a) distributed - one process per box scales nicely; (b) robust - script down, DB down, no problem: files just queue up; (c) simple - the script logic is trivial: - while 1 - for each file in the XML CDR directory - open it - parse it (XML::Simple for us) - insert it in to the DB - delete it - sleep for a couple of seconds Two error cases: can't parse or can't find data which should be there: move the file in to another directory to be examined by real eyes; DB insert fails: break out of inner loop and it'll be retried after a short pause.
--Dave
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- David Knell, Director, 3C Limited T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031 http://www.3c.co.uk
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 <http://iax:guest@conference.freeswitch.org/888> googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Don't worry, I don't think you are undermining anything. It's just a discussion thread. If Yossi gets mod_cdr back up to speed we will happily accept it. BTW, *hint* You use the perl script to turn the current batch of xml files into a single file or array of sql stmts. then you pass that on to a single insert / update transaction. Your realtime insert into the db idea will suffer the same slow effect of single transaction per insert which is your bottleneck. But it will scale rather far before it becomes a problem. The bigger issue is the chicken or the egg on what to do when the db is down. File system is the safest place to put the data cos if it goes, so does you box and there is little chance of dispare apart from filling it up and you have the ability to reprocess the files again if the db dies. Single file per record solves file locking problems and the xml data is so big anyway you would want to seggregate it. Like I said I don't go on giving out hints all the time so that's the last from me for now. ;) On Thu, Oct 30, 2008 at 3:27 PM, regs@kinetix.gr <regs@kinetix.gr> wrote:
Don't get me wrong. I am not trying to undermine the excellent work that has already been done on the event handling modules of FS.
I myself am a big fun of modular constructs in order to achieve something complicated (e.g. the UNIX way).
But there are some situations that we will always need some extra features in order to accomplish our goal.
As I wrote before, my current system is using the cdr-push (from the gateway side) method to gather my cdrs. Many billing systems and many terminating providers are satisfied with that model alone. I am not. In the end of each hour I am gathering all the cdrs and checking them one by one against my database in order to verify that ALL of my cdrs were handled by my radius servers.
In my future system (where a more batch-like mode is preferable) I am "forced" to use a cdr-pull method. My billing system will be responisble for "pulling" the cdrs from the gateways and then process them. So if we have a look at the event handling weaponry of FS the following modules cannot meet me needs :
mod_event_multicast mod_event socket mod_radius_cdr mod_xmpp_event
because they all rely on a mechanism where the gateway is pushing my cdrs to my billing system (and you need a checking mechanism to verify that all the cdrs were handled).
So now, lets have a look at the alternatives :
mod_xml_cdr : this was my first choice as it had all the info needed (even more). But the cdr per file approach proved out to be inadequate in terms of performance. I wrote a few lines of code (in perl) that did a listing of the directory, parsed the cdrs using XML::Simple (without even doing anything meaningful, e.g. checks, inserts to a database etc.) and I was not able to make my system parse more than a 30 cdrs/sec (reference 1000 USD average system, not drawing any conclusions, don't get me wrong). Also while the parser was running I had a high cpu utilization (as expected) that was competing with my FS service. Wouldn't that make a bad impact of my freeswitch performance?
so I went to a different module
mod_cdr_csv : the text .csv format with the plethora of attributes was perfect for my application but it would be a daunting task for me to make a script tha purges the Master csv while it was being used by FS. I googled the issue and found about the HUP method and the rotate directive in the config. So far so good. But it lacked some functionality.
All I was trying to state in my previous messages was that it would be nice if the rotation was being initiated by FS, and maybe have different behavior depending on another directive or something. Let me explain what I mean.
There are some people who don't care about what cdrs exist in a given rotated log file, as long as a) no cdrs get lost in the process of rotation , b) no duplicates exist. I can see that the HUP method satisfied this need.
There is another group of people that DO care about what cdrs are in a rotated log file. E.g. : The file with the name Master.csv.2008-01-01-09-00-00 would only contain cdrs that were terminated from "2008-01-01 09:00:00" to "2008-01-01 09:59:59" if an hourly setup was desirable or from "2008-01-01 09:00:00" to "2008-01-01 09:04:59" if a five-minutes setup was desirable. That need is not covered by the current HUP method because some cdrs might "escape" from the next to the previous file due to the fact that there is a delay between cron executing the HUP and FS doing the file log rotation.
In simple words : I am getting my job done with FS they way it already is (no question about that). BUT, I (or someone with different needs) could do his/her job better if some minor features were present.
I am not saying that you should should embrace modules that meet the average needs yet sacrifice flexibility. I am merely suggesting extending the flexibility of the already existing ones. Put some more lego tiles in your box set :)
Michael Collins wrote:
/me sends Anthony's post to the printer to be laminated and framed… J
------------------------------------------------------------------------
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Anthony Minessale *Sent:* Thursday, October 30, 2008 6:10 AM *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] mod_cdr revival (or new module maybe)
Actually,
The goal is to not limit the functionality by over thinking how things will be used but to provide building blocks to make much more possible. Our analogy for this is that if you take the common lego reference, If you have something cool built out of legos but they are super glued together it limits what you *could* have done had then not been.
The system that David described is indeed ideal. I have mentioned it more than once and its no coincidence that FreeSWITCH plays into that model to a tee. I do not mention it very often because I think understanding that concept alone is valuable advice that I'd prefer not to waste on deaf ears so I recommend you heed his suggestion.
Logging direct to DB is not a sin or anything but it's just not very reliable. We have so many ways to process call records as it is. you can use the cdr_csv if you want legacy support from asterisk that you can eventually tweak with templatable output, you can even make a template that generates the log as ready to roll sql statements you can just cat into mysql.
We have event_socket + channel_hangup event which has all the same info that the cdr_csv uses to generate the file. We have the XML cdr that is a 3 dimensional view of the call with an account of nearly everything that happend including each application executed, branching records when the call is transferred complete with a timestamp for each occurance.
--"However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc) so that it meets the peculiarities of every different project."
A single cron entry that does kill -HUP `cat freeswitch.pid` and you have automatic rotation in unix. In windows there are several cron equivs and instead of kill you can use an xml-rpc script to call the FSAPI call "fsctl send_sighup" also you can schedule it to happen automaticly with the FSAPI
sched_api @300 mygroup fsctl send_sighup
A script connected to event socket subscribed to channel_hangup can act as your DB cdr engine and has the bonus that it could connect to N boxes at once as well as be able to tell if the system was in trouble by listening for system heartbeat events and or socket disconnect.
Moral of the story, there is more already there than first appears. I've always hated the M$ paper clip so I strive not to bring him back. "It looks like you are trying to run a VoIP termination company! Shall I assume how you want your call details?"
On Thu, Oct 30, 2008 at 7:43 AM, regs@kinetix.gr <mailto:regs@kinetix.gr> <regs@kinetix.gr <mailto:regs@kinetix.gr>>
wrote:
I'll try some tests with various combinations first and then decide what's best.
I have to say that I was astonished to find out that freeswitch had so many event handlers from the very beginning. However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc) so that it meets the peculiarities of every different project. That would make a big difference compared to other softswitch solutions where the lack of such features prohibits people from using them (especially in the carrier grade level). User feedback (and wish lists) is the key for the success of an open source (or not) project...
Thank you all for your replies. You' ve been very helpful.
David Knell wrote:
The sleep's done each time the directory's empty, not each time a file's written. File open and close are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you process it.
We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also on the local machine, from which they're simply copied to the main CDR processor.
It performance simply isn't worth worrying about.
--Dave
"sleep for a couple of seconds"
But then you could only insert 1800 cdrs per hour... If I was to insert 36000 cdrs per hour this means that I have to open parse close 10 files per second. Imagine the I/O penalty just for opening - closing
the file.
(the persing is the same for both situations)
David Knell wrote:
regs@kinetix.gr <mailto:regs@kinetix.gr> wrote:
Yes, the xml files give you tons of info... but isn't it a little insufficient - performance wise - to open and close so many files in such a little time. In a PBX environment that wouldn't be an issue but if we get to the small-voip-carrier level (some thousand cdrs per hour) that could slow things down considerably, wouldn't it?
Not that you'd notice. We run XML CDR to database scripting on each box that we use for switching, and it's a pretty trivial task compared with switching all that media. Doing it this way is:- (a) distributed - one process per box scales nicely; (b) robust - script down, DB down, no problem: files just queue up; (c) simple - the script logic is trivial: - while 1 - for each file in the XML CDR directory - open it - parse it (XML::Simple for us) - insert it in to the DB - delete it - sleep for a couple of seconds Two error cases: can't parse or can't find data which should be there: move the file in to another directory to be examined by real eyes; DB insert fails: break out of inner loop and it'll be retried after a short pause.
--Dave
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto: Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- David Knell, Director, 3C Limited T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031 http://www.3c.co.uk
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto: Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> <mailto:MSN%3Aanthony_minessale@hotmail.com<MSN%253Aanthony_minessale@hotmail.com>
GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> <mailto:PAYPAL%3Aanthony.minessale@gmail.com<PAYPAL%253Aanthony.minessale@gmail.com>
IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> <mailto:sip%3A888@conference.freeswitch.org<sip%253A888@conference.freeswitch.org>
iax:guest@conference.freeswitch.org/888 <http://iax:guest@conference.freeswitch.org/888> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org<googletalk%253Aconf%252B888@conference.freeswitch.org>
pstn:213-799-1400
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Yes, I agree. But one could use the two methods combined (csv or xml + db) for redundancy. Is there any consideration regarding automatic log rotation (e.g. hourly, or user specified) without the need of a HUP? Now, that could make things a lot easier for the development of an external csv to db aggregation script because the script would read from a closed (not used by freeswitch at the time) CDRs file. And the developer could be sure that the cdrs contained in that file would have a hangup timestamp that could be described by the filename (e.g. 20080101_010000.csv). Michael Jerris wrote:
Unsure at this time. There has been some work on mod_cdr_odbc. We generally advise against direct to db cdr methods without a very robust backup method for when the db is down.
On Oct 29, 2008, at 9:57 AM, "regs@kinetix.gr" <regs@kinetix.gr> wrote:
Hi,
I saw in the wiki that the mod_cdr module is now unsupported. There is also a note about a revival of the module. I would like to ask the following :
What is the current state of the revival process? (should we expect something in the near future?)
Will it have the same functionality as before (DB support for instance)?
Are there any plans for a brand new database specific event handler module?
It would be great if there was one so that developers (especially those who develop billing applications) would not have to create their own hacks (cron scripts etc.)
Thank you for your time,
-- ------------------------------------------- Apostolos Pantsiopoulos Kinetix Tele.com Support Center
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (6)
-
Anthony Minessale
-
David Knell
-
Michael Collins
-
Michael Jerris
-
regs＠kinetix.gr
-
Shawn Lewis