FreeSwitch Out of threads!
Hello, I decided to do a bit of "stability" testing leaving 100 sessions calling a playback "tone_stream://$${tetris}" on my dual-core CPU with 4G of RAM during a few hours. After 5 hours working flawelessly, I started getting lots of: 2010-10-04 23:53:14.188596 [CRIT] switch_event.c:342 Out of threads! Later combined with: 2010-10-04 23:59:02.692961 [CRIT] switch_event.c:325 Event system overloading Both messages (the second one at an increased rate) followed by a few: 2010-10-05 00:25:43.711929 [CRIT] switch_time.c:755 Virtual Migration Detected! Syncing Clock Just before a crash where this time I did not get a core (??)... Any idea what I could have done wrong? François. PS: This was NOT intended to be a stress test, I just wanted to check how FS would handle at low-medium load during a few hours.
François, are the 100 sessions the same from the beginning to the end ? If yes, I am afraid this is going to be considered as a stress test, as this would not happen with live traffic. David Ponzone Direction Technique email: david.ponzone@ipeva.fr tel: 01 74 03 18 97 gsm: 06 66 98 76 34 Service Client IPeva tel: 0811 46 26 26 www.ipeva.fr - www.ipeva-studio.com Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. IPeva décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur. Le 05/10/2010 à 10:19, François Delawarde a écrit :
Hello,
I decided to do a bit of "stability" testing leaving 100 sessions calling a playback "tone_stream://$${tetris}" on my dual-core CPU with 4G of RAM during a few hours.
After 5 hours working flawelessly, I started getting lots of:
2010-10-04 23:53:14.188596 [CRIT] switch_event.c:342 Out of threads!
Later combined with:
2010-10-04 23:59:02.692961 [CRIT] switch_event.c:325 Event system overloading
Both messages (the second one at an increased rate) followed by a few:
2010-10-05 00:25:43.711929 [CRIT] switch_time.c:755 Virtual Migration Detected! Syncing Clock
Just before a crash where this time I did not get a core (??)...
Any idea what I could have done wrong?
François. PS: This was NOT intended to be a stress test, I just wanted to check how FS would handle at low-medium load during a few hours.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Did you increase the number of file descriptors? Regards HASSAN On 2010-10-05, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
Hello,
I decided to do a bit of "stability" testing leaving 100 sessions calling a playback "tone_stream://$${tetris}" on my dual-core CPU with 4G of RAM during a few hours.
After 5 hours working flawelessly, I started getting lots of:
2010-10-04 23:53:14.188596 [CRIT] switch_event.c:342 Out of threads!
Later combined with:
2010-10-04 23:59:02.692961 [CRIT] switch_event.c:325 Event system overloading
Both messages (the second one at an increased rate) followed by a few:
2010-10-05 00:25:43.711929 [CRIT] switch_time.c:755 Virtual Migration Detected! Syncing Clock
Just before a crash where this time I did not get a core (??)...
Any idea what I could have done wrong?
François. PS: This was NOT intended to be a stress test, I just wanted to check how FS would handle at low-medium load during a few hours.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sent from my mobile device
Please use 64bit and ulimit -c unlimited ulimit -d unlimited ulimit -f unlimited ulimit -i unlimited ulimit -n 999999 ulimit -q unlimited ulimit -u unlimited ulimit -v unlimited ulimit -x unlimited ulimit -s 240 ulimit -l unlimited before start freeswitch 2010/10/5 Nyamul Hassan <mnhassan@usa.net>:
Did you increase the number of file descriptors?
Regards HASSAN
On 2010-10-05, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
Hello,
I decided to do a bit of "stability" testing leaving 100 sessions calling a playback "tone_stream://$${tetris}" on my dual-core CPU with 4G of RAM during a few hours.
After 5 hours working flawelessly, I started getting lots of:
2010-10-04 23:53:14.188596 [CRIT] switch_event.c:342 Out of threads!
Later combined with:
2010-10-04 23:59:02.692961 [CRIT] switch_event.c:325 Event system overloading
Both messages (the second one at an increased rate) followed by a few:
2010-10-05 00:25:43.711929 [CRIT] switch_time.c:755 Virtual Migration Detected! Syncing Clock
Just before a crash where this time I did not get a core (??)...
Any idea what I could have done wrong?
François. PS: This was NOT intended to be a stress test, I just wanted to check how FS would handle at low-medium load during a few hours.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sent from my mobile device
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I maintain "only" 100 sessions at all times, with a duration of 20-30s, and no more than 10cps, so it is NOT a stress test, come on! I mean this would not even load an Asterisk server, and loading the server was not an objective, just to maintain low-medium load over a period of time. It worked very well during 5h. Why would I need to ulimit the crap out of my server then? :-) François. On Tue, 2010-10-05 at 16:10 +0700, Dome Charoenyost wrote:
Please use 64bit and ulimit -c unlimited ulimit -d unlimited ulimit -f unlimited ulimit -i unlimited ulimit -n 999999 ulimit -q unlimited ulimit -u unlimited ulimit -v unlimited ulimit -x unlimited ulimit -s 240 ulimit -l unlimited
before start freeswitch
2010/10/5 Nyamul Hassan <mnhassan@usa.net>:
Did you increase the number of file descriptors?
Regards HASSAN
On 2010-10-05, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
Hello,
I decided to do a bit of "stability" testing leaving 100 sessions calling a playback "tone_stream://$${tetris}" on my dual-core CPU with 4G of RAM during a few hours.
After 5 hours working flawelessly, I started getting lots of:
2010-10-04 23:53:14.188596 [CRIT] switch_event.c:342 Out of threads!
Later combined with:
2010-10-04 23:59:02.692961 [CRIT] switch_event.c:325 Event system overloading
Both messages (the second one at an increased rate) followed by a few:
2010-10-05 00:25:43.711929 [CRIT] switch_time.c:755 Virtual Migration Detected! Syncing Clock
Just before a crash where this time I did not get a core (??)...
Any idea what I could have done wrong?
François. PS: This was NOT intended to be a stress test, I just wanted to check how FS would handle at low-medium load during a few hours.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sent from my mobile device
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I maintain "only" 100 sessions at all times, with a duration of 20-30s, and no more than 10cps, so it is NOT a stress test, come on!
I mean this would not even load an Asterisk server, and loading the server was not an objective, just to maintain low-medium load over a period of time. It worked very well during 5h.
Why would I need to ulimit the crap out of my server then? :-)
Don't feel alone, you're not the only one. We're getting the same pair of messages. We know what it means, we have just not found the cause of it. In our case happens quite fast (about an hour IIRC) when doing stress testing of about 900 calls at a time lasting 10 seconds each at a rate of about 40cps. Moises Silva Senior Software Engineer Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
We learned from another thread about how Sangoma regularly tests 32 E1 setups in the lab, and were preparing a concept paper to show a telecom that traditional vendors can be replaced using open source even for large scale implementations. Does this mean we should rethink our strategy? Regards HASSAN On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote:
On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I maintain "only" 100 sessions at all times, with a duration of 20-30s, and no more than 10cps, so it is NOT a stress test, come on!
I mean this would not even load an Asterisk server, and loading the server was not an objective, just to maintain low-medium load over a period of time. It worked very well during 5h.
Why would I need to ulimit the crap out of my server then? :-)
Don't feel alone, you're not the only one.
We're getting the same pair of messages. We know what it means, we have just not found the cause of it. In our case happens quite fast (about an hour IIRC) when doing stress testing of about 900 calls at a time lasting 10 seconds each at a rate of about 40cps.
Moises Silva Senior Software Engineer Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sent from my mobile device
On Tue, Oct 5, 2010 at 10:35 AM, Nyamul Hassan <mnhassan@usa.net> wrote:
We learned from another thread about how Sangoma regularly tests 32 E1 setups in the lab, and were preparing a concept paper to show a telecom that traditional vendors can be replaced using open source even for large scale implementations. Does this mean we should rethink our strategy?
FreeSWITCH is under constant development and the performance behavior may change from the git of one month ago, to the latest one. In short, stress testing is tricky, it's just matter of finding the right balance. Do not expect blindly checking out latest freeswitch git any day and always have bug free software (if such thing exists I'd like to see it). You need to do your homework for you particular implementation. Moises Silva Senior Software Engineer Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
I wouldn't jump to conclusions that fast, it's just a bug with current git head, probably only happening in very specific cases. If I learned anything from hollywood movies, it would be that bugs or bad people always end up being tracked down and destroyed. François. On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote:
We learned from another thread about how Sangoma regularly tests 32 E1 setups in the lab, and were preparing a concept paper to show a telecom that traditional vendors can be replaced using open source even for large scale implementations. Does this mean we should rethink our strategy?
Regards HASSAN
On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote:
On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I maintain "only" 100 sessions at all times, with a duration of 20-30s, and no more than 10cps, so it is NOT a stress test, come on!
I mean this would not even load an Asterisk server, and loading the server was not an objective, just to maintain low-medium load over a period of time. It worked very well during 5h.
Why would I need to ulimit the crap out of my server then? :-)
Don't feel alone, you're not the only one.
We're getting the same pair of messages. We know what it means, we have just not found the cause of it. In our case happens quite fast (about an hour IIRC) when doing stress testing of about 900 calls at a time lasting 10 seconds each at a rate of about 40cps.
Moises Silva Senior Software Engineer Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
If there is a bug why are you not opening it on JIRA? http://jira.freeswitch.org We are doing work on improving the event system to be faster and use less memory. If it caused some problems, that is not uncommon, software development requires testing. The right thing to do is to report it not to make underhanded comments about rethinking your strategy. Ideally I would like to visit the machine while it's in that state or otherwise I would like you to get a gcore while it's int that state, open it in gdb and produce a "thread apply all bt" trace in text format and attach it to the JIRA. On Tue, Oct 5, 2010 at 9:55 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I wouldn't jump to conclusions that fast, it's just a bug with current git head, probably only happening in very specific cases.
If I learned anything from hollywood movies, it would be that bugs or bad people always end up being tracked down and destroyed.
François.
On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote:
We learned from another thread about how Sangoma regularly tests 32 E1 setups in the lab, and were preparing a concept paper to show a telecom that traditional vendors can be replaced using open source even for large scale implementations. Does this mean we should rethink our strategy?
Regards HASSAN
On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote:
On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I maintain "only" 100 sessions at all times, with a duration of 20-30s, and no more than 10cps, so it is NOT a stress test, come on!
I mean this would not even load an Asterisk server, and loading the server was not an objective, just to maintain low-medium load over a period of time. It worked very well during 5h.
Why would I need to ulimit the crap out of my server then? :-)
Don't feel alone, you're not the only one.
We're getting the same pair of messages. We know what it means, we have just not found the cause of it. In our case happens quite fast (about an hour IIRC) when doing stress testing of about 900 calls at a time lasting 10 seconds each at a rate of about 40cps.
Moises Silva Senior Software Engineer Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
BTW, If you are any less than this revision don't bother to report it until you have tested latest GIT commit b02c69bba9f46cad37225a0986cae068f24dcd81 Author: Anthony Minessale <anthm@freeswitch.org> Date: Wed Sep 22 18:14:24 2010 -0500 On Tue, Oct 5, 2010 at 10:26 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
If there is a bug why are you not opening it on JIRA? http://jira.freeswitch.org
We are doing work on improving the event system to be faster and use less memory.
If it caused some problems, that is not uncommon, software development requires testing. The right thing to do is to report it not to make underhanded comments about rethinking your strategy.
Ideally I would like to visit the machine while it's in that state or otherwise I would like you to get a gcore while it's int that state, open it in gdb and produce a "thread apply all bt" trace in text format and attach it to the JIRA.
On Tue, Oct 5, 2010 at 9:55 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I wouldn't jump to conclusions that fast, it's just a bug with current git head, probably only happening in very specific cases.
If I learned anything from hollywood movies, it would be that bugs or bad people always end up being tracked down and destroyed.
François.
On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote:
We learned from another thread about how Sangoma regularly tests 32 E1 setups in the lab, and were preparing a concept paper to show a telecom that traditional vendors can be replaced using open source even for large scale implementations. Does this mean we should rethink our strategy?
Regards HASSAN
On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote:
On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I maintain "only" 100 sessions at all times, with a duration of 20-30s, and no more than 10cps, so it is NOT a stress test, come on!
I mean this would not even load an Asterisk server, and loading the server was not an objective, just to maintain low-medium load over a period of time. It worked very well during 5h.
Why would I need to ulimit the crap out of my server then? :-)
Don't feel alone, you're not the only one.
We're getting the same pair of messages. We know what it means, we have just not found the cause of it. In our case happens quite fast (about an hour IIRC) when doing stress testing of about 900 calls at a time lasting 10 seconds each at a rate of about 40cps.
Moises Silva Senior Software Engineer Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
GIT from 1st of October. François. On Tue, 2010-10-05 at 10:28 -0500, Anthony Minessale wrote:
BTW,
If you are any less than this revision don't bother to report it until you have tested latest GIT
commit b02c69bba9f46cad37225a0986cae068f24dcd81 Author: Anthony Minessale <anthm@freeswitch.org> Date: Wed Sep 22 18:14:24 2010 -0500
On Tue, Oct 5, 2010 at 10:26 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
If there is a bug why are you not opening it on JIRA? http://jira.freeswitch.org
We are doing work on improving the event system to be faster and use less memory.
If it caused some problems, that is not uncommon, software development requires testing. The right thing to do is to report it not to make underhanded comments about rethinking your strategy.
Ideally I would like to visit the machine while it's in that state or otherwise I would like you to get a gcore while it's int that state, open it in gdb and produce a "thread apply all bt" trace in text format and attach it to the JIRA.
On Tue, Oct 5, 2010 at 9:55 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I wouldn't jump to conclusions that fast, it's just a bug with current git head, probably only happening in very specific cases.
If I learned anything from hollywood movies, it would be that bugs or bad people always end up being tracked down and destroyed.
François.
On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote:
We learned from another thread about how Sangoma regularly tests 32 E1 setups in the lab, and were preparing a concept paper to show a telecom that traditional vendors can be replaced using open source even for large scale implementations. Does this mean we should rethink our strategy?
Regards HASSAN
On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote:
On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I maintain "only" 100 sessions at all times, with a duration of 20-30s, and no more than 10cps, so it is NOT a stress test, come on!
I mean this would not even load an Asterisk server, and loading the server was not an objective, just to maintain low-medium load over a period of time. It worked very well during 5h.
Why would I need to ulimit the crap out of my server then? :-)
Don't feel alone, you're not the only one.
We're getting the same pair of messages. We know what it means, we have just not found the cause of it. In our case happens quite fast (about an hour IIRC) when doing stress testing of about 900 calls at a time lasting 10 seconds each at a rate of about 40cps.
Moises Silva Senior Software Engineer Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
if you are on centos or something like it. you can run this from the FS build root. ./support-d/fscore_pb gcore <your nick> while FS is running in this state. That will give you a URL you can tell me, it will upload the trace I need to see right to our pastebin. If you can get it that way, and add our ssh key from ./support-d/shinzon.pub, install it and email or irc pm me the hostname. And I can examine it. On Tue, Oct 5, 2010 at 10:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
GIT from 1st of October.
François.
On Tue, 2010-10-05 at 10:28 -0500, Anthony Minessale wrote:
BTW,
If you are any less than this revision don't bother to report it until you have tested latest GIT
commit b02c69bba9f46cad37225a0986cae068f24dcd81 Author: Anthony Minessale <anthm@freeswitch.org> Date: Wed Sep 22 18:14:24 2010 -0500
On Tue, Oct 5, 2010 at 10:26 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
If there is a bug why are you not opening it on JIRA? http://jira.freeswitch.org
We are doing work on improving the event system to be faster and use less memory.
If it caused some problems, that is not uncommon, software development requires testing. The right thing to do is to report it not to make underhanded comments about rethinking your strategy.
Ideally I would like to visit the machine while it's in that state or otherwise I would like you to get a gcore while it's int that state, open it in gdb and produce a "thread apply all bt" trace in text format and attach it to the JIRA.
On Tue, Oct 5, 2010 at 9:55 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I wouldn't jump to conclusions that fast, it's just a bug with current git head, probably only happening in very specific cases.
If I learned anything from hollywood movies, it would be that bugs or bad people always end up being tracked down and destroyed.
François.
On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote:
We learned from another thread about how Sangoma regularly tests 32 E1 setups in the lab, and were preparing a concept paper to show a telecom that traditional vendors can be replaced using open source even for large scale implementations. Does this mean we should rethink our strategy?
Regards HASSAN
On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote:
On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote: > I maintain "only" 100 sessions at all times, with a duration of 20-30s, > and no more than 10cps, so it is NOT a stress test, come on! > > I mean this would not even load an Asterisk server, and loading the > server was not an objective, just to maintain low-medium load over a > period of time. It worked very well during 5h. > > Why would I need to ulimit the crap out of my server then? :-)
Don't feel alone, you're not the only one.
We're getting the same pair of messages. We know what it means, we have just not found the cause of it. In our case happens quite fast (about an hour IIRC) when doing stress testing of about 900 calls at a time lasting 10 seconds each at a rate of about 40cps.
Moises Silva Senior Software Engineer Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Can you update first to this revision of HEAD or later? commit 80883ebeb81c73a3dc5c4ee65050d57e2c6ccf00 Author: Anthony Minessale <anthm@freeswitch.org> Date: Tue Oct 5 11:11:21 2010 -0500 I added a small patch that I think may be related to your problem. On Tue, Oct 5, 2010 at 10:59 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
if you are on centos or something like it. you can run this from the FS build root.
./support-d/fscore_pb gcore <your nick>
while FS is running in this state.
That will give you a URL you can tell me, it will upload the trace I need to see right to our pastebin.
If you can get it that way, and add our ssh key from ./support-d/shinzon.pub, install it and email or irc pm me the hostname. And I can examine it.
On Tue, Oct 5, 2010 at 10:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
GIT from 1st of October.
François.
On Tue, 2010-10-05 at 10:28 -0500, Anthony Minessale wrote:
BTW,
If you are any less than this revision don't bother to report it until you have tested latest GIT
commit b02c69bba9f46cad37225a0986cae068f24dcd81 Author: Anthony Minessale <anthm@freeswitch.org> Date: Wed Sep 22 18:14:24 2010 -0500
On Tue, Oct 5, 2010 at 10:26 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
If there is a bug why are you not opening it on JIRA? http://jira.freeswitch.org
We are doing work on improving the event system to be faster and use less memory.
If it caused some problems, that is not uncommon, software development requires testing. The right thing to do is to report it not to make underhanded comments about rethinking your strategy.
Ideally I would like to visit the machine while it's in that state or otherwise I would like you to get a gcore while it's int that state, open it in gdb and produce a "thread apply all bt" trace in text format and attach it to the JIRA.
On Tue, Oct 5, 2010 at 9:55 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I wouldn't jump to conclusions that fast, it's just a bug with current git head, probably only happening in very specific cases.
If I learned anything from hollywood movies, it would be that bugs or bad people always end up being tracked down and destroyed.
François.
On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote:
We learned from another thread about how Sangoma regularly tests 32 E1 setups in the lab, and were preparing a concept paper to show a telecom that traditional vendors can be replaced using open source even for large scale implementations. Does this mean we should rethink our strategy?
Regards HASSAN
On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote: > On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde > <fdelawarde@wirelessmundi.com> wrote: >> I maintain "only" 100 sessions at all times, with a duration of 20-30s, >> and no more than 10cps, so it is NOT a stress test, come on! >> >> I mean this would not even load an Asterisk server, and loading the >> server was not an objective, just to maintain low-medium load over a >> period of time. It worked very well during 5h. >> >> Why would I need to ulimit the crap out of my server then? :-) > > Don't feel alone, you're not the only one. > > We're getting the same pair of messages. We know what it means, we > have just not found the cause of it. In our case happens quite fast > (about an hour IIRC) when doing stress testing of about 900 calls at a > time lasting 10 seconds each at a rate of about 40cps. > > Moises Silva > Senior Software Engineer > Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON > L3R 9R6 Canada > t. 1 905 474 1990 x128 | e. moy@sangoma.com > > _______________________________________________ > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
I will certainly tomorrow first thing! Thanks, François. On Tue, 2010-10-05 at 11:17 -0500, Anthony Minessale wrote:
Can you update first to this revision of HEAD or later?
commit 80883ebeb81c73a3dc5c4ee65050d57e2c6ccf00 Author: Anthony Minessale <anthm@freeswitch.org> Date: Tue Oct 5 11:11:21 2010 -0500
I added a small patch that I think may be related to your problem.
On Tue, Oct 5, 2010 at 10:59 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
if you are on centos or something like it. you can run this from the FS build root.
./support-d/fscore_pb gcore <your nick>
while FS is running in this state.
That will give you a URL you can tell me, it will upload the trace I need to see right to our pastebin.
If you can get it that way, and add our ssh key from ./support-d/shinzon.pub, install it and email or irc pm me the hostname. And I can examine it.
On Tue, Oct 5, 2010 at 10:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
GIT from 1st of October.
François.
On Tue, 2010-10-05 at 10:28 -0500, Anthony Minessale wrote:
BTW,
If you are any less than this revision don't bother to report it until you have tested latest GIT
commit b02c69bba9f46cad37225a0986cae068f24dcd81 Author: Anthony Minessale <anthm@freeswitch.org> Date: Wed Sep 22 18:14:24 2010 -0500
On Tue, Oct 5, 2010 at 10:26 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
If there is a bug why are you not opening it on JIRA? http://jira.freeswitch.org
We are doing work on improving the event system to be faster and use less memory.
If it caused some problems, that is not uncommon, software development requires testing. The right thing to do is to report it not to make underhanded comments about rethinking your strategy.
Ideally I would like to visit the machine while it's in that state or otherwise I would like you to get a gcore while it's int that state, open it in gdb and produce a "thread apply all bt" trace in text format and attach it to the JIRA.
On Tue, Oct 5, 2010 at 9:55 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I wouldn't jump to conclusions that fast, it's just a bug with current git head, probably only happening in very specific cases.
If I learned anything from hollywood movies, it would be that bugs or bad people always end up being tracked down and destroyed.
François.
On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote: > We learned from another thread about how Sangoma regularly tests 32 E1 > setups in the lab, and were preparing a concept paper to show a > telecom that traditional vendors can be replaced using open source > even for large scale implementations. Does this mean we should rethink > our strategy? > > Regards > HASSAN > > > On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote: > > On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde > > <fdelawarde@wirelessmundi.com> wrote: > >> I maintain "only" 100 sessions at all times, with a duration of 20-30s, > >> and no more than 10cps, so it is NOT a stress test, come on! > >> > >> I mean this would not even load an Asterisk server, and loading the > >> server was not an objective, just to maintain low-medium load over a > >> period of time. It worked very well during 5h. > >> > >> Why would I need to ulimit the crap out of my server then? :-) > > > > Don't feel alone, you're not the only one. > > > > We're getting the same pair of messages. We know what it means, we > > have just not found the cause of it. In our case happens quite fast > > (about an hour IIRC) when doing stress testing of about 900 calls at a > > time lasting 10 seconds each at a rate of about 40cps. > > > > Moises Silva > > Senior Software Engineer > > Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON > > L3R 9R6 Canada > > t. 1 905 474 1990 x128 | e. moy@sangoma.com > > > > _______________________________________________ > > FreeSWITCH-users mailing list > > FreeSWITCH-users@lists.freeswitch.org > > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > > http://www.freeswitch.org > > >
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
I can't reproduce so far after updating to HEAD, I'll post a Jira issue if it happens again. How do you guys do to always resolve all my problems in a few hours or days at max? I won't believe it anymore if you say you used to be an Asterisk bug marshal (I'm used to wait months for issues to be attended there)... :-) Thanks, François. On Tue, 2010-10-05 at 11:17 -0500, Anthony Minessale wrote:
Can you update first to this revision of HEAD or later?
commit 80883ebeb81c73a3dc5c4ee65050d57e2c6ccf00 Author: Anthony Minessale <anthm@freeswitch.org> Date: Tue Oct 5 11:11:21 2010 -0500
I added a small patch that I think may be related to your problem.
On Tue, Oct 5, 2010 at 10:59 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
if you are on centos or something like it. you can run this from the FS build root.
./support-d/fscore_pb gcore <your nick>
while FS is running in this state.
That will give you a URL you can tell me, it will upload the trace I need to see right to our pastebin.
If you can get it that way, and add our ssh key from ./support-d/shinzon.pub, install it and email or irc pm me the hostname. And I can examine it.
On Tue, Oct 5, 2010 at 10:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
GIT from 1st of October.
François.
On Tue, 2010-10-05 at 10:28 -0500, Anthony Minessale wrote:
BTW,
If you are any less than this revision don't bother to report it until you have tested latest GIT
commit b02c69bba9f46cad37225a0986cae068f24dcd81 Author: Anthony Minessale <anthm@freeswitch.org> Date: Wed Sep 22 18:14:24 2010 -0500
On Tue, Oct 5, 2010 at 10:26 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
If there is a bug why are you not opening it on JIRA? http://jira.freeswitch.org
We are doing work on improving the event system to be faster and use less memory.
If it caused some problems, that is not uncommon, software development requires testing. The right thing to do is to report it not to make underhanded comments about rethinking your strategy.
Ideally I would like to visit the machine while it's in that state or otherwise I would like you to get a gcore while it's int that state, open it in gdb and produce a "thread apply all bt" trace in text format and attach it to the JIRA.
On Tue, Oct 5, 2010 at 9:55 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I wouldn't jump to conclusions that fast, it's just a bug with current git head, probably only happening in very specific cases.
If I learned anything from hollywood movies, it would be that bugs or bad people always end up being tracked down and destroyed.
François.
On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote: > We learned from another thread about how Sangoma regularly tests 32 E1 > setups in the lab, and were preparing a concept paper to show a > telecom that traditional vendors can be replaced using open source > even for large scale implementations. Does this mean we should rethink > our strategy? > > Regards > HASSAN > > > On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote: > > On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde > > <fdelawarde@wirelessmundi.com> wrote: > >> I maintain "only" 100 sessions at all times, with a duration of 20-30s, > >> and no more than 10cps, so it is NOT a stress test, come on! > >> > >> I mean this would not even load an Asterisk server, and loading the > >> server was not an objective, just to maintain low-medium load over a > >> period of time. It worked very well during 5h. > >> > >> Why would I need to ulimit the crap out of my server then? :-) > > > > Don't feel alone, you're not the only one. > > > > We're getting the same pair of messages. We know what it means, we > > have just not found the cause of it. In our case happens quite fast > > (about an hour IIRC) when doing stress testing of about 900 calls at a > > time lasting 10 seconds each at a rate of about 40cps. > > > > Moises Silva > > Senior Software Engineer > > Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON > > L3R 9R6 Canada > > t. 1 905 474 1990 x128 | e. moy@sangoma.com > > > > _______________________________________________ > > FreeSWITCH-users mailing list > > FreeSWITCH-users@lists.freeswitch.org > > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > > http://www.freeswitch.org > > >
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Think of FreeSWITCH as our personal solution to our unsolvable bugs over there. On Thu, Oct 7, 2010 at 7:35 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I can't reproduce so far after updating to HEAD, I'll post a Jira issue if it happens again.
How do you guys do to always resolve all my problems in a few hours or days at max? I won't believe it anymore if you say you used to be an Asterisk bug marshal (I'm used to wait months for issues to be attended there)... :-)
Thanks, François.
On Tue, 2010-10-05 at 11:17 -0500, Anthony Minessale wrote:
Can you update first to this revision of HEAD or later?
commit 80883ebeb81c73a3dc5c4ee65050d57e2c6ccf00 Author: Anthony Minessale <anthm@freeswitch.org> Date: Tue Oct 5 11:11:21 2010 -0500
I added a small patch that I think may be related to your problem.
On Tue, Oct 5, 2010 at 10:59 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
if you are on centos or something like it. you can run this from the FS build root.
./support-d/fscore_pb gcore <your nick>
while FS is running in this state.
That will give you a URL you can tell me, it will upload the trace I need to see right to our pastebin.
If you can get it that way, and add our ssh key from ./support-d/shinzon.pub, install it and email or irc pm me the hostname. And I can examine it.
On Tue, Oct 5, 2010 at 10:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
GIT from 1st of October.
François.
On Tue, 2010-10-05 at 10:28 -0500, Anthony Minessale wrote:
BTW,
If you are any less than this revision don't bother to report it until you have tested latest GIT
commit b02c69bba9f46cad37225a0986cae068f24dcd81 Author: Anthony Minessale <anthm@freeswitch.org> Date: Wed Sep 22 18:14:24 2010 -0500
On Tue, Oct 5, 2010 at 10:26 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
If there is a bug why are you not opening it on JIRA? http://jira.freeswitch.org
We are doing work on improving the event system to be faster and use less memory.
If it caused some problems, that is not uncommon, software development requires testing. The right thing to do is to report it not to make underhanded comments about rethinking your strategy.
Ideally I would like to visit the machine while it's in that state or otherwise I would like you to get a gcore while it's int that state, open it in gdb and produce a "thread apply all bt" trace in text format and attach it to the JIRA.
On Tue, Oct 5, 2010 at 9:55 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote: > I wouldn't jump to conclusions that fast, it's just a bug with current > git head, probably only happening in very specific cases. > > If I learned anything from hollywood movies, it would be that bugs or > bad people always end up being tracked down and destroyed. > > François. > > > On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote: >> We learned from another thread about how Sangoma regularly tests 32 E1 >> setups in the lab, and were preparing a concept paper to show a >> telecom that traditional vendors can be replaced using open source >> even for large scale implementations. Does this mean we should rethink >> our strategy? >> >> Regards >> HASSAN >> >> >> On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote: >> > On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde >> > <fdelawarde@wirelessmundi.com> wrote: >> >> I maintain "only" 100 sessions at all times, with a duration of 20-30s, >> >> and no more than 10cps, so it is NOT a stress test, come on! >> >> >> >> I mean this would not even load an Asterisk server, and loading the >> >> server was not an objective, just to maintain low-medium load over a >> >> period of time. It worked very well during 5h. >> >> >> >> Why would I need to ulimit the crap out of my server then? :-) >> > >> > Don't feel alone, you're not the only one. >> > >> > We're getting the same pair of messages. We know what it means, we >> > have just not found the cause of it. In our case happens quite fast >> > (about an hour IIRC) when doing stress testing of about 900 calls at a >> > time lasting 10 seconds each at a rate of about 40cps. >> > >> > Moises Silva >> > Senior Software Engineer >> > Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON >> > L3R 9R6 Canada >> > t. 1 905 474 1990 x128 | e. moy@sangoma.com >> > >> > _______________________________________________ >> > FreeSWITCH-users mailing list >> > FreeSWITCH-users@lists.freeswitch.org >> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users >> > http://www.freeswitch.org >> > >> > > > _______________________________________________ > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Ok, I just wanted to ask before opening any JIRA issues, it could have been some hidden config I missed. I will try to reproduce, learn what a gcore is and how to extract it. Moises, is there a Jira already open for this issue? François. On Tue, 2010-10-05 at 10:26 -0500, Anthony Minessale wrote:
If there is a bug why are you not opening it on JIRA? http://jira.freeswitch.org
We are doing work on improving the event system to be faster and use less memory.
If it caused some problems, that is not uncommon, software development requires testing. The right thing to do is to report it not to make underhanded comments about rethinking your strategy.
Ideally I would like to visit the machine while it's in that state or otherwise I would like you to get a gcore while it's int that state, open it in gdb and produce a "thread apply all bt" trace in text format and attach it to the JIRA.
On Tue, Oct 5, 2010 at 9:55 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I wouldn't jump to conclusions that fast, it's just a bug with current git head, probably only happening in very specific cases.
If I learned anything from hollywood movies, it would be that bugs or bad people always end up being tracked down and destroyed.
François.
On Tue, 2010-10-05 at 20:35 +0600, Nyamul Hassan wrote:
We learned from another thread about how Sangoma regularly tests 32 E1 setups in the lab, and were preparing a concept paper to show a telecom that traditional vendors can be replaced using open source even for large scale implementations. Does this mean we should rethink our strategy?
Regards HASSAN
On 2010-10-05, Moises Silva <moises.silva@gmail.com> wrote:
On Tue, Oct 5, 2010 at 5:45 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote:
I maintain "only" 100 sessions at all times, with a duration of 20-30s, and no more than 10cps, so it is NOT a stress test, come on!
I mean this would not even load an Asterisk server, and loading the server was not an objective, just to maintain low-medium load over a period of time. It worked very well during 5h.
Why would I need to ulimit the crap out of my server then? :-)
Don't feel alone, you're not the only one.
We're getting the same pair of messages. We know what it means, we have just not found the cause of it. In our case happens quite fast (about an hour IIRC) when doing stress testing of about 900 calls at a time lasting 10 seconds each at a rate of about 40cps.
Moises Silva Senior Software Engineer Sangoma Technologies Inc. | 100 Renfrew Drive, Suite 100, Markham ON L3R 9R6 Canada t. 1 905 474 1990 x128 | e. moy@sangoma.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
what cpu? /b On Oct 5, 2010, at 3:19 AM, François Delawarde wrote:
Hello,
I decided to do a bit of "stability" testing leaving 100 sessions calling a playback "tone_stream://$${tetris}" on my dual-core CPU with 4G of RAM during a few hours.
After 5 hours working flawelessly, I started getting lots of:
2010-10-04 23:53:14.188596 [CRIT] switch_event.c:342 Out of threads!
Later combined with:
2010-10-04 23:59:02.692961 [CRIT] switch_event.c:325 Event system overloading
Both messages (the second one at an increased rate) followed by a few:
2010-10-05 00:25:43.711929 [CRIT] switch_time.c:755 Virtual Migration Detected! Syncing Clock
AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ see: http://pastebin.freeswitch.org/14135 François. On Tue, 2010-10-05 at 09:09 -0500, Brian West wrote:
what cpu?
/b
On Oct 5, 2010, at 3:19 AM, François Delawarde wrote:
Hello,
I decided to do a bit of "stability" testing leaving 100 sessions calling a playback "tone_stream://$${tetris}" on my dual-core CPU with 4G of RAM during a few hours.
After 5 hours working flawelessly, I started getting lots of:
2010-10-04 23:53:14.188596 [CRIT] switch_event.c:342 Out of threads!
Later combined with:
2010-10-04 23:59:02.692961 [CRIT] switch_event.c:325 Event system overloading
Both messages (the second one at an increased rate) followed by a few:
2010-10-05 00:25:43.711929 [CRIT] switch_time.c:755 Virtual Migration Detected! Syncing Clock
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (7)
-
Anthony Minessale
-
Brian West
-
David Ponzone
-
Dome Charoenyost
-
François Delawarde
-
Moises Silva
-
Nyamul Hassan