[Freeswitch-users] Load test - performance not even matching Asterisk

[J. Javier Maestro]

On Oct Thu 09 2008 16:36, Jon Bruel wrote:
> I have been asked by Mark Crane and others to contribute more positively
> by joining the community, and I would be glad to do so. As with anyone I
> have my own agenda being to create a virtual PBX suite - using FS or
> what ever I find most suitable. At this stage, I have added a load
> testing page to the wiki, a modest first step to my contribution.  

Thanks a lot for your contribution :)

> A few words about my findings regarding load tests: On the surface the
> FS compares with the Asterisk: +/- 20% in terms of CPU load. The test
> may not credit the FS for possible better resilience. Knowing Asterisk I
> also know its instability under load, and with the architecture of the
> FS, this may be resolved by design, and I'm looking forward to verifying
> that. So I'll - in between the other tasks - continue with load tests
> which attempt to simulate real life situations, with complex dial plans
> and with erroneous internets. Ideas are welcome!

I am not a Load Test expert, but from an engineering point of view, I rather
first go for a "baseline suite" that would perform load testing under the
best conditions possible (that is, an average dial plan, and certainly not an
"erroneous internet" --I suppose you mean a lossy connection :-?).

This will give a reference result of the capabilities of FS (and any other
soft switch/pbx that can run the tests). Then, you can begin to push FS to
the limits, which will certainly test its resilience, architecture, etc.


If you go first for the real world, worst-case scenario, you might not get an
objective result. Imagine a test scenario such as the following: a web
browser that is incredibly fast rendering well-formed XML pages (such as
XHTML) but barks when rendering a malformed page. On the other hand, a crappy
browser, say with respect to speed or standards, but that has a best-effort
approach and manages to render, say 90% of a malformed page.

The first is a really good, speedy, browser, while the other is a crappy but
certainly resilient, best-effort, one. Broad, different results that have to
be viewed in their own frame of reference.


I would also try to approach the tests with an "atomic" point of view. That
is, if you go for a worst-case, real life, dense-in-features, scenario you
might find a multi-variable bug or whatever that might break FS and will be
hard to track because the test is itself multivariable, etc.

If you build tests by small, "atomic", steps, up to your real-life scenario,
where you to hit a nasty performance bug (or any other bug by this matter) it
will surely be much easier to track the reason triggering.


Finally, it would be wonderful to end up having the foundations for a load
test framework that would be able to test different switches / pbx
capabilities, much like the existing testing frameworks. Perhaps a nice
starting point would be 

  http://en.wikipedia.org/wiki/Load_testing
  http://en.wikipedia.org/wiki/Performance_testing
  http://en.wikipedia.org/wiki/Reliability_testing
  http://en.wikipedia.org/wiki/Volume_testing

  http://en.wikipedia.org/wiki/Category:Load_testing_tools
  

So these are my 2 cents :D and again, take all this with a grain of salt
since IANALTE 8)

Cheers,

-- 
J. Javier Maestro  <jjmaestro at nosys.es>
Socio Consultor   -    Nosys AJjV S.L.