http://en.wikipedia.org/wiki/Software_testing#Input_combinations_and_preconditions
"A very fundamental problem with software testing is that testing under all combinations of inputs and preconditions (initial state) is not feasible, even with a simple product. This means that the number of defects in a software product can be very large and defects that occur infrequently are difficult to find in testing. More significantly, non-functional dimensions of quality (how it is supposed to be versus what it is supposed to do)—usability, scalability, performance, compatibility, reliability—can be highly subjective; something that constitutes sufficient value to one person may be intolerable to another.

Software testing is a very difficult thing. Testing every possible combination of actions and specifically covering the edge cases most likely to cause problems is pretty much impossible - to run through every possible action for something as large as FS would literally take longer than the age of the universe. As a result a script to test for all possible errors is pretty much impossible, especially for new functionality that'll be added before the script's updated to test them. There will always be some bugs, users testing and reporting is the best way to detect ones that the developers have missed. Generally FS is pretty good at avoiding them.

Testing yourself is always a good idea before rolling out to a production system. That'll show any obvious ones up that can be fixed quickly.

-Steve


On 4 February 2011 15:13, Madovsky <infos@madovsky.org> wrote:
FS team needs user as users need FS team...
maybe think to a script (developped by all possible users)
that test every new functionality...

----- Original Message -----
From: "Anthony Minessale" <anthony.minessale@gmail.com>
To: "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org>
Sent: Friday, February 04, 2011 9:58 AM
Subject: Re: [Freeswitch-users] Auto test FS install - now necessary?


> I've been doing this a long time and it boils down to:
>
> 50% superstition/paranoia.
> 50% regressions from commits.
>
> Stop and think. Without people updating everyday and telling us when
> something is not right, we may never find out.
> Engaging in the community and this mailing list and irc is a community
> effort and its sometimes a BETA program too.
> Just like if this was a cool video game, you get to play it before
> everyone else.
> It's hard to follow every possible execution path and get into every
> nook and cranny of 3 million lines of code.
> Even with the best unit testing int the world, we can't do it without
> real people testing.
> If you stop updating or persuade others to stop, now we are in real
> trouble.
>
> Also, pay attention to the GIT commit messages:
>
> This particular problem (which is fixed now btw) was committed with a
> warning to please test.
> There was a bigger problem earlier this week that resulted in some
> minority invasive changes and its easy to catch the problems and in
> the end the software is now more stable.
>
> commit f60fdf653dd2d7f8d3eaa6a9086e1f68bd993c59
> Author: Anthony Minessale <anthm@freeswitch.org>
> Date:   Wed Feb 2 16:22:43 2011 -0600
>
>    fix possible bad pointer in global vars (please test)
>
>
>
> On Thu, Feb 3, 2011 at 11:46 PM, Avi Marcus <avi@avimarcus.net> wrote:
>> Maybe I'm just being paranoid, but I've been seeing a lot of "I just
>> updated
>> to latest git and X broke"
>> And that x is always something else...
>> I'm not a seasoned coder on this magnitude, but perhaps we can develop a
>> test suite (at least part of one) to test that a fresh install is
>> functioning?
>> A FULL test would probably be a huge undertaking and involve recqual for
>> testing media, too... Also, each mod would need to have it's own test
>> suite.
>> Due to the rapid updating of FreeSWITCH this not be what you want to put
>> your effort into, but...
>> Thoughts?
>>
>> On Feb 4, 2011 4:04 AM, "Anirudha Shimpi" <anu@familytv.com> wrote:
>>> Just downloaded (from git) and installed FS 1.0.7, however, every time I
>>> do
>>> originate user/1001 &park, FS crashes. Running Centos 5.5 64 bit.
>>>
>>
>> _______________________________________________
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users@lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale@hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com
> IRC: irc.freenode.net #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888@conference.freeswitch.org
> googletalk:conf+888@conference.freeswitch.org
> pstn:+19193869900
>
> _______________________________________________
> FreeSWITCH-users mailing list
> FreeSWITCH-users@lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org


_______________________________________________
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org