[Freeswitch-users] Auto test FS install - now necessary?

[Steven Ayre]

http://en.wikipedia.org/wiki/Software_testing#Input_combinations_and_preconditions
"A very fundamental problem with software testing is that testing under all
combinations of inputs and preconditions (initial state) is not feasible,
even with a simple product. This means that the number of defects in a
software product can be very large and defects that occur infrequently are
difficult to find in testing. More significantly, non-functional dimensions
of quality (how it is supposed to be versus what it is supposed to
do)—usability, scalability, performance, compatibility, reliability—can be
highly subjective; something that constitutes sufficient value to one person
may be intolerable to another.

Software testing is a very difficult thing. Testing every possible
combination of actions and specifically covering the edge cases most likely
to cause problems is pretty much impossible - to run through every possible
action for something as large as FS would literally take longer than the age
of the universe. As a result a script to test for all possible errors is
pretty much impossible, especially for new functionality that'll be added
before the script's updated to test them. There will always be some bugs,
users testing and reporting is the best way to detect ones that the
developers have missed. Generally FS is pretty good at avoiding them.

Testing yourself is always a good idea before rolling out to a production
system. That'll show any obvious ones up that can be fixed quickly.

-Steve


On 4 February 2011 15:13, Madovsky <infos at madovsky.org> wrote:

> FS team needs user as users need FS team...
> maybe think to a script (developped by all possible users)
> that test every new functionality...
>
> ----- Original Message -----
> From: "Anthony Minessale" <anthony.minessale at gmail.com>
> To: "FreeSWITCH Users Help" <freeswitch-users at lists.freeswitch.org>
> Sent: Friday, February 04, 2011 9:58 AM
> Subject: Re: [Freeswitch-users] Auto test FS install - now necessary?
>
>
> > I've been doing this a long time and it boils down to:
> >
> > 50% superstition/paranoia.
> > 50% regressions from commits.
> >
> > Stop and think. Without people updating everyday and telling us when
> > something is not right, we may never find out.
> > Engaging in the community and this mailing list and irc is a community
> > effort and its sometimes a BETA program too.
> > Just like if this was a cool video game, you get to play it before
> > everyone else.
> > It's hard to follow every possible execution path and get into every
> > nook and cranny of 3 million lines of code.
> > Even with the best unit testing int the world, we can't do it without
> > real people testing.
> > If you stop updating or persuade others to stop, now we are in real
> > trouble.
> >
> > Also, pay attention to the GIT commit messages:
> >
> > This particular problem (which is fixed now btw) was committed with a
> > warning to please test.
> > There was a bigger problem earlier this week that resulted in some
> > minority invasive changes and its easy to catch the problems and in
> > the end the software is now more stable.
> >
> > commit f60fdf653dd2d7f8d3eaa6a9086e1f68bd993c59
> > Author: Anthony Minessale <anthm at freeswitch.org>
> > Date:   Wed Feb 2 16:22:43 2011 -0600
> >
> >    fix possible bad pointer in global vars (please test)
> >
> >
> >
> > On Thu, Feb 3, 2011 at 11:46 PM, Avi Marcus <avi at avimarcus.net> wrote:
> >> Maybe I'm just being paranoid, but I've been seeing a lot of "I just
> >> updated
> >> to latest git and X broke"
> >> And that x is always something else...
> >> I'm not a seasoned coder on this magnitude, but perhaps we can develop a
> >> test suite (at least part of one) to test that a fresh install is
> >> functioning?
> >> A FULL test would probably be a huge undertaking and involve recqual for
> >> testing media, too... Also, each mod would need to have it's own test
> >> suite.
> >> Due to the rapid updating of FreeSWITCH this not be what you want to put
> >> your effort into, but...
> >> Thoughts?
> >>
> >> On Feb 4, 2011 4:04 AM, "Anirudha Shimpi" <anu at familytv.com> wrote:
> >>> Just downloaded (from git) and installed FS 1.0.7, however, every time
> I
> >>> do
> >>> originate user/1001 &park, FS crashes. Running Centos 5.5 64 bit.
> >>>
> >>
> >> _______________________________________________
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >>
> >>
> >
> >
> >
> > --
> > Anthony Minessale II
> >
> > FreeSWITCH http://www.freeswitch.org/
> > ClueCon http://www.cluecon.com/
> > Twitter: http://twitter.com/FreeSWITCH_wire
> >
> > AIM: anthm
> > MSN:anthony_minessale at hotmail.com <MSN%3Aanthony_minessale at hotmail.com>
> > GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com<PAYPAL%3Aanthony.minessale at gmail.com>
> > IRC: irc.freenode.net #freeswitch
> >
> > FreeSWITCH Developer Conference
> > sip:888 at conference.freeswitch.org <sip%3A888 at conference.freeswitch.org>
> > googletalk:conf+888 at conference.freeswitch.org<googletalk%3Aconf%2B888 at conference.freeswitch.org>
> > pstn:+19193869900
> >
> > _______________________________________________
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
>
>
> _______________________________________________
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20110204/d5119e78/attachment-0001.html