<html>

  <head>

    <meta content="text/html; charset=windows-1252"

      http-equiv="Content-Type">

  </head>

  <body bgcolor="#FFFFFF" text="#000000">

    Yes, to clarify the sync_clock was a different issue it appear in

    this tread to discard that could be the cause behind from the main

    problem on cpu peaks, and it doesn't.<br>

    <br>

    My initial problem off cpu peaks, the main reason behind looks like

    that malloc calls are locking the cpu after a while  of running fs,

    or something is triggering block in fs.  <br>

    I've putt all the info in jira, I'm still searching and trying to

    reproduce the issue in a local machine.<br>

    <br>

    using tcmalloc implementation solves the issue, and so far as been

    quite mornings. :)<br>

    <br>

    Regards,<br>

    António<br>

    <br>

    <div class="moz-cite-prefix">On 03/02/2017 10:07 PM, Steven Ayre

      wrote:<br>

    </div>

    <blockquote

cite="mid:CAFiqYumf-TebN4MFnzcpZNbV+qTdxTtFvK4=e3yL+M72r+TNog@mail.gmail.com"

      type="cite">

      <div dir="ltr">Should add that we don't see the cpu behaviour

        you're seeing. We only sync clock because the monotonic clock

        drifts away from the NTP one on some of our servers by up to

        300ms. Usually it's only an issue if FreeSWITCH starts on boot

        before NTP has synced.

        <div><br>

        </div>

        <div>tcmalloc is presumably solving your cpu issue and sync

          clock working around your separate clock sync issue. If you're

          running sync clock frequently enough it's probably safe.

          enable-use-system-time is essentially the extreme end of that.

          The reason to normally prefer sync_clock_when_idle is so as

          not to affect CDRs of current calls by creating big jumps in

          timestamp/duration, but if you're never letting them drift far

          apart that shouldn't become an issue.</div>

      </div>

      <div class="gmail_extra"><br>

        <div class="gmail_quote">On 1 March 2017 at 13:57, Antonio Silva

          <span dir="ltr">&lt;<a moz-do-not-send="true"

              href="mailto:asilva@wirelessmundi.com" target="_blank">asilva@wirelessmundi.com</a>&gt;</span>

          wrote:<br>

          <blockquote class="gmail_quote" style="margin:0 0 0

            .8ex;border-left:1px #ccc solid;padding-left:1ex">

            <div bgcolor="#FFFFFF" text="#000000"> thsk steven, i've to

              do some test first as well. <br>

              <br>

              As for the main problem, i put back the sync clock every

              night and do far no peaks on cpu sys. So the solution for

              me was to put fs compiled with tcmalloc.<br>

              <br>

              Regards,<br>

              António

              <div>

                <div class="h5"><br>

                  <br>

                  <div class="m_8524991909929835003moz-cite-prefix">On

                    03/01/2017 01:02 PM, Steven Ayre wrote:<br>

                  </div>

                  <blockquote type="cite">

                    <div dir="ltr">We're planning on rolling it out but

                      haven't yet done so. Need to schedule some testing

                      on it first.

                      <div><br>

                      </div>

                      <div>I suspect the risk he refers to is if you're

                        manually changing the system clock you can cause

                        large clock jumps. That can affect CDR

                        timestamps, either billing for longer periods

                        than you should or getting negative call

                        durations.</div>

                      <div><br>

                      </div>

                      <div>If you're using the NTP daemon to get

                        constant adjustments, it's configured correctly,

                        it starts on boot, is trustworthy, refuses to

                        make large clock changes and you're monitoring

                        it properly you'll probably be fine. I'd suggest

                        using UTC for the system clock too, to guarantee

                        you're not seeing any DST jumps.</div>

                      <div><br>

                      </div>

                      <div>Steve</div>

                    </div>

                    <div class="gmail_extra"><br>

                      <div class="gmail_quote">On 24 February 2017 at

                        18:02, Antonio Silva <span dir="ltr">&lt;<a

                            moz-do-not-send="true"

                            href="mailto:asilva@wirelessmundi.com"

                            target="_blank">asilva@wirelessmundi.com</a>&gt;</span>

                        wrote:<br>

                        <blockquote class="gmail_quote" style="margin:0

                          0 0 .8ex;border-left:1px #ccc

                          solid;padding-left:1ex">

                          <div bgcolor="#FFFFFF" text="#000000"> Hi

                            Steven,<br>

                            <br>

                            Thanks for the tips. <br>

                            <br>

                            i don't use sync_clock_when_idle because it

                            waits until there is 0 calls, if you never

                            hit 0 calls then the clock is never sync, i

                            wanted to be sure that is done.<br>

                            <br>

                            didn't know about 

                            "enable-use-system-time",  just see the

                            commit and there is a note from Anthony:

                            "add enable-use-system-time param to

                            switch.conf.xml <b>use at your own risk</b>",

                            are you using it? does it affect

                            performance? <br>

                            <br>

                            I've to tried it. <br>

                            <br>

                            <br>

                            About cpu sys to high and getting block,

                            using tcmalloc seams to solve it, today

                            everything went ok, no peaks in cpu sys. <br>

                            Next week i'll put back the sync_clock.<br>

                            <br>

                            Regards,

                            <div>

                              <div class="m_8524991909929835003h5"><br>

                                <br>

                                <div

                                  class="m_8524991909929835003m_1244224680982037732moz-cite-prefix">On

                                  02/24/2017 02:31 PM, Steven Ayre

                                  wrote:<br>

                                </div>

                                <blockquote type="cite">

                                  <div dir="ltr">NTP adjusts the system

                                    clock while FS initialises a

                                    monotonic clock on start. FS doesn't

                                    see adjustments for drift by NTP.

                                    <div><br>

                                    </div>

                                    <div>sync_clock_when_idle is better

                                      than sync_clock.<br>

                                    </div>

                                    <div><br>

                                    </div>

                                    <div>You can also use

                                      the enable-use-system-time option

                                      which'll get it to use the system

                                      (and therefore NTP) time.</div>

                                  </div>

                                  <div class="gmail_extra"><br>

                                    <div class="gmail_quote">On 23

                                      February 2017 at 23:14, Antonio

                                      Silva <span dir="ltr">&lt;<a

                                          moz-do-not-send="true"

                                          href="mailto:asilva@wirelessmundi.com"

                                          target="_blank">asilva@wirelessmundi.com</a>&gt;</span>

                                      wrote:<br>

                                      <blockquote class="gmail_quote"

                                        style="margin:0 0 0

                                        .8ex;border-left:1px #ccc

                                        solid;padding-left:1ex">

                                        <div bgcolor="#FFFFFF"

                                          text="#000000"> i had a issue

                                          with cdr timestamp and the

                                          system date, after uptime of

                                          140 days fs clock was behind

                                          1min. <br>

                                          It has something between

                                          system clock and ntp, so to

                                          force fs to keep up i force

                                          the sync_clock.

                                          <div>

                                            <div

                                              class="m_8524991909929835003m_1244224680982037732h5"><br>

                                              <br>

                                              <br>

                                              <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063moz-cite-prefix">On

                                                02/24/2017 12:06 AM,

                                                Brian West wrote:<br>

                                              </div>

                                              <blockquote type="cite">

                                                <div dir="ltr">First why

                                                  did you ever call

                                                  sync_clock?</div>

                                                <div class="gmail_extra"><br>

                                                  <div

                                                    class="gmail_quote">On

                                                    Thu, Feb 23, 2017 at

                                                    4:51 PM, Antonio

                                                    Silva <span

                                                      dir="ltr">&lt;<a

                                                        moz-do-not-send="true"

href="mailto:asilva@wirelessmundi.com" target="_blank">asilva@wirelessmundi.com</a>&gt;</span>

                                                    wrote:<br>

                                                    <blockquote

                                                      class="gmail_quote"

                                                      style="margin:0 0

                                                      0

                                                      .8ex;border-left:1px

                                                      #ccc

                                                      solid;padding-left:1ex">

                                                      <div

                                                        bgcolor="#FFFFFF"

                                                        text="#000000">

                                                        Hi Brian,<br>

                                                        <br>

                                                        Not for today,

                                                        the first time

                                                        it happens was

                                                        my first though

                                                        something with

                                                        the traffic, but

                                                        i couldn't find

                                                        any association

                                                        with the type of

                                                        traffic.

                                                        Actually after

                                                        restart fs i

                                                        receive traffic

                                                        from the same

                                                        endpoints, same

                                                        type of calls

                                                        and the issue

                                                        don't reproduce

                                                        until next

                                                        morning...<br>

                                                        <br>

                                                        There was

                                                        something i was

                                                        doing at 00

                                                        every day, the

                                                        command "fsctl

                                                        sync_clock"...

                                                        well i just

                                                        disabled it

                                                        today. I've to

                                                        wait until

                                                        tomorrow to

                                                        check the

                                                        results.  <br>

                                                        <br>

                                                        I just install

                                                        fs compiled with

                                                        tcmalloc, so if

                                                        there is no

                                                        problems i will

                                                        put back the

                                                        "sync_clock "

                                                        command to

                                                        really check if

                                                        it could be the

                                                        cause. <br>

                                                        <br>

                                                        Right now i'm

                                                        quite lost on

                                                        what to do

                                                        more...

                                                        <div>

                                                          <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063h5"><br>

                                                          <br>

                                                          <br>

                                                          <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732moz-cite-prefix">On

                                                          02/23/2017

                                                          11:37 PM,

                                                          Brian West

                                                          wrote:<br>

                                                          </div>

                                                          <blockquote

                                                          type="cite">

                                                          <div dir="ltr">Do

                                                          you have pcaps

                                                          of the traffic

                                                          that may

                                                          trigger this

                                                          behavior?

                                                          <div><br>

                                                          </div>

                                                          <div>/b</div>

                                                          <div><br>

                                                          </div>

                                                          </div>

                                                          <div

                                                          class="gmail_extra"><br>

                                                          <div

                                                          class="gmail_quote">On

                                                          Thu, Feb 23,

                                                          2017 at 2:58

                                                          PM, Antonio

                                                          Silva <span

                                                          dir="ltr">&lt;<a

moz-do-not-send="true" href="mailto:asilva@wirelessmundi.com"

                                                          target="_blank">asilva@wirelessmundi.com</a>&gt;</span>

                                                          wrote:<br>

                                                          <blockquote

                                                          class="gmail_quote"

style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

                                                          <div

                                                          bgcolor="#FFFFFF"

                                                          text="#000000">

                                                          Thanks Juan,

                                                          that's it, i

                                                          was missing

                                                          the dev pkg.

                                                          <div>

                                                          <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732h5"><br>

                                                          <br>

                                                          <br>

                                                          <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841moz-cite-prefix">On

                                                          02/23/2017

                                                          08:34 PM, Juan

                                                          Pablo Godoy

                                                          Macari wrote:<br>

                                                          </div>

                                                          <blockquote

                                                          type="cite">

                                                          <div dir="ltr">Antonio,

                                                          you should

                                                          install this:

                                                          <div><br>

                                                          </div>

                                                          <div>apt-get

                                                          install

                                                          libgoogle-perftools-dev<br>

                                                          </div>

                                                          <div><br>

                                                          </div>

                                                          <div>Regards,</div>

                                                          </div>

                                                          <div

                                                          class="gmail_extra"><br>

                                                          <div

                                                          class="gmail_quote">2017-02-23

                                                          16:26

                                                          GMT-03:00 Juan

                                                          Pablo Godoy

                                                          Macari <span

                                                          dir="ltr">&lt;<a

moz-do-not-send="true" href="mailto:jpgodoy@redvoiss.net"

                                                          target="_blank">jpgodoy@redvoiss.net</a>&gt;</span>:<br>

                                                          <blockquote

                                                          class="gmail_quote"

style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

                                                          <div dir="ltr">Try

this: ./configure LIBS="-ltcmalloc"</div>

                                                          <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841HOEnZb">

                                                          <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841h5">

                                                          <div

                                                          class="gmail_extra"><br>

                                                          <div

                                                          class="gmail_quote">2017-02-23

                                                          16:19

                                                          GMT-03:00

                                                          Antonio Silva

                                                          <span

                                                          dir="ltr">&lt;<a

moz-do-not-send="true" href="mailto:asilva@wirelessmundi.com"

                                                          target="_blank">asilva@wirelessmundi.com</a>&gt;</span>:<br>

                                                          <blockquote

                                                          class="gmail_quote"

style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

                                                          <div

                                                          bgcolor="#FFFFFF"

                                                          text="#000000">

                                                          Hi,<br>

                                                          <br>

                                                          Did you

                                                          compile fs in

                                                          debian with

                                                          tcmalloc? <br>

                                                          <br>

                                                          didn't work

                                                          for me... do i

                                                          miss

                                                          something?<br>

                                                          <br>

                                                          <br>

                                                          <a

                                                          moz-do-not-send="true"

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-abbreviated"

href="mailto:root@dev:/usr/src/freeswitch.git#" target="_blank">root@dev:/usr/src/freeswitch.g<wbr>it#</a>

                                                          apt-get

                                                          install

                                                          libtcmalloc-minimal4<br>

                                                          <a

                                                          moz-do-not-send="true"

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-abbreviated"

href="mailto:root@dev:/usr/src/freeswitch.git#" target="_blank">root@dev:/usr/src/freeswitch.g<wbr>it#</a>

                                                          export

                                                          LIBS=-ltcmalloc<br>

                                                          <a

                                                          moz-do-not-send="true"

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-abbreviated"

href="mailto:root@dev:/usr/src/freeswitch.git#" target="_blank">root@dev:/usr/src/freeswitch.g<wbr>it#</a>

                                                          ./configure<br>

                                                          checking for a

                                                          BSD-compatible

                                                          install...

                                                          /usr/bin/install

                                                          -c<br>

                                                          checking

                                                          whether build

                                                          environment is

                                                          sane... yes<br>

                                                          checking for a

                                                          thread-safe

                                                          mkdir -p...

                                                          /bin/mkdir -p<br>

                                                          checking for

                                                          gawk... no<br>

                                                          checking for

                                                          mawk... mawk<br>

                                                          checking

                                                          whether make

                                                          sets

                                                          $(MAKE)... yes<br>

                                                          checking

                                                          whether make

                                                          supports

                                                          nested

                                                          variables...

                                                          yes<br>

                                                          checking

                                                          whether make

                                                          supports

                                                          nested

                                                          variables...

                                                          (cached) yes<br>

                                                          checking build

                                                          system type...

x86_64-unknown-linux-gnu<br>

                                                          checking host

                                                          system type...

x86_64-unknown-linux-gnu<br>

                                                          checking for

                                                          style of

                                                          include used

                                                          by make... GNU<br>

                                                          checking for

                                                          gcc... gcc<br>

                                                          checking

                                                          whether the C

                                                          compiler

                                                          works... no<br>

                                                          configure:

                                                          error: in

`/usr/src/freeswitch.git':<br>

                                                          configure:

                                                          error: C

                                                          compiler

                                                          cannot create

                                                          executables

                                                          <div>

                                                          <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682h5"><br>

                                                          <br>

                                                          <br>

                                                          <br>

                                                          <br>

                                                          <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-cite-prefix">On

                                                          02/23/2017

                                                          07:56 PM,

                                                          Antonio Silva

                                                          wrote:<br>

                                                          </div>

                                                          <blockquote

                                                          type="cite">

                                                          Hi Juan,<br>

                                                          <br>

                                                          Thanks for the

                                                          tips!!<br>

                                                          <br>

                                                          Unfortunately

                                                          i cannot test

                                                          with -nosql

                                                          option, i need

                                                          to have the

                                                          sip_registration

                                                          fill to be

                                                          able to send

                                                          options to

                                                          natted

                                                          endpoints. But

                                                          i don't think

                                                          the issue is

                                                          in db, i use

                                                          postgres and

                                                          to get better

                                                          performance

                                                          from it i run

                                                          it on tmpfs.<br>

                                                          <br>

                                                          <br>

                                                          I read about

                                                          tcmalloc, but

                                                          from jira

                                                          FS-7315, i get

                                                          the idea that

                                                          it didn't

                                                          change

                                                          anything.. but

                                                          i guess i can

                                                          try it and

                                                          post the

                                                          results, from

                                                          the bt i do

                                                          see some calls

                                                          been done to

                                                          malloc, but

                                                          not sure if

                                                          they are

                                                          locking or

                                                          not.<br>

                                                          <br>

                                                          Regards,<br>

                                                          <br>

                                                          <br>

                                                          <div

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-cite-prefix">On

                                                          02/23/2017

                                                          06:52 PM, Juan

                                                          Pablo Godoy

                                                          Macari wrote:<br>

                                                          </div>

                                                          <blockquote

                                                          type="cite">

                                                          <div dir="ltr">Hello

                                                          Antonio,

                                                          <div><br>

                                                          </div>

                                                          <div>I had the

                                                          same issue, to

                                                          fix it i tried

                                                          many things

                                                          written in

                                                          this mail

                                                          list. Try

                                                          running FS

                                                          with -nosql

                                                          option, if

                                                          that doesn't

                                                          work recompile

                                                          FS with

                                                          tcmalloc (from

                                                          google's

                                                          perfomance

                                                          tools).</div>

                                                          <div><br>

                                                          </div>

                                                          <div>That did

                                                          it for me.</div>

                                                          <div><br>

                                                          </div>

                                                          <div>Regards,</div>

                                                          <div>Juan

                                                          Pablo.</div>

                                                          </div>

                                                          <div

                                                          class="gmail_extra"><br>

                                                          <div

                                                          class="gmail_quote">2017-02-23

                                                          13:26

                                                          GMT-03:00

                                                          Antonio Silva

                                                          <span

                                                          dir="ltr">&lt;<a

moz-do-not-send="true" href="mailto:asilva@wirelessmundi.com"

                                                          target="_blank">asilva@wirelessmundi.com</a>&gt;</span>:<br>

                                                          <blockquote

                                                          class="gmail_quote"

style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>

                                                          <br>

                                                          i'm having an

                                                          issue with FS

                                                          that gets lock

                                                          every morning 

                                                          with cpu sys<br>

                                                          hitting 80%,

                                                          because is

                                                          limited in fs

                                                          with min-idle,

                                                          it could reach

                                                          the<br>

                                                          100%, so the

                                                          calls are

                                                          rejected by

                                                          load, to solve

                                                          the problem i

                                                          kill fs<br>

                                                          and start it

                                                          again.<br>

                                                          <br>

                                                          I discard a

                                                          load issue,

                                                          because after

                                                          restarting

                                                          freeswitch the

                                                          system<br>

                                                          increases in

                                                          number of

                                                          sessions/cps

                                                          and no

                                                          problems at

                                                          all until next<br>

                                                          day.... it's

                                                          given me

                                                          crazy... i've

                                                          enough memory,

                                                          lots of

                                                          cores...<br>

                                                          after restart

                                                          fs the system

                                                          stays at less

                                                          than 30% of

                                                          cpu usage.<br>

                                                          <br>

                                                          <br>

                                                          Right now i

                                                          have hpet as

                                                          clocksource,

                                                          transcoding is

                                                          minimal (use<br>

                                                          inheric_codec,

                                                          only happens

                                                          because of

                                                          different

                                                          ptimes with

                                                          some<br>

                                                          endpoints) .<br>

                                                          <br>

                                                          <br>

                                                          anyone having

                                                          similar

                                                          problem? or

                                                          any tip i can

                                                          do to debug

                                                          this issue?<br>

                                                          <br>

                                                          <br>

                                                          For info the

                                                          Jira ticket

                                                          can be located

                                                          at<br>

                                                          <a

                                                          moz-do-not-send="true"

href="https://freeswitch.org/jira/browse/FS-9957" rel="noreferrer"

                                                          target="_blank">https://freeswitch.org/jira/br<wbr>owse/FS-9957</a>,<br>

                                                          <br>

                                                          i've done a

                                                          gcore, top,

                                                          htop when the

                                                          system is with

                                                          high cpu sys

                                                          but i<br>

                                                          can't figure

                                                          the thread

                                                          that is

                                                          causing the

                                                          issue...

                                                          initial i

                                                          thought<br>

                                                          it was because

                                                          of module lua

                                                          (most of my

                                                          logic is done

                                                          there), but

                                                          today<br>

                                                          only 2 threads

                                                          where running

                                                          mod_lua when

                                                          it happen.<br>

                                                          <br>

                                                          <br>

                                                          Appreciate the

                                                          help.<br>

                                                          <br>

                                                          <br>

                                                          --<br>

                                                          <br>

                                                          Saludos /

                                                          Regards /

                                                          Cumprimentos,<br>

                                                          António silva<br>

                                                          <br>

                                                          <br>

______________________________<wbr>______________________________<wbr>_____________<br>

                                                          Professional

                                                          FreeSWITCH

                                                          Consulting

                                                          Services:<br>

                                                          <a

                                                          moz-do-not-send="true"

href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>

                                                          <a

                                                          moz-do-not-send="true"

href="http://www.freeswitchsolutions.com" rel="noreferrer"

                                                          target="_blank">http://www.freeswitchsolutions<wbr>.com</a><br>

                                                          <br>

                                                          Official

                                                          FreeSWITCH

                                                          Sites<br>

                                                          <a

                                                          moz-do-not-send="true"

href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>

                                                          <a

                                                          moz-do-not-send="true"

href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.o<wbr>rg</a><br>

                                                          <a

                                                          moz-do-not-send="true"

href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>

                                                          <br>

FreeSWITCH-users mailing list<br>

                                                          <a

                                                          moz-do-not-send="true"

href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswi<wbr>tch.org</a><br>

                                                          <a

                                                          moz-do-not-send="true"

href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users"

                                                          rel="noreferrer"

target="_blank">http://lists.freeswitch.org/ma<wbr>ilman/listinfo/freeswitch-user<wbr>s</a><br>

                                                          UNSUBSCRIBE:<a

moz-do-not-send="true"

                                                          href="http://lists.freeswitch.org/mailman/options/freeswitch-users"

rel="noreferrer" target="_blank">http://lists.frees<wbr>witch.org/mailman/options/free<wbr>switch-users</a><br>

                                                          <a

                                                          moz-do-not-send="true"

href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a></blockquote>

                                                          </div>

                                                          <br>

                                                          </div>

                                                          <br>

                                                          <fieldset

class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301mimeAttachmentHeader"></fieldset>

                                                          <br>

                                                          <pre>______________________________<wbr>______________________________<wbr>_____________

Professional FreeSWITCH Consulting Services: 

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-abbreviated" href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions<wbr>.com</a>

Official FreeSWITCH Sites

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.o<wbr>rg</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a>

FreeSWITCH-users mailing list

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-abbreviated" href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswi<wbr>tch.org</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/ma<wbr>ilman/listinfo/freeswitch-user<wbr>s</a>

UNSUBSCRIBE:<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.frees<wbr>witch.org/mailman/options/free<wbr>switch-users</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a></pre>

      </blockquote>

      <fieldset class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301mimeAttachmentHeader"></fieldset>

      <pre>______________________________<wbr>______________________________<wbr>_____________

Professional FreeSWITCH Consulting Services: 

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-abbreviated" href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions<wbr>.com</a>

Official FreeSWITCH Sites

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.o<wbr>rg</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a>

FreeSWITCH-users mailing list

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-abbreviated" href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswi<wbr>tch.org</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/ma<wbr>ilman/listinfo/freeswitch-user<wbr>s</a>

UNSUBSCRIBE:<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.frees<wbr>witch.org/mailman/options/free<wbr>switch-users</a>

<a moz-do-not-send="true" class="m_8524991909929835003m_1244224680982037732m_3869087452762594063m_-2504150484991105732m_-763834884867468841m_-1241466475139863682m_-847135978965072301moz-txt-link-freetext" href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a></pre>

    </blockquote>

  </div></div></div>

______________________________<wbr>______________________________<wbr>_____________