Hold on, this doesn't make any sense.. your logs show that the error was triggered by sofia_glue.c:6605, but your file only contains 3017 lines?

Looking at the source code, older versions of FS contains >6000 lines of code in sofia_glue.c, where as newer versions contain approx 3000 lines.

Can you confirm that you have done a clean build, and that the FS binary you are executing is indeed the one you are building?

Cal

On Mon, Jul 22, 2013 at 10:25 PM, Victor Chukalovskiy <victor.chukalovskiy@gmail.com> wrote:
In all profiles it's commented out:
<!--<param name="manage-shared-appearance" value="true"/>-->
Assuming default is off, it's off.

BTW, sofia_glue.c only got 3017 lines. Please clarify.


      On 13-07-22 04:42 PM, Anthony Minessale wrote:

    

    

      
do you have manage-shared-appearance off too cos
        that re-enables presence?
        


        

        
edit sofia_glue.c on line 6614 and add a print to see what
          the sql is that is being executed.

        


        

        
switch_log_printf(SWITCH_CHANNEL_LOG, SWITCH_LOG_ERROR,
          "SQL: [%s]\n", sql);

        

        


        

        
then make mod_sofia-install

        


        

        
and relaunch and try to catch a sample of what it's doing.

        


        

        


        

        


        

      

      


        

        
On Mon, Jul 22, 2013 at 3:27 PM, Victor
          Chukalovskiy <victor.chukalovskiy@gmail.com>
          wrote:

          

            

              
Anthony, 

                

                Thank you, this is a very useful command. I see a bunch
                of records like:

                

                db="sofia_reg_<profile_name>"

                        Type: CORE_DB

                        Last used: 10

                        Flags: Unlocked, Detached(0)

                        Creator: sofia_glue.c:6605

                        Last User: sofia_glue.c:6605

                

                Appears that these DBs are in use and user/creator is
                sofia_glue.c:6605. Same as in error log:
                


                  [WARNING] sofia_glue.c:6605 Max handles 50

                  

                

                And this is not a cored.db... it's sitting on the disk
                drive still. Is there a way to setup sip_profiles not to
                create any sofia_reg_...DBs?

                

                Re: vmware setup. It's good to know it's not optimal,
                however it gives rack space, administrator's time, and $
                savings compared to dedicated servers.

                

                Should it be Ok to stick with VM setup provided enough
                CPU and RAM is allocated to each FS guest? Are there
                reasons not to run FS on VM other than CPU and RAM
                allocation?

                

                Thank you,

                Victor

                

                

              

              

                

                  

                    
Its a large number of handles to have
                      open all in sofia if you are not doing anything
                      


                      

                      
db_cache status

                      


                      

                      
from cli can show where those handles come
                        from and what db they are using.

                      
Clearly the box is being overloaded past its
                        limits.

                      


                      

                      
This is why we don't endorse virtual servers.

                      


                      

                    

                    


                      

                      
On Mon, Jul 22, 2013 at
                        2:49 PM, Victor Chukalovskiy <victor.chukalovskiy@gmail.com>
                        wrote:

                        

                          

                            
Disabled in all profiles:

                              <param name="manage-presence"
                              value="false"/>
                              

                                


                                  

                                  On 13-07-22 03:40 PM, Anthony
                                  Minessale wrote:

                                

                              

                            

                            

                              

                                

                                  
Did you turn of the
                                    presence in the sofia profile, that
                                    is enabled by default and very
                                    costly.  comment out manage-presence

                                  


                                    

                                    
On Mon, Jul
                                      22, 2013 at 2:32 PM, Victor
                                      Chukalovskiy <victor.chukalovskiy@gmail.com>
                                      wrote:

                                      

                                        

                                          
Michael, Ken, 

                                            

                                            Thank you for the response.
                                            Firstly, let me clarify
                                            setup for each of DBs in
                                            use:

                                            

                                            core_db: sqlite,
                                            /dev/shm/core.db (RAM) -
                                            using default 50 max handles

                                            mod_sofia db:
                                            sqlite, /opt/fs/db (HDD) 

                                            mod_lcr db: Postgres

                                            mod_limit: hash
                                            backend

                                            

                                            Note: I believe we rarely
                                            use mod_sofia DB's...this is
                                            because we don't use any
                                            registrations, acl only.
                                            When I list sofia
                                            per-profile db files on
                                            production box, most of them
                                            has not been edited for a
                                            couple days. Are my
                                            conclusions correct here, or
                                            do I miss something? Maybe
                                            they are accessed a lot but
                                            only for reading?

                                            

                                            I understand recommendations
                                            from Ken and from Michaels
                                            are along the lines of
                                            increasing core db and sofia
                                            db throughput. However here
                                            is what puzzles me:

                                            -the db used most
                                            extensively is core_db. It'
                                            already in the ram.

                                            -we don't have much load on
                                            sofia dbs (or am I mistaken
                                            in my logic?)

                                            -error log started with:
                                            


                                              

                                              [CRIT] switch_time.c:967
                                              Virtual Migration
                                              Detected! Syncing Clock

                                            

                                            
 [WARNING]
                                              switch_scheduler.c:115
                                              Task was executed late by
                                              19 seconds 1 heartbeat
                                              (core)

                                              

                                            

                                            Does it mean that there were
                                            abnormal events prior
                                            hitting 50 max handles
                                            limit?

                                            

                                            Thank you,

                                            -Victor
                                            

                                              


                                                

                                                On 13-07-22 02:52 PM,
                                                Anthony Minessale wrote:

                                              

                                            

                                          

                                          

                                            

                                              

                                                
Check the
                                                  odbcinst.ini settings
                                                  from the wiki or use
                                                  native pg mode.
                                                  
increase the max
                                                    number of pg
                                                    connections on your
                                                    server and turn up
                                                    FS to match
                                                    in max-db-handles in
                                                    switch.conf.xml

                                                  


                                                  

                                                  
You are already
                                                    beating it up pretty
                                                    good esp for a fake
                                                    server though.  2500
                                                    is a lot.

                                                  


                                                  

                                                

                                                


                                                  

                                                  
On
                                                    Mon, Jul 22, 2013 at
                                                    1:44 PM, Ken Rice <krice@freeswitch.org>
                                                    wrote:

                                                    
Mount

                                                      the entire
                                                      freeswitch/db dir
                                                      as tmpfs... most
                                                      of these errors
                                                      are for

                                                      the sofia DB
                                                      getting backed
                                                      up...

                                                      

                                                      The coredb is a
                                                      completely
                                                      different sqlite
                                                      db from the sofia
                                                      db

                                                      

                                                      On 7/22/13 12:54
                                                      PM, "Victor
                                                      Chukalovskiy" <victor.chukalovskiy@gmail.com>

                                                      wrote:

                                                      

                                                        


                                                          > Hello,

                                                          >

                                                          > Need some
                                                          help finding
                                                          the bottleneck
                                                          here. Running
                                                          1.2.stable on
                                                          a

                                                          > decent
                                                          multi-core
                                                          vmware
                                                          platform. No
                                                          RTP, SIP
                                                          signaling
                                                          only. Around

                                                          > 2500
                                                          sessions and
                                                          30 average CPS
                                                          load.
                                                          Essentially,
                                                          it's a
                                                          multi-profile

                                                          > .xml
                                                          config that
                                                          uses mod_lcr
                                                          extensively
                                                          (postgres DB).
                                                          Yesterday it

                                                          > stopped
                                                          processing
                                                          calls with the
                                                          following log
                                                          output:

                                                          >

                                                          >
                                                          2013-07-21
                                                          06:00:37.642678
                                                          [CRIT]
                                                          switch_time.c:967
                                                          Virtual
                                                          Migration

                                                          > Detected!
                                                          Syncing Clock

                                                          >
                                                          2013-07-21
                                                          06:01:07.141734
                                                          [WARNING]
                                                          switch_scheduler.c:115
                                                          Task was

                                                          > executed
                                                          late by 19
                                                          seconds 1
                                                          heartbeat
                                                          (core)

                                                          >
                                                          2013-07-21
                                                          06:02:32.402668
                                                          [CRIT]
                                                          switch_time.c:967
                                                          Virtual
                                                          Migration

                                                          > Detected!
                                                          Syncing Clock

                                                          >
                                                          2013-07-21
                                                          06:04:42.499744
                                                          [WARNING]
                                                          switch_scheduler.c:115
                                                          Task was

                                                          > executed
                                                          late by 115
                                                          seconds 1
                                                          heartbeat
                                                          (core)

                                                          >
                                                          2013-07-21
                                                          06:04:42.499744
                                                          [WARNING]
                                                          switch_scheduler.c:115
                                                          Task was

                                                          > executed
                                                          late by 74
                                                          seconds 2
                                                          check_ip
                                                          (core)

                                                          >
                                                          2013-07-21
                                                          06:05:15.102683
                                                          [CRIT]
                                                          switch_time.c:967
                                                          Virtual
                                                          Migration

                                                          > Detected!
                                                          Syncing Clock

                                                          >
                                                          2013-07-21
                                                          06:05:44.961004
                                                          [WARNING]
                                                          switch_scheduler.c:115
                                                          Task was

                                                          > executed
                                                          late by 22
                                                          seconds 1
                                                          heartbeat
                                                          (core)

                                                          >
                                                          2013-07-21
                                                          06:05:44.961004
                                                          [WARNING]
                                                          switch_scheduler.c:115
                                                          Task was

                                                          > executed
                                                          late by 2
                                                          seconds 2
                                                          check_ip
                                                          (core)

                                                          >
                                                          2013-07-21
                                                          06:05:49.322672
                                                          [WARNING]
                                                          mod_lcr.c:448
                                                          Max handles 50

                                                          > exceeded,
                                                          blocking....

                                                          >
                                                          2013-07-21
                                                          06:05:50.962987
                                                          [WARNING]
                                                          sofia_glue.c:6605
                                                          Max handles 50

                                                          > exceeded,
                                                          blocking....

                                                          >
                                                          2013-07-21
                                                          06:05:50.962987
                                                          [WARNING]
                                                          sofia_glue.c:6605
                                                          Max handles 50

                                                          > exceeded,
                                                          blocking....

                                                          >
                                                          2013-07-21
                                                          06:05:50.962987
                                                          [WARNING]
                                                          sofia_glue.c:6605
                                                          Max handles 50

                                                          > exceeded,
                                                          blocking....

                                                          >
                                                          ................................

                                                          >
                                                          2013-07-21
                                                          06:05:53.962672
                                                          [WARNING]
                                                          sofia_glue.c:6605
                                                          Max handles 50

                                                          > exceeded,
                                                          blocking....

                                                          >
                                                          2013-07-21
                                                          06:05:54.422671
                                                          [ERR]
                                                          mod_lcr.c:448
                                                          Error
                                                          connecting

                                                          >
                                                          2013-07-21
                                                          06:05:54.422671
                                                          [WARNING]
                                                          mod_lcr.c:1779
                                                          LCR lookup
                                                          failed

                                                          > for
                                                          14389955082

                                                          >
                                                          2013-07-21
                                                          06:05:54.962671
                                                          [WARNING]
                                                          sofia_glue.c:6605
                                                          Max handles 50

                                                          > exceeded,
                                                          blocking....

                                                          >
                                                          2013-07-21
                                                          06:05:54.962671
                                                          [WARNING]
                                                          sofia_glue.c:6605
                                                          Max handles 50

                                                          > exceeded,
                                                          blocking....

                                                          >
                                                          ................................

                                                          >
                                                          2013-07-21
                                                          06:05:55.962758
                                                          [WARNING]
                                                          sofia_glue.c:6605
                                                          Max handles 50

                                                          > exceeded,
                                                          blocking....

                                                          >
                                                          2013-07-21
                                                          06:05:56.062671
                                                          [ERR]
                                                          sofia_glue.c:6605
                                                          Error
                                                          connecting

                                                          >
                                                          2013-07-21
                                                          06:05:56.062671
                                                          [ERR]
                                                          sofia_glue.c:6606
                                                          Error Opening
                                                          DB

                                                          >
                                                          2013-07-21
                                                          06:05:56.062671
                                                          [WARNING]
                                                          sofia_glue.c:6605
                                                          Max handles 50

                                                          > exceeded,
                                                          blocking....

                                                          >
                                                          2013-07-21
                                                          06:05:56.062671
                                                          [ERR]
                                                          sofia_glue.c:6605
                                                          Error
                                                          connecting

                                                          >
                                                          2013-07-21
                                                          06:05:56.062671
                                                          [ERR]
                                                          sofia_glue.c:6606
                                                          Error Opening
                                                          DB

                                                          >

                                                          >
                                                          FreeSWITCH
                                                          stayed
                                                          unusable since
                                                          then until it
                                                          was restarted.

                                                          >

                                                          > FYI core
                                                          DB is already
                                                          in the RAM:

                                                          > <param
                                                          name="core-db-name"

                                                          value="/dev/shm/core.db"

                                                          />

                                                          >

                                                          >

                                                          > Thank
                                                          you,

                                                          > Victor

                                                          >

                                                          >
_________________________________________________________________________

                                                          >
                                                          Professional
                                                          FreeSWITCH
                                                          Consulting
                                                          Services:

                                                          > consulting@freeswitch.org

                                                          > http://www.freeswitchsolutions.com

                                                          >

                                                          >
                                                          FreeSWITCH-powered
                                                          IP PBX: The
                                                          CudaTel
                                                          Communication
                                                          Server

                                                          > http://www.cudatel.com

                                                          >

                                                          > Official
                                                          FreeSWITCH
                                                          Sites

                                                          > http://www.freeswitch.org

                                                          > http://wiki.freeswitch.org

                                                          > http://www.cluecon.com

                                                          >

                                                          >
                                                          FreeSWITCH-users
                                                          mailing list

                                                          > FreeSWITCH-users@lists.freeswitch.org

                                                          > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users


                                                          >
                                                          UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users


                                                          > http://www.freeswitch.org

                                                          

                                                        

                                                      

                                                      --

                                                          Ken

                                                          http://www.FreeSWITCH.org

                                                          http://www.ClueCon.com

                                                          http://www.OSTAG.org

                                                          irc.freenode.net
                                                          #freeswitch

                                                        
                                                      

                                                        


                                                          

                                                          

_________________________________________________________________________

                                                          Professional
                                                          FreeSWITCH
                                                          Consulting
                                                          Services:

                                                          consulting@freeswitch.org

                                                          http://www.freeswitchsolutions.com

                                                          

                                                          FreeSWITCH-powered
                                                          IP PBX: The
                                                          CudaTel
                                                          Communication
                                                          Server

                                                          http://www.cudatel.com

                                                          

                                                          Official
                                                          FreeSWITCH
                                                          Sites

                                                          http://www.freeswitch.org

                                                          http://wiki.freeswitch.org

                                                          http://www.cluecon.com

                                                          

                                                          FreeSWITCH-users
                                                          mailing list

                                                          FreeSWITCH-users@lists.freeswitch.org

                                                          http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

                                                          UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users


                                                          http://www.freeswitch.org

                                                        

                                                      

                                                    

                                                  

                                                  

                                                  

                                                  


                                                  

                                                  -- 

                                                  Anthony Minessale II

                                                  

                                                  FreeSWITCH http://www.freeswitch.org/

                                                  ClueCon http://www.cluecon.com/

                                                  Twitter: http://twitter.com/FreeSWITCH_wire

                                                  

                                                  AIM: anthm

                                                  MSN:anthony_minessale@hotmail.com

                                                  GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com

                                                  IRC: irc.freenode.net
                                                  #freeswitch

                                                  

                                                  FreeSWITCH Developer
                                                  Conference

                                                  sip:888@conference.freeswitch.org

                                                  googletalk:conf+888@conference.freeswitch.org

                                                  pstn:+19193869900 

                                                

                                                

                                                

                                                
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


                                              

                                              

                                            

                                          

                                        

                                        

_________________________________________________________________________

                                        Professional FreeSWITCH
                                        Consulting Services:

                                        consulting@freeswitch.org

                                        http://www.freeswitchsolutions.com

                                        

                                        FreeSWITCH-powered IP PBX: The
                                        CudaTel Communication Server

                                        http://www.cudatel.com

                                        

                                        Official FreeSWITCH Sites

                                        http://www.freeswitch.org

                                        http://wiki.freeswitch.org

                                        http://www.cluecon.com

                                        

                                        FreeSWITCH-users mailing list

                                        FreeSWITCH-users@lists.freeswitch.org

                                        http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

                                        UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

                                        http://www.freeswitch.org

                                        

                                      

                                    

                                    

                                    

                                    


                                    

                                    -- 

                                    Anthony Minessale II

                                    

                                    FreeSWITCH http://www.freeswitch.org/

                                    ClueCon http://www.cluecon.com/

                                    Twitter: http://twitter.com/FreeSWITCH_wire

                                    

                                    AIM: anthm

                                    MSN:anthony_minessale@hotmail.com

                                    GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com

                                    IRC: irc.freenode.net
                                    #freeswitch

                                    

                                    FreeSWITCH Developer Conference

                                    sip:888@conference.freeswitch.org

                                    googletalk:conf+888@conference.freeswitch.org

                                    pstn:+19193869900 

                                  

                                  

                                  

                                  
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


                                

                                

                              

                            

                          

                          

_________________________________________________________________________

                          Professional FreeSWITCH Consulting Services:

                          consulting@freeswitch.org

                          http://www.freeswitchsolutions.com

                          

                          FreeSWITCH-powered IP PBX: The CudaTel
                          Communication Server

                          http://www.cudatel.com

                          

                          Official FreeSWITCH Sites

                          http://www.freeswitch.org

                          http://wiki.freeswitch.org

                          http://www.cluecon.com

                          

                          FreeSWITCH-users mailing list

                          FreeSWITCH-users@lists.freeswitch.org

                          http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

                          UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

                          http://www.freeswitch.org

                          

                        

                      

                      

                      

                      


                      

                      -- 

                      Anthony Minessale II

                      

                      FreeSWITCH http://www.freeswitch.org/

                      ClueCon http://www.cluecon.com/

                      Twitter: http://twitter.com/FreeSWITCH_wire

                      

                      AIM: anthm

                      MSN:anthony_minessale@hotmail.com

                      GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com

                      IRC: irc.freenode.net
                      #freeswitch

                      

                      FreeSWITCH Developer Conference

                      sip:888@conference.freeswitch.org

                      googletalk:conf+888@conference.freeswitch.org

                      pstn:+19193869900 

                    

                    

                    

                    
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


                  

                  

                

              

            

            

_________________________________________________________________________

            Professional FreeSWITCH Consulting Services:

            consulting@freeswitch.org

            http://www.freeswitchsolutions.com

            

            FreeSWITCH-powered IP PBX: The CudaTel Communication Server

            http://www.cudatel.com

            

            Official FreeSWITCH Sites

            http://www.freeswitch.org

            http://wiki.freeswitch.org

            http://www.cluecon.com

            

            FreeSWITCH-users mailing list

            FreeSWITCH-users@lists.freeswitch.org

            http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

            UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

            http://www.freeswitch.org

            

          

        

        

        

        


        

        -- 

        Anthony Minessale II

        

        FreeSWITCH http://www.freeswitch.org/

        ClueCon http://www.cluecon.com/

        Twitter: http://twitter.com/FreeSWITCH_wire

        

        AIM: anthm

        MSN:anthony_minessale@hotmail.com

        GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com

        IRC: irc.freenode.net
        #freeswitch

        

        FreeSWITCH Developer Conference

        sip:888@conference.freeswitch.org

        googletalk:conf+888@conference.freeswitch.org

        pstn:+19193869900
      

      

      

      

      
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


    

    

  



_________________________________________________________________________

Professional FreeSWITCH Consulting Services:

consulting@freeswitch.org

http://www.freeswitchsolutions.com



FreeSWITCH-powered IP PBX: The CudaTel Communication Server

http://www.cudatel.com



Official FreeSWITCH Sites

http://www.freeswitch.org

http://wiki.freeswitch.org

http://www.cluecon.com



FreeSWITCH-users mailing list

FreeSWITCH-users@lists.freeswitch.org

http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

http://www.freeswitch.org