[Freeswitch-users] Crash on incoming E1 call

Robin Gilks robin.gilks at taitradio.com
Thu Jul 5 05:18:08 MSD 2012


Current running from commit 04bd463d12bf4c0116bec3cee537749b2040ed40

This occurs on all incoming calls from a Digium Wildcard TE210P with a
zaptel E1 interface with libpri.

I've tried updating to dahdi-linux-complete-2.6.1+2.6.1.tar.gz but
failed miserably - couldn't even get the E1 link itself to come up :(

I've captured a backtrace of the event on the zaptel drivers,
hopefully useful to someone.

gdb /usr/local/freeswitch_TR/bin/freeswitch -x ${HOME}/gdbcommands


$ cat gdbcommands
handle SIGPIPE nostop noprint
handle SIG33 nostop noprint
set logging on
set pagination off
set breakpoint pending on
run
thread apply all bt full
set logging off

Attached is the output of the backtrace.



-- 
Robin Gilks

-- 

------------------------------
This email, including any attachments, is only for the intended recipient. 
It is subject to copyright, is confidential and may be the subject of legal 
or other privilege, none of which is waived or lost by reason of this 
transmission.
If you are not an intended recipient, you may not use, disseminate, 
distribute or reproduce such email, any attachments, or any part thereof. 
If you have received a message in error, please notify the sender 
immediately and erase all copies of the message and any attachments.
Unfortunately, we cannot warrant that the email has not been altered or 
corrupted during transmission nor can we guarantee that any email or any 
attachments are free from computer viruses or other conditions which may 
damage or interfere with recipient data, hardware or software. The 
recipient relies upon its own procedures and assumes all risk of use and of 
opening any attachments.
------------------------------
-------------- next part --------------
Function "qFatal" not defined.
Breakpoint 1 (qFatal) pending.
[Thread debugging using libthread_db enabled]

Program exited normally.
[Thread debugging using libthread_db enabled]
Executing new program: /usr/local/freeswitch_TR/bin/freeswitch
[Thread debugging using libthread_db enabled]
[New Thread 0xb7f8eb90 (LWP 3171)]
[New Thread 0xb7f52b90 (LWP 3172)]
[New Thread 0xb7e51b90 (LWP 3173)]
[New Thread 0xb7e15b90 (LWP 3174)]
[Thread 0xb7e15b90 (LWP 3174) exited]
[New Thread 0xb7dd9b90 (LWP 3175)]
[Thread 0xb7dd9b90 (LWP 3175) exited]
[New Thread 0xb7dd9b90 (LWP 3176)]
[New Thread 0xb7e15b90 (LWP 3177)]
[New Thread 0xb7cd7b90 (LWP 3178)]
[New Thread 0xb7c9bb90 (LWP 3179)]
[Thread 0xb7c9bb90 (LWP 3179) exited]
[New Thread 0xb7c9bb90 (LWP 3180)]
[New Thread 0xb7acdb90 (LWP 3181)]
[New Thread 0xb7a91b90 (LWP 3182)]
[New Thread 0xb7a19b90 (LWP 3184)]
[New Thread 0xb7a55b90 (LWP 3183)]
[New Thread 0xb79ddb90 (LWP 3185)]
[New Thread 0xb796fb90 (LWP 3186)]
[New Thread 0xb7933b90 (LWP 3187)]
[New Thread 0xb78c5b90 (LWP 3188)]
[New Thread 0xb7889b90 (LWP 3189)]
[New Thread 0xb781bb90 (LWP 3190)]
[New Thread 0xb77dfb90 (LWP 3191)]
[New Thread 0xb77a3b90 (LWP 3192)]
[New Thread 0xb7735b90 (LWP 3193)]
[New Thread 0xb76f9b90 (LWP 3194)]
[New Thread 0xb76bdb90 (LWP 3195)]
[New Thread 0xb7681b90 (LWP 3196)]
[Thread 0xb76f9b90 (LWP 3194) exited]
[New Thread 0xb76f9b90 (LWP 3197)]
[Thread 0xb7681b90 (LWP 3196) exited]
[Thread 0xb76bdb90 (LWP 3195) exited]
[Thread 0xb76f9b90 (LWP 3197) exited]
[New Thread 0xb76f9b90 (LWP 3198)]
[New Thread 0xb7681b90 (LWP 3199)]
[New Thread 0xb76bdb90 (LWP 3200)]
[New Thread 0xb7645b90 (LWP 3201)]
[New Thread 0xb7609b90 (LWP 3202)]
[New Thread 0xb75cdb90 (LWP 3203)]
[New Thread 0xb7591b90 (LWP 3204)]
[Thread 0xb75cdb90 (LWP 3203) exited]
[Thread 0xb7591b90 (LWP 3204) exited]
[New Thread 0xb7591b90 (LWP 3207)]
[Thread 0xb7591b90 (LWP 3207) exited]
[New Thread 0xb7591b90 (LWP 3208)]
[New Thread 0xb75cdb90 (LWP 3209)]
[Thread 0xb7591b90 (LWP 3208) exited]
[Thread 0xb75cdb90 (LWP 3209) exited]
[New Thread 0xb75cdb90 (LWP 3210)]
[New Thread 0xb7591b90 (LWP 3211)]
[Thread 0xb7591b90 (LWP 3211) exited]
[New Thread 0xb7591b90 (LWP 3229)]
[Thread 0xb7591b90 (LWP 3229) exited]
[New Thread 0xb7591b90 (LWP 3249)]
[Thread 0xb7591b90 (LWP 3249) exited]
[New Thread 0xb7591b90 (LWP 3250)]
[New Thread 0xb7555b90 (LWP 3251)]
[New Thread 0xb73ffb90 (LWP 3252)]
[Thread 0xb73ffb90 (LWP 3252) exited]
[Thread 0xb7555b90 (LWP 3251) exited]
[Thread 0xb7591b90 (LWP 3250) exited]
[New Thread 0xb7591b90 (LWP 3256)]
[Thread 0xb7591b90 (LWP 3256) exited]
[New Thread 0xb7591b90 (LWP 3257)]
[New Thread 0xb7555b90 (LWP 3258)]
[Thread 0xb7555b90 (LWP 3258) exited]
[Thread 0xb7591b90 (LWP 3257) exited]
[New Thread 0xb7591b90 (LWP 3259)]
[Thread 0xb7591b90 (LWP 3259) exited]
[New Thread 0xb7591b90 (LWP 3260)]
[New Thread 0xb7555b90 (LWP 3261)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb7591b90 (LWP 3260)]
0x0061ecfe in teletone_dtmf_get (dtmf_detect_state=0xb7b24d24, buf=0xb758e4a0 "0", dur=0x50) at libs/libteletone/src/libteletone_detect.c:466
466		*dur = dtmf_detect_state->dur;

Thread 51 (Thread 0xb7555b90 (LWP 3261)):
#0  0x002c19f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637daa in apr_thread_cond_wait (cond=0x8120040, mutex=0x8120010) at locks/unix/thread_cond.c:68
        rv = -4
#2  0x00571644 in switch_thread_cond_wait (cond=0x8120040, mutex=0x8120010) at src/switch_apr.c:374
No locals.
#3  0x00616c6e in timer_next (timer=0xb7be2574) at src/switch_time.c:676
        private_info = 0xb7be6cf8
        cond_index = 30
#4  0x00588ffc in switch_core_timer_next (timer=0xf95) at src/switch_core_timer.c:74
        __func__ = "switch_core_timer_next"
#5  0x005ca0e8 in rtp_common_read (rtp_session=0xb7bd2320, payload_type=0xb7bce37c "", flags=0xb7bce390, io_flags=0) at src/switch_rtp.c:3193
        do_cng = 0
        read_pretriggered = 1
        session = 0xb7bc6300
        channel = 0xb7bcba38
        bytes = 172
        rtcp_bytes = 0
        status = SWITCH_STATUS_SUCCESS
        poll_status = SWITCH_STATUS_SUCCESS
        rtcp_status = 3081764880
        rtcp_poll_status = <value optimized out>
        check = 0
        ret = <value optimized out>
        sleep_mss = 30000
        poll_loop = 0
        fdr = 0
        rtcp_fdr = 0
        hot_socket = 0
        read_loops = 0
        __func__ = "rtp_common_read"
#6  0x005cb663 in switch_rtp_zerocopy_read_frame (rtp_session=0xb7bd2320, frame=0xb7bce354, io_flags=0) at src/switch_rtp.c:3875
        bytes = <value optimized out>
#7  0x00755815 in sofia_read_frame (session=0xb7bc6300, frame=0xb7555198, flags=0, stream_id=0) at mod_sofia.c:1069
        status = SWITCH_STATUS_GENERR
        channel = 0xb7bcba38
        sanity = <value optimized out>
        rtcp_frame = {report_count = 6301, packet_type = 87, ssrc = 3082605288, ntp_msw = 7452332, ntp_lsw = 3075812680, timestamp = 5745299, packet_count = 3082605288, octect_count = 7452332, nb_reports = 3075812680, reports = {{ssrc = 5870656, fraction = 88 'X', lost = 1, highest_sequence_number_received = 2274, jitter = 5864607, lsr = 0, dlsr = 0}, {ssrc = 1623859, fraction = 172 '\254', lost = 0, highest_sequence_number_received = 3082576640, jitter = 3075812856, lsr = 6270647, dlsr = 3082598968}, {ssrc = 63, fraction = 1 '\001', lost = 7435646, highest_sequence_number_received = 0, jitter = 3075814268, lsr = 1452897, dlsr = 3075812732}, {ssrc = 3075812840, fraction = 2 '\002', lost = 3082598968, highest_sequence_number_received = 7327234, jitter = 34, lsr = 0, dlsr = 0}, {ssrc = 0, fraction = 0 '\000', lost = 0, highest_sequence_number_received = 0, jitter = 0, lsr = 0, dlsr = 0}}}
        __PRETTY_FUNCTION__ = "sofia_read_frame"
        __func__ = "sofia_read_frame"
#8  0x005a47bd in switch_core_session_read_frame (session=0xb7bc6300, frame=0xb7555198, flags=0, stream_id=0) at src/switch_core_io.c:191
        ptr = <value optimized out>
        status = SWITCH_STATUS_SUCCESS
        need_codec = <value optimized out>
        perfect = <value optimized out>
        do_bugs = 0
        do_resample = 0
        is_cng = 0
        codec_impl = {codec_type = 3082608520, ianacode = 172 '\254', iananame = 0xb7553f38 "h?U\267P\220Y", fmtp = 0x71b6ac "\270\063\036", samples_per_second = 3082576640, actual_samples_per_second = 0, bits_per_second = -1219150024, microseconds_per_packet = 5701956, samples_per_packet = 3082608472, decoded_bytes_per_packet = 3075817308, encoded_bytes_per_packet = 3075817272, number_of_channels = 172 '\254', codec_frames_per_packet = -1212390656, init = 0x71b6ac <sqlite3_last_insert_rowid+1751760>, encode = 0xb7553f68, decode = 0x599050 <switch_core_session_dequeue_event+64>, destroy = 0xb7bcdf58, codec_id = 3075817308, impl_id = 3075817320, next = 0x578440}
        flag = 0
        i = <value optimized out>
        __PRETTY_FUNCTION__ = "switch_core_session_read_frame"
        __func__ = "switch_core_session_read_frame"
#9  0x005d04ab in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:482
        status = <value optimized out>
        event = 0x6f90cb
        data = 0xb7be7170
        stream_id = 0
        pre_b = 0
        ans_a = 1
        ans_b = 1
        originator = 0
        input_callback = 0
        msg = {from = 0x0, message_id = SWITCH_MESSAGE_REDIRECT_AUDIO, numeric_arg = 0, string_arg = 0x0, string_arg_size = 0, pointer_arg = 0x0, pointer_arg_size = 0, numeric_reply = 0, string_reply = 0x0, string_reply_size = 0, pointer_reply = 0x0, pointer_reply_size = 0, flags = 0, _file = 0x0, _func = 0x0, _line = 0, string_array_arg = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, delivery_time = 0}
        user_data = 0x0
        chan_a = 0xb7bcba38
        chan_b = 0x80fd9d8
        read_frame = 0x0
        session_a = 0xb7bc6300
        session_b = 0x80f82a0
        read_frame_count = 3
        app_name = <value optimized out>
        app_arg = <value optimized out>
        hook_var = <value optimized out>
        silence_codec = {codec_interface = 0x0, implementation = 0x0, fmtp_in = 0x0, fmtp_out = 0x0, flags = 0, memory_pool = 0x0, private_info = 0x0, agreed_pt = 0 '\000', mutex = 0x0, next = 0x0}
        silence_frame = {codec = 0x0, source = 0x0, packet = 0x0, packetlen = 0, extra_data = 0x0, data = 0x0, datalen = 0, buflen = 0, samples = 0, rate = 0, payload = 0 '\000', timestamp = 0, seq = 0, ssrc = 0, m = SWITCH_FALSE, flags = 0}
        silence_data = {0 <repeats 2048 times>}
        silence_var = 0x0
        silence_val = 0
        bypass_media_after_bridge = 0
        bridge_answer_timeout = <value optimized out>
        sent_update = 0
        answer_limit = 0
        exec_app = 0x0
        exec_data = 0x0
        vid_thread = 0x0
        vh = {session_a = 0x0, session_b = 0x0, up = 0}
        vid_launch = 0
        __func__ = "audio_bridge_thread"
        __PRETTY_FUNCTION__ = "audio_bridge_thread"
#10 0x005d1d91 in audio_bridge_on_exchange_media (session=0xb7bc6300) at src/switch_ivr_bridge.c:645
        channel = 0xb7bcba38
        bd = 0xb7be7170
        state = <value optimized out>
        var = <value optimized out>
        __func__ = "audio_bridge_on_exchange_media"
#11 0x005a0af5 in switch_core_session_run (session=0xb7bc6300) at src/switch_core_state_machine.c:443
        global_proceed = <value optimized out>
        proceed = 1
        ptr = <value optimized out>
        rstatus = <value optimized out>
        state = CS_EXCHANGE_MEDIA
        endstate = <value optimized out>
        endpoint_interface = <value optimized out>
        driver_state_handler = 0x8ca6e0
        application_state_handler = <value optimized out>
        new_loops = 60000
        __PRETTY_FUNCTION__ = "switch_core_session_run"
        __func__ = "switch_core_session_run"
#12 0x0059e6ca in switch_core_session_thread (thread=0xb7bcf020, obj=0xb7bc6300) at src/switch_core_session.c:1412
        session = 0xb7bc6300
        event = <value optimized out>
        event_str = 0x0
        val = <value optimized out>
        __func__ = "switch_core_session_thread"
        __PRETTY_FUNCTION__ = "switch_core_session_thread"
#13 0x0063e0f6 in dummy_worker (opaque=0xb7bcf020) at threadproc/unix/thread.c:138
No locals.
#14 0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#15 0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 50 (Thread 0xb7591b90 (LWP 3260)):
#0  0x0061ecfe in teletone_dtmf_get (dtmf_detect_state=0xb7b24d24, buf=0xb758e4a0 "0", dur=0x50) at libs/libteletone/src/libteletone_detect.c:466
No locals.
#1  0x009329ca in zap_channel_read (zchan=0xb7b24cc8, data=0x827310c, datalen=0xb758e5d8) at src/zap_io.c:2435
        sln_buf = "\000z \372\000\212\000\212\200\340\000\067\000?\200\022\200\347\300\367\000'\000\061\300\364\000\236\000\222\200\353\000f\000z\000J\000\311\000\206\000\232\210\377\000R\000R\200\025\000\337\000\333P\374\300\017\020\375\000\337\000\337\240\006\000\065\000;\300\r\000\321\000\276\000?\023\000\063\000'\350\000\200\346\200\347\240\372\240\a`\005\210\376\220\374\020\002\340\005p\003\b\377H\376\210\377H\377\210\377H\377\210\377\310\377\310\377\310\377\310\377\310\377\310\377\370\377\370\377\370\377\070\000\370\377\070\000\070\000\070\000\070\000\070\000\070\000\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377\370\377", '\000' <repeats 703 times>
        sln = 0xb758e160
        slen = 160
        digit_str = "0", '\000' <repeats 78 times>
        status = ZAP_SUCCESS
        codec_func = <value optimized out>
        max = 4096
        i = <value optimized out>
        __PRETTY_FUNCTION__ = "zap_channel_read"
        __func__ = "zap_channel_read"
#2  0x003488a5 in channel_read_frame (session=0x80f82a0, frame=0xb7590968, flags=0, stream_id=0) at /home/taitadmin/freeswitch/src/mod/../../libs/openzap/mod_openzap/mod_openzap.c:707
        channel = 0x80fd9d8
        len = 160
        wflags = <value optimized out>
        dtmf = '\000' <repeats 127 times>
        status = <value optimized out>
        total_to = 240
        chunk = 40
        do_break = 0
        __PRETTY_FUNCTION__ = "channel_read_frame"
        __func__ = "channel_read_frame"
#3  0x005a47bd in switch_core_session_read_frame (session=0x80f82a0, frame=0xb7590968, flags=0, stream_id=0) at src/switch_core_io.c:191
        ptr = <value optimized out>
        status = SWITCH_STATUS_SUCCESS
        need_codec = <value optimized out>
        perfect = <value optimized out>
        do_bugs = 0
        do_resample = 0
        is_cng = 0
        codec_impl = {codec_type = 135618968, ianacode = 172 '\254', iananame = 0xb758f708 "8\367X\267P\220Y", fmtp = 0x71b6ac "\270\063\036", samples_per_second = 135234208, actual_samples_per_second = 1, bits_per_second = -1218906360, microseconds_per_packet = 5701956, samples_per_packet = 135618920, decoded_bytes_per_packet = 3076060972, encoded_bytes_per_packet = 3076060936, number_of_channels = 172 '\254', codec_frames_per_packet = 135234208, init = 0x71b6ac <sqlite3_last_insert_rowid+1751760>, encode = 0xb758f738, decode = 0x599050 <switch_core_session_dequeue_event+64>, destroy = 0x8156168, codec_id = 3076060972, impl_id = 3076060984, next = 0x578440}
        flag = 0
        i = <value optimized out>
        __PRETTY_FUNCTION__ = "switch_core_session_read_frame"
        __func__ = "switch_core_session_read_frame"
#4  0x005d04ab in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:482
        status = <value optimized out>
        event = 0x6e8ce4
        data = 0xb7be7328
        stream_id = 0
        pre_b = 0
        ans_a = 1
        ans_b = 1
        originator = 1
        input_callback = 0
        msg = {from = 0x0, message_id = SWITCH_MESSAGE_REDIRECT_AUDIO, numeric_arg = 0, string_arg = 0x0, string_arg_size = 0, pointer_arg = 0x0, pointer_arg_size = 0, numeric_reply = 0, string_reply = 0x0, string_reply_size = 0, pointer_reply = 0x0, pointer_reply_size = 0, flags = 0, _file = 0x0, _func = 0x0, _line = 0, string_array_arg = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, delivery_time = 0}
        user_data = 0x0
        chan_a = 0x80fd9d8
        chan_b = 0xb7bcba38
        read_frame = 0x0
        session_a = 0x80f82a0
        session_b = 0xb7bc6300
        read_frame_count = 8
        app_name = <value optimized out>
        app_arg = <value optimized out>
        hook_var = <value optimized out>
        silence_codec = {codec_interface = 0x0, implementation = 0x0, fmtp_in = 0x0, fmtp_out = 0x0, flags = 0, memory_pool = 0x0, private_info = 0x0, agreed_pt = 0 '\000', mutex = 0x0, next = 0x0}
        silence_frame = {codec = 0x0, source = 0x0, packet = 0x0, packetlen = 0, extra_data = 0x0, data = 0x0, datalen = 0, buflen = 0, samples = 0, rate = 0, payload = 0 '\000', timestamp = 0, seq = 0, ssrc = 0, m = SWITCH_FALSE, flags = 0}
        silence_data = {0 <repeats 2048 times>}
        silence_var = 0x1 <Address 0x1 out of bounds>
        silence_val = 0
        bypass_media_after_bridge = 0
        bridge_answer_timeout = <value optimized out>
        sent_update = 1
        answer_limit = 0
        exec_app = 0x0
        exec_data = 0x0
        vid_thread = 0x0
        vh = {session_a = 0x0, session_b = 0x0, up = 0}
        vid_launch = 0
        __func__ = "audio_bridge_thread"
        __PRETTY_FUNCTION__ = "audio_bridge_thread"
#5  0x005d39b7 in switch_ivr_multi_threaded_bridge (session=0x80f82a0, peer_session=0xb7bc6300, input_callback=0, session_data=0x0, peer_session_data=0x0) at src/switch_ivr_bridge.c:1360
        app = <value optimized out>
        data = <value optimized out>
        a_leg = 0x0
        caller_channel = 0x80fd9d8
        peer_channel = 0xb7bcba38
        status = 25
        state = <value optimized out>
        event = 0x0
        br = 1
        var = <value optimized out>
        cause = <value optimized out>
        msg = {from = 0x6f8db0 "src/switch_ivr_bridge.c", message_id = SWITCH_MESSAGE_INDICATE_BRIDGE, numeric_arg = 0, string_arg = 0xb7be7468 "12532daa-c63c-11e1-86f5-af2d94b5a6fd", string_arg_size = 0, pointer_arg = 0x0, pointer_arg_size = 0, numeric_reply = 0, string_reply = 0x0, string_reply_size = 0, pointer_reply = 0x0, pointer_reply_size = 0, flags = 0, _file = 0x0, _func = 0x0, _line = 0, string_array_arg = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, delivery_time = 0}
        __func__ = "switch_ivr_multi_threaded_bridge"
#6  0x004426c1 in audio_bridge_function (session=0x80f82a0, data=0xb7b96fb8 "sofia/gateway/fxoline3/8569") at /home/taitadmin/freeswitch/src/mod/applications/mod_dptools/mod_dptools.c:3120
        channel = <value optimized out>
        peer_channel = 0xb7bcba38
        a_key = 0x0
        ok = <value optimized out>
        func = 0
        b_key = 0x0
        caller_channel = 0x80fd9d8
        peer_session = 0xb7bc6300
        transfer_on_fail = 0x0
        tof_data = <value optimized out>
        tof_array = {0x0, 0x0, 0x0, 0x0}
        v_campon = <value optimized out>
        v_campon_retries = <value optimized out>
        v_campon_sleep = 0x400 <Address 0x400 out of bounds>
        v_campon_timeout = 0xb7b774c0 "sofia/gateway/fxoline3/8569"
        v_campon_fallback_exten = 0xb7590b08 ""
        cause = SWITCH_CAUSE_SUCCESS
        campon_retries = 7452332
        campon_timeout = 28
        campon_sleep = <value optimized out>
        fail = <value optimized out>
        thread_started = -1218901128
        stake = {session = 0x0, running = 0, do_xfer = 0, moh = 0x0}
        moh = 0x1 <Address 0x1 out of bounds>
        thread = 0x0
        thd_attr = 0x0
        camp_data = 0xb7b03e00 "CANCEL"
        status = SWITCH_STATUS_SUCCESS
        camp_loops = -1212718032
        __func__ = "audio_bridge_function"
#7  0x0059d51b in switch_core_session_exec (session=0x80f82a0, application_interface=0xb7b80dc8, arg=0x8275728 "sofia/gateway/${distributor(cisco_fxo)}/8569") at src/switch_core_session.c:2332
        log = <value optimized out>
        lp = <value optimized out>
        event = 0x0
        var = <value optimized out>
        channel = 0x80fd9d8
        expanded = 0xb7b96fb8 "sofia/gateway/fxoline3/8569"
        app = 0x4443b8 "bridge"
        app_uuid_var = <value optimized out>
        msg = {from = 0x6f02ed "src/switch_core_session.c", message_id = SWITCH_MESSAGE_INDICATE_APPLICATION_EXEC, numeric_arg = 0, string_arg = 0x0, string_arg_size = 0, pointer_arg = 0x0, pointer_arg_size = 0, numeric_reply = 0, string_reply = 0x0, string_reply_size = 0, pointer_reply = 0x0, pointer_reply_size = 0, flags = 0, _file = 0x0, _func = 0x0, _line = 0, string_array_arg = {0x4443b8 "bridge", 0xb7b96fb8 "sofia/gateway/fxoline3/8569", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, delivery_time = 0}
        delim = <value optimized out>
        scope = 0
        uuid_str = "125325f8-c63c-11e1-86f2-af2d94b5a6fd\000{Z5\000\020\000\000\000\\F{\231\000\000\000\000@\021(\000 \000\000\000\030\000\000\000@\021(\000\370\020Y\267\001\000\000\000\033\000\000\000\002\000\001\001?\025\b \000\000\000\300\004(\000\\\031o\000  o\000\000\000\000\000\\F{\231\n\304\004\000\230\020Y\267\276\060\031\000\020\215\024\000\364\377'\000\230.\037\b\364\377'\000@\021(\000\000\000\000\000\270\020Y\267\276\060\031\000\020\215\024\000\254\266q\000\310\020Y\267\254\266q\000\n\000\000\000\301\204n\000\350\020Y\267\221\342d\000\000\000\000\000\301\204n\000e\000\000\000\254\266q\000\020?\267\000\000\000\000\370\020Y\267+Ed\000`\177\267\267 W'\b\a\000\000\000\305\037W\000\020\215\024\000\254\266q\000("...
        app_uuid = 0xb7591003 "125325f8-c63c-11e1-86f2-af2d94b5a6fd"
        __PRETTY_FUNCTION__ = "switch_core_session_exec"
        __func__ = "switch_core_session_exec"
#8  0x0059dc7e in switch_core_session_execute_application_get_flags (session=0x80f82a0, app=0x8275720 "bridge", arg=<value optimized out>, flags=0x0) at src/switch_core_session.c:2207
        application_interface = 0xb7b80dc8
        status = SWITCH_STATUS_SUCCESS
        __func__ = "switch_core_session_execute_application_get_flags"
#9  0x005a147c in switch_core_session_run (session=0x80f82a0) at src/switch_core_state_machine.c:217
        global_proceed = <value optimized out>
        proceed = <value optimized out>
        ptr = <value optimized out>
        rstatus = <value optimized out>
        state = <value optimized out>
        endstate = <value optimized out>
        endpoint_interface = <value optimized out>
        driver_state_handler = 0x356360
        application_state_handler = <value optimized out>
        new_loops = 60000
        __PRETTY_FUNCTION__ = "switch_core_session_run"
        __func__ = "switch_core_session_run"
#10 0x0059e6ca in switch_core_session_thread (thread=0x82754b8, obj=0x80f82a0) at src/switch_core_session.c:1412
        session = 0x80f82a0
        event = <value optimized out>
        event_str = 0x0
        val = <value optimized out>
        __func__ = "switch_core_session_thread"
        __PRETTY_FUNCTION__ = "switch_core_session_thread"
#11 0x0063e0f6 in dummy_worker (opaque=0x82754b8) at threadproc/unix/thread.c:138
No locals.
#12 0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#13 0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 39 (Thread 0xb75cdb90 (LWP 3210)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x0063fc79 in apr_sleep (t=100000) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 58000}
#2  0x0061416f in do_sleep (t=100000) at src/switch_time.c:171
        ts = {tv_sec = -1218653288, tv_nsec = 0}
#3  0x0047798a in vm_event_thread_run (thread=0x822bd28, obj=0x0) at /home/taitadmin/freeswitch/src/mod/applications/mod_voicemail/mod_voicemail.c:3886
        pop = 0xb7b78a20
        __func__ = "vm_event_thread_run"
#4  0x0063e0f6 in dummy_worker (opaque=0x822bd28) at threadproc/unix/thread.c:138
No locals.
#5  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 33 (Thread 0xb7609b90 (LWP 3202)):
#0  0x002c4781 in read () from /lib/libpthread.so.0
No symbol table info available.
#1  0x006b6cd3 in read_char (el=0xb7b971d8, cp=0xb760934b "\n\254\266q") at read.c:294
        num_read = -1212583088
        tried = 0
#2  0x006b67bc in el_getc (el=0xb7b971d8, cp=0xb760934b "\n\254\266q") at read.c:362
        num_read = -1218407605
        ma = 0xb7b97478
#3  0x006b68ff in read_getcmd (el=0xb7b971d8, nread=0xb7609398) at read.c:241
No locals.
#4  el_gets (el=0xb7b971d8, nread=0xb7609398) at read.c:497
        cmdnum = 1 '\001'
        num = 1
        ch = 10 '\n'
#5  0x005845ab in console_thread (thread=0xb7bb2a40, obj=0xb7bb28e8) at src/switch_console.c:1026
        arg = 1
        count = 1
        line = <value optimized out>
        pool = 0xb7bb28e8
        __func__ = "console_thread"
        __PRETTY_FUNCTION__ = "console_thread"
#6  0x0063e0f6 in dummy_worker (opaque=0xb7bb2a40) at threadproc/unix/thread.c:138
No locals.
#7  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#8  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 32 (Thread 0xb7645b90 (LWP 3201)):
#0  0x002c19f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637daa in apr_thread_cond_wait (cond=0xb7ba77a8, mutex=0xb7ba7778) at locks/unix/thread_cond.c:68
        rv = -4
#2  0x0062e99c in apr_queue_pop (queue=0xb7ba7748, data=0xb7645398) at misc/apr_queue.c:276
        rv = 0
#3  0x005701f4 in switch_queue_pop (queue=0xb7ba7748, data=0xb7645398) at src/switch_apr.c:1040
No locals.
#4  0x005b8649 in chat_thread_run (thread=0xb7bae738, obj=0xb7ba7748) at src/switch_loadable_module.c:625
        pop = 0x5b85cb
        __func__ = "chat_thread_run"
#5  0x0063e0f6 in dummy_worker (opaque=0xb7bae738) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 31 (Thread 0xb76bdb90 (LWP 3200)):
#0  0x002c19f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637daa in apr_thread_cond_wait (cond=0x811f000, mutex=0x811efd0) at locks/unix/thread_cond.c:68
        rv = -4
#2  0x0062e99c in apr_queue_pop (queue=0x811efa0, data=0xb76bd398) at misc/apr_queue.c:276
        rv = 0
#3  0x005701f4 in switch_queue_pop (queue=0x811efa0, data=0xb76bd398) at src/switch_apr.c:1040
No locals.
#4  0x005b8649 in chat_thread_run (thread=0xb7ba7728, obj=0x811efa0) at src/switch_loadable_module.c:625
        pop = 0x5b85cb
        __func__ = "chat_thread_run"
#5  0x0063e0f6 in dummy_worker (opaque=0xb7ba7728) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 30 (Thread 0xb7681b90 (LWP 3199)):
#0  0x002c499e in accept () from /lib/libpthread.so.0
No symbol table info available.
#1  0x0063ce7d in apr_socket_accept (new=0xb768133c, sock=0x8291220, connection_context=0x82930a0) at network_io/unix/sockets.c:187
No locals.
#2  0x00570c1b in switch_socket_accept (new_sock=0xb768133c, sock=0x8291220, pool=0x82930a0) at src/switch_apr.c:700
No locals.
#3  0x0032c65f in mod_event_socket_runtime () at /home/taitadmin/freeswitch/src/mod/event_handlers/mod_event_socket/mod_event_socket.c:2829
        pool = 0x8291098
        listener_pool = 0x82930a0
        rv = <value optimized out>
        sa = 0x8291168
        inbound_socket = 0x8293170
        listener = 0x0
        x = <value optimized out>
        errs = 0
        __func__ = "mod_event_socket_runtime"
#4  0x005b6d8a in switch_loadable_module_exec (thread=0x811ef50, obj=0x811ed40) at src/switch_loadable_module.c:98
        status = 4294966784
        module = 0xb7b0ac08
        __PRETTY_FUNCTION__ = "switch_loadable_module_exec"
        __func__ = "switch_loadable_module_exec"
#5  0x0063e0f6 in dummy_worker (opaque=0x811ef50) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 29 (Thread 0xb76f9b90 (LWP 3198)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x0063fc79 in apr_sleep (t=9562) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 9000}
#2  0x0061416f in do_sleep (t=9562) at src/switch_time.c:171
        ts = {tv_sec = 822547974, tv_nsec = -1217424712}
#3  0x0061674c in softtimer_runtime () at src/switch_time.c:939
        timediff = 4294966782
        too_late = 20000000
        current_ms = 1490
        x = <value optimized out>
        tick = 29
        ts = <value optimized out>
        last = 1341449712812517
        fwd_errs = 0
        rev_errs = 0
        time_sync = 4
        __func__ = "softtimer_runtime"
#4  0x005b6d8a in switch_loadable_module_exec (thread=0x811ece8, obj=0x811ea48) at src/switch_loadable_module.c:98
        status = 4294966782
        module = 0x811ff30
        __PRETTY_FUNCTION__ = "switch_loadable_module_exec"
        __func__ = "switch_loadable_module_exec"
#5  0x0063e0f6 in dummy_worker (opaque=0x811ece8) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 24 (Thread 0xb7735b90 (LWP 3193)):
#0  0x002c19f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637daa in apr_thread_cond_wait (cond=0x82400f8, mutex=0x82400c8) at locks/unix/thread_cond.c:68
        rv = -4
#2  0x00571644 in switch_thread_cond_wait (cond=0x82400f8, mutex=0x82400c8) at src/switch_apr.c:374
No locals.
#3  0x00ed1752 in timer_thread_run (thread=0x8240208, obj=0x0) at mod_spandsp_fax.c:207
        timer = {interval = 20, flags = 1, samples = 160, samplecount = 160, timer_interface = 0x811fe98, memory_pool = 0x82477e8, private_info = 0x8247878, diff = 0, tick = 0}
        pvt = 0x0
        __func__ = "timer_thread_run"
#4  0x0063e0f6 in dummy_worker (opaque=0x8240208) at threadproc/unix/thread.c:138
No locals.
#5  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 23 (Thread 0xb77a3b90 (LWP 3192)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x0063fc79 in apr_sleep (t=1000000) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 590000}
#2  0x0061416f in do_sleep (t=1000000) at src/switch_time.c:171
        ts = {tv_sec = 7452332, tv_nsec = -1216728328}
#3  0x0044ef68 in node_thread_run (thread=0x820b5e8, obj=0x820aef0) at /home/taitadmin/freeswitch/src/mod/applications/mod_fifo/mod_fifo.c:1901
        ppl_waiting = 0
        consumer_total = 0
        idle_consumers = 0
        node = 0x0
        last = 0x820efb8
        cur_priority = 1
        __func__ = "node_thread_run"
#4  0x0063e0f6 in dummy_worker (opaque=0x820b5e8) at threadproc/unix/thread.c:138
No locals.
#5  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 22 (Thread 0xb77dfb90 (LWP 3191)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x0063fc79 in apr_sleep (t=100000) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 93000}
#2  0x0061416f in do_sleep (t=100000) at src/switch_time.c:171
        ts = {tv_sec = -1216482408, tv_nsec = 0}
#3  0x007ab2c1 in sofia_presence_event_thread_run (thread=0x81352c8, obj=0x0) at sofia_presence.c:1395
        count = 0
        pop = 0xb7b8cee0
        __func__ = "sofia_presence_event_thread_run"
#4  0x0063e0f6 in dummy_worker (opaque=0x81352c8) at threadproc/unix/thread.c:138
No locals.
#5  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 21 (Thread 0xb781bb90 (LWP 3190)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x00382a41 in lpwrap_one_loop (spri=0x812d098) at src/ozmod/ozmod_libpri/lpwrap_pri.c:233
        rfds = {fds_bits = {0, 0, 2048, 0 <repeats 29 times>}}
        efds = {fds_bits = {0, 0, 2048, 0 <repeats 29 times>}}
        now = {tv_sec = 0, tv_usec = 70000}
        next = <value optimized out>
        event = <value optimized out>
        handler = <value optimized out>
        sel = <value optimized out>
        __func__ = "lpwrap_one_loop"
#2  0x00382b6f in lpwrap_run_pri (spri=0x812d098) at src/ozmod/ozmod_libpri/lpwrap_pri.c:274
        ret = 0
        __func__ = "lpwrap_run_pri"
#3  0x003812e1 in zap_libpri_run (me=0x81299c0, obj=0xb7b208e0) at src/ozmod/ozmod_libpri/ozmod_libpri.c:1059
        isdn_data = 0x812d050
        i = 31
        x = 1
        down = 0
        got_d = 1
        __func__ = "zap_libpri_run"
#4  0x00937b4a in thread_launch (args=0x81299c0) at src/zap_threadmutex.c:90
        exit_val = <value optimized out>
#5  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 20 (Thread 0xb7889b90 (LWP 3189)):
#0  0x002c1d08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637d59 in apr_thread_cond_timedwait (cond=0x8151490, mutex=0x8151460, timeout=1000000) at locks/unix/thread_cond.c:89
        rv = <value optimized out>
        then = 1341449713192582
        abstime = {tv_sec = 1341449713, tv_nsec = 192582000}
#2  0x0062e896 in apr_queue_pop_timeout (queue=0x8151430, data=0xb7889394, timeout=1000000) at misc/apr_queue.c:339
        rv = 0
#3  0x005701c2 in switch_queue_pop_timeout (queue=0x8151430, data=0xb7889394, timeout=4294967292) at src/switch_apr.c:1045
No locals.
#4  0x007716ca in sofia_profile_worker_thread_run (thread=0x8151410, obj=0x8150588) at sofia.c:1795
        sleepy_time = <value optimized out>
        ireg_loops = 2
        gateway_loops = 0
        pop = 0x81f1d58
        sql_len = 32768
        sqlbuf = 0x816fbe8 "delete from sip_subscriptions where call_id='72a10054-40de-1230-1f82-bc305be6cb52';\n"
        sql = 0x0
        last_commit = 1341449712182571
        last_check = 1341449712182571
        len = 0
        statements = 0
        __PRETTY_FUNCTION__ = "sofia_profile_worker_thread_run"
        __func__ = "sofia_profile_worker_thread_run"
#5  0x0063e0f6 in dummy_worker (opaque=0x8151410) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 19 (Thread 0xb78c5b90 (LWP 3188)):
#0  0x001fb17c in epoll_wait () from /lib/libc.so.6
No symbol table info available.
#1  0x0085ca30 in su_epoll_port_wait_events (self=0x81646f8, tout=1000) at su_epoll_port.c:495
        j = <value optimized out>
        n = 135719856
        events = <value optimized out>
        index = <value optimized out>
        version = 3
        __PRETTY_FUNCTION__ = "su_epoll_port_wait_events"
#2  0x0086257d in su_base_port_run (self=0x81646f8) at su_base_port.c:349
        tout = 1000
        tout2 = 0
        __PRETTY_FUNCTION__ = "su_base_port_run"
#3  0x00869189 in su_port_run (self=0x8161bb8) at su_port.h:326
No locals.
#4  su_root_run (self=0x8161bb8) at su_root.c:819
        __PRETTY_FUNCTION__ = "su_root_run"
#5  0x0085f257 in su_pthread_port_clone_main (varg=0xb7a54f9c) at su_pthread_port.c:334
        arg = <value optimized out>
        task = {{sut_port = 0x81646f8, sut_root = 0x8161bb8}}
        zap = 0
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 18 (Thread 0xb7933b90 (LWP 3187)):
#0  0x002c1d08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637d59 in apr_thread_cond_timedwait (cond=0x813d440, mutex=0x813d410, timeout=1000000) at locks/unix/thread_cond.c:89
        rv = <value optimized out>
        then = 1341449713539554
        abstime = {tv_sec = 1341449713, tv_nsec = 539554000}
#2  0x0062e896 in apr_queue_pop_timeout (queue=0x813d3e0, data=0xb7933394, timeout=1000000) at misc/apr_queue.c:339
        rv = 0
#3  0x005701c2 in switch_queue_pop_timeout (queue=0x813d3e0, data=0xb7933394, timeout=4294967292) at src/switch_apr.c:1045
No locals.
#4  0x007716ca in sofia_profile_worker_thread_run (thread=0x813d3c0, obj=0x813c5d0) at sofia.c:1795
        sleepy_time = <value optimized out>
        ireg_loops = 0
        gateway_loops = 0
        pop = 0x0
        sql_len = 32768
        sqlbuf = 0x8165108 ""
        sql = 0x0
        last_commit = 1341449711532619
        last_check = 1341449711532619
        len = 0
        statements = 0
        __PRETTY_FUNCTION__ = "sofia_profile_worker_thread_run"
        __func__ = "sofia_profile_worker_thread_run"
#5  0x0063e0f6 in dummy_worker (opaque=0x813d3c0) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 17 (Thread 0xb796fb90 (LWP 3186)):
#0  0x001fb17c in epoll_wait () from /lib/libc.so.6
No symbol table info available.
#1  0x0085ca30 in su_epoll_port_wait_events (self=0x814ffe0, tout=1000) at su_epoll_port.c:495
        j = <value optimized out>
        n = 135590504
        events = <value optimized out>
        index = <value optimized out>
        version = 3
        __PRETTY_FUNCTION__ = "su_epoll_port_wait_events"
#2  0x0086257d in su_base_port_run (self=0x814ffe0) at su_base_port.c:349
        tout = 1000
        tout2 = 0
        __PRETTY_FUNCTION__ = "su_base_port_run"
#3  0x00869189 in su_port_run (self=0x812de38) at su_port.h:326
No locals.
#4  su_root_run (self=0x812de38) at su_root.c:819
        __PRETTY_FUNCTION__ = "su_root_run"
#5  0x0085f257 in su_pthread_port_clone_main (varg=0xb7a90f9c) at su_pthread_port.c:334
        arg = <value optimized out>
        task = {{sut_port = 0x814ffe0, sut_root = 0x812de38}}
        zap = 0
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 16 (Thread 0xb79ddb90 (LWP 3185)):
#0  0x002c1d08 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637d59 in apr_thread_cond_timedwait (cond=0x8139c18, mutex=0x8139be8, timeout=1000000) at locks/unix/thread_cond.c:89
        rv = <value optimized out>
        then = 1341449713533559
        abstime = {tv_sec = 1341449713, tv_nsec = 533559000}
#2  0x0062e896 in apr_queue_pop_timeout (queue=0x8139bb8, data=0xb79dd394, timeout=1000000) at misc/apr_queue.c:339
        rv = 0
#3  0x005701c2 in switch_queue_pop_timeout (queue=0x8139bb8, data=0xb79dd394, timeout=4294967292) at src/switch_apr.c:1045
No locals.
#4  0x007716ca in sofia_profile_worker_thread_run (thread=0x8139b98, obj=0x8138490) at sofia.c:1795
        sleepy_time = <value optimized out>
        ireg_loops = 29
        gateway_loops = 0
        pop = 0x0
        sql_len = 32768
        sqlbuf = 0x81594f0 ""
        sql = 0x0
        last_commit = 1341449712532555
        last_check = 1341449712532555
        len = 0
        statements = 0
        __PRETTY_FUNCTION__ = "sofia_profile_worker_thread_run"
        __func__ = "sofia_profile_worker_thread_run"
#5  0x0063e0f6 in dummy_worker (opaque=0x8139b98) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 15 (Thread 0xb7a55b90 (LWP 3183)):
#0  0x001fb17c in epoll_wait () from /lib/libc.so.6
No symbol table info available.
#1  0x0085ca30 in su_epoll_port_wait_events (self=0xb7b0e920, tout=1000) at su_epoll_port.c:495
        j = <value optimized out>
        n = 0
        events = <value optimized out>
        index = <value optimized out>
        version = 1
        __PRETTY_FUNCTION__ = "su_epoll_port_wait_events"
#2  0x00862374 in su_base_port_step (self=0xb7b0e920, tout=1000) at su_base_port.c:467
        now = <value optimized out>
        __PRETTY_FUNCTION__ = "su_base_port_step"
#3  0x00869540 in su_port_step (self=0xb7b0ed78, tout=1000) at su_port.h:340
No locals.
#4  su_root_step (self=0xb7b0ed78, tout=1000) at su_root.c:858
        __PRETTY_FUNCTION__ = "su_root_step"
#5  0x0077bcb1 in sofia_profile_thread_run (thread=0x8151320, obj=0x8150588) at sofia.c:2223
        profile = <value optimized out>
        node = 0x89bb91
        s_event = 0x0
        sanity = <value optimized out>
        worker_thread = 0x8151410
        st = SWITCH_STATUS_SUCCESS
        __func__ = "sofia_profile_thread_run"
        __PRETTY_FUNCTION__ = "sofia_profile_thread_run"
#6  0x0063e0f6 in dummy_worker (opaque=0x8151320) at threadproc/unix/thread.c:138
No locals.
#7  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#8  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 14 (Thread 0xb7a19b90 (LWP 3184)):
#0  0x001fb17c in epoll_wait () from /lib/libc.so.6
No symbol table info available.
#1  0x0085ca30 in su_epoll_port_wait_events (self=0x8150068, tout=1000) at su_epoll_port.c:495
        j = <value optimized out>
        n = 135625752
        events = <value optimized out>
        index = <value optimized out>
        version = 3
        __PRETTY_FUNCTION__ = "su_epoll_port_wait_events"
#2  0x0086257d in su_base_port_run (self=0x8150068) at su_base_port.c:349
        tout = 1000
        tout2 = 0
        __PRETTY_FUNCTION__ = "su_base_port_run"
#3  0x00869189 in su_port_run (self=0x81566b0) at su_port.h:326
No locals.
#4  su_root_run (self=0x81566b0) at su_root.c:819
        __PRETTY_FUNCTION__ = "su_root_run"
#5  0x0085f257 in su_pthread_port_clone_main (varg=0xb7accf9c) at su_pthread_port.c:334
        arg = <value optimized out>
        task = {{sut_port = 0x8150068, sut_root = 0x81566b0}}
        zap = 0
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 13 (Thread 0xb7a91b90 (LWP 3182)):
#0  0x001fb17c in epoll_wait () from /lib/libc.so.6
No symbol table info available.
#1  0x0085ca30 in su_epoll_port_wait_events (self=0x8142bb8, tout=1000) at su_epoll_port.c:495
        j = <value optimized out>
        n = 0
        events = <value optimized out>
        index = <value optimized out>
        version = 1
        __PRETTY_FUNCTION__ = "su_epoll_port_wait_events"
#2  0x00862374 in su_base_port_step (self=0x8142bb8, tout=1000) at su_base_port.c:467
        now = <value optimized out>
        __PRETTY_FUNCTION__ = "su_base_port_step"
#3  0x00869540 in su_port_step (self=0x8145f58, tout=1000) at su_port.h:340
No locals.
#4  su_root_step (self=0x8145f58, tout=1000) at su_root.c:858
        __PRETTY_FUNCTION__ = "su_root_step"
#5  0x0077bcb1 in sofia_profile_thread_run (thread=0x813d2e0, obj=0x813c5d0) at sofia.c:2223
        profile = <value optimized out>
        node = 0x89bb91
        s_event = 0x0
        sanity = <value optimized out>
        worker_thread = 0x813d3c0
        st = SWITCH_STATUS_SUCCESS
        __func__ = "sofia_profile_thread_run"
        __PRETTY_FUNCTION__ = "sofia_profile_thread_run"
#6  0x0063e0f6 in dummy_worker (opaque=0x813d2e0) at threadproc/unix/thread.c:138
No locals.
#7  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#8  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 12 (Thread 0xb7acdb90 (LWP 3181)):
#0  0x001fb17c in epoll_wait () from /lib/libc.so.6
No symbol table info available.
#1  0x0085ca30 in su_epoll_port_wait_events (self=0x813c408, tout=1000) at su_epoll_port.c:495
        j = <value optimized out>
        n = 0
        events = <value optimized out>
        index = <value optimized out>
        version = 1
        __PRETTY_FUNCTION__ = "su_epoll_port_wait_events"
#2  0x00862374 in su_base_port_step (self=0x813c408, tout=1000) at su_base_port.c:467
        now = <value optimized out>
        __PRETTY_FUNCTION__ = "su_base_port_step"
#3  0x00869540 in su_port_step (self=0x812d818, tout=1000) at su_port.h:340
No locals.
#4  su_root_step (self=0x812d818, tout=1000) at su_root.c:858
        __PRETTY_FUNCTION__ = "su_root_step"
#5  0x0077bcb1 in sofia_profile_thread_run (thread=0x8139128, obj=0x8138490) at sofia.c:2223
        profile = <value optimized out>
        node = 0x89bb91
        s_event = 0x0
        sanity = <value optimized out>
        worker_thread = 0x8139b98
        st = SWITCH_STATUS_SUCCESS
        __func__ = "sofia_profile_thread_run"
        __PRETTY_FUNCTION__ = "sofia_profile_thread_run"
#6  0x0063e0f6 in dummy_worker (opaque=0x8139128) at threadproc/unix/thread.c:138
No locals.
#7  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#8  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 11 (Thread 0xb7c9bb90 (LWP 3180)):
#0  0x002c19f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637daa in apr_thread_cond_wait (cond=0xb7afedd0, mutex=0xb7afeda0) at locks/unix/thread_cond.c:68
        rv = -4
#2  0x0062e99c in apr_queue_pop (queue=0xb7afed70, data=0xb7c9b398) at misc/apr_queue.c:276
        rv = 0
#3  0x005701f4 in switch_queue_pop (queue=0xb7afed70, data=0xb7c9b398) at src/switch_apr.c:1040
No locals.
#4  0x0078a04f in sofia_msg_thread_run (thread=0x8134bd0, obj=0xb7afed70) at sofia.c:1386
        pop = 0xb7b8b538
        my_id = <value optimized out>
        __func__ = "sofia_msg_thread_run"
#5  0x0063e0f6 in dummy_worker (opaque=0x8134bd0) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 9 (Thread 0xb7cd7b90 (LWP 3178)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x0063fc79 in apr_sleep (t=500000) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 327000}
#2  0x0061416f in do_sleep (t=500000) at src/switch_time.c:171
        ts = {tv_sec = 135612016, tv_nsec = 5959643}
#3  0x005af344 in switch_scheduler_task_thread (thread=0x8117e70, obj=0x0) at src/switch_scheduler.c:171
        __func__ = "switch_scheduler_task_thread"
#4  0x0063e0f6 in dummy_worker (opaque=0x8117e70) at threadproc/unix/thread.c:138
No locals.
#5  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 8 (Thread 0xb7e15b90 (LWP 3177)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x0063fc79 in apr_sleep (t=1000000) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 824000}
#2  0x0061416f in do_sleep (t=1000000) at src/switch_time.c:171
        ts = {tv_sec = 136186440, tv_nsec = 7474448}
#3  0x0059130f in switch_core_sql_db_thread (thread=0xb7d39b08, obj=0x0) at src/switch_core_sqldb.c:979
        sec = 10
        reg_sec = 10
#4  0x0063e0f6 in dummy_worker (opaque=0xb7d39b08) at threadproc/unix/thread.c:138
No locals.
#5  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 7 (Thread 0xb7dd9b90 (LWP 3176)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x0063fc79 in apr_sleep (t=200000) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 60000}
#2  0x0061416f in do_sleep (t=200000) at src/switch_time.c:171
        ts = {tv_sec = 137175627, tv_nsec = 0}
#3  0x005972c5 in switch_core_sql_thread (thread=0xb7d39ae8, obj=0x0) at src/switch_core_sqldb.c:1133
        pop = 0x0
        iterations = 3
        trans = 1 '\001'
        len = 517
        sql_len = 105306
        sqlbuf = 0x82d20a0 ""
        sql = <value optimized out>
        save_sql = 0x0
        lc = <value optimized out>
        wrote = 1
        do_sleep = 1
        auto_pause = 0
        __PRETTY_FUNCTION__ = "switch_core_sql_thread"
        __func__ = "switch_core_sql_thread"
#4  0x0063e0f6 in dummy_worker (opaque=0xb7d39ae8) at threadproc/unix/thread.c:138
No locals.
#5  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 4 (Thread 0xb7e51b90 (LWP 3173)):
#0  0x002c19f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637daa in apr_thread_cond_wait (cond=0x8065640, mutex=0x8065610) at locks/unix/thread_cond.c:68
        rv = -4
#2  0x0062e99c in apr_queue_pop (queue=0x80655e0, data=0xb7e51398) at misc/apr_queue.c:276
        rv = 0
#3  0x005701f4 in switch_queue_pop (queue=0x80655e0, data=0xb7e51398) at src/switch_apr.c:1040
No locals.
#4  0x006088e0 in log_thread (t=0xb7eb3ae0, obj=0x0) at src/switch_log.c:294
        pop = 0x0
        node = 0x0
        binding = 0x0
        __func__ = "log_thread"
#5  0x0063e0f6 in dummy_worker (opaque=0xb7eb3ae0) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 3 (Thread 0xb7f52b90 (LWP 3172)):
#0  0x002c19f0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00637daa in apr_thread_cond_wait (cond=0x8060258, mutex=0x8060228) at locks/unix/thread_cond.c:68
        rv = -4
#2  0x0062e99c in apr_queue_pop (queue=0x80601f8, data=0xb7f52398) at misc/apr_queue.c:276
        rv = 0
#3  0x005701f4 in switch_queue_pop (queue=0x80601f8, data=0xb7f52398) at src/switch_apr.c:1040
No locals.
#4  0x005c058f in switch_event_dispatch_thread (thread=0x8065440, obj=0x80601f8) at src/switch_event.c:264
        pop = 0x0
        event = 0x0
        my_id = 0
        __func__ = "switch_event_dispatch_thread"
#5  0x0063e0f6 in dummy_worker (opaque=0x8065440) at threadproc/unix/thread.c:138
No locals.
#6  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#7  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 2 (Thread 0xb7f8eb90 (LWP 3171)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x0063fc79 in apr_sleep (t=1000000) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 411000}
#2  0x0061416f in do_sleep (t=1000000) at src/switch_time.c:171
        ts = {tv_sec = 0, tv_nsec = -1208425608}
#3  0x0058bfc6 in pool_thread (thread=0xb7fbfe38, obj=0x0) at src/switch_core_memory.c:555
No locals.
#4  0x0063e0f6 in dummy_worker (opaque=0xb7fbfe38) at threadproc/unix/thread.c:138
No locals.
#5  0x002bd6e1 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x001faaee in clone () from /lib/libc.so.6
No symbol table info available.

Thread 1 (Thread 0xb7ff39c0 (LWP 3168)):
#0  0x001f34b7 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x0063fc79 in apr_sleep (t=1000000) at time/unix/time.c:246
        tv = {tv_sec = 0, tv_usec = 637000}
#2  0x0061416f in do_sleep (t=1000000) at src/switch_time.c:171
        ts = {tv_sec = 245760, tv_nsec = -1213202416}
#3  0x00582cc7 in switch_console_loop () at src/switch_console.c:1144
        arg = 1
        thread = 0xb7bb2a40
        thd_attr = 0xb7bb2978
        pool = 0xb7bb28e8
        __func__ = "switch_console_loop"
        __PRETTY_FUNCTION__ = "switch_console_loop"
#4  0x005a94c8 in switch_core_runtime_loop (bg=0) at src/switch_core.c:870
No locals.
#5  0x0804acb9 in main (argc=1, argv=0xbfffeb24) at src/switch.c:936
        pid_path = "/usr/local/freeswitch_TR/run/freeswitch.pid", '\000' <repeats 212 times>
        pid_buffer = "3168", '\000' <repeats 27 times>
        old_pid_buffer = '\000' <repeats 31 times>
        pid_len = 4
        old_pid_len = 1385573
        err = 0x6f7fcc "Success"
        nf = 0
        runas_user = 0x0
        runas_group = 0x0
        nc = 0
        pid = <value optimized out>
        x = <value optimized out>
        opts = <value optimized out>
        opts_str = '\000' <repeats 1023 times>
        local_argv = {0xbfffec24 "/usr/local/freeswitch_TR/bin/freeswitch", 0x0 <repeats 1023 times>}
        local_argc = 1
        arg_argv = {0x0 <repeats 128 times>}
        alt_dirs = 0
        log_set = 0
        run_set = 0
        do_kill = 0
        known_opt = 0
        priority = 0
        do_wait = 0
        flags = 68225
        ret = <value optimized out>
        destroy_status = <value optimized out>
        fd = 0x8054548
        pool = 0x8054508
        rlp = {rlim_cur = 245760, rlim_max = 245760}
        waste = 0


Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list