Hi,

FreeSwitch software working well in a few days (~3 - 5 days), then new incoming call requests are accepted since FreeSwitch is blocked !! Ongoing calls continue their session, their calls seems not effected, but new calls are not accepted. I got FreeSwitch snapshot and analyzed it in GDB.

I have 601 therads & most of them are waiting

Thread 0x7f16bc55f700 (LWP 28544) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

When i apply "thread apply all bt" in gdb, I see most of the threads try to push events into queue (switch_queue_push )

Thread 600 (Thread 0x7f16bc55f700 (LWP 28544)):

#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

#1 0x00007f180cf9b87d in apr_thread_cond_wait (cond=<optimized out>, mutex=<optimized out>) at locks/unix/thread_cond.c:68

#2 0x00007f180cf92dd0 in apr_queue_push (queue=queue@entry=0x7f180db157a8, data=data@entry=0x7f16d3d5ec20) at misc/apr_queue.c:166

#3 0x00007f180cc958fb in switch_queue_push (queue=0x7f180db157a8, data=data@entry=0x7f16d3d5ec20) at src/switch_apr.c:1134

#4 0x00007f180cd17850 in switch_event_queue_dispatch_event (eventp=0x7f16bc55ec48) at src/switch_event.c:384

#5 switch_event_fire_detailed (file=file@entry=0x7f180cfb07ea "src/switch_channel.c", func=func@entry=0x7f180cfb2ba0 <__func__.18348> "switch_channel_perform_set_running_state", line=line@entry=2260, event=event@entry=0x7f16bc55ec48, user_data=user_data@entry=0x0) at src/switch_event.c:1986

#6 0x00007f180cc9f118 in switch_channel_perform_set_running_state (channel=0x7f17e3e7de00, state=CS_NEW, file=0x7f180cfbc590 "src/switch_core_state_machine.c", func=<optimized out>, line=543) at src/switch_channel.c:2260

#7 0x00007f180ccc87d0 in switch_core_session_run (session=0x7f17e3e7fd28) at src/switch_core_state_machine.c:543

#8 0x00007f180ccc36de in switch_core_session_thread (thread=<optimized out>, obj=0x7f17e3e7fd28) at src/switch_core_session.c:1629

#9 0x00007f180ccbf47d in switch_core_session_thread_pool_worker (thread=0x7f17e3e9abb0, obj=0x80) at src/switch_core_session.c:1692

#10 0x00007f180cfa1910 in dummy_worker (opaque=0x7f17e3e9abb0) at threadproc/unix/thread.c:151

#11 0x00007f180c1e0064 in start_thread (arg=0x7f16bc55f700) at pthread_create.c:309

#12 0x00007f180b8b862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

More interesting thing is below, when I look up event type, approximately all of them are "SWITCH_EVENT_CHANNEL_STATE" and switch_queue (i think sofia_module queue is used in this scenario ) become full !!! nelts (number of elements ) and bounds values are equal, and there are 553 (full_waiters) waiters try to push , but no body try to consume it (empty_waiters = 0)

(gdb) print *(switch_queue_t *) 0x7f180db157a8

$1 = {

data = 0x7f1805cfe038,

nelts = 50000,

in = 43000,

out = 43000,

bounds = 50000,

full_waiters = 553,

empty_waiters = 0,

one_big_mutex = 0x7f180db157e8,

not_empty = 0x7f180db15838,

not_full = 0x7f180db15890,

terminated = 0

}

(gdb) print *(switch_event_t *) 0x7f16d3d5ec20

$1 = {

event_id = SWITCH_EVENT_CHANNEL_STATE,

priority = SWITCH_PRIORITY_NORMAL,

owner = 0x0,

subclass_name = 0x0,

headers = 0x7f16d3d5f750,

last_header = 0x7f16d3d601d0,

body = 0x0,

bind_user_data = 0x0,

event_user_data = 0x0,

key = 0,

next = 0x0,

flags = 0

}

Why i am gonna getting this state?

Any thoughts, tips, tricks would be much appreciated.

Regards,

Angel