You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).
Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.
FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.
Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.
I have run of out of possible explanations, so calling you guys for help.
Thank you.