I'm having the same issue when connecting to the ESL from Django through Apache WSGI.I increased the number of processes with the processes=n option in the WSGIDaemonProcess directive and it first seems to work but then I realized that it works only sometimes and I also notice it slows down all the web server.
When ESLconnection is called, the Apache process/thread freezes and doesn't respond anything to the request so I couldn't obtained any debug trace of what is really happening. Using the Django built-in server all works OK so it seems a problem between WSGI and Freeswitch Python ESL.
Any help would be really appreciated!