I have a similar scenario, and my solution was to create several conferences. I have one master conference, that serves to connect all sub-conferences. So when a user calls in to the conference I add him to a sub conference, then I do an outdial to an extension that joins the sub conference to the master conference (this connection should not be muted). Once the sub conference has 250 participants, I create a new sub conference. I control all of this via ESL, this also requires some additional logic to handle conference commands, as I need to make sure it propogates to all sub conferences.
The advantage of this approach is that instead of having one loop running on 1 CPU core looping through all listeners we now have multiple conference loops running on different cores. The issue with large conferences is that the conference loop runs every 20 ms by default and the main conference loop is not (and probably can't be) multi-threaded.
Since on any large conference, there will most likely be only a handful of speakers, the best solution would probably be to optimize mod_conference to have a separate list of unmuted participants and only loop through those, but I was afraid to undertake such a change myself.