Check out the wiki to see the conference API, check the count of members on a conference using that API from the dialplan, and if it's at the max then use Application=respond, data=486, to blow off the call with a busy.

Sorry I don't have the details for you, but I wanted to point you in the right direction ASAP.

--Dave

From: Daniel Eiland [mailto:daniel.eiland@gmail.com]
To: freeswitch-users@lists.freeswitch.org
Sent: Tue, 09 Oct 2012 11:27:12 -0700
Subject: [Freeswitch-users] Reject incoming call (namely session) on conference member limit

Hey folks,

I was poking around the mod_conference settings and saw that I can limit the maximum members in a conference (via the max-members parameter). However, once this limit is reached, the next incoming call is still answered and played a tone (designated via max-members-sound) before it is simply hung-up.

In the SIP world, this looks like an INVITE-200-ACK sequence followed by a BYE-200 sequence. Is there anyway I can skip the BYE-200 sequence and simply reject the call outright with say a 486 response to the INVITE?

Thanks,
Daniel