Why are Sports Leagues generally exempt from Affirmative Action policies?

It's basically a business of private contractors. You have to be the best to be offered a contract, and there are very few people who have the skills to offer. There are millions of people all qualified for office jobs, but only a select handful of people who are qualified to be professional athletes.