What I read that quote is saying is that if he wishes things and declines things, he might as well be a slave. Not that slavery is what Stoics think is a natural way of the human spirit.
"Show me a man who is not a slave; one is a slave to lust, another to greed, another to ambition, and all...