I suppose that's right if you're talking about Sylvester the Jester, but I think the reverse is actually true. Magicians constantly demonstrate their control over the uncontrollable. If we look at magicians like Lennart Green, his act is all about order being derived out of total chaos. Indeed he makes this explicit in his patter about chaos theory and it's a point he lectures on in discussing his material with magicians. Lennart's the most obvious example, but the same is actually true for most magicians.
Even an act like Cardini's, in which "stuff happens" to the magician apparently beyond his control, is an example of this. That's because the audience knows that the secret actor behind the madness - the one controlling events - is the magician. Most people get this and admire the magician's skill, but for those that don't Cardini put in a few sign posts so that even the most obtuse person wouldn't have to struggle with the obvious. The best example of this is the fact that Cardini ended his billiard ball sequence with the stark, unadorned manipulation of a single ball. No magic, just skill and performed out of character - reminding everyone that the agency controlling the fabric of time and space is the magician.
So, an alternative (not contrarian) take on this.