I was thinking...
You have to give even the stupidest-looking contestants a bit of credit. They're racing against time through a huge vertical labyrinth, while forcing forth their memory of the brief introductions to the temple's rooms, under absolutely extreme pressure. We've all been nervious before and know its consequences... perhaps some of the contestants just couldn't stand it?
Heck, the temple is designed to be intimidating. The guards don't help it either. The contestants, while some obviously brighter than others, are all under that same pressure to do well and win the vacation.