NOTE: This topic was split from the "My dream temple run" topic in the Treasure Room.
Yeah, you really can't tweak the speeds too much before things get unrealistic. Just two or three notches in either direction, at the most. 
Heh... yeah, there are definitely some elements of the game that could be refined, like that. The way it works in-game, in case you're wondering, is as follows:
1. There is a "difficulty factor" which is set to something like 195 by default (it's 150 in "easier" mode and 240 in "harder" mode, but not many people know where the cheats menu is to change this setting).
2. The skill of each player ranges from 50 to 150.
3. Subtract the player's skill value from the difficulty factor, then divide this by 100. This number gets multiplied by all the nominal objective completion times in the temple to determine how long something takes. If you work out the numbers, the fastest possible player will have a speed that is about three times the slowest possible player (on the default difficulty). This seemed realistic to me when I was making the game, but it probably could have used some more refinement.
If you want to know some more secrets about the game, just ask. There are a ton of little quirks about the game I'd love to share with people if they're interested.