I think the main problem is with his face - when trying to look angry he looks like he's trying a to pass an uncomfortably large stool, when happy he has this creepy half-smile which never quite reaches his eyes, like he's pretending to understand the joke. And the rest of the time he just looks startled.

