i think he was more annoyed that he'd not seen it coming.
after all, hed seen off the master, the timelords, gallifrey, daleks, cybermen, in fact, just about everything, to find the most innocent person he'd have expected would bring about his demise
after all, if it was the master or a cyberman stuck in there, he'd have let him die, wouldnt he?