Why fight him in whitehall when he can take him to Scotland where he knew he had the advantage.
Because he didn't have the advantage there. The enemy had attack helicopters and a large supply of goons and weapons. Bond had a shotgun and an old gameskeeper (or whatever he was). Let's not forget how badly it turned out for him: he failed in his primary goal.