This is weird, I don't know if it's the acting, the script or the editing but none of it hangs together. It's a lot of separate scenes trying to tell a story and failing.
Somehow I have to agree with you.
Why was he so agressive at first with Miss Towler when she went to his house? Is he a bit of a grump and a tyrant under that charming outerfacade?