I wouldn't exclude the possibility that there's another - as yet unknown - party involved.
If the Silence were present when plastic Rory was created, that would mean they were also "seen" by Daleks, Cybermen and all the other races and species that were also there. Making the Silence so powerful that not even the most powerful species in the universe could detect them - including some that reply on inanimate enhancements - is maybe a step too far (and raises questions about why they picked on our relatively primative planet in the first place when so much tech was already there for the taking - including a Tardis).
We still don't know why they "needed" a 1960's spacesuit? Apparently, they didn't need the rocket or a trip to the moon* and it's fair to assume that they got to Earth using tech so why even involve humans in building it? It's almost as if they made the suit and then had to wait around till humans had one like it - like your wife hanging on to 300 old frocks - in case they ever some back into fashion
There's still a big question mark over the killing of "Joy". She wouldn't be forgotten and someone disappearing after passing through security at the Whitehouse would be noticed very quickly (plus there was "residue" - she wasn't completely phyiscally erased). It's possible that the Silence could brainwash the security guards into not noticing that didn't leave - or even erase here from everyone's memory - but that still leaves at least one of them going round that bathroom with a mop and bucket

It also seems out of character for the Silence to kill someone - totally unnecessary and risks bring attention to themselves (a guard was stood just outside the door) - unless it was to make some point to Amy - to instill fear that would remain after the actual memory had faded. It stands out for me that they didn't take any violent action against the Doctor and his team despite obviously seeing that they presented far greater threats than some short-sighted old woman who just wanted to take a leak.
My take on the Silence is that they have been waiting for a specific event that couldn't be forced - such as the birth of some Golden Child. That implies either propecy or time travel (or both with the ability to time travel being "lost" or removed away along the way). There's a popular plot-line in sci-fi (and Von Daniken style books) that the Earth was "seeded" by an alien race - either as a social experiment, to continue a species/expand territory or to create a specific event at a fixed point in our evolution. Dr Who (the TV series) has always tapped into this sort of populat story line and just tweaked it a litte - and I think that's what series 5 and 6 have been about.
Given that Moffat seems to be undoing a lot of the changes made by RTD, I'm going for the Silence being proto-Time Lords or at least a species planted by the Time Lords as a kind of reset switch or safety net.
*Allowing for possible logical flaws in the story but - everything onboard Apollo was weighed to check payload wasn't exceeded - you might not "see" them, but the Silence have mass and that means they have weight. Logically, it was impossible for a Silence to get aboard Apollo 11 - (a mobile, empty suit doesn't work as they didn't wear helmets during take-off and flight)
Just chewing things over