I think it is fair, because Louis would vote to keep his own act.
The Conway Sisters aside (and Simon made the right decision at that time and gained himself some points with the public!) the judges always vote for their own.
Which, the majority of the time, is quite right too - the judges are there to look out for their category, supposedly.
If it is NOT a group in the bottom two, I do think it's possible it might work out unfairly for the one who goes. Because, if Louis took it to deadlock, the public vote might have sent someone else home.
If that happens we will have to wait for some "leaked" voting figures to establish if the outcome was fair or not. And that will depend on whether you believe these figures or not.