They didn't swap the teams much in the early series - normally only when the teams weren't balanced. The first time they changed people without an imbalance was in the ninth episode of Series 2, when Paul and Ruth were swapped. I presumed that was a gender thing - there were six candidates left, four men and two women, so it made more sense for each team to have two men and one woman.
Sometimes they swap people if they can't work with each other. The only time that that has been explicitly stated to be the reason was in Series 1 with Paul and Saira, but I think that there may have been other occasions as well. For example, in Series 8 Katie was moved to the boys' team in Week 3 - in both the previous tasks, she had fallen out with the other girls quite heavily, so I felt like that was a way of giving her a fairer shot. Also there have been times when someone has been challenged in the second week or so by switching teams and being made the Project Manager, such as Stella in Series 6 and Tim in Series 9.