I don't understand why you can't join your previous company once again. There are lots of advantages, viz, you know the work culture, environment, and people very well. This is the place where you reinvent yourself, and you can see the impact also. If you are joining in a senior position, you know what should be/can be done to improvise on certain matters.
As long as perceptions of the colleagues are concerned, it is a case of seeking others' approval. If you belong to that personality, this will remain true even if you join work at a new place. Justify your action to yourself, and the outcome will justify the rest all.