Despite the great significance of Hajj, it is sufficient to do it only one time in your whole life. Islam is easy and takes into consideration the burdens and difficulties that pilgrims go through:
Abu Hurayrah (may Allah be pleased with him) narrated that one day the Prophet (s.a.w) addressed people saying: "O people! Allah has prescribed Hajj upon you, so perform it."
A man asked, 'Every year, O Messenger of Allah?'

The Prophet (s.a.w) kept silent. When the man repeated his question thrice, the Prophet (s.a.w) said: "Had I answered in the affirmative, it would have become a (yearly) obligation, and this would have been beyond your capacity."

Then he added: "Leave me alone so long as I leave you alone (i.e. do not ask questions about things I didn't mention). What caused the destruction of the people prior to you was that they used to ask so many questions, and disagree with their Prophets. So when I command you to do something, do it to the extent of your ability, and if I forbid you from doing something, avoid it."