In an open letter to the WHO, 150 leading international scientists, doctors and medical ethicists said new findings about the mosquito-borne disease made it "unethical" for Rio 2016 to go ahead as planned.
Representatives of institutions such as Oxford University and Harvard and Yale universities in the United States cite the failure of a mosquito-eradication programme in Brazil, and the country's "weakened" health system as reasons to move or postpone the Games in "the name of public health".
"An unnecessary risk is posed when 500,000 foreign tourists from all countries attend the Games, potentially acquire that strain, and return home to places where it can become endemic," the letter reads.
The biggest risk, it claims, is if athletes contract the virus and return home to poor countries that have not yet suffered a Zika outbreak.
In a statement, the WHO said: "Brazil is one of almost 60 countries and territories which to date report continuing transmission of Zika by mosquitoes.
"People continue to travel between these countries and territories for a variety of reasons.
"The best way to reduce risk of disease is to follow public health travel advice."
The letter also expresses concern that the WHO has a conflict of interest because of its partnership with the International Olympic Committee (IOC), which has said it sees no reason to delay or move the Games because of Zika.
Earlier this month, scientists in Brazil warned that Zika could be even more dangerous than first thought.
Experts believe it could be behind several damaging neurological conditions - in addition to microcephaly - where babies are born with abnormally small heads because of restricted brain development - and Guillain-Barré syndrome, a neurological disorder which causes muscle weakness.
Dr Tom Frieden, the head of the US Centers for Disease Control and Prevention (CDC), which confirmed the link between Zika and microcephaly, says "there is no public health reason to cancel or delay the Olympics".
He has called on the US, however, to act faster to prevent pregnant women contracting Zika.
This comes amid congressional deadlock over the release of $1.9 billion (£1.3 billion/€1.7 billion) in funding.
Pregnant women are at particular risk due to a link with microcephaly.
WHO, whose director general Margaret Chan declared the Zika virus a global public health emergency earlier this year, has advised women who are pregnant to not attend the Games.