Malaysia and Indonesia have vowed to turn back any boats carrying migrants that enter their waters, as the UN calls for "a massive international search and rescue operation" in the Andaman Sea and Straits of Malacca.