Jayantha Dhanapala announced his withdrawal after finishing last in three informal polls among members of the UN Security Council over who will succeed Annan, who steps down at the end of the year.
"Sri Lanka ... has now decided not to further pursue the candidature of Ambassador Jayantha Dhanapala in the interest of ensuring a consensus in electing an Asian candidate," the foreign ministry said.
The government said it hoped its action would help Ban, the South Korean foreign minister and front-runner for the UN's top job.
Annan has served two five-year terms and his successor is widely expected to come from Asia under an unwritten rule of regional rotation.
Dhanapala, 67, is an adviser to Sri Lankan President Mahinda Rajapakse.
His exit leaves Ban and five other candidates in the race.
The others are India's Shashi Tharoor, Annan's under secretary general for communications; Latvian President Vaira Vike-Freiberga; Surakiart Sathirathai, a deputy prime minister of the ousted Thai r