South African novelist Damon Galgut pulled out explicitly because of the campaign, while Turkish Nobel laureate Orhan Pamuk and fellow writer Kiran Desai withdrew last week for reasons which remain unclear.
But in Galle, a picturesque colonial-era fort town where the festival began on Wednesday, others including the organisers questioned the logic of targeting an independently-run festival that promotes open debate.
"To call for a boycott of the festival is an act of silencing that I find totally unacceptable," said Sunila Abeysekara, a prominent Sri Lankan human rights and media freedom activist, during a panel discussion on Thursday.
Nigerian novelist Chimamanda Ngozi Adichie said it was obvious that free speech was limited in Sri Lanka, but this did not mean authors should stay away.
"My take is that the way to deal with bad speech is to talk about it," commented the author of "Half of a Yellow Sun", which was shortlisted for the 2007 Orange Prize.
"Literature discussions