Justice Minister and SLMC leader Rauf Hakeem told reporters Rajapakse's administration was at fault for letting the bloodshed escalate.
"We are convinced that the government did not do anything to prevent the violence against the Muslims," said Hakeem, who is the most senior Muslim in Rajapakse's cabinet.
Rajapakse, who was visiting Bolivia when the violence broke out, travelled to the resort region of Beruwala shortly after his return to the island and met local community leaders from both sides.
Hundreds of troops were deployed to help police after the extremist Buddhist Force, or BBS, marched in neighbouring Alutgama on Sunday, with clashes breaking out as it claimed its procession was stoned by Muslims.
At least four people were killed and nearly 80 seriously wounded, while doz