Malaysian troops are negotiating with about 100 men from the Philippines who have identified themselves as the 'royal army' of the Sulu Sultanate, which has a historic land claim to the area, say police.
The Philippines massacre of 46 people on Monday on Mindanao appears to have been politically motivated, with fingers pointing to a powerful local clan.
Fighting began after the Supreme Court halted a historic deal to expand a Muslim autonomous zone in the region.
The Supreme Court prevented the signing of a territorial accord between the state and MILF, a rebel group, Monday. Opponents had called the deal unconstitutional.