Multimode instability in optical bistability