then call the routine:
B = fmincon(@(B) norm(X - X*B*B','fro'),B0,[],[],[],[],[],[],@nonLinCon)
with B0 being a good guess on what the answer will be.
Also, you need to understand that this algorithms tries to find a local minimum, which may not be the solution you ultimately want. For instance: