Choosing an order parameter is an art. Usually it's a new phase which we don't understand yet, and guessing the order parameter is a piece of figuring out what's going on. Also, there is often more than one sensible choice. In magnets, for example, one can treat M as a fixed-length vector in , labelling the different broken symmetry states. This is the best choice at low temperatures, where we study the elementary excitations and topological defects. For studying the transition from low to high temperatures, when the magnetization goes to zero, it is better to consider M as a vector of varying length (a vector in three dimensional space). Finding the simplest description for your needs is often the key to the problem.
Statistical Mechanics: Entropy, Order Parameters, and Complexity, now available at Oxford University Press (USA, Europe).