The phrases “on the contrary” and “to the contrary” are used to reply to an opposing point.
When a distinction is being made that does not involve opposition of this sort, “in contrast” is appropriate.
Here’s a simple test: if you could possibly substitute “that’s wrong” the phrase you want is “on the contrary” or “to the contrary.” If not, then use “in contrast.”
