We compare open- and closed-response versions of two questions that measure students’ reasoning on the Physics Measurement Questionnaire, same mean different spread (SMDS), and different mean same spread (DMSS). We conducted an experiment in one class (𝑁 =248) to test if asking students for their reasoning on an open-response question changes their answer to the closed-response question. To test whether answering the open-response question alters students’ closed-response answers, we prompted half of the students to explain their reasoning before seeing the closed-response options, while the other half went straight to the closed-response reasoning options. We find no effect of asking students their reasoning in an open-response question on their subsequent closed-response answers. Additionally, we investigated whether we measure students’ reasoning differently on the closed-response versus open-response versions of the question by evaluating the consistency between the students’ open-response and closed-response answers. Although we measure similarities in students’ reasoning across the two methods, the differences indicate that the open-response and closed-response questions provide different information. The results inform the need for critical judgment when deciding between open- and closed-response assessment formats.