There is increasing interest in deploying machine learning models at scale for multi-institutional studies in physics education research. Here we investigate the efficacy of applying machine learning models to institutions outside of their training set, using natural language processing to code open-ended survey responses. We find that, in general, changing institutional contexts can affect machine learning estimates of code frequencies: either previously documented sources of uncertainty increase in magnitude, new unknown sources of uncertainty emerge, or both.

Within physics education research (PER), a growing body of literature investigates using natural language processing machine learning algorithms to apply coding schemes to student writing. The aspiration is that this form of measurement may be more efficient and consistent than similar measurements made with human analysis, allowing larger and broader data sets to be analyzed. In our work, we are harnessing recent innovations in Large Language Models (LLMs) such as BERT and LLaMA to learn complex coding scheme rules.