Damage caused by freezing wet, porous materials is a widespread problem but is hard to predict or control. Here, we show that polycrystallinity significantly speeds up the stress buildup process that underpins this damage. Unfrozen water in grain-boundary grooves feeds ice growth at temperatures below the freezing temperature, leading to fast stress buildup. These stresses can build up to levels that can easily break many brittle materials. The dynamics of the process are very variable, which we ascribe to local differences in ice-grain orientation and to the surprising mobility of many grooves - which further accelerates stress buildup. Our Letter will help understand how freezing damage occurs and in developing accurate models and effective damage-mitigation strategies. © 2023 American Physical Society.