Topological transport phenomena in magnetic materials are a major topic of current condensed matter research. One of the most widely studied phenomena is the topological Hall effect (THE), which is generated via spin-orbit interactions between conduction electrons and topological spin textures such as skyrmions. We report a comprehensive set of Hall effect and magnetization measurements on epitaxial films of the prototypical ferromagnetic metal SrRuO3 the magnetic and transport properties of which were systematically modulated by varying the concentration of Ru vacancies. We observe Hall effect anomalies that closely resemble signatures of the THE, but a quantitative analysis demonstrates that they result from inhomogeneities in the ferromagnetic magnetization caused by a nonrandom distribution of Ru vacancies. As such inhomogeneities are difficult to avoid and are rarely characterized independently, our results call into question the identification of topological spin textures in numerous prior transport studies of quantum materials, heterostructures, and devices. Firm conclusions regarding the presence of such textures must meet stringent conditions such as probes that couple directly to the noncollinear magnetization on the atomic scale. © 2020 authors. Published by the American Physical Society.