This was surprising because  structs are value types and so not normally heap allocated unless some autoboxing happens, so after a bit of digging it turned out that by default, if you use a struct as a key in a Dictionary it gets boxed for equality operations.

Implementing IEquatable quickly fixed this issue.