I was profiling when i noticed a large number of allocations of a type i recognized to be a struct.
This was surprising because structs are value types and so not normally heap allocated unless some autoboxing happens, so after a bit of digging it turned out that by default, if you use a struct as a key in a Dictionary it gets boxed for equality operations.
Implementing IEquatable quickly fixed this issue.