Text this: Fast and precise inference on diffusivity in interacting particle systems