Yes the lightness of the loader means the retention really doesn't need to be very strong.
On my last skirmish I needed easy access to my speed loader as I was running just two 30rnd mags with my GBBR, so I stuffed it into a spare plastic FAST mag pouch that was already on my belt. I didn't lose it. It wasn't very tight held, so at lunchtime I wrapped some cloth tape around it so it gripped in the pouch a little better.
As a bonus it made it a little less rattly too.