I've had exactly the same problem with a few of my mags.
On some M4 mags, the cutout in the side of the mag is positioned just a mm or so too high at the top, so the top of the mag release lets it drop just enough so that the end of the hop unit does not fully push back the tab that holds the BBs in the mag.
Three possible solutions: 1) buy and fit a new mag release button; 2) scrape off the paint on the top lip of the mag release button, then drop a bit of solder onto it. Once cold, file it down flat so that you have raised the top about 1mm; 3) glue a thin piece of plastic under the top edge of each of the new mag retention holes.
1) is the most expensive, but probably the quickest and easiest.
2) should solve the problem for the least amount of cash, provided you have a soldering iron and can solder.
3) is probably the most time consuming and may mean the mags won't hold in any other rifle.