Wear what you like. If anyone else doesn't like it, that's their problem. I will happily mix things up from time to time, I quite often wear an OD vest with black combat pants and black top when doing CQB, because it is practical and the camo doesn't matter much.
Yes some people will freak out when they see that kind of thing going on and start lecturing on it, but using odd uniforms and what is to hand is common in real conflicts: Kriegsmarine submariners in WW2 famously wore abandoned British Army Battledress blouses and pants left behind when the BEF bailed out of Dunkirk, and that was not simply a different colour, it was the enemy's uniform, although in fairness, it was very similar to the proper Kriegsmarine submariner's jacket.