Wear Ambiguous colors
The general pallet of colors that exist today is gigantic. Just go to your local hardware store and take a look at the paint isle to see how many different blues there are.
With that the average person will usually label colors with as little effort as possible usually lumping them together in either red blue green yellow etc. giving no mind to the shades in between unless they work in a profession that deals with colors to that degree. At most the average person may toss a dark, or light in front of it too narrow it down slightly.
To aide in blending in, or maybe a better way to describe this would be make you harder to identify, is rather easy in theory. Now in practices it is going to depend on where you are going to be operating/traveling and what the color pallet of the area is. No mater how ambiguous your gray shirt may be if everyone around you is wearing brown. It’s not going to work.
Start out by doing your research and see what the general look of the area is. Figure out the type of clothing first then play with the colors. Are they generally wearing jeans? Khakis? Gym shorts? Sports jerseys? T-shirts? what colors? Brands, Logos, and patterns will all play a part here as well.
Figure out the type/style of clothing and then play with the colors. If you’re in a business area you will likely see more suits, khakis and polos, and likely in blacks, grays, whites, and maybe some blues.
In this example you would likely want to wear a suit or at least business casual clothing and choose colors that can be mistaken for multiple others. In this instance lets say you need to wear a suit you would most likely find your self looking for something that is either black (if thats the dominate color) or a color some where in between gray and blue if there is more of a mix.
By doing this your either blending in with the crowd by wearing black or if their is more of a mix of colors your wearing a color that can be mistaken for another color by either the person identifying you to someone or the person who you are being identified to.
Say you are being followed by hostile surveillance and your tail is handing you off to another tail. If he describes you as wearing gray, but the new tail looks at you and thinks you are wearing blue he may pass you by or think someone near by wearing gray is his actual target.
That was a highly simplistic explanation of it but you get the idea (I hope). The more ambiguous the color, providing it blends in with the general pallet of the area, the higher likelihood that anyone describing you may confuse anyone they are talking to due to personal color bias of the describer and the listener.