You can use YUV color space, it takes into account this factor :
| Y' | | 0.299 0.587 0.114 | | R |
| U | = | -0.14713 -0.28886 0.436 | | G |
| V | | 0.615 -0.51499 -0.10001 | | B |
You can also use the CIE color space for this purpose.
EDIT:
I shall mention that YUV color space is an inexpensive approximation that can be computed via simple formulas. But it is not perceptually uniform. Perceptually uniform means that a change of the same amount in a color value should produce a change of about the same visual importance.
If you need a more precise and rigourous metric you must definitely consider CIELAB color space or an another perceptually uniform space (even if there are no simple formulas for conversion).
Human perception is weaker in chroma than intensity.
For example, in commercial video, the YCbCr/YPbPr color spaces (also called Y'UV) reduces the resolution of the chroma info but preserves the luma (Y). In digital video compression such as 4:2:0 and 4:2:2 reduces the chroma bitrate due to relatively weaker perception.
I believe that you can calculate a distance function giving higher priority over luma (Y) and less priority over chroma.
Also, under low intensity, human vision is practically black-and-white. Therefore, the priority function is non-linear in that for low luma (Y) you put less and less weight on chroma.
Color perception is not Euclidean. Any distance formula will be both good enough and terrible at the same time. Any measure based on Euclidean distance (RGB, HSV, Luv, Lab, ...) will be good enough for similar colors, showing aqua being close to teal. But for non-close values it gets to be arbitrary. For instance, is red closer to green or to blue?
The XYZ and RGB systems are far from
exhibiting perceptual uniformity.
Finding a transformation of XYZ into a
reasonably perceptually-uniform space
consumed a decade or more at the CIE
and in the end no single system could
be agreed.
I would recommend using CIE94 (DeltaE-1994), it's said to be a decent representation of the human color perception. I've used it quite a bit in my computer-vision related applications, and I am rather happy with the result.
It's however rather computational expensive to perform such a comparison:
In case that resource disappears the author's conclusion is that the best low-cost approximation to the distance between two RGB colours can be achieved using this formula (in C code).
typedef struct {
unsigned char r, g, b;
} RGB;
double ColourDistance(RGB e1, RGB e2)
{
long rmean = ( (long)e1.r + (long)e2.r ) / 2;
long r = (long)e1.r - (long)e2.r;
long g = (long)e1.g - (long)e2.g;
long b = (long)e1.b - (long)e2.b;
return sqrt((((512+rmean)*r*r)>>8) + 4*g*g + (((767-rmean)*b*b)>>8));
}