there are two different possibilities combining colors:

1. additive mixing (like RGB)

2. subtractive mixing (like CMYK)

So in subtractive color mixing the result is what you expected, but there is no blue, instead there is cyan:

Yellow + cyan = green

In general subtractive color mixing is just "taking away" (filtering) from white while additive color mixing is adding up from black. (base colors of subtractive are inverse from additive: red ->cyan; green->magenta; blue->yellow)

So if you start with white screen applying filters:

min( white (255,255,255), yellow (255,255,0), cyan (0,255,255)) = green (0,255,0)

The correct answer is NO, because there is no correct working model of how "color mixing in the real world" really works. It is FAR too complex and conditional and not really at all like the simple Red-Blue-Yellow stuff that we learned in school (it in fact requires all of Chemistry and a lot of Physics and Biology to resolve).

However, the simplistic answer is: YES, use subtractive mixing rather than Additive mixing.

The color-mixing that we learned in grade school is based on pigment combinations which are a form of subtractive color mixing (very simplistically). That is the more colors that we add together, the darker it becomes because each pigment subtracts a little bit more light.

On the other hand, almost all computer color-schemes are additive in that they are based on combining light waves (very simplistically), so they get brighter, because each color adds a little bit more light.

The RGB+ scheme is somewhat, the additive complement to the subtractive scheme that we learned in most US elementary schools (which is RBY-). However, they do not match up exactly and it can be difficult to convert between them (researching now ...)

OK, if you just want to switch from additive combinations in RGB to subtractive ones, you can use the following reverse-bayesan type formula to combine two colors:

NewColor.R = (Color1.R * Color2.R)/255
NewColor.G = (Color1.G * Color2.G)/255
NewColor.B = (Color1.B * Color2.B)/255

Adjusting for the difference in the chromatic poles (G to Y, then back to G) is a lot harder ...

It has been pointed out that this produces Black for the example problem, and technically this is correct for a true subtractive system, however, if you want more diluting/subtractive system, you could try this instead:

NewColor.R = 255 - SQRT(((255-Color1.R)^2 + (255-Color2.R)^2)/2)
NewColor.G = 255 - SQRT(((255-Color1.G)^2 + (255-Color2.G)^2)/2)
NewColor.B = 255 - SQRT(((255-Color1.B)^2 + (255-Color2.B)^2)/2)

This produces a dark grey instead of Black. But to get Yellow or anything close, you still have to fix the color-scheme's pole-alignment problem.

I think your problem with combining hues is that you are doing it by adding together the two angles and dividing by two. As you've noticed, the result often makes no sense. I think you'd be better off converting the angles to Cartesian coordinates on the unit circle, averaging those, and finding the angle of the resulting point (ignoring the magnitude).

There is code to mix colors in a realistic way in krita: https://projects.kde.org/projects/calligra/repository/revisions/master/show/krita/plugins/extensions/painterlyframework.

Note that the code including the illuminants file is GPLv2+. It can convert from RGB to wavelengths, do composition and convert back.

Check this implementation for additive, substractive and other mixing alghoritms.

Is fully functional (writed in java), so you can test whatever colors you need to mix, and see if it fit your needs.

As other responses pointed, Blue + Yellow (exactly Cyan + Yellow) is Green on substractive CMYK alghoritm. See by yourself

One way to do subtractive mixture of RGB colors is to first convert the RGB colors to spectral reflectance curves. The conversion is fairly simple, and once you've done it, you can do a true subtractive mixture of the reflectance curves, and then convert the result back to RGB. There is another similar question: stackoverflow.com/questions/10254022/, where this process is discussed in more detail.

Wondering if calculation of inversion of the RGB value work. Since it's about subtraction of lights, technically the subtraction part can be calculated by simple math.

For example cyan + yellow

cyan = 0x00ffff yellow = 0xffff00

Their inversions are 0xff0000 and 0x0000ff, meaning they absorbed red and blue lights completely. Their 1:1 mixture should absorb half of red and blue lights (since the other half of the mixture can still reflects some red and blue light), which is consistent with (0xff0000 + 0x00ffff) / 2 = 0x7f007f. Now we subtract the value from 0xffffff we have 0x80ff80 which is green!

There is a novel (Feb 2022) technique showcased in 2 minute papers.

Autors call it the "Mixbox", and the technique is open source (algorithm can be found on github, mind the license though). Currently Rebelle 5 is implementing this color mixing technique, more programs may follow in the nearest future.

Imo the strongest point of this is its simplicity in implementation and how easily you can use it out of the box. Their paper is also (supposedly -- I didn't read it) well written.

I would like to add that I have creted a library in python that handles this problem in a way that spares you from having to implement complex interpolation and color system convertion functions. It is called colorir.

Example of blue + yellow:

PolarGrad(["0000ff", "ffff00"]).perc(0.5)

Gives you: "#00c5a1"

Behind the curtains, colorir interpolates the colors in HCLuv, a perceptually uniform color system. This does not mimic a "real" mixture of colors in the sense that it wont account for the subtractive properties of real pigments (i.e. the final mixed color will not look darker than the originals). It will, however, mix the colors as in finding what the almost exact average of the visual properties of the two colors look like. The following steps are all handled automatically by the PolarGrad class:

1. Interpret the value of the input colors (in this case the hex rgb values "0000ff" - blue and "ffff00" - yellow)
2. Convert these colors to a perceptually uniform color coordinate system (by default HCLuv, but other color spaces can be used)
3. Sample a color from the middle of the gradient (0.5)
4. Convert the sampled color back to RGB or other format that you may want

Mixing Colors like Pigments

1. Remove white from all colors, keeping the white parts and color parts
2. Average the RGB values of the white parts removed from the colors
3. Average the RGB values of the color parts
4. Take out the white from the averaged color parts
5. Half the white value removed and add that value to the Green of the averaged color parts
6. Add the averaged white parts back in and make whole number

Javascript Implementation

// rgbColor is an array of colors,
// where each color is an array with the three color values (RGB, 0 to 255)
function mixRGB(rgbColors) {
function sumColors(summedColors, nextColor) {
return summedColors.map(
(summedColor, i) => nextColor[i] + summedColor,
);
}


function averageColors(colors) {
return colors
.reduce(sumColors, [0, 0, 0])
.map(c => c/colors.length);
}


// Remove white from all colors
const whiteParts = [];
const colorParts = [];
rgbColors.forEach(color => {
const whiteVal = Math.min(...color);
whiteParts.push([whiteVal, whiteVal, whiteVal]);
colorParts.push(color.map(val => val - whiteVal));
});
        

// Average the whites from each selection
const averagedWhite = averageColors(whiteParts);
// Average all non-white colors
let averagedColor = averageColors(colorParts);


// Take out the white from the averaged colors
const whitePart = Math.min(...averagedColor);
averagedColor = averagedColor.map(color => color - whitePart);


// Half the white value removed and add that value to the Green
averagedColor[1] += (whitePart / 2);
        

// Add the averaged white back in and make whole number
averagedColor = averagedColor.map((color, i) => Math.floor(color + averagedWhite[i]));
    

return averagedColor;
}

I built a color game based off this you can try