It's as on this diagram top left :

L/R/GND/MIC
MIC is unconnected as it's not used. Maybe it should be connected to ground too. But not sure why.
From this picture
I see that some earphones have reversed standard L/R/MIC/GND. So maybe your earphones are like these.
If so soldering the unused pin to ground will solve the problem.
It is most strange that you have different behaviour between emulation station menu and ingame as the selection of speaker/earphones has nothing to do with the processor. It's mechanical inside the jack connector. When the headphone is inserted it opens a circuit preventing sound from going to speaker.
The layout