Specifying various fonts with various scales with fontspec

The fontspec package allows us to define something like:

\setmainfont{texgyrepagella-regular.otf}[
    BoldFont = texgyrepagella-bold.otf ,
    ItalicFont = texgyrepagella-italic.otf ,
    BoldItalicFont = texgyrepagella-bolditalic.otf ]

It also has a scale option.

I want to combine the two, and say something like: The main font is X.otf with scale 1, and if the main font is bold, use font Y.otf with scale 0.8. How can I achieve this?

Note that this scaling should only be applied when the main font is in bold, not for all bold fonts in the text.