I might be missing something here, but rather than changing the configuration each time, might another approach be to use .Include() on only those queries where you want to eager load?
Suppose we have a Product class which has a navigation property to a Colour class, you might load the Colour for a Product like this -