Nope, you're not imagining it. It's due to build up of moisture/bacteria in the vents. When it gets fully cold or hot the smell goes away, but will continue to build up when there is condensation and no air moving through. Have heard it referred to as "bum smell" but that's a bit harsh, lol.
If you can find an easy way to get rid of it, good luck. There are some commercial products available that squirt down the vents to kill the stuff that causes it, but not readily available from searches I have found.