the reason itd sound bad would be due to the way the sound travels...
the sub would produce sound waves going in all directions...
however what would be happening is that the sound waves bouncing off the boot and back into the cabin would destructively interfere with the sound waves already travelling to the front - meaning they arent in phase.
the reason itd sound better with the boot open is that the sound waves travelling towards the boot would be dissipated out of there.
all you need to get the sound waves in phase is to move your sub around in the boot to find the right position.