I'd reckon on it being the resistor block that controls the fan speed - if the resistive wire is broken, it can make intermittent contact - you can check if this is the fault by turning the fan to HI and if it comes on straight away it will NOT be the resistor block. The fact that it takes a few seconds to turn on without the A/C rules out low gas IMO. Could also be a bad fuse or relay.
Welcome to the forum too.
"If anyone disagrees with anything I say, I am quite prepared not only to retract it, but also to deny under oath that I ever said it." --T. Lehrer
"The secret of success is sincerity. Once you can fake that you've got it made." -- Jean Giraudoux