G'day guys.
Hmmm, I wonder if you might be looking at a slightly flaky ground. I'm not all that familiar with the digital amps, but one thing I would imagine is that they would require a particularly good ground. Even more critical than that for a more "conventional" amp. If this is causing some kind of interference radiation it might be overloading the front end of the receiver. What gauge is your power and ground cables? If it's 8G, I'd suggest that's probably too light for the 5 channel jobby. Need at least 4G. Oh, it's probably worthwhile checking the antenna ground point too - if the antenna cable isn't properly shielded it can cause problems. You are aware that there's a join in the antenna cable, right? Behind the glovebox, there's a line plug/socket combo to extend the antenna cable to the head unit. Make sure that's fully plugged in.
Re the hum problem when the rear channels are plugged - again I'm not sure, but it's possible that the 5 channel digital unit has a different input grounding arrangement, and when you connect both front AND rear, we're forming a ground loop through the shield of the two sets of RCA cables. It might be necessary to use an isolation transformer on one pair of channels.
As Garth pointed out though, if you can't get it to work in your install, then you should be able to return it - or at least exchange it as potentially faulty. Dig your heels in, they should look after you.
Gotta go guys, later!