Yeah clothes brush works a treat, or one of those slightly sticky roller things that pick up the lint or even just a very slightly damp rag, the moisture will help pick it up. You could also use a vacuum with the brush attachment to brush it up as its sucking.
Maybe you should consider getting some throwover car seat covers, I'm lucky I don't have that prob, gotta love leather