Having used both Samsung and pure Android. I can say the biggest issue I had was TouchWiz, which annoys the hell out of me. If you're ok with the interface, and can afford it, the Note 3 is the better phone - but it's an entirely different price point.
The Nexus 5 is excellent bang for buck...