You can take video/photo of anybody who is on public property as long as you are either on public property too, or on your own property. If you're on someone else's property, the property owner/s can request you stop filming or taking photos.
As for the voice recording the conversation, you can do this without telling him, but unless you are going to take it to the police, you are not legally allowed to let anyone listen to it, nor are you allowed to tell anybody the gist of the conversation. You need his permission to show or convey the conversation to anyone who was not a party in it. You are allowed to show it to police in 'pursuit of your legal rights' or something like that though.
I'd say, get video footage via motion-detecting webcam, and ask him about it. After you do, tell him you have footage of him stealing the mail, and if he tries to shrug it off, remind him that it is considered theft, and ask if he would really like to be charged with that for something as silly as stealing mail because the stupid twit can't let a grudge go.