It will be the actuator. I got the same problem. I pulled the door apart and found its the little lever on the actuator where the control arm connects to, thats connected to the arm on the key mechanism. The actuator unit isnt serviceable from memory, but i have heard of people butchering them and fixing them. But for me I cant be bothered. Il just replace it as one, one day. For now it still works despite some force. So far iv only bent one key.
Good luck.