It looks to me that you pulled the insert out of the knob.
That's why it was glued in.
On my VT when I took mine off, that came off as part of the knob. Oh, the knobs are not glued on.
While pulling the knob up, slightly twist it from side to side.
With the gear knob removed properly, the gear stick is metal underneath with a notch cut out on top of it to stop the knob turning.
You will find that, the piece of plastic left on your gear stick IS part of the knob and will come off and inside that piece of plastic, there is a key that fits into the notch of the gear stick. You may need to use some force, twist and lift.
Maybe use some multi grips or similar, as you don't have much to hold on to.
I think you will find that if you don't wreck it getting it off, you will be able to glue it back into the knob.
I think in your case, there was too much twist and not enough pull.