Filling the hole with MDF will be much sturdier, and you'll get stronger and more accurate midbass.
-After dynamatting the outer skin, make some templates of the service holes with cardboard - make sure your templates are a few mm smaller then the hole. It has to fit in the hole, not on it.
-Trace around your templates onto 3-6mm MDF and cut them out with a jigsaw. edit: then paint them to prevent moisture rotting them.
-Next, cover the service holes with duct tape. It's a good idea to leave a bit of "slack" in the tape, this will make more sense in the next step. Make sure there aren't any gaps and the tape makes a good seal.
-Next place your MDF piece on the tape/hole, and put a layer of tape over the top securing it in place.
-At this point test fit your door trim - it may foul the MDF requiring some adjustments (loosen the tape, push it in a bit and stick it back down).
-If it all fits well, then you are ready to cover the whole lot with dynamat. It's a good idea to get a permanent marker and roughly trace the shapes of the service holes on your dynamat as you are sticking it on, so you know where to cut if you ever need to access the inside of the door.