I’ve often found you don’t even need the hammer aspect, just using a mason bit will work, especially for softer materials. It’ll take longer, but if it’s only for the finish, it’s faster than fixing a blowout like this.

Alternatively turn off the hammer for the finish, or finish the hole from this side (since you have the pilot hole). This works well on any material, for example to prevent blowouts on wood or flash on metal.