I have done this through the years on everything from PTO output shafts, to cheap weed wackers. It seems most of the time the brass ring on the oil seal gets distorted while pushing/pounding in. If there was any scoring in the bore at all, including minor scratching, its going to leak there.
I am not a fan of silicone, but as long as you are very careful with it, it should be fine. Next step for me would be to change the bearing to a sealed type, use capillary thread locker carefully around the outter rim of the bearing (to help that edge seal), then a smear of three bond and a new oil seal.
If that does not seal, I believe Harley makes a cookie sheet just for this.