I've done quite a lot of guides on alloy heads recently, British and Japanese (not mine guv). I think I've applied a bit more heat than you maybe . . . I've cooked the heads in a preheated oven at 200 C + for a good 45 minutes. With really close-fitting double diameter drifts made to suit each size, I've whacked quite firmly with a large (as in 1.5lb+ ball peen) hammer - one good well-aimed whack from a decent weapon on a firmly-fitting tool being worth a million whacks from a lighter hammer on a slackly-fitting drift. If the guide hasn't started to shift immediately (like first or second stroke), I've applied a bit of gas torch up the port and round about to get a bit more heat in, and tried again. So far so good. Some of the guides have been bronze, some chilled cast-iron. In most cases, once the guide has started, it'll carry on moving - but if a cast iron guide is rusty or pitted on the 'port' side, it needs cleaning up by whatever means, or you end up risking damaging the hole in the head and needing an oversize guide and some remedial boring..
It ain't a favourite job, that's for sure. I think you did right to grind away laboriously, thus avoiding damage to the head, but I'm guessing you could have been more cavalier with the heat.