G'day John,
after checking Goldy's suggestions and also the cam plate detent spring and plunger, you still have the problem. I'd look at the selector forks.
I had a similar problem that started that way and over time got worse (mine was 3rd and 4th). Eventually tracked it to a bent fork, by swapping them over and moved the problem to 1st and 2nd.
Cheers