I would have to carefully look at the parts book to say for sure, But I am sorting out clutch bits for a few projects at the moment but in the past, It seems to me certain BSAs and some Triumphs used a brass thrust washer on the back of the clutch baskets.
I have little experience on a plunger style clutch but trying to assemble one from parts for a project so will be working on that tonight in the shop, But in the past I had similar issues with Unit style clutches (BSA & Triumph) and took some fettling to get a good assy that did not wobble and have excess play, as well as line up the chain run also. If you shim or run too thick a thrust washer, or dont get the parts set up just right it can misalign with bad results, Too loose on the clutch center and main shaft is disconcerting as well, But it just seemed to be a lot of work to get right. Once you do, it pays to take a photo of the parts used as well as measurements so its repeatable,
It would be nice if we could just order new parts and it all fits perfectly, but hahaha! Not on a old British bike. Parts should come with a label "Some fettling required"