Jules, bergs has given you the answer, here's the how and why.
The layshaft has a circlip, in a groove at the blind bush end. This circlip acts as a stop to position the layshaft pinion on the layshaft such that the pinion flat base is level with, or more usually, slightly proud of the layshaft end. This is the datum that positions the layshaft in relation to everything else. This pinion face runs against the top hat face of the blind bush. You can see now that layshaft and loose first gear need a spacer to keep everything within bounds. The spacer is the washer that goes between the layshaft shoulder and first gear cog. No shim or washer is fitted at the blind bush end. Adjusting the thickness of this washer (Part no. 67-3251) controls the layshaft endfloat when the inner cover is replaced. With new bushes, etc a new standard washer should be good enough. Depending on what you have, it may be easier to make one.
The layshaft has a very slight radius where the diameter changes. The correct washer has the inner edge relieved to clear this radius. Maybe your washer is not original. It should be hardened.
Endfloat on the layshaft will depend on the position of the fixed pinion on the layshaft, the amount of wear on the top hat faces of the layshaft bushes, face wear on the first gear cog, the thickness of the washer and also the thickness of the gasket between the inner cover and the gearbox casing. We assume the basic castings are standard.
With old machines that have been through mis -guided hands there is plenty of scope for things to be not quite right.
Plunger and swing arm layshafts are different. Plunger layshaft has a plain blind bush end. S/A has a scroll. Circlip groove is nearer the end of the shaft on plunger, the pinion gears have the corresponding location step deep in the pinion (plunger) or at the dog tips (S/A)
Both shafts are dimensionally the same, so sometimes get mixed up. Check you have the right parts.
Check the pinion is on up to the circlip on the layshaft. If all in order, leave the pinion in place, but if it has to come off, removing the pinion requires a lot of force. Knocking the pinion off the shaft with a steel tube is how the drive dogs get broken off, so use a proper press. One root of the pinion gear teeth has an oil hole, this corresponds with an oilway to the centre of the layshaft.
Swarfy.
Additional. Looking at the second picture I reckon I can see the circlip, at the tip of the pinion dogs, so that should be a S/A layshaft. Even the oil holes line up! Pinion looks to have the later type of round profile drive dogs.