The ball bearing assembly is there to support the chain wheel, not the center; that's supported by the gearbox mainshaft.
There should be no wobble in the clutch assembly until the clutch lever is pulled in. If it otherwise wobbles (when the clutch is engaged), then the centre is loose on the mainshaft or the output bearing in the gearbox is shot or loose in gearbox bore, in which instance something is amiss.
David