This may not be the correct way but I have always just tightened it until there was no play in the bearings when you rock the wheel back and forth and then back the nut off to the next notch for the cotter pin to go through. Never had a bearing failure doing it this way.
I basically do the same. I spin the rotor/drum and tighten the nut untill the rotor stops, then back it off. I'll do this 3 times to "seat" the bearing. the third time, I'll just back it off 1 flat of the nut and install the cotter pin.