Retaining Skills Under Distribution Shifts: Sequential Bayesian Inference, Reinforcement Learning And Applications