Pre-training. eg train on general pictures before specific stuff. means you fit many parameters for detecting edges etc firsts
Reduce the learning rate.
Replace last layer (softmax) for new problem.
Freeze feature learning of early layers.
When retraining model on new data, model may forget answers to old data.