This project investigates the effect of different learning rate decaying strategies on deep neural network training, in terms of both convergence time and final accuracy. MNIST and CIFAR-10 datasets are used as examples. Details of the model framework can be checked in the report appendix and in the github notebooks.