REINFORCEMENT LEARNING FOR WEIGHT INITIALIZATION

Abstract

Effective neural network weight initialization is crucial for successful training, yet standard methods often rely on assumptions violated by modern architectures and advanced activation functions like Swish. This dissertation details a study investigating the feasibility of using Reinforcement Learning (RL) to tune a scaling factor for He initialization when employing Swish activations. An RL agent explored different scaling factors, evaluating them by training a small convolutional neural network on CIFAR-10 for a 7-epoch proxy task. Over 200 episodes, the RL agent demonstrated learning, converging towards a specific range of scaling factors that optimized the 7-epoch validation accuracy. This preliminary investigation highlights the functionality of the RL framework for initialization tuning and underscores the importance of evaluating the fidelity of short-term proxy tasks in predicting longer-term training performance, informing subsequent research into more complex symbolic initialization discovery.