Optimizing Model Hyperparameters

Overview

Teaching: 15 min
Exercises: 120 min

Questions

• How does MAST-ML enable users to modify the hyperparameters of a model?

• Buidling off of the idea of using a grid search of hyperparameters, how can users converge on an optimized model?

Objectives

• Students learn about how model hyperparameters can affect performance and are introduced to some basic ideas on how these hyperparameters can be optimized, namely the grid search method.

• Students explore various methods of searching for a global minimum in model errors and avoiding getting trapped in a local minimum.

• Students run a neural network within MAST-ML. They can describe what they are, how to use them, and how to optimize them.

Background

During other modules included in this curriculum we’ve introduced the basic idea of optimizing models through modifying hyperparameters of the model. Hyperparameters are parameters that affect how the model learns and put constraints on the learning process. In other activities this optimization step has usually been limited to a single optimization step in which a grid of an example hyperparameter is generated and the best performing model from that grid is identified as the optimized model.

In this module we’ll spend a bit more time looking in more detail at an optimization procedure that goes beyond the very basic introduction. The main activity of this module will use the MAST-ML software, and we’ll be primarily working with the multi-layer perceptron neural network model from Scikit-learn.

Readings

If you haven’t worked with neural networks before I’d recommend reviewing this very nice introduction series from the youtube channel 3blue1brown which not only covers the basic structure of neural networks but also how they learn while walking through a nice example problem.

Activity

This activity is broken down into two parts. The first is an introduction to Neural Networks which we will then use in the second part to explore optimizing a neural network.

1. For an interactive way get to know neural networks better also visit this neural network demo by tensorflow
   1.1. Complete the activities related to the playground included in the ML crash course by google
   1.2. (Optional) If you are looking to practice some programming you can also continue on to the related “programming exercise” that they include which will link you to google colab and provide its own instructions.
2. Colab activity We will be using Google Colab to perform all of our computing so to begin download the the
   .ipynb notebook file,
   the bandgap_data_v2.csv,
   2.1. First start by uploading the .ipynb file to Colab
   2.2. Then upload the data files to a folder in your google drive called “MASTML_colab”
   2.3. The rest of the instructions for this activity are included within the notebook itself.

Note if you have completed previous modules

if you just completed the modifying workflows activity you may have this setup already and do not need to download or upload the data files as they already exist in your MASTML_colab folder

Key Points

• A sequential grid search of candidate hyperparameter values can progressively search for the best combination of model hyperparameters.

• Often time multiple grid searches are required to fully explore a hyperparameter space in sufficient detail.

• If exploring higher numbers of hyperparameters (>4 or so) it may be better to use more sophisticated search techniques due to computational constraints.