Classical simulation methods as well as machine learning methods (e.g. neural networks) have a large number of hyperparameters that significantly determine the accuracy, efficiency, and transferability of the method.
In classical simulations, the hyperparameters are usually determined by adaptation to measured values.
Esp. in neural networks, the hyperparameters determine the network architecture: number and type of layers, number of neurons, activation functions, measures against overfitting etc.
The most common methods to determine hyperparameters are intuitive testing, grid search or random search.
The tool OmniOpt performs hyperparameter optimization within a broad range of applications as classical simulations or machine learning algorithms.
Omniopt is robust and it checks and installs all dependencies automatically and fixes many problems in the background.
While Omniopt optimizes, no further intervention is required.
# Hyperparameter Optimization (OmniOpt)
Classical simulation methods as well as machine learning methods (e.g. neural networks) have
a large number of hyperparameters that significantly determine the accuracy, efficiency, and
transferability of the method. In classical simulations, the hyperparameters are usually
determined by adaptation to measured values. Esp. in neural networks, the hyperparameters
determine the network architecture: number and type of layers, number of neurons, activation
functions, measures against overfitting etc. The most common methods to determine hyperparameters
are intuitive testing, grid search or random search.
The tool OmniOpt performs hyperparameter optimization within a broad range of applications as
classical simulations or machine learning algorithms.
Omniopt is robust and it checks and installs all dependencies automatically and fixes many
problems in the background. While Omniopt optimizes, no further intervention is required.
You can follow the ongoing stdout (standard output) live in the console.
Omniopt’s overhead is minimal and virtually imperceptible.
## Quickstart with OmniOpt
The following instructions demonstrate the basic usage of OmniOpt on the ZIH system, based on the hyperparameter optimization for a neural network.
The following instructions demonstrate the basic usage of OmniOpt on the ZIH system, based
on the hyperparameter optimization for a neural network.
The typical OmniOpt workflow comprises at least the following steps:
...
...
@@ -21,24 +28,33 @@ The typical OmniOpt workflow comprises at least the following steps:
### Prepare Application Script and Software Environment
The following example application script was created from [https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html](https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html){:target="_blank"} as a starting point.
Therein, a neural network is trained on the MNIST Fashion dataset.
The following example application script was created from
Therein, a neural network is trained on the MNIST Fashion dataset.
There are three script preparation steps for OmniOpt:
+ Changing hard-coded hyperparameters (chosen here: batch size, epochs, size of layer 1 and 2) into command line parameters.
Esp. for this example, the Python module argparse (see the docs at [https://docs.python.org/3/library/argparse.html](https://docs.python.org/3/library/argparse.html){:target="_blank"}) is used.
+ Changing hard-coded hyperparameters (chosen here: batch size, epochs, size of layer 1 and 2)
into command line parameters.
Esp. for this example, the Python module `argparse` (see the docs at
There are many ways for parsing arguments into Python scripts.
The most easiest approach is the sys module (see [https://www.geeksforgeeks.org/how-to-use-sys-argv-in-python/](https://www.geeksforgeeks.org/how-to-use-sys-argv-in-python/){:target="_blank"}), which would be fully sufficient for usage with OmniOpt.
Nevertheless, this basic approach has no consistency checks or error handling etc.
+ Mark the output of the optimization target (chosen here: average loss) by prefixing it with the RESULT string.
There are many ways for parsing arguments into Python scripts.
