The Pyomo Configuration System

The Pyomo config system provides a set of three classes (ConfigDict, ConfigList, and ConfigValue) for managing and documenting structured configuration information and user input. The system is based around the ConfigValue class, which provides storage for a single configuration entry. ConfigValue objects can be grouped using two containers (ConfigDict and ConfigList), which provide functionality analogous to Python’s dict and list classes, respectively.

At its simplest, the Config system allows for developers to specify a dictionary of documented configuration entries, allow users to provide values for those entries, and retrieve the current values:

>>> from pyomo.common.config import (
...     ConfigDict, ConfigList, ConfigValue
... )
>>> config = ConfigDict()
>>> config.declare('filename', ConfigValue(
...     default=None,
...     domain=str,
...     description="Input file name",
... ))
<pyomo.common.config.ConfigValue object at ...>
>>> config.declare("bound tolerance", ConfigValue(
...     default=1E-5,
...     domain=float,
...     description="Bound tolerance",
...     doc="Relative tolerance for bound feasibility checks"
... ))
<pyomo.common.config.ConfigValue object at ...>
>>> config.declare("iteration limit", ConfigValue(
...     default=30,
...     domain=int,
...     description="Iteration limit",
...     doc="Number of maximum iterations in the decomposition methods"
... ))
<pyomo.common.config.ConfigValue object at ...>
>>> config['filename'] = 'tmp.txt'
>>> print(config['filename'])
tmp.txt
>>> print(config['iteration limit'])
30

For convenience, ConfigDict objects support read/write access via attributes (with spaces in the declaration names replaced by underscores):

>>> print(config.filename)
tmp.txt
>>> print(config.iteration_limit)
30
>>> config.iteration_limit = 20
>>> print(config.iteration_limit)
20

Domain validation

All Config objects support a domain keyword that accepts a callable object (type, function, or callable instance). The domain callable should take data and map it onto the desired domain, optionally performing domain validation (see ConfigValue, ConfigDict, and ConfigList for more information). This allows client code to accept a very flexible set of inputs without “cluttering” the code with input validation:

>>> config.iteration_limit = 35.5
>>> print(config.iteration_limit)
35
>>> print(type(config.iteration_limit).__name__)
int

In addition to common types (like int, float, bool, and str), the config system profides a number of custom domain validators for common use cases:

Bool(val)	Domain validator for bool-like objects.
Integer(val)	Domain validation function admitting integers
PositiveInt(val)	Domain validation function admitting strictly positive integers
NegativeInt(val)	Domain validation function admitting strictly negative integers
NonNegativeInt(val)	Domain validation function admitting integers >= 0
NonPositiveInt(val)	Domain validation function admitting integers <= 0
PositiveFloat(val)	Domain validation function admitting strictly positive numbers
NegativeFloat(val)	Domain validation function admitting strictly negative numbers
NonPositiveFloat(val)	Domain validation function admitting numbers less than or equal to 0
NonNegativeFloat(val)	Domain validation function admitting numbers greater than or equal to 0
In(domain[, cast])	Domain validation class admitting a Container of possible values
InEnum(domain)	Domain validation class admitting an enum value/name.
IsInstance(*bases[, document_full_base_names])	Domain validator for type checking.
ListOf(itemtype[, domain, string_lexer])	Domain validator for lists of a specified type
Module([basePath, expandPath])	Domain validator for modules.
Path([basePath, expandPath])	Domain validator for a path-like object.
PathList([basePath, expandPath])	Domain validator for a list of path-like objects.
DynamicImplicitDomain(callback)	Implicit domain that can return a custom domain based on the key.

Configuring class hierarchies

A feature of the Config system is that the core classes all implement __call__, and can themselves be used as domain values. Beyond providing domain verification for complex hierarchical structures, this feature allows ConfigDicts to cleanly support the configuration of derived objects. Consider the following example:

>>> class Base(object):
...     CONFIG = ConfigDict()
...     CONFIG.declare('filename', ConfigValue(
...         default='input.txt',
...         domain=str,
...     ))
...     def __init__(self, **kwds):
...         c = self.CONFIG(kwds)
...         c.display()
...
>>> class Derived(Base):
...     CONFIG = Base.CONFIG()
...     CONFIG.declare('pattern', ConfigValue(
...         default=None,
...         domain=str,
...     ))
...
>>> tmp = Base(filename='foo.txt')
filename: foo.txt
>>> tmp = Derived(pattern='.*warning')
filename: input.txt
pattern: .*warning

Here, the base class Base declares a class-level attribute CONFIG as a ConfigDict containing a single entry (filename). The derived class (Derived) then starts by making a copy of the base class’ CONFIG, and then defines an additional entry (pattern). Instances of the base class will still create c instances that only have the single filename entry, whereas instances of the derived class will have c instances with two entries: the pattern entry declared by the derived class, and the filename entry “inherited” from the base class.

An extension of this design pattern provides a clean approach for handling “ephemeral” instance options. Consider an interface to an external “solver”. Our class implements a solve() method that takes a problem and sends it to the solver along with some solver configuration options. We would like to be able to set those options “persistently” on instances of the interface class, but still override them “temporarily” for individual calls to solve(). We implement this by creating copies of the class’s configuration for both specific instances and for use by each solve() call:

>>> class Solver(object):
...     CONFIG = ConfigDict()
...     CONFIG.declare('iterlim', ConfigValue(
...         default=10,
...         domain=int,
...     ))
...     def __init__(self, **kwds):
...         self.config = self.CONFIG(kwds)
...     def solve(self, model, **options):
...         config = self.config(options)
...         # Solve the model with the specified iterlim
...         config.display()
...
>>> solver = Solver()
>>> solver.solve(None)
iterlim: 10
>>> solver.config.iterlim = 20
>>> solver.solve(None)
iterlim: 20
>>> solver.solve(None, iterlim=50)
iterlim: 50
>>> solver.solve(None)
iterlim: 20

Interacting with argparse

In addition to basic storage and retrieval, the Config system provides hooks to the argparse command-line argument parsing system. Individual Config entries can be declared as argparse arguments using the declare_as_argument() method. To make declaration simpler, the declare() method returns the declared Config object so that the argument declaration can be done inline:

>>> import argparse
>>> config = ConfigDict()
>>> config.declare('iterlim', ConfigValue(
...     domain=int,
...     default=100,
...     description="iteration limit",
... )).declare_as_argument()
<pyomo.common.config.ConfigValue object at ...>
>>> config.declare('lbfgs', ConfigValue(
...     domain=bool,
...     description="use limited memory BFGS update",
... )).declare_as_argument()
<pyomo.common.config.ConfigValue object at ...>
>>> config.declare('linesearch', ConfigValue(
...     domain=bool,
...     default=True,
...     description="use line search",
... )).declare_as_argument()
<pyomo.common.config.ConfigValue object at ...>
>>> config.declare('relative tolerance', ConfigValue(
...     domain=float,
...     description="relative convergence tolerance",
... )).declare_as_argument('--reltol', '-r', group='Tolerances')
<pyomo.common.config.ConfigValue object at ...>
>>> config.declare('absolute tolerance', ConfigValue(
...     domain=float,
...     description="absolute convergence tolerance",
... )).declare_as_argument('--abstol', '-a', group='Tolerances')
<pyomo.common.config.ConfigValue object at ...>

The ConfigDict can then be used to initialize (or augment) an argparse ArgumentParser object:

>>> parser = argparse.ArgumentParser("tester")
>>> config.initialize_argparse(parser)

Key information from the ConfigDict is automatically transferred over to the ArgumentParser object:

>>> print(parser.format_help())
usage: tester [-h] [--iterlim INT] [--lbfgs] [--disable-linesearch]
              [--reltol FLOAT] [--abstol FLOAT]
...
  -h, --help            show this help message and exit
  --iterlim INT         iteration limit
  --lbfgs               use limited memory BFGS update
  --disable-linesearch  [DON'T] use line search

Tolerances:
  --reltol FLOAT, -r FLOAT
                        relative convergence tolerance
  --abstol FLOAT, -a FLOAT
                        absolute convergence tolerance

Parsed arguments can then be imported back into the ConfigDict:

>>> args=parser.parse_args(['--lbfgs', '--reltol', '0.1', '-a', '0.2'])
>>> args = config.import_argparse(args)
>>> config.display()
iterlim: 100
lbfgs: true
linesearch: true
relative tolerance: 0.1
absolute tolerance: 0.2

Accessing user-specified values

It is frequently useful to know which values a user explicitly set, and which values a user explicitly set but have never been retrieved. The configuration system provides two generator methods to return the items that a user explicitly set (user_values()) and the items that were set but never retrieved (unused_user_values()):

>>> print([val.name() for val in config.user_values()])
['lbfgs', 'relative tolerance', 'absolute tolerance']
>>> print(config.relative_tolerance)
0.1
>>> print([val.name() for val in config.unused_user_values()])
['lbfgs', 'absolute tolerance']

Generating output & documentation

Configuration objects support three methods for generating output and documentation: display(), generate_yaml_template(), and generate_documentation(). The simplest is display(), which prints out the current values of the configuration object (and if it is a container type, all of it’s children). generate_yaml_template() is similar to display(), but also includes the description fields as formatted comments.

>>> solver_config = config
>>> config = ConfigDict()
>>> config.declare('output', ConfigValue(
...     default='results.yml',
...     domain=str,
...     description='output results filename'
... ))
<pyomo.common.config.ConfigValue object at ...>
>>> config.declare('verbose', ConfigValue(
...     default=0,
...     domain=int,
...     description='output verbosity',
...     doc='This sets the system verbosity.  The default (0) only logs '
...     'warnings and errors.  Larger integer values will produce '
...     'additional log messages.',
... ))
<pyomo.common.config.ConfigValue object at ...>
>>> config.declare('solvers', ConfigList(
...     domain=solver_config,
...     description='list of solvers to apply',
... ))
<pyomo.common.config.ConfigList object at ...>
>>> config.display()
output: results.yml
verbose: 0
solvers: []
>>> print(config.generate_yaml_template())
output: results.yml  # output results filename
verbose: 0           # output verbosity
solvers: []          # list of solvers to apply

It is important to note that both methods document the current state of the configuration object. So, in the example above, since the solvers list is empty, you will not get any information on the elements in the list. Of course, if you add a value to the list, then the data will be output:

>>> tmp = config()
>>> tmp.solvers.append({})
>>> tmp.display()
output: results.yml
verbose: 0
solvers:
  -
    iterlim: 100
    lbfgs: true
    linesearch: true
    relative tolerance: 0.1
    absolute tolerance: 0.2
>>> print(tmp.generate_yaml_template())
output: results.yml          # output results filename
verbose: 0                   # output verbosity
solvers:                     # list of solvers to apply
  -
    iterlim: 100             # iteration limit
    lbfgs: true              # use limited memory BFGS update
    linesearch: true         # use line search
    relative tolerance: 0.1  # relative convergence tolerance
    absolute tolerance: 0.2  # absolute convergence tolerance

The third method (generate_documentation()) behaves differently. This method is designed to generate reference documentation. For each configuration item, the doc field is output. If the item has no doc, then the description field is used.

List containers have their domain documented and not their current values. The documentation can be configured through optional arguments. The defaults generate LaTeX documentation:

>>> print(config.generate_documentation())
\begin{description}[topsep=0pt,parsep=0.5em,itemsep=-0.4em]
  \item[{output}]\hfill
    \\output results filename
  \item[{verbose}]\hfill
    \\This sets the system verbosity.  The default (0) only logs warnings and
    errors.  Larger integer values will produce additional log messages.
  \item[{solvers}]\hfill
    \\list of solvers to apply
  \begin{description}[topsep=0pt,parsep=0.5em,itemsep=-0.4em]
    \item[{iterlim}]\hfill
      \\iteration limit
    \item[{lbfgs}]\hfill
      \\use limited memory BFGS update
    \item[{linesearch}]\hfill
      \\use line search
    \item[{relative tolerance}]\hfill
      \\relative convergence tolerance
    \item[{absolute tolerance}]\hfill
      \\absolute convergence tolerance
  \end{description}
\end{description}