Template Examples

These examples demonstrate how the confuse templates work to validate configuration values.

Sequence

A Sequence template allows validation of a sequence of configuration items that all must match a subtemplate. The items in the sequence can be simple values or more complex objects, as defined by the subtemplate. When the view is defined in multiple sources, the highest priority source will override the entire list of items, rather than appending new items to the list from lower sources. If the view is not defined in any source of the configuration, an empty list will be returned.

As an example of using the Sequence template, consider a configuration that includes a list of servers, where each server is required to have a host string and an optional port number that defaults to 80. For this example, an initial configuration file named servers_example.yaml has the following contents:

servers:
  - host: one.example.com
  - host: two.example.com
    port: 8000
  - host: three.example.com
    port: 8080

Validation of this configuration could be performed like this:

>>> import confuse
>>> import pprint
>>> source = confuse.YamlSource("servers_example.yaml")
>>> config = confuse.RootView([source])
>>> template = {
...     "servers": confuse.Sequence(
...         {
...             "host": str,
...             "port": 80,
...         }
...     ),
... }
>>> valid_config = config.get(template)
>>> pprint.pprint(valid_config)
{'servers': [{'host': 'one.example.com', 'port': 80},
             {'host': 'two.example.com', 'port': 8000},
             {'host': 'three.example.com', 'port': 8080}]}

The list of items in the initial configuration can be overridden by setting a higher priority source. Continuing the previous example:

>>> config.set(
...     {
...         "servers": [
...             {"host": "four.example.org"},
...             {"host": "five.example.org", "port": 9000},
...         ],
...     }
... )
>>> updated_config = config.get(template)
>>> pprint.pprint(updated_config)
{'servers': [{'host': 'four.example.org', 'port': 80},
             {'host': 'five.example.org', 'port': 9000}]}

If the requested view is missing, Sequence returns an empty list:

>>> config.clear()
>>> config.get(template)
{'servers': []}

However, if an item within the sequence does not match the subtemplate provided to Sequence, then an error will be raised:

>>> config.set(
...     {"servers": [{"host": "bad_port.example.net", "port": "default"}]}
... )
>>> try:
...     config.get(template)
... except confuse.ConfigError as err:
...     print(err)
servers#0.port: must be a number

Note

A python list is not the shortcut for defining a Sequence template but will instead produce a OneOf template. For example, config.get([str]) is equivalent to config.get(confuse.OneOf([str])) and not config.get(confuse.Sequence(str)).

MappingValues

A MappingValues template allows validation of a mapping of configuration items where the keys can be arbitrary but all the values need to match a subtemplate. Use cases include simple user-defined key:value pairs or larger configuration blocks that all follow the same structure, but where the keys naming each block are user-defined. In addition, individual items in the mapping can be overridden and new items can be added by higher priority configuration sources. This is in contrast to the Sequence template, in which a higher priority source overrides the entire list of configuration items provided by a lower source.

In the following example, a hypothetical todo list program can be configured with user-defined colors and category labels. Colors are required to be in hex format. For each category, a description is required and a priority level is optional, with a default value of 0. An initial configuration file named todo_example.yaml has the following contents:

colors:
  red: '#FF0000'
  green: '#00FF00'
  blue: '#0000FF'
categories:
  default:
    description: Things to do
  high:
    description: These are important
    priority: 50
  low:
    description: Will get to it eventually
    priority: -10

Validation of this configuration could be performed like this:

>>> import confuse
>>> import pprint
>>> source = confuse.YamlSource("todo_example.yaml")
>>> config = confuse.RootView([source])
>>> template = {
...     "colors": confuse.MappingValues(
...         confuse.String(pattern="#[0-9a-fA-F]{6,6}")
...     ),
...     "categories": confuse.MappingValues(
...         {
...             "description": str,
...             "priority": 0,
...         }
...     ),
... }
>>> valid_config = config.get(template)
>>> pprint.pprint(valid_config)
{'categories': {'default': {'description': 'Things to do', 'priority': 0},
                'high': {'description': 'These are important', 'priority': 50},
                'low': {'description': 'Will get to it eventually',
                        'priority': -10}},
 'colors': {'blue': '#0000FF', 'green': '#00FF00', 'red': '#FF0000'}}

Items in the initial configuration can be overridden and the mapping can be extended by setting a higher priority source. Continuing the previous example:

>>> config.set(
...     {
...         "colors": {
...             "green": "#008000",
...             "orange": "#FFA500",
...         },
...         "categories": {
...             "urgent": {
...                 "description": "Must get done now",
...                 "priority": 100,
...             },
...             "high": {
...                 "description": "Important, but not urgent",
...                 "priority": 20,
...             },
...         },
...     }
... )
>>> updated_config = config.get(template)
>>> pprint.pprint(updated_config)
{'categories': {'default': {'description': 'Things to do', 'priority': 0},
                'high': {'description': 'Important, but not urgent',
                         'priority': 20},
                'low': {'description': 'Will get to it eventually',
                        'priority': -10},
                'urgent': {'description': 'Must get done now',
                           'priority': 100}},
 'colors': {'blue': '#0000FF',
            'green': '#008000',
            'orange': '#FFA500',
            'red': '#FF0000'}}

If the requested view is missing, MappingValues returns an empty dict:

>>> config.clear()
>>> config.get(template)
{'colors': {}, 'categories': {}}

However, if an item within the mapping does not match the subtemplate provided to MappingValues, then an error will be raised:

>>> config.set(
...     {
...         "categories": {
...             "no_description": {
...                 "priority": 10,
...             },
...         },
...     }
... )
>>> try:
...     config.get(template)
... except confuse.ConfigError as err:
...     print(err)
categories.no_description.description not found

Filename

A Filename template validates a string as a filename, which is normalized and returned as an absolute, tilde-free path. By default, relative path values that are provided in config files are resolved relative to the application’s configuration directory, as returned by Configuration.config_dir(), while relative paths from command-line options are resolved from the current working directory. However, these default relative path behaviors can be changed using the cwd, relative_to, in_app_dir, or in_source_dir parameters to the Filename template. In addition, relative path resolution for an entire source file can be changed by creating a ConfigSource with the base_for_paths parameter set to True. Setting the behavior at the source-level can be useful when all Filename templates should be relative to the source. The template-level parameters provide more fine-grained control.

While the directory used for resolving relative paths can be controlled, the Filename template should not be used to guarantee that a file is contained within a given directory, because an absolute path may be provided and will not be subject to resolution. In addition, Filename validation only ensures that the filename is a valid path on the platform where the application is running, not that the file or any parent directories exist or could be created.

Note

Running the example below will create the application config directory ~/.config/ExampleApp/ on MacOS and Unix machines or %APPDATA%\ExampleApp\ on Windows machines. The filenames in the sample output will also be different on your own machine because the paths to the config files and the current working directory will be different.

For this example, we will validate a configuration with filenames that should be resolved as follows:

• library: a filename that should always be resolved relative to the application’s config directory

• media_dir: a directory that should always be resolved relative to the source config file that provides that value

• photo_dir and video_dir: subdirectories that should be resolved relative of the value of media_dir

• temp_dir: a directory that should be resolved relative to /tmp/

• log: a filename that follows the default Filename template behavior

The initial user config file will be at ~/.config/ExampleApp/config.yaml, where it will be discovered automatically using the Search Paths, and has the following contents:

library: library.db
media_dir: media
photo_dir: my_photos
video_dir: my_videos
temp_dir: example_tmp
log: example.log

Validation of this initial user configuration could be performed as follows:

>>> import confuse
>>> import pprint
>>> config = confuse.Configuration("ExampleApp", __name__)  # Loads user config
>>> print(config.config_dir())  # Application config directory
/home/user/.config/ExampleApp
>>> template = {
...     "library": confuse.Filename(in_app_dir=True),
...     "media_dir": confuse.Filename(in_source_dir=True),
...     "photo_dir": confuse.Filename(relative_to="media_dir"),
...     "video_dir": confuse.Filename(relative_to="media_dir"),
...     "temp_dir": confuse.Filename(cwd="/tmp"),
...     "log": confuse.Filename(),
... }
>>> valid_config = config.get(template)
>>> pprint.pprint(valid_config)
{'library': '/home/user/.config/ExampleApp/library.db',
 'log': '/home/user/.config/ExampleApp/example.log',
 'media_dir': '/home/user/.config/ExampleApp/media',
 'photo_dir': '/home/user/.config/ExampleApp/media/my_photos',
 'temp_dir': '/tmp/example_tmp',
 'video_dir': '/home/user/.config/ExampleApp/media/my_videos'}

Because the user configuration file config.yaml was in the application’s configuration directory of /home/user/.config/ExampleApp/, all of the filenames are below /home/user/.config/ExampleApp/ except for temp_dir, whose template used the cwd parameter. However, if the following YAML file is then loaded from /var/tmp/example/config.yaml as a higher-level source, some of the paths will no longer be relative to the application config directory:

library: new_library.db
media_dir: new_media
photo_dir: new_photos
# video_dir: my_videos  # Not overridden
temp_dir: ./new_example_tmp
log: new_example.log

Continuing the example code from above:

>>> config.set_file("/var/tmp/example/config.yaml")
>>> updated_config = config.get(template)
>>> pprint.pprint(updated_config)
{'library': '/home/user/.config/ExampleApp/new_library.db',
 'log': '/home/user/.config/ExampleApp/new_example.log',
 'media_dir': '/var/tmp/example/new_media',
 'photo_dir': '/var/tmp/example/new_media/new_photos',
 'temp_dir': '/tmp/new_example_tmp',
 'video_dir': '/var/tmp/example/new_media/my_videos'}

Now, the media_dir and its subdirectories are relative to the directory containing the new source file, because the media_dir template used the in_source_dir parameter. However, log remains in the application config directory because it uses the default Filename template behavior. The base directories for the library and temp_dir items are also not affected.

If the previous YAML file is instead loaded with the base_for_paths parameter set to True, then a default Filename template will use that config file’s directory as the base for resolving relative paths:

>>> config.set_file("/var/tmp/example/config.yaml", base_for_paths=True)
>>> updated_config = config.get(template)
>>> pprint.pprint(updated_config)
{'library': '/home/user/.config/ExampleApp/new_library.db',
 'log': '/var/tmp/example/new_example.log',
 'media_dir': '/var/tmp/example/new_media',
 'photo_dir': '/var/tmp/example/new_media/new_photos',
 'temp_dir': '/tmp/new_example_tmp',
 'video_dir': '/var/tmp/example/new_media/my_videos'}

The filename for log is now within the directory containing the new source file. However, the directory for the library file has not changed since its template uses the in_app_dir parameter, which takes precedence over the source’s base_for_paths setting. The template-level cwd parameter, used with temp_dir, also takes precedence over the source setting.

For configuration values set from command-line options, relative paths will be resolved from the current working directory by default, but the cwd, relative_to, and in_app_dir template parameters alter that behavior. Continuing the example code from above, command-line options are mimicked here by splitting a mock command line string and parsing it with argparse:

>>> import os
>>> print(os.getcwd())  # Current working directory
/home/user
>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument("--library")
>>> parser.add_argument("--media_dir")
>>> parser.add_argument("--photo_dir")
>>> parser.add_argument("--temp_dir")
>>> parser.add_argument("--log")
>>> cmd_line = (
...     "--library cmd_line_library --media_dir cmd_line_media "
...     "--photo_dir cmd_line_photo --temp_dir cmd_line_tmp "
...     "--log cmd_line_log"
... )
>>> args = parser.parse_args(cmd_line.split())
>>> config.set_args(args)
>>> config_with_cmdline = config.get(template)
>>> pprint.pprint(config_with_cmdline)
{'library': '/home/user/.config/ExampleApp/cmd_line_library',
 'log': '/home/user/cmd_line_log',
 'media_dir': '/home/user/cmd_line_media',
 'photo_dir': '/home/user/cmd_line_media/cmd_line_photo',
 'temp_dir': '/tmp/cmd_line_tmp',
 'video_dir': '/home/user/cmd_line_media/my_videos'}

Now the log and media_dir paths are relative to the current working directory of /home/user, while the photo_dir and video_dir paths remain relative to the updated media_dir path. The library and temp_dir paths are still resolved as before, because those templates used in_app_dir and cwd, respectively.

If a configuration value is provided as an absolute path, the path will be normalized but otherwise unchanged. Here is an example of overridding earlier values with absolute paths:

>>> config.set(
...     {
...         "library": "~/home_library.db",
...         "media_dir": "/media",
...         "video_dir": "/video_not_under_media",
...         "temp_dir": "/var/./remove_me/..//tmp",
...         "log": "/var/log/example.log",
...     }
... )
>>> absolute_config = config.get(template)
>>> pprint.pprint(absolute_config)
{'library': '/home/user/home_library.db',
 'log': '/var/log/example.log',
 'media_dir': '/media',
 'photo_dir': '/media/cmd_line_photo',
 'temp_dir': '/var/tmp',
 'video_dir': '/video_not_under_media'}

The paths for library and temp_dir have been normalized, but are not impacted by their template parameters. Since photo_dir was not overridden, the previous relative path value is now being resolved from the new media_dir absolute path. However, the video_dir was set to an absolute path and is no longer a subdirectory of media_dir.

Path

A Path template works the same as a Filename template, but returns a pathlib.Path object instead of a string. Using the same initial example as above for Filename but with Path templates gives the following:

>>> import confuse
>>> import pprint
>>> config = confuse.Configuration("ExampleApp", __name__)
>>> print(config.config_dir())  # Application config directory
/home/user/.config/ExampleApp
>>> template = {
...     "library": confuse.Path(in_app_dir=True),
...     "media_dir": confuse.Path(in_source_dir=True),
...     "photo_dir": confuse.Path(relative_to="media_dir"),
...     "video_dir": confuse.Path(relative_to="media_dir"),
...     "temp_dir": confuse.Path(cwd="/tmp"),
...     "log": confuse.Path(),
... }
>>> valid_config = config.get(template)
>>> pprint.pprint(valid_config)
{'library': PosixPath('/home/user/.config/ExampleApp/library.db'),
 'log': PosixPath('/home/user/.config/ExampleApp/example.log'),
 'media_dir': PosixPath('/home/user/.config/ExampleApp/media'),
 'photo_dir': PosixPath('/home/user/.config/ExampleApp/media/my_photos'),
 'temp_dir': PosixPath('/tmp/example_tmp'),
 'video_dir': PosixPath('/home/user/.config/ExampleApp/media/my_videos')}

Optional

While many templates like Integer and String can be configured to return a default value if the requested view is missing, validation with these templates will fail if the value is left blank in the YAML file or explicitly set to null in YAML (ie, None in python). The Optional template can be used with other templates to allow its subtemplate to accept null as valid and return a default value. The default behavior of Optional allows the requested view to be missing, but this behavior can be changed by passing allow_missing=False, in which case the view must be present but its value can still be null. In all cases, any value other than null will be passed to the subtemplate for validation, and an appropriate ConfigError will be raised if validation fails. Optional can also be used with more complex templates like MappingTemplate to make entire sections of the configuration optional.

Consider a configuration where log can be set to a filename to enable logging to that file or set to null or not included in the configuration to indicate logging to the console. All of the following are valid configurations using the Optional template with Filename as the subtemplate:

>>> import sys
>>> import confuse
>>> def get_log_output(config):
...     output = config["log"].get(confuse.Optional(confuse.Filename()))
...     if output is None:
...         return sys.stderr
...     return output
>>> config = confuse.RootView([])
>>> config.set({"log": "/tmp/log.txt"})  # `log` set to a filename
>>> get_log_output(config)
'/tmp/log.txt'
>>> config.set({"log": None})  # `log` set to None (ie, null in YAML)
>>> get_log_output(config)
<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
>>> config.clear()  # Clear config so that `log` is missing
>>> get_log_output(config)
<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>

However, validation will still fail with Optional if a value is given that is invalid for the subtemplate:

>>> config.set({"log": True})
>>> try:
...     get_log_output(config)
... except confuse.ConfigError as err:
...     print(err)
log: must be a filename, not bool

And without wrapping the Filename subtemplate in Optional, null values are not valid:

>>> config.set({"log": None})
>>> try:
...     config["log"].get(confuse.Filename())
... except confuse.ConfigError as err:
...     print(err)
log: must be a filename, not NoneType

If a program wants to require an item to be present in the configuration, while still allowing null to be valid, pass allow_missing=False when creating the Optional template:

>>> def get_log_output_no_missing(config):
...     output = config["log"].get(
...         confuse.Optional(confuse.Filename(), allow_missing=False)
...     )
...     if output is None:
...         return sys.stderr
...     return output
>>> config.set({"log": None})  # `log` set to None is still OK...
>>> get_log_output_no_missing(config)
<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
>>> config.clear()  # but `log` missing now raises an error
>>> try:
...     get_log_output_no_missing(config)
... except confuse.ConfigError as err:
...     print(err)
log not found

The default value returned by Optional can be set explicitly by passing a value to its default parameter. However, if no explicit default is passed to Optional and the subtemplate has a default value defined, then Optional will return the subtemplate’s default value. For subtemplates that do not define default values, like MappingTemplate, None will be returned as the default unless an explicit default is provided.

In the following example, Optional is used to make an Integer template more lenient, allowing blank values to validate. In addition, the entire extra_config block can be left out without causing validation errors. If we have a file named optional.yaml with the following contents:

favorite_number: # No favorite number provided, but that's OK
# This part of the configuration is optional. Uncomment to include.
# extra_config:
#   fruit: apple
#   number: 10

Then the configuration can be validated as follows:

>>> import confuse
>>> source = confuse.YamlSource("optional.yaml")
>>> config = confuse.RootView([source])
>>> config["favorite_number"].get(confuse.Optional(5))
5
>>> config["favorite_number"].get(confuse.Optional(confuse.Integer(5)))
5
>>> config["favorite_number"].get(confuse.Optional(int, default=5))
5
>>> config["favorite_number"].get(confuse.Optional(5, default="five"))
'five'
>>> extra_config = config["extra_config"].get(
...     confuse.Optional(
...         {"fruit": str, "number": int},
...     )
... )
>>> print(extra_config is None)
True
>>> config["extra_config"].get(
...     confuse.Optional(
...         {"fruit": str, "number": int},
...         default={},
...     )
... )
{}
>>> config["extra_config"].get(
...     confuse.Optional(
...         {"fruit": str, "number": int},
...         default={"fruit": "orange", "number": 3},
...     )
... )
{'fruit': 'orange', 'number': 3}

Without the Optional template wrapping the Integer, the blank value in the YAML file will cause an error:

>>> try:
...     config["favorite_number"].get(5)
... except confuse.ConfigError as err:
...     print(err)
favorite_number: must be a number

If the extra_config for this example configuration is supplied, it must still match the subtemplate. Therefore, this will fail:

>>> config.set({"extra_config": {}})
>>> try:
...     config["extra_config"].get(
...         confuse.Optional(
...             {"fruit": str, "number": int},
...         )
...     )
... except confuse.ConfigError as err:
...     print(err)
extra_config.fruit not found

But this override of the example configuration will validate:

>>> config.set({"extra_config": {"fruit": "banana", "number": 1}})
>>> config["extra_config"].get(
...     confuse.Optional(
...         {"fruit": str, "number": int},
...     )
... )
{'fruit': 'banana', 'number': 1}