QuickFig - Quick and Painless Config Parser for Python

A lightweight schema-supporting config parser for projects.

QuickFig Release Notes

0.3.0
- Add support setting parameters via environment variables
- Fix bug causing issues finding definitions in section mode
0.2.0
- Add support for Python 2.7
0.1.1
- Initial version

Quick Intro to QuickFig

QuickFig is a quick and easy to use library for configuring your application using a Yaml file. QuickFig allows you to set up a loose schema with default values and data types that reduces error checking and config processing code in your app.

I have written something like this for almost every project I have done and it always take longer than it should, so I decided to write it as a library and share it.

Features

Set a loose schema for the configuration. While parameters do not require a schema defined, it helps as it will take care of the conversion for it.
- Example Schema:
```
  schema = {
            'runtime.debug': {
                'desc': "DEBUG Mode',
                'type': 'bool', 
                'default': 'false', 
                'env': [ 'MYAPP_DEBUG', 'DEBUG']},
            'myapp.float': {
                'type': 'float', 
                'default': 1.2},
            'myapp.str': {
                'type': 'str',
                'default': "string value"}
  }
        
  config = QuickFig(definition=schema)
```
  The above defines parameter ‘runtime.debug’ which is a boolan that defaults to False if not defined, you can set it by setting env variables MYAPP_DEBUG or DEBUG, but if both are set, value in MYAPP_DEBUG will win. It also defines “myapp.float” and myapp.str as a float and a string respectively
- A Schema for a parameter can define:
  - desc - Description of the parameter
  - type - Parameter Type
    - Pre-Defined Data Types (Additional types may be added dynamically as needed):
      - str - Strings (default if not set)
      - bool - Boolean
      - int - Integer
      - float - Floating point
      - list - A List of values
      - dict - A key-value dictionary
  - default - Default value for parameter
  - env - Environment variable(s) that can be used if parameter is not set
    - If at least one env variable is set and parameter is not defined in the config file, the first environment variable that is defined will be used.
- When Schema is not defined, QuickFig will take a best guest at parameter type based on current value.

Read a YAML configuration file

The file format can be of any depth and may include parameters with or without a schema

Example: (Using same schema)

Config File: (config.yaml) myapp: str: My String float: 9.2

Code:

  ## Env variable DEBUG is set to "yes"
  config = QuickFig(definition=schema)

  # No data yet, only schema default
  assert config.myapp.str == "string value"

  #Load config from file
  config.quickfig_load_from_file('config.yaml')

  # Data from config file:
  assert config.myapp.str == "My String"

  # Data From env variable DEBUG
  # Note that it got converted to a boolean from string
  assert config.runtime.debug == True

Set parameter overrides at runtime that set the value regardless of env variables or values in config files. This is handy if you want to allow, for example, user to override a config file parameter from command line
- Example:
```
  #Note that value can be a string or a boolean
  overrides={'runtime.debug': 'yes'}
  config = QuickFig(definition=schema, overrides=overrides)

  assert config.runtime.debug = True
```

Store all of this configuration in an object that allows:

Access to read/get any value using dotter notation:

Example:

  config = QuickFig(definitions=schema)

  # No data yet, only schema default
  assert config.myapp.str == "string value"

  #set the value
  config.set("myapp.str", "new_value")

  # Get the value using dict-like get(key, default_value)
  assert config.get("myapp.str", "default_value") == "new_value"

Access parameters using property notation (setting this way is not supported):

Example:

  config = QuickFig(definitions=schema)

  # create a sub-config for section `myapp`
  myapp_config = config.section('myapp')

  #set the value
  myapp_config.set("str", "new_value")

  # Get the value using full config object
  assert config.myapp.str == "new_value"

  # Get the value using full sub-config object
  assert myapp_config.str == "new_value"

Creation of a filtered sub-config object that allows access to child nodes without using full path:

Example:

  config = QuickFig(definitions=schema)

  # No data yet, only schema default
  assert config.myapp.str == "string value"

  #set the value
  config.set("myapp.str", "new_value")

  # Get the value using property notation
  assert config.myapp.str == "new_value"

Usage

Concepts

Key concept in QuickFig is that while the config file can be netsted, it can always be flattended using the dotted notation - so that something like this:

    myapp:
        section_1:
           parameter: value
        section_2:
           parameter: another value

can also be represented as:

    myapp.section_1.parameter: value
    myapp.section_2.parameter: value

While the main config file can be in either format - the schema and overrides use the dotted notation

The goal is to make the config as seamless as possible to reduce validation and management code in your app.

Examples:

Full Example:

(This example exists in examples directory)

Config file: (full_example.conf)

 # Config file for full_example
  app:
    component1:
      enabled: true
      host: yahoo.com
    component2:
      enabled: false
      delay: 1.0

Code: (full_example.py)

  #!/usr/bin/env python3
  """
  Full Example of using QuickFig
  """
  from argparse import ArgumentParser, Action, ArgumentError
  from argparse import RawDescriptionHelpFormatter
  import os
  import sys
    
  from quickfig import QuickFig
  import yaml
    
  __version__ = 1.0
  __updated__ = ""
    
  SCHEMA_YAML = '''
  debug:
    desc: General Debug Flag
    type: bool
    default: false
    env:
      - APP_DEBUG
      - DEBUG
    
  app.component1.enabled:
    desc: Component 1 Enabled Flag
    type: bool
    default: false
    
  app.component1.host:
    desc: Component 1 Hostname
    type: str
    default: google.com
    
  app.component1.port:
    desc: Component 1 Port Number
    type: int
    default: 443
    
  app.component2.enabled:
    desc: Component 2 Enabled Flag
    type: bool
    default: no
    
  app.component2.delay:
    desc: Delay in seconds for component 2 to startup
    type: float
    default: 0.5
  '''
    
  SCHEMA = yaml.safe_load(SCHEMA_YAML)
    
    
  class StoreNameValuePair(Action):
      ''' Action to store an section.option = value pair into the  '''
    
      def __call__(self, parser, namespace, values, option_string=None):
          for value in values:
              try:
                  (key, value) = value.split("=", 2)
              except ValueError:
                  raise ArgumentError(self,
                                      "Could not parse argument %s as section.option=value format" % value)
              config = getattr(namespace, self.dest) or {}
              config[key] = value
              setattr(namespace, self.dest, config)
    
    
  def sep(text=""):
      ''' Print a separator '''
      print('{:^60}'.format("-" * 60))
      print('---- {:^50} ----'.format(text))
      print('{:^60}'.format("-" * 60))
    
    
  def component1(config, debug):
      ''' Run component 1 '''
      print("Starting component 1")
      print("Connecting to: %s:%s" % (config.host, config.port))
      if debug:
          print("Component 1 Config:\n%s" % config)
    
    
  def component2(config, debug):
      ''' Run component 2 '''
      print("Starting component 2")
      print("Delay is: %s" % config.delay)
      if debug:
          print("Component 2 Config:\n%s" % config)
    
    
  def run():
      program_version = "v%s" % __version__
      program_build_date = str(__updated__)
      program_version_message = '%%(prog)s %s (%s)' % (
          program_version, program_build_date)
    
      parser = ArgumentParser()
      parser.add_argument("-D", dest="debug", action="store_true",
                          help="set debug mode [default: %(default)s]")
    
      parser.add_argument("-P", dest="options", metavar="option=value",
                          action=StoreNameValuePair, nargs="+",
                          help="Override config parameters, use option=value syntax")
    
      parser.add_argument('-V', '--version', action='version',
                          version=program_version_message)
    
      # Process arguments
      args = parser.parse_args()
    
      # Determine overrides based on cli args
      overrides = {}
      if args.options:
          overrides.update(args.options)
      if args.debug:
          overrides['debug'] = True
    
      # Create our config
      config = QuickFig(definitions=SCHEMA, overrides=overrides)
    
      # load the config
      config.quickfig_load_from_file("full_example.conf")
      sep("Starting...")
    
      print("Debug Mode is: %s" % ("on" if config.debug else "off"))
    
      if config.debug:
          print("Total Config: \n%s" % config)
    
      # run component1 if enabled
      if config.app.component1.enabled:
          sep("Component 1")
          # Run component 1 and give it only the config it needs
          component1(config.section('app.component1'), config.debug)
    
      # run component2 if enabled
      if config.app.component2.enabled:
          sep("Component 2")
          # Run component 1 and give it only the config it needs
          component2(config.section('app.component2'), config.debug)
      sep("Finished")
    
    
  if __name__ == "__main__":
      run()

Sample Runs:

Basic (no arguments)

  ./full_example.py
  ------------------------------------------------------------
  ----                    Starting...                     ----
  ------------------------------------------------------------
  Debug Mode is: off
  ------------------------------------------------------------
  ----                    Component 1                     ----
  ------------------------------------------------------------
  Starting component 1
  Connecting to: yahoo.com:443
  ------------------------------------------------------------
  ----                      Finished                      ----
  ------------------------------------------------------------

As per config, debug is off, component 1 is enabled, component 2 is not

Override component 2 enable from command line (-P app.component2.enabled=yes)

  ./full_example.py -P app.component2.enabled=yes
  ------------------------------------------------------------
  ----                    Starting...                     ----
  ------------------------------------------------------------
  Debug Mode is: off
  ------------------------------------------------------------
  ----                    Component 1                     ----
  ------------------------------------------------------------
  Starting component 1
  Connecting to: yahoo.com:443
  ------------------------------------------------------------
  ----                    Component 2                     ----
  ------------------------------------------------------------
  Starting component 2
  Delay is: 1.0
  ------------------------------------------------------------
  ----                      Finished                      ----
  ------------------------------------------------------------

Enable Debug Mode (-D)

 ./full_example.py -D | sed "s/^/            /"
 ------------------------------------------------------------
 ----                    Starting...                     ----
 ------------------------------------------------------------
 Debug Mode is: on
 Total Config: 
 #QuickFig Config
        
 # Component 1 Enabled Flag (Default: 'False')
 app.component1.enabled = True
        
 # Component 1 Hostname (Default: 'google.com')
 app.component1.host = yahoo.com
        
 # Component 1 Port Number (Default: '443')
 app.component1.port = 443
        
 # Delay in seconds for component 2 to startup (Default: '0.5')
 app.component2.delay = 1.0
        
 # Component 2 Enabled Flag (Default: 'False')
 app.component2.enabled = False
        
 # General Debug Flag (Default: 'False')
 debug = True
        
 #End QuickFig Config
        
 ------------------------------------------------------------
 ----                    Component 1                     ----
 ------------------------------------------------------------
 Starting component 1
 Connecting to: yahoo.com:443
 Component 1 Config:
 #QuickFig Config
 #
 # Path: app.component1
 #
        
 # Component 1 Enabled Flag (Default: 'False')
 enabled = True
        
 # Component 1 Hostname (Default: 'google.com')
 host = yahoo.com
        
 # Component 1 Port Number (Default: '443')
 port = 443
        
 #End QuickFig Config
        
 ------------------------------------------------------------
 ----                      Finished                      ----
 ------------------------------------------------------------

Enable debug mode via env variable

  ------------------------------------------------------------
  ----                    Starting...                     ----
  ------------------------------------------------------------
  Debug Mode is: on
  Total Config: 
  #QuickFig Config
            
  # Component 1 Enabled Flag (Default: 'False')
  app.component1.enabled = True
            
  # Component 1 Hostname (Default: 'google.com')
  app.component1.host = yahoo.com
            
  # Component 1 Port Number (Default: '443')
  app.component1.port = 443
            
  # Delay in seconds for component 2 to startup (Default: '0.5')
  app.component2.delay = 1.0
            
  # Component 2 Enabled Flag (Default: 'False')
  app.component2.enabled = False
            
  # General Debug Flag (Default: 'False')
  debug = True
            
  #End QuickFig Config
            
  ------------------------------------------------------------
  ----                    Component 1                     ----
  ------------------------------------------------------------
  Starting component 1
  Connecting to: yahoo.com:443
  Component 1 Config:
  #QuickFig Config
  #
  # Path: app.component1
  #
            
  # Component 1 Enabled Flag (Default: 'False')
  enabled = True
            
  # Component 1 Hostname (Default: 'google.com')
  host = yahoo.com
            
  # Component 1 Port Number (Default: '443')
  port = 443
            
  #End QuickFig Config
            
  ------------------------------------------------------------
  ----                      Finished                      ----
  ------------------------------------------------------------

API:

Main Class: QuickFig()

This is the main class when using QuickFig.

QuickFig() Constructor

Keyword arguments:
- definitions - schema as a dictionary
  - definitions should be flat(dotted notation) dictionary of key to definition dict
- config - config as a python dictionary object
  - Dictionary may be nested or flat using dots
- overrides - overrides as a dictionary object
  - overrides should be flat dictionary of key to value
- resolver - an optional instance of class ConfDataTypeResolver().
  - If not specified, uses default resolver.
  - Main reason to specify is to add custom type resolution without affecting default resolver

QuickFig() Instance Methods

quickfig_load(config) - load configuration provided by parameter config
quickfig_load_from_file(filename, warn=False) - load configuration from file whose name is provided by filename parameter. If warn is true, throw a warning into the log if file not present or cannot read.
section(name) - get a filtered version of the config with name as prefix. Returns instance of QuickFigNode() class
set(key, value) - set parameter specified by key to value. key is a dotted notation representation of a parameter name.
get(key, default_value=None) - get parameter specified by key(dotted notation). if not set, return default_value, if specified. Note, default value specified by schema will have priority over default_value
get_data_type(key, test_value="") - get data type for parameter specified by key. If parameter is not set and has no schema, use value of test_value to guess at the data type. Returns object of class QuickFigDataType()
get_definition(key, test_value="", default_dtype=None)) - get definition for parameter specified bykey. Use test_valueto guess at data type if no definition exists and parameter is not set. ifdefault_dtype` is specified, use that data type instead of guessing.

Class: QuickFigNode()

This class appears similar to QuickFig, but is not instanciated directly. Instead it is created via section(name) method on either QuickFig() or QuickFigNode() instances)

QuickFigNode() Instance Methods

section(name) - get a filtered version of the config with name as prefix. name is combined with current prefix, if one is already defined. Returns an instance of QuickFigNode() class
set(key, value) - set parameter specified by key to value. key is a dotted notation representation of a parameter name.
get(key, default_value=None) - get parameter specified by key(dotted notation). if not set, return default_value, if specified. Note, default value specified by schema will have priority over default_value
get_data_type(key, test_value="") - get data type for parameter specified by key. If parameter is not set and has no schema, use value of test_value to guess at the data type. Returns object of class QuickFigDataType()
get_definition(key, test_value="", default_dtype=None)) - get definition for parameter specified bykey. Use test_valueto guess at data type if no definition exists and parameter is not set. ifdefault_dtype` is specified, use that data type instead of guessing.