Parameters

Class for parameter handling. ParameterContainer is based on dict and supports reading and writing to YAML files.

Example YAML file:

active_set: SET1

sets:
    - set_id: SET1
      processor1:
        method: special_method_2
    - set_id: SET2
      processor1:
        method: special_method_3

defaults:
    flow:
        task1: true
        task2: true
        task3: true

    processor1:
        method: special_method_1
        field1: 44100
        field2: 22050

    processor1_method_parameters:
        special_method_1:
            field1: 'test1'

        special_method_2:
            field1: 'test2'

        special_method_3:
            field1: 'test3'

    processor2:
        recipe: special_method_1;special_method_2;special_method_3

    processor2_method_parameters:
        special_method_1:
            field1: 'test1'

        special_method_2:
            field1: 'test2'

        special_method_3:
            field1: 'test3'

Once ParameterContainer.process() is called:

  1. active_set field is used to select parameter set to override parameters in the defaults block. After this parameter container contains only parameters inside defaults block with overrides.

  2. Each main level section (flow, processor, processor_method_parameters in the example above) are processed one by one.

    • If section contains method-field, parameters are copied from [SECTION_NAME]_method_parameters under parameters-field.
    • If section contains recipe-field, recipe is first parsed and parameters are copied from [SECTION_NAME]_method_parameters under parameters-field.

Parameters after processing:

flow:
    task1: true
    task2: true
    task3: true

processor1:
    _hash: 1d511b716b3cd075fbc752750b0c5932
    method: special_method_2
    field1: 44100
    field2: 22050
    parameters:
        field1: 'test2'

processor2:
    _hash: f17897bd2a133d1c1d1c853e491d2a3a
    recipe:
        - method: special_method_1
        - method: special_method_2
        - method: special_method_3

    special_method_1;special_method_2;special_method_3
    parameters:
        special_method_1:
            field1: 'test1'

        special_method_2:
            field1: 'test2'

        special_method_3:
            field1: 'test3'

Recipe

Recipe special field can be used to select multiple methods. It is specially useful for constructing feature matrix from multiple sources. Method blocks in the string are delimited with ; (e.g. method1;method2;method1).

Individual items in this list can be formatted following way:

  • [method_name (string)] => full vector
  • [method_name (string)]=[start index (int)]-[end index (int)] => default channel 0 and vector [start:end]
  • [method_name (string)]=[channel (int)]:[start index (int)]-[end index (int)] => specified channel and vector [start:end]
  • [method_name (string)]=1,2,3,4,5 => vector [1,2,3,4,4]
  • [method_name (string)]=0 => specified channel and full vector

Paths and parameter hash

Parameters under each section is used to form parameter hash. ParameterContainer’s property dcase_framework.parameters.ParameterContainer.path_structure defines how these section wise parameter hashes are used to form storage paths for each section. The main idea is that when parameters change path will change and when the parameters are the same path is the same allowing reusing already stored data (process with correct parameters).

Path structure

Example definition for path structure.

self.path_structure = {
    'feature_extractor': [
        'feature_extractor.parameters.*'
    ],
    'feature_normalizer': [
        'feature_extractor.parameters.*'
    ],
    'learner': [
        'feature_extractor',
        'feature_normalizer',
        'feature_aggregator',
        'learner'
    ],
    'recognizer': [
        'feature_extractor',
        'feature_normalizer',
        'feature_aggregator',
        'learner',
        'recognizer'
    ],
    'evaluator': [
    ]
}

One can use wild card for lists (e.g. feature_extractor.parameters.*), in this case each item in the list is producing individual path. This can be used to make paths, for examples, for each feature extractor separately.

This will lead following paths:

feature_extractor/feature_extractor_68a40f5e3b77df9564aaa68c92e95be9/
feature_extractor/feature_extractor_74c5e3ce692f5973c5071c1cf0a89ee0/
feature_extractor/feature_extractor_661304966061610bc09744166b10f76e/

feature_normalizer/feature_extractor_68a40f5e3b77df9564aaa68c92e95be9/
feature_normalizer/feature_extractor_74c5e3ce692f5973c5071c1cf0a89ee0/
feature_normalizer/feature_extractor_661304966061610bc09744166b10f76e/

learner/feature_extractor_5ca1f32c65b3eea59e1bb27b09b747ea/feature_normalizer_67b9b20ff555e8eaee22f5e50695df8b/feature_aggregator_baaf606d9ac1eaca43a6a24b599998a9/learner_624a422b47a32e20b90ad6e6151057f8

recognizer/feature_extractor_5ca1f32c65b3eea59e1bb27b09b747ea/feature_normalizer_67b9b20ff555e8eaee22f5e50695df8b/feature_aggregator_baaf606d9ac1eaca43a6a24b599998a9/learner_624a422b47a32e20b90ad6e6151057f8/recognizer_08c503973f61ef4c4c5f7c56709d801c

Parameter section used to form hash will be saved in each sub folder (parameters.yaml) to make it easier handle files manually if needed.

Hash

Parameter hash value is md5 hash for stringified parameter dict of the section, with a few clean ups helping to keep hash compatible when extending parameter selection in the section later. Following rules are used:

  • If section contains field enable with value False all other fields inside this section are excluded from the parameter hash calculation. This will make the hash robust if section is not used but still unused parameters are changed.
  • If section contains fields with value False, this field is excluded from the parameter hash calculation. This will enable to add new flag parameters, without changing hash, just define the new flag so that previous behaviour is happening when this field is set to false.
  • If section contains any of the non_hashable_fields fields, those are excluded from the parameter hash calculation. These fields are set when ParameterContainer is constructed, and they usually are fields used to print various values to the console. These fields do not change the system output to be saved onto disk, and hence they are excluded from hash.

Use dcase_framework.parameters.ParameterContainer.non_hashable_fields to exclude fields from hash. use dcase_framework.parameters.ParameterContainer.control_sections to omit hash calculation for parameter sections which do not needed it.

ParameterContainer

Usage examples:

1
2
3
4
5
6
7
8
# Load parameters
params = ParameterContainer().load(filename='parameters.yaml')
# Process parameters
params.process()
# Print parameters
print(params)
# Get parameters
value = get_path('section1.parameter1')
ParameterContainer(\*args, \*\*kwargs) Constructor
ParameterContainer.load(\*args, \*\*kwargs) Load file
ParameterContainer.save(\*args, \*\*kwargs) Save file
ParameterContainer.exists() Checks that file exists
ParameterContainer.get_path(dotted_path[, ...]) Get value from nested dict with dotted path
ParameterContainer.show() Print container content
ParameterContainer.log([level]) Log container content
ParameterContainer.override(override) Override container content recursively.
ParameterContainer.process([...]) Process parameters
ParameterContainer.process_method_parameters(section) Process methods and recipes in the section
ParameterContainer.get_hash([data]) Get unique hash string (md5) for given parameter dict