SetProgramOptions Python Module

DEPRECATION NOTICE

This package was forked by the original author and is now maintained under the name ActiveConfigProgramOptions (GitLab, PyPI, Docs). Users of SetProgramOptions should switch to the new package.

Table of Contents:

Indices and Tables

Overview

The SetProgramOptions package extends the ConfigParserEnhanced package by adding additional operations for handling command-line options.

The primary use case that provided the impetus to develop SetProgramOptions was to support complex configuration environments for a software project that is tested on a variety of platforms and architectures, including GPUs and HPC systems. This project is several million lines of code and has hundreds of CMake options in its configuration space.

We developed SetProgramOptions and SetProgramOptions to allow our build system to use optimized .ini files to manage our configuration space.

This package includes two classes:

  1. SetProgramOptions - A general purpose command line handler that handles generic command line options.

  2. SetProgramOptionsCMake - A subclass of SetProgramOptions, this class further extends SetProgramOptions by adding CMake-specific operations to provide ease of use for CMake specific options. It also adds an additional generator option to allow the generation of either bash style command line options or a CMake source fragment file.

An example .ini file using SetProgramOptions might look like:

1[Directory *nix]
2opt-set ls

This configuration is the SetProgramOptions version of a hello world example. Here, the opt-set ls option is specifying a single command line option which in this case is the command ls.

We can expand this to add additional entries:

1[Directory *nix]
2opt-set ls
3opt-set -l
4opt-set -r
5opt-set -t
6opt-remove -r

When processed, this example would result in a concactenated string containing the command ls -l -t. We threw in the opt-remove -r operation which removed the -r entry.

For more details on how this is used, see the Examples section below.

Examples

Here we show a few examples of how SetProgramOptions and SetProgramOptionsCMake can be used.

Example 1

In example-01 we have a fairly simple .ini file that demostrates utilization of the use operation that comes built in with ConfigParserEnhanced. In this example we are going to demonstrate how the opt-set operations in our .ini file can be used to generate a custom bash command with customizable argument sets added.

In this case, we will process the [MY_LS_COMMAND] section to generate a bash command that would generate a directory listing in list form by reverse time order of last modified with a custom timestamp format.

While this example is quite simple we can see how a complex environment in a DevOps setting might use this to bundle “common” operations to reduce the amount of copying and pasting that is used.

example-01.ini

 1#
 2# example-01.ini
 3#
 4[LS_COMMAND]
 5opt-set ls
 6
 7[LS_LIST_TIME_REVERSED]
 8opt-set "-l -t -r"
 9
10[LS_CUSTOM_TIME_STYLE]
11opt-set --time-style : "+%%Y-%%m-%%d %%H:%%M:%%S"
12
13[MY_LS_COMMAND]
14use LS_COMMAND
15use LS_LIST_TIME_REVERSED
16use LS_CUSTOM_TIME_STYLE

example-01.py

 1#!/usr/bin/env python3
 2# -*- mode: python; py-indent-offset: 4; py-continuation-offset: 4 -*-
 3from pathlib import Path
 4import setprogramoptions
 5
 6print(80*"-")
 7print(f"- {Path(__file__).name}")
 8print(80*"-")
 9
10filename = "example-01.ini"
11popts = setprogramoptions.SetProgramOptions(filename)
12
13section = "MY_LS_COMMAND"
14popts.parse_section(section)
15bash_options = popts.gen_option_list(section, generator="bash")
16print(" ".join(bash_options))
17
18print("Done")

Console Output

1--------------------------------------------------------------------------------
2- example-01.py
3--------------------------------------------------------------------------------
4ls -l -t -r --time-style="+%Y-%m-%d %H:%M:%S"
5Done

Example 2

Example 2 demonstrates the use of SetProgramOptionsCMake which is a subclass of SetProgramOptions and implements operations for handling CMake variables in a .ini file. SetProgramOptionsCMake also introduces a new “CMake fragment” generator and implements logic to maintain consistency of behavior between generated BASH scripts and CMake fragments with knoweldge of how CMake treats -D operations at the command line versus set() operations inside a CMake script.

Here we show the new operation opt-set-cmake-var which has the form opt-set-cmake-var VARNAME <OPTIONAL PARAMS> : VALUE. Details and a comprehensive list of the optional parameters can be found in the SetProgramOptionsCMake Class Reference page.

The main elements this example will demonstrate is how to use opt-set-cmake-var operations and how they can be used.

This example is fairly straightforward since the .ini file options being defined will generate the same output for both generator types.

example-02.ini

 1#
 2# example-02.ini
 3#
 4[CMAKE_COMMAND]
 5opt-set cmake
 6
 7[CMAKE_GENERATOR_NINJA]
 8opt-set -G : Ninja
 9
10[MYPROJ_OPTIONS]
11opt-set-cmake-var  MYPROJ_CXX_FLAGS       STRING       : "-O0 -fopenmp"
12opt-set-cmake-var  MYPROJ_ENABLE_OPTION_A BOOL   FORCE : ON
13opt-set-cmake-var  MYPROJ_ENABLE_OPTION_B BOOL         : ON
14
15[MYPROJ_SOURCE_DIR]
16opt-set /path/to/source/dir
17
18[MYPROJ_CONFIGURATION_NINJA]
19use CMAKE_COMMAND
20use CMAKE_GENERATOR_NINJA
21use MYPROJ_OPTIONS
22use MYPROJ_SOURCE_DIR

example-02.py

 1#!/usr/bin/env python3
 2# -*- mode: python; py-indent-offset: 4; py-continuation-offset: 4 -*-
 3from pathlib import Path
 4import setprogramoptions
 5
 6def print_separator(label):
 7    print("")
 8    print(f"{label}")
 9    print("-"*len(label))
10    return
11
12print(80*"-")
13print(f"- {Path(__file__).name}")
14print(80*"-")
15
16filename = "example-02.ini"
17popts = setprogramoptions.SetProgramOptionsCMake(filename)
18
19section = "MYPROJ_CONFIGURATION_NINJA"
20popts.parse_section(section)
21
22# Generate BASH output
23print_separator("Generate Bash Output")
24bash_options = popts.gen_option_list(section, generator="bash")
25print(" \\\n   ".join(bash_options))
26
27# Generate a CMake Fragment
28print_separator("Generate CMake Fragment")
29cmake_options = popts.gen_option_list(section, generator="cmake_fragment")
30print("\n".join(cmake_options))
31
32print("\nDone")

Console Output

 1--------------------------------------------------------------------------------
 2- example-02.py
 3--------------------------------------------------------------------------------
 4
 5Generate Bash Output
 6--------------------
 7cmake \
 8   -G=Ninja \
 9   -DMYPROJ_CXX_FLAGS:STRING="-O0 -fopenmp" \
10   -DMYPROJ_ENABLE_OPTION_A:BOOL=ON \
11   -DMYPROJ_ENABLE_OPTION_B:BOOL=ON \
12   /path/to/source/dir
13
14Generate CMake Fragment
15-----------------------
16set(MYPROJ_CXX_FLAGS "-O0 -fopenmp" CACHE STRING "from .ini configuration")
17set(MYPROJ_ENABLE_OPTION_A ON CACHE BOOL "from .ini configuration" FORCE)
18set(MYPROJ_ENABLE_OPTION_B ON CACHE BOOL "from .ini configuration")
19
20Done

Example 3

The last example we show here is a bit more complicated and gets into some of the differences in how CMake treats a command line variable being set via a -D option versus a set() operation within a CMakeLists.txt file.

The script prints out a notice explaining the nuance but the general idea is that CMake treats options provided by a -D option at the command line as though they are CACHE variables with the FORCE option enabled. This is designed to allow command-line parameters to generally take precedence over what a CMakeLists.txt file might set as though it’s a user-override. The added FORCE option also ensures that if there are multiple -D options setting the same variable the last one will win.

This can have a subtle yet profound effect on how we must process our opt-set-cmake-var operations within a .ini file if our goal is to ensure that the resulting CMakeCache.txt file generated by a CMake run would be the same for both bash and cmake fragment generators.

In this case, in order to have a variable be set by the bash generator it must be a CACHE variable – which can be accomplished by either adding a TYPE or a FORCE option. We will note here that if FORCE is given without a TYPE then we use the default type of STRING.

If the same CMake variable is being assigned, such as in a case where we have a section that is updating a flag, then the FORCE option must be present on the second and all subsequent occurrences of opt-set-cmake-var or the bash generator will skip over that assignment since a non-forced set() operation in CMake would not overwrite an existing cache var. This situation can occur frequently if our .ini file(s) are structured to have some common configuration option set and then a specialization which updates one of the arguments. One example of this kind of situation might be where we have a specialization that adds OpenMP and we would want to add the -fopenmp flags to our linker flags.

example-03.ini

 1#
 2# example-03.ini
 3#
 4[TEST_VAR_EXPANSION_COMMON]
 5opt-set-cmake-var CMAKE_CXX_FLAGS STRING : "${LDFLAGS|ENV} -foo"
 6
 7
 8[TEST_VAR_EXPANSION_UPDATE_01]
 9opt-set cmake
10use TEST_VAR_EXPANSION_COMMON
11# This will be skipped by the BASH generator without a FORCE option added
12opt-set-cmake-var CMAKE_CXX_FLAGS STRING: "${CMAKE_CXX_FLAGS|CMAKE} -bar"

example-03.py

 1#!/usr/bin/env python3
 2# -*- mode: python; py-indent-offset: 4; py-continuation-offset: 4 -*-
 3from pathlib import Path
 4from pprint import pprint
 5import setprogramoptions
 6
 7def print_separator(label):
 8    print("")
 9    print(f"{label}")
10    print("-"*len(label))
11    return
12
13def test_setprogramoptions(filename="config.ini"):
14    print(f"filename: {filename}")
15
16    section_name = "TEST_VAR_EXPANSION_UPDATE_01"
17    print(f"section_name = {section_name}")
18
19    parser = setprogramoptions.SetProgramOptionsCMake(filename=filename)
20    parser.debug_level = 0
21    parser.exception_control_level = 4
22    parser.exception_control_compact_warnings = True
23
24    data = parser.configparserenhanceddata[section_name]
25    print_separator(f"parser.configparserenhanceddata[{section_name}]")
26    pprint(data, width=120)
27
28    print_separator("Show parser.options")
29    pprint(parser.options, width=200, sort_dicts=False)
30
31    print_separator("Bash Output")
32    print("Note: The _second_ assignment to `CMAKE_CXX_FLAGS` is skipped by a BASH generator")
33    print("      without a `FORCE` option since by definition all CMake `-D` options on a ")
34    print("      BASH command line are both CACHE and FORCE. Within a CMake source fragment")
35    print("      changing an existing CACHE var requires a FORCE option to be set so we should")
36    print("      skip the second assignment to maintain consistency between the bash and cmake")
37    print("      fragment generators with respect to the CMakeCache.txt file that would be")
38    print("      generated.")
39    print("      The `WARNING` message below is terse since it's in compact form -- disable")
40    print("      the `exception_control_compact_warnings` flag to get the full warning message.")
41    print("")
42    option_list = parser.gen_option_list(section_name, generator="bash")
43    print("")
44    print(" \\\n    ".join(option_list))
45
46    print_separator("CMake Fragment")
47    option_list = parser.gen_option_list(section_name, generator="cmake_fragment")
48    if len(option_list) > 0:
49        print("\n".join(option_list))
50    else:
51        print("-")
52    print("")
53
54    return 0
55
56
57def main():
58    """
59    main app
60    """
61    print(80*"-")
62    print(f"- {Path(__file__).name}")
63    print(80*"-")
64    test_setprogramoptions(filename="example-03.ini")
65    return 0
66
67
68if __name__ == "__main__":
69    main()
70    print("Done.")

Console Output

 1--------------------------------------------------------------------------------
 2- example-03.py
 3--------------------------------------------------------------------------------
 4filename: example-03.ini
 5section_name = TEST_VAR_EXPANSION_UPDATE_01
 6
 7parser.configparserenhanceddata[TEST_VAR_EXPANSION_UPDATE_01]
 8-------------------------------------------------------------
 9{}
10
11Show parser.options
12-------------------
13{'TEST_VAR_EXPANSION_UPDATE_01': [{'type': ['opt_set'], 'value': None, 'params': ['cmake']},
14                                  {'type': ['opt_set_cmake_var'], 'value': '${LDFLAGS|ENV} -foo', 'params': ['CMAKE_CXX_FLAGS', 'STRING']},
15                                  {'type': ['opt_set_cmake_var'], 'value': '${CMAKE_CXX_FLAGS|CMAKE} -bar', 'params': ['CMAKE_CXX_FLAGS', 'STRING']}]}
16
17Bash Output
18-----------
19Note: The _second_ assignment to `CMAKE_CXX_FLAGS` is skipped by a BASH generator
20      without a `FORCE` option since by definition all CMake `-D` options on a 
21      BASH command line are both CACHE and FORCE. Within a CMake source fragment
22      changing an existing CACHE var requires a FORCE option to be set so we should
23      skip the second assignment to maintain consistency between the bash and cmake
24      fragment generators with respect to the CMakeCache.txt file that would be
25      generated.
26      The `WARNING` message below is terse since it's in compact form -- disable
27      the `exception_control_compact_warnings` flag to get the full warning message.
28
29!! EXCEPTION SKIPPED (WARNING : ValueError) @ File "/Users/wcmclen/Library/Python/3.9/lib/python/site-packages/setprogramoptions/SetProgramOptionsCMake.py", line 294, in _program_option_handler_opt_set_cmake_var_bash
30
31cmake \
32    -DCMAKE_CXX_FLAGS:STRING="${LDFLAGS} -foo"
33
34CMake Fragment
35--------------
36set(CMAKE_CXX_FLAGS "$ENV{LDFLAGS} -foo" CACHE STRING "from .ini configuration")
37set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -bar" CACHE STRING "from .ini configuration")
38
39Done.

We will note here that the CMake fragment generator will still generate all of the commands. In this case the second set() command would be ignored by CMake since it’s not FORCEd but the main take-away here is that the bash generator omitted the second -D operation since that would be a FORCE operation by default which is not representative of what was specified in the .ini file.