SetProgramOptions Python Module
DEPRECATION NOTICE
This package was forked by the original author and is now maintained under
the name ActiveConfigProgramOptions
(GitLab,
PyPI,
Docs).
Users of SetProgramOptions
should switch to the new package.
Indices and Tables
Overview
The SetProgramOptions
package extends the
ConfigParserEnhanced
package by adding additional operations for handling command-line
options.
The primary use case that provided the impetus to develop SetProgramOptions was to support complex configuration environments for a software project that is tested on a variety of platforms and architectures, including GPUs and HPC systems. This project is several million lines of code and has hundreds of CMake options in its configuration space.
We developed SetProgramOptions and SetProgramOptions to allow our build system to use optimized .ini files to manage our configuration space.
This package includes two classes:
SetProgramOptions - A general purpose command line handler that handles generic command line options.
SetProgramOptionsCMake - A subclass of SetProgramOptions, this class further extends SetProgramOptions by adding CMake-specific operations to provide ease of use for CMake specific options. It also adds an additional generator option to allow the generation of either bash style command line options or a CMake source fragment file.
An example .ini file using SetProgramOptions
might look like:
1[Directory *nix]
2opt-set ls
This configuration is the SetProgramOptions version of a hello world example.
Here, the opt-set ls
option is specifying a single command line option
which in this case is the command ls.
We can expand this to add additional entries:
1[Directory *nix]
2opt-set ls
3opt-set -l
4opt-set -r
5opt-set -t
6opt-remove -r
When processed, this example would result in a concactenated string containing
the command ls -l -t
. We threw in the opt-remove -r
operation which
removed the -r entry.
For more details on how this is used, see the Examples section below.
Examples
Here we show a few examples of how SetProgramOptions
and
SetProgramOptionsCMake
can be used.
Example 1
In example-01 we have a fairly simple .ini file that demostrates utilization
of the use
operation that comes built in with
ConfigParserEnhanced.
In this example we are going to demonstrate how the opt-set
operations
in our .ini file can be used to generate a custom bash command with
customizable argument sets added.
In this case, we will process the [MY_LS_COMMAND]
section to generate
a bash command that would generate a directory listing in list form by
reverse time order of last modified with a custom timestamp format.
While this example is quite simple we can see how a complex environment in a DevOps setting might use this to bundle “common” operations to reduce the amount of copying and pasting that is used.
example-01.ini
1#
2# example-01.ini
3#
4[LS_COMMAND]
5opt-set ls
6
7[LS_LIST_TIME_REVERSED]
8opt-set "-l -t -r"
9
10[LS_CUSTOM_TIME_STYLE]
11opt-set --time-style : "+%%Y-%%m-%%d %%H:%%M:%%S"
12
13[MY_LS_COMMAND]
14use LS_COMMAND
15use LS_LIST_TIME_REVERSED
16use LS_CUSTOM_TIME_STYLE
example-01.py
1#!/usr/bin/env python3
2# -*- mode: python; py-indent-offset: 4; py-continuation-offset: 4 -*-
3from pathlib import Path
4import setprogramoptions
5
6print(80*"-")
7print(f"- {Path(__file__).name}")
8print(80*"-")
9
10filename = "example-01.ini"
11popts = setprogramoptions.SetProgramOptions(filename)
12
13section = "MY_LS_COMMAND"
14popts.parse_section(section)
15bash_options = popts.gen_option_list(section, generator="bash")
16print(" ".join(bash_options))
17
18print("Done")
Console Output
1--------------------------------------------------------------------------------
2- example-01.py
3--------------------------------------------------------------------------------
4ls -l -t -r --time-style="+%Y-%m-%d %H:%M:%S"
5Done
Example 2
Example 2 demonstrates the use of SetProgramOptionsCMake
which is a subclass of SetProgramOptions
and implements
operations for handling CMake variables in a .ini file.
SetProgramOptionsCMake also introduces a new “CMake fragment”
generator and implements logic to maintain consistency of
behavior between generated BASH scripts and CMake fragments
with knoweldge of how CMake treats -D
operations at the
command line versus set()
operations inside a CMake script.
Here we show the new operation opt-set-cmake-var
which
has the form opt-set-cmake-var VARNAME <OPTIONAL PARAMS> : VALUE
.
Details and a comprehensive list of the optional parameters can
be found in the SetProgramOptionsCMake
Class Reference page.
The main elements this example will demonstrate is how to use
opt-set-cmake-var
operations and how they can be used.
This example is fairly straightforward since the .ini file options being defined will generate the same output for both generator types.
example-02.ini
1#
2# example-02.ini
3#
4[CMAKE_COMMAND]
5opt-set cmake
6
7[CMAKE_GENERATOR_NINJA]
8opt-set -G : Ninja
9
10[MYPROJ_OPTIONS]
11opt-set-cmake-var MYPROJ_CXX_FLAGS STRING : "-O0 -fopenmp"
12opt-set-cmake-var MYPROJ_ENABLE_OPTION_A BOOL FORCE : ON
13opt-set-cmake-var MYPROJ_ENABLE_OPTION_B BOOL : ON
14
15[MYPROJ_SOURCE_DIR]
16opt-set /path/to/source/dir
17
18[MYPROJ_CONFIGURATION_NINJA]
19use CMAKE_COMMAND
20use CMAKE_GENERATOR_NINJA
21use MYPROJ_OPTIONS
22use MYPROJ_SOURCE_DIR
example-02.py
1#!/usr/bin/env python3
2# -*- mode: python; py-indent-offset: 4; py-continuation-offset: 4 -*-
3from pathlib import Path
4import setprogramoptions
5
6def print_separator(label):
7 print("")
8 print(f"{label}")
9 print("-"*len(label))
10 return
11
12print(80*"-")
13print(f"- {Path(__file__).name}")
14print(80*"-")
15
16filename = "example-02.ini"
17popts = setprogramoptions.SetProgramOptionsCMake(filename)
18
19section = "MYPROJ_CONFIGURATION_NINJA"
20popts.parse_section(section)
21
22# Generate BASH output
23print_separator("Generate Bash Output")
24bash_options = popts.gen_option_list(section, generator="bash")
25print(" \\\n ".join(bash_options))
26
27# Generate a CMake Fragment
28print_separator("Generate CMake Fragment")
29cmake_options = popts.gen_option_list(section, generator="cmake_fragment")
30print("\n".join(cmake_options))
31
32print("\nDone")
Console Output
1--------------------------------------------------------------------------------
2- example-02.py
3--------------------------------------------------------------------------------
4
5Generate Bash Output
6--------------------
7cmake \
8 -G=Ninja \
9 -DMYPROJ_CXX_FLAGS:STRING="-O0 -fopenmp" \
10 -DMYPROJ_ENABLE_OPTION_A:BOOL=ON \
11 -DMYPROJ_ENABLE_OPTION_B:BOOL=ON \
12 /path/to/source/dir
13
14Generate CMake Fragment
15-----------------------
16set(MYPROJ_CXX_FLAGS "-O0 -fopenmp" CACHE STRING "from .ini configuration")
17set(MYPROJ_ENABLE_OPTION_A ON CACHE BOOL "from .ini configuration" FORCE)
18set(MYPROJ_ENABLE_OPTION_B ON CACHE BOOL "from .ini configuration")
19
20Done
Example 3
The last example we show here is a bit more complicated and gets into
some of the differences in how CMake treats a command line variable
being set via a -D
option versus a set()
operation within a
CMakeLists.txt file.
The script prints out a notice explaining the nuance but the general
idea is that CMake treats options provided by a -D
option at the
command line as though they are CACHE
variables with the FORCE
option enabled. This is designed to allow command-line parameters to
generally take precedence over what a CMakeLists.txt file might set
as though it’s a user-override. The added FORCE
option also ensures
that if there are multiple -D
options setting the same variable the
last one will win.
This can have a subtle yet profound effect on how we must process our
opt-set-cmake-var
operations within a .ini file if our goal is to
ensure that the resulting CMakeCache.txt
file generated by a CMake
run would be the same for both bash and cmake fragment generators.
In this case, in order to have a variable be set by the bash generator
it must be a CACHE
variable – which can be accomplished by either
adding a TYPE or a FORCE option.
We will note here that if FORCE
is given without a TYPE then we
use the default type of STRING.
If the same CMake variable is being assigned, such as in a case where
we have a section that is updating a flag, then the FORCE option must
be present on the second and all subsequent occurrences of opt-set-cmake-var
or the bash generator will skip over that assignment since a non-forced
set()
operation in CMake would not overwrite an existing cache var.
This situation can occur frequently if our .ini file(s) are structured to
have some common configuration option set and then a specialization which
updates one of the arguments. One example of this kind of situation might
be where we have a specialization that adds OpenMP and we would want to add
the -fopenmp
flags to our linker flags.
example-03.ini
1#
2# example-03.ini
3#
4[TEST_VAR_EXPANSION_COMMON]
5opt-set-cmake-var CMAKE_CXX_FLAGS STRING : "${LDFLAGS|ENV} -foo"
6
7
8[TEST_VAR_EXPANSION_UPDATE_01]
9opt-set cmake
10use TEST_VAR_EXPANSION_COMMON
11# This will be skipped by the BASH generator without a FORCE option added
12opt-set-cmake-var CMAKE_CXX_FLAGS STRING: "${CMAKE_CXX_FLAGS|CMAKE} -bar"
example-03.py
1#!/usr/bin/env python3
2# -*- mode: python; py-indent-offset: 4; py-continuation-offset: 4 -*-
3from pathlib import Path
4from pprint import pprint
5import setprogramoptions
6
7def print_separator(label):
8 print("")
9 print(f"{label}")
10 print("-"*len(label))
11 return
12
13def test_setprogramoptions(filename="config.ini"):
14 print(f"filename: {filename}")
15
16 section_name = "TEST_VAR_EXPANSION_UPDATE_01"
17 print(f"section_name = {section_name}")
18
19 parser = setprogramoptions.SetProgramOptionsCMake(filename=filename)
20 parser.debug_level = 0
21 parser.exception_control_level = 4
22 parser.exception_control_compact_warnings = True
23
24 data = parser.configparserenhanceddata[section_name]
25 print_separator(f"parser.configparserenhanceddata[{section_name}]")
26 pprint(data, width=120)
27
28 print_separator("Show parser.options")
29 pprint(parser.options, width=200, sort_dicts=False)
30
31 print_separator("Bash Output")
32 print("Note: The _second_ assignment to `CMAKE_CXX_FLAGS` is skipped by a BASH generator")
33 print(" without a `FORCE` option since by definition all CMake `-D` options on a ")
34 print(" BASH command line are both CACHE and FORCE. Within a CMake source fragment")
35 print(" changing an existing CACHE var requires a FORCE option to be set so we should")
36 print(" skip the second assignment to maintain consistency between the bash and cmake")
37 print(" fragment generators with respect to the CMakeCache.txt file that would be")
38 print(" generated.")
39 print(" The `WARNING` message below is terse since it's in compact form -- disable")
40 print(" the `exception_control_compact_warnings` flag to get the full warning message.")
41 print("")
42 option_list = parser.gen_option_list(section_name, generator="bash")
43 print("")
44 print(" \\\n ".join(option_list))
45
46 print_separator("CMake Fragment")
47 option_list = parser.gen_option_list(section_name, generator="cmake_fragment")
48 if len(option_list) > 0:
49 print("\n".join(option_list))
50 else:
51 print("-")
52 print("")
53
54 return 0
55
56
57def main():
58 """
59 main app
60 """
61 print(80*"-")
62 print(f"- {Path(__file__).name}")
63 print(80*"-")
64 test_setprogramoptions(filename="example-03.ini")
65 return 0
66
67
68if __name__ == "__main__":
69 main()
70 print("Done.")
Console Output
1--------------------------------------------------------------------------------
2- example-03.py
3--------------------------------------------------------------------------------
4filename: example-03.ini
5section_name = TEST_VAR_EXPANSION_UPDATE_01
6
7parser.configparserenhanceddata[TEST_VAR_EXPANSION_UPDATE_01]
8-------------------------------------------------------------
9{}
10
11Show parser.options
12-------------------
13{'TEST_VAR_EXPANSION_UPDATE_01': [{'type': ['opt_set'], 'value': None, 'params': ['cmake']},
14 {'type': ['opt_set_cmake_var'], 'value': '${LDFLAGS|ENV} -foo', 'params': ['CMAKE_CXX_FLAGS', 'STRING']},
15 {'type': ['opt_set_cmake_var'], 'value': '${CMAKE_CXX_FLAGS|CMAKE} -bar', 'params': ['CMAKE_CXX_FLAGS', 'STRING']}]}
16
17Bash Output
18-----------
19Note: The _second_ assignment to `CMAKE_CXX_FLAGS` is skipped by a BASH generator
20 without a `FORCE` option since by definition all CMake `-D` options on a
21 BASH command line are both CACHE and FORCE. Within a CMake source fragment
22 changing an existing CACHE var requires a FORCE option to be set so we should
23 skip the second assignment to maintain consistency between the bash and cmake
24 fragment generators with respect to the CMakeCache.txt file that would be
25 generated.
26 The `WARNING` message below is terse since it's in compact form -- disable
27 the `exception_control_compact_warnings` flag to get the full warning message.
28
29!! EXCEPTION SKIPPED (WARNING : ValueError) @ File "/Users/wcmclen/Library/Python/3.9/lib/python/site-packages/setprogramoptions/SetProgramOptionsCMake.py", line 294, in _program_option_handler_opt_set_cmake_var_bash
30
31cmake \
32 -DCMAKE_CXX_FLAGS:STRING="${LDFLAGS} -foo"
33
34CMake Fragment
35--------------
36set(CMAKE_CXX_FLAGS "$ENV{LDFLAGS} -foo" CACHE STRING "from .ini configuration")
37set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -bar" CACHE STRING "from .ini configuration")
38
39Done.
We will note here that the CMake fragment generator will still generate
all of the commands. In this case the second set()
command would be ignored
by CMake since it’s not FORCEd but the main take-away here is that the
bash generator omitted the second -D
operation since that would be
a FORCE
operation by default which is not representative of what was
specified in the .ini file.