The CLI Flag System

devapps uses absl flags to get configured via the CLI or environ.

Application modules and packages define the flags they require or support within their source code themselves.

Dependent on which of these modules are imported at a certain time after startup (before the call to app.run), then these are the flags presented to the user when he calls -h|--help|--helpful|-hf.

Set flag values are then globals throughout the application.

This makes a lot of sense when a package has a lot of varying use cases, with certain modules sometimes needed or not.

Note

It is allowed to do FLG.foo=bar after startup - but considered bad practice.

Flag Definitions via Nested Class Trees

In devapps, while fully supporting the standard absl mechanics (flags.DEFINE_string) we also allow to defined them in class name spaces:

These are e.g. the flags of the project module:

from devapp.app import FLG, app, run_app, do, system

class Flags:
    # short codes built dynamically (conflict resolved) for all flags:
    autoshort = '' # You could give a short code prefix here

    # flag name:
    class force:
        # CLI help string:
        n = 'Assume y on all questions. Required when started w/o a tty'
        d = False # default

    class init_at:
        n = 'Set up project in given directory. env vars / relative dirs supported.'
        d = ''

    class init_create_unit_files:
        n = 'List service unit files you want to have created for systemctl --user'
        d = []

    class init_resource_match:
        n = 'Install only matching resources. Example: -irm "redis, hub"'
        d = []

    class list_resources_files:
        n = 'Show available definition files.'
        d = False

    class edit_matching_resource_file:
        n = 'Open resource files in $EDITOR, matching given string in their content'
        d = ''

    (...)

def run():
    if FLG.init_at:
        # structlog call:
        app.info('Re-initializing project', location=FLG.init_at)
        (...)

# if flags argument to run_app is given it will implicitly call devapp.tools.define_flags:
main = lambda: run_app(run, flags=Flags)

Full Control

module import does not cause flags already to be defined, the parent class is just a namespace without any magic.

Instead there is special call devapp.tools.define_flags(my_flags_class), which calls absl.define_<type> for any of the inner classes.

Handing the flags via the flags argument into run_app will issue that call.

A call to help then lists the flags on the CLI:

-h: Module Help

This lists the supported flags for the module whose main method is called:

$ ops project -h

Creating A Project With Resources
────────────────────────────────────────────────────────────────────────────────

This plugin is helper for creating a project directory, incl. required local
resources. Your system remains unchanged, except <project_dir> and
<conda_prefix>.

It provides an install action (implicitely by providing the
--init_resource_match or --init_at switch)

Default action is: list (show installable resources, -m <match> filters).

At install we will (re-)initialize a "project_dir", at location given with
--init_at (default: '.'), incl:

❖ Installing available resources, like databases and tools within a given
  directory (conda_prefix)

❖ Creating resource start wrappers in <project_dir>/bin

❖ Generating default config when required

❖ Optionally generating systemd unit files (e.g. via: --init_create_all_units)

❖ Instances support: export <name>_instances=x before running and you'll get x
  systemd units created, for startable commands.

  Example: export client_instances=10; ops p -irm client -icau
  (Name of a resource: ops p [-m <match>])

❖ Any other parametrization: Via environ variables Check key environ vars in
  list output and also doc text.

Privilege escalation is not required for any of these steps.


Main command line flags [matching ops_devapp.project]:
appc    add_post_process_cmd     ''
Add this to all commands which have systemd service units. Intended for output redirection. Not applied when stdout is a tty.
Example: -appc='2>&1 | rotatelogs -e -n1 "$logfile" 1M' ($logfile defined in wrapper -> use single quotes).
Tip: Use rotatelogs only on powers of 10 - spotted problems with 200M. Use 100M or 1G in that case. 
cp      conda_prefix             /home/runner/miniconda3

    Resources install location, except filesystem based ones. Env vars resolved.

    Aliases:
    - local|l: <project_dir>/conda
    - default|d: $HOME/miniconda3 (default path of conda)
    - current|c: Any current conda_prefix set when running.

    Note: Installing resources outside the project keeps the project relocatable and resources reusable for other products.

damsu   delete_all_matching_service_unit_files ''
This removes all matching unit files calling devapp service wrappers. Say "service" to match all 
di      dev_install              False
Set the project up in developer mode - incl. make and poetry file machinery 
emrf    edit_matching_resource_file ''
Open resource files in $EDITOR, matching given string in their content 
f       force                    False
Assume y on all questions. Required when started w/o a tty 
fr      force_reinstall          False
Do not only install resources detected uninstalled but reinstall all 
fd      fs_dir                   default

    Filesystem based resource location. Env vars resolved.
    Aliases:
    - local|l: <project_dir>/fs
    - default|d: $HOME/miniconda3/fs (default path of conda)
    - conda|c: Within conda_prefix/fs

ia      init_at                  ''
Set up project in given directory. env vars / relative dirs supported. Sets install action implicitly 
icau    init_create_all_units    False
If set we create unit files for ALL service type resources 
icuf    init_create_unit_files   ''
List service unit files you want to have created for systemctl --user.  
Valid: Entries in rsc.provides, rsc.cmd or rsc.exe (i.e. the filename of the wrapper in bin dir). Not: rsc.name.
irm     init_resource_match      ''
Like resource match but implies install action 
i       install                  ACTION
Install 
is      install_state            False
show install state infos 
l       list                     ACTION*
Show available definition files. 
lrf     list_resources_files     ACTION
Alias for list action 
        log_resources_fully      False
Always output all settings of resources when logging 
m       resource_match           ''
Provide a match string for actions. Examples: -irm "redis, hub" or -irm '!mysql, !redis' (! negates). 
s       system                   ''
Set to a server for system mode API (e.g. lc hub(s))

-hf [match string]: List ALL (matching) flags. E.g. -hf or -hf log.


-hf: All Flags

-hf <match> gives help for ALL flags imported, here with a match:

$ ops project -hf log | grep -A 100 'All supported'
All supported command line flags [matching log]:
absl.logging
        alsologtostderr          False
also log to stderr? 
        log_dir                  ''
directory to write logfiles into 
        logger_levels            ''
Specify log level of loggers. The format is a CSV list of `name:level`. Where `name` is the logger name used with `logging.getLogger()`, and `level` is a level name  (INFO, DEBUG, etc). e.g. `myapp.foo:INFO,other.logger:DEBUG` 
        logtostderr              False
Should only log to stderr? 
        showprefixforinfo        True
If False, do not prepend prefix to info messages when it's logged to stderr, --verbosity is set to INFO level, and python logging is used. 
        stderrthreshold          fatal
log messages at this level, or more severe, to stderr in addition to the logfile.  Possible values are 'debug', 'info', 'warning', 'error', and 'fatal'.  Obsoletes --alsologtostderr. Using --alsologtostderr cancels the effect of this flag. Please also note that this flag is subject to --verbosity and requires logfile not be stderr. 
v       verbosity                -1
Logging verbosity level. Messages logged at this level or lower will be included. Set to 1 for debug logging. If the flag was not set or supplied, the value will be changed from the default of -1 (warning) to 0 (info) after flags are parsed. 
devapp.tools
dasdi   sensitive_data_identifiers pass.*|.*secret.*
Regexp which helps identify keys carrying sensitive information (for filtering out of logs). Case insensitive matching. 
structlogging.sl
latn    log_add_thread_name      False
Add name of thread 
ldcs    log_dev_coljson_style    dark
 Pygments style for colorized json. To use the 16 base colors and leave it to the terminal palette how to render choose light or dark <abap|algol|algol_nu|arduino|autumn|borland|bw|colorful|dark|default|dracula|emacs|friendly|friendly_grayscale|fruity|gruvbox-dark|gruvbox-light|igor|inkpot|light|lilypond|lovelace|manni|material|monokai|murphy|native|one-dark|paraiso-dark|paraiso-light|pastie|perldoc|rainbow_dash|rrt|sas|solarized-dark|solarized-light|stata|stata-dark|stata-light|tango|trac|vim|vs|xcode|zenburn>
ldfc    log_dev_fmt_coljson      json,payload
List of keys to log as json. 
lf      log_fmt                  auto
Force a log format. 0: off, 1: auto, 2: plain, 3: plain_no_colors, 4: json. Note: This value can be set away from auto via export log_fmt as well. 
ll      log_level                20
Log level (10: debug, 20: info, ...). You may also say log_level=error 
lsf     log_stack_filter         fn not contains frozen and fn not contains /rx/
When logging error tracebacks this is an optional filter. Keywords:fn: filename, frame: frame nr, line: line nr, name: name of callable  
Example: fn contains project and frame lt 1 (axiros/pycond expression)
lsmf    log_stack_max_frames     3
Maximum Frames Shown in Terminal Stack Traces 
ltln    log_thread_local_names   False
Prefer thread local logger_name, when set 
ltf     log_time_fmt             %m-%d %H:%M:%S
Log time format. Shortcuts: "ISO", "dt" 
ops_devapp.project
appc    add_post_process_cmd     ''
Add this to all commands which have systemd service units. Intended for output redirection. Not applied when stdout is a tty.
Example: -appc='2>&1 | rotatelogs -e -n1 "$logfile" 1M' ($logfile defined in wrapper -> use single quotes).
Tip: Use rotatelogs only on powers of 10 - spotted problems with 200M. Use 100M or 1G in that case. 
        log_resources_fully      False
Always output all settings of resources when logging 


Flag Types

$ cat /tmp/flagtest.py
#!/usr/bin/env python
from devapp.app import app, run_app, FLG

class Flags:
    autoshort = '' # enabling short forms, prefixed with '', i.e. not prefixed

    class my_bool:
        d = False

    class my_float:
        d = 1.1

    class my_int:
        d = 1

    class my_multi_str:
        t = 'multi_string'  # can supply more than once, any value

    class my_list:
        s = 'x'   # non auto short form
        t = list  # comma sepped values into a list 
        d =  'foo, bar'

    class my_opt:
        t = ['foo', 'bar']  # can pick exactly one within the list (enum)
        d =  'foo'

    class my_opt_multi:
        t = ('a', 'b', 'c') # now we can select more than one within the tuple (multi_enum)
        d =  'a'

    class my_str:
        d = 'foo' # most easy way

    class my_str_detailed:
        '''Detailed help'''
        n = '''Options (multiline help)
        - opt1: foo
        - opt2: bar
        '''
        s = False # disable short
        d = 'opt1'

    class my_condition:
        # will be parsed into an axiros/pycond filter, incl. the condition (list).
        t = 'pycond'
        d = 'fn not contains frozen and fn not contains /rx/'


# Print out all FLG vals.
# Normal (global) app access e.g. like : if FLG.my_int > 42:
flg = lambda: [(k, getattr(FLG, k)) for k in FLG if k.startswith('my_')]
run = lambda: app.info('Flag values (CLI over defaults):', json=flg())

if __name__ == '__main__':
    # supplying the flags keyword implicitly calls devapp.tools.define_flags on them:
    run_app(run, flags=Flags)

With this

$ /tmp/flagtest.py -h
__main__
mb    my_bool         False       My_bool                                       
mc    my_condition    fn not..    My_condition (axiros/pycond expression)       
mf    my_float        1.1         My_float                                      
mi    my_int          1           My_int                                        
x     my_list         foo,bar     My_list                                       
mms   my_multi_str    ['']        My_multi_str; repeat this option to specify.. 
mo    my_opt          foo          My_opt<foo|bar>                              
mom   my_opt_multi    ['a']       My_opt_multi; repeat this option to specify.. 
ms    my_str          foo         My_str                                        
      my_str_detailed opt1        Options (multiline help) - opt1: foo - opt2:..

-hf [match string]: List ALL (matching) flags. E.g. -hf or -hf log.

$ /tmp/flagtest.py -mo baz || true
FATAL Flags parsing error: flag --my_opt=baz: value should be one of <foo|bar>
Pass --helpshort or --helpfull to see help on flags.
$ /tmp/flagtest.py -ms a -mb -mf 42.1 -mi 42 -mms a -mms b -mo bar -mom b -mom c -x a,b -ms b -lf plain
07-25 16:06:03 [info     ] Flag values (CLI over defaults): [flagtest] json=[
  [
    "my_list",
    [
      "a",
      "b"
    ]
  ],
  [
    "my_bool",
    true
  ],
  [
    "my_condition",
    [
      "<function build_pycond_flag_expr.<locals>.<lambda> at 0x7fcc09a830e0>",
      "fn not contains frozen and fn not contains /rx/"
    ]
  ],
  [
    "my_float",
    42.1
  ],
  [
    "my_int",
    42
  ],
  [
    "my_multi_str",
    [
      "a",
      "b"
    ]
  ],
  [
    "my_opt",
    "bar"
  ],
  [
    "my_opt_multi",
    [
      "b",
      "c"
    ]
  ],
  [
    "my_str",
    "b"
  ],
  [
    "my_str_detailed",
    "opt1"
  ]
]


Hint

Note my_str defined twice in the example - last wins (except when defined multi_string) -> you can preparametrize apps in wrappers and still overwrite flags when calling the wrapper.

E.g. in the wrapper you have -ll 20 while in the call you say -ll 10 to have debug logging for a certain run.

Environ Flags

Adding --environ_flags causes the app to check the process environ first(!), for any flag value.

Setting project directory and log level via environ

$ export init_at="$HOME/foo"; export log_level=30
$
$ ops project -ia /tmp --environ_flags

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│                                                                              │
│ /home/runner/work/devapps/devapps/src/devapp/app.py:509 in run_phase_2       │
│                                                                              │
│   506 │   │   │   # main = lambda: run_app(Action, flags=Flags, wrapper=clea │
│   507 │   │   │   if FLG.dirwatch:                                           │
│   508 │   │   │   │   signal.signal(reload_signal, reload_handler)           │
│ ❱ 509 │   │   │   res = wrapper(main) if wrapper else main()                 │
│   510 │   │   │   if FLG.dirwatch:                                           │
│   511 │   │   │   │   app.info('Keep running, dirwatch is set')              │
│   512 │   │   │   │   while 1:                                               │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │            args = [                                                      │ │
│ │                   │                                                      │ │
│ │                   '/home/runner/miniconda3/envs/devapps_py3.7/bin/ops',  │ │
│ │                   │   'project'                                          │ │
│ │                   ]                                                      │ │
│ │              ex = DieNow('Not exists', {'dir': '/home/runner/foo'})      │ │
│ │ flags_validator = None                                                   │ │
│ │          kw_log = {}                                                     │ │
│ │             log = <AXLogger(context={}, processors=[<function            │ │
│ │                   filter_by_level at 0x7fa9487c0290>,                    │ │
│ │                   <structlog.processors.TimeStamper object at            │ │
│ │                   0x7fa948329cd0>, <function add_log_level at            │ │
│ │                   0x7fa9489599e0>, <function censor_passwords at         │ │
│ │                   0x7fa9487c0320>, <function add_logger_name at          │ │
│ │                   0x7fa9487c03b0>, <function                             │ │
│ │                   stack_info.<locals>._stack_info at 0x7fa949f2a440>,    │ │
│ │                   <structlogging.renderers.ThemeableConsoleRenderer      │ │
│ │                   object at 0x7fa94869b9d0>])>                           │ │
│ │            main = <function run at 0x7fa9486888c0>                       │ │
│ │            name = 'project'                                              │ │
│ │            post = None                                                   │ │
│ │             res = None                                                   │ │
│ │     watcher_pid = None                                                   │ │
│ │         wrapper = None                                                   │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ /home/runner/work/devapps/devapps/src/devapp/plugins/ops_devapp/project/__in │
│ it__.py:301 in run                                                           │
│                                                                              │
│   298 │   d = os.path.abspath(d)                                             │
│   299 │   d = d[:-1] if d.endswith('/') else d                               │
│   300 │   if not exists(d):                                                  │
│ ❱ 301 │   │   app.die('Not exists', dir=d)                                   │
│   302 │   do(os.chdir, d)                                                    │
│   303 │   d = FLG.init_at = os.path.abspath('.')                             │
│   304 │   project.set_project_dir(dir=d)                                     │
│                                                                              │
│ ╭──────── locals ────────╮                                                   │
│ │ d = '/home/runner/foo' │                                                   │
│ │ m = ''                 │                                                   │
│ ╰────────────────────────╯                                                   │
│                                                                              │
│ /home/runner/work/devapps/devapps/src/devapp/app.py:166 in die               │
│                                                                              │
│   163 │                                                                      │
│   164 │   def die(msg, **kw):                                                │
│   165 │   │   """Application decided to bail out"""                          │
│ ❱ 166 │   │   raise DieNow(msg, kw)                                          │
│   167 │                                                                      │
│   168 │   app.die = die                                                      │
│   169 │   app.name = name                                                    │
│                                                                              │
│ ╭───────────── locals ──────────────╮                                        │
│ │  kw = {'dir': '/home/runner/foo'} │                                        │
│ │ msg = 'Not exists'                │                                        │
│ ╰───────────────────────────────────╯                                        │
╰──────────────────────────────────────────────────────────────────────────────╯
DieNow: ('Not exists', {'dir': '/home/runner/foo'})

07-25 16:06:09 [error    ] Not exists                     [project] dir=/home/runner/foo
$

Environ over CLI

Please note again that the environ value does overwrite the CLI value, when environ_flags is explicitly set. On the cli, you'd have to use the unset command first.

Flagsets

You can store full sets of flags in files and refer to them via the absl standard --flagfile=.... flag.

Using Flags in pytest

When the started process is pytest, then the environ_flags flag is set to true. Means you can export non default flag values before starting pytest like so:

export my_flag=myval && pytest -xs test/test_my_test.py

Action Flags

Example:

$ cat /tmp/action_flagtest.py
#!/usr/bin/env python
from devapp.app import app, run_app, FLG


class Flags:
    autoshort=''

    class force:
        d = False

    class Actions:
        class install:
            d = False

            class verbose:
                s = 'iv' # no auto for nested flags
                d = False

        class run:
            d = True # default


class ActionNS:
    def _pre():
        print('pre')

    def _post():
        print('post')

    def run():
        print('running', FLG.force)

    def install():
        print('installing', FLG.force, FLG.install_verbose)

if __name__ == '__main__':
    run_app(ActionNS, flags=Flags)

Test it:

$ /tmp/action_flagtest.py -h
__main__
f     force           False       Force                                         
i     install         ACTION      Install                                       
r     run             ACTION*     Run

-hf [match string]: List ALL (matching) flags. E.g. -hf or -hf log.

$ /tmp/action_flagtest.py
pre
running False
post
$ /tmp/action_flagtest.py install -iv
pre
installing False True
post
$ /tmp/action_flagtest.py -f install --install_verbose
pre
installing True True
post
$ /tmp/action_flagtest.py run --install_verbose=True || true
FATAL Flags parsing error: Unknown command line flag 'install_verbose'
Pass --helpshort or --helpfull to see help on flags.

Mind the concatenation of action and flag name for the nested property verbose within def install() action.