Procman

A lightweight process manager for user scripts.

But what is it, really?

Procman is a tool for running multiple external processes and multiplexing their output to the console. It also cleans up and stops the whole stack if any one of the running processes stops or dies on its own.

Procman is loosely based on Honcho and uses Honcho’s process manager API. Honcho (and Foreman, and Heroku) parses a Procfile to run an application stack, usually with a specific .env file. See the Introduction in the Honcho documentation for an overview with an example Procfile.

Procman provides an easy way to run such applications using a single YAML configuration file instead. Procman includes a functional example config with a small demo Flask app that uses redis to store a web counter (I did say small/simple…). The example app requires both redis-py (which in turn requires redis) and flask. Note installing Procman with the [examples] flag will install the python deps but you must install redis using your own platform tools (eg, apt or brew).

Procman is tested on the 3 primary GH runner platforms, so as long as you have a new-ish Python and a sane shell environment it should Just Work.

Note Procman only supports Python 3.6+.

Quick Start

Procman is mainly configuration-driven via YAML config files; the included example can be displayed and copied via command-line options (see the Usage section below). To create your own configuration, you need at least one script to run and a place to put it (see Configuration settings for more details).

The current version supports minimal command options and there are no required arguments:

(dev) user@host $ procman
usage: procman [-h] [-v] [-d] [-c RUNFOR] [-D] [-t] [--version] [-S]

Process manager for user scripts

options:
  -h, --help            show this help message and exit
  -v, --verbose         Display more processing info (default: False)
  -d, --dump-config     dump active yaml configuration to stdout (default:
                        False)
  -c RUNFOR, --countdown RUNFOR
                        Runtime STOP timer in seconds - 0 means run until
                        whenever (default: 0)
  -D, --demo            Run demo config (default: False)
  -t, --test            Run sanity checks (default: False)
  --version             show program's version number and exit
  -S, --show            Display user data paths (default: False)

No cfg file found; use the --demo arg or create a cfg file

Usage

To get started, clone this repository, then follow the virtual environment install steps below. Procman uses pathlib to find user paths, which you can view below after running procman --show. Run procman --dump-config to view the active YAML configuration.

To create your own default config file in the working directory, the local copy must be named .procman.yaml. To get a copy of the example configuration file, do:

$ cd path/to/work/dir/
$ procman --dump-config > .procman.yaml
$ $EDITOR .procman.yaml
$ procman --dump-config  # you should see your config settings

If needed, you can also create additional procman config files to override your default project configuration. These alternate config files can have arbitrary names (ending in ‘.yml’ or ‘.yaml’) but we recommend using something like procman-dev-myproject.yml or similar. Since only one configuration can be “active”, the non-default config file must be set via the environment variable PROCMAN_CFG, eg:

$ procman --dump-config > procman-develop.yml
$ $EDITOR procman-develop.yml  # set alternate scripts, other options
$ PROCMAN_CFG="procman-develop.yml" procman --verbose

Configuration settings

Using your preferred editor, edit/add process blocks to scripts as shown in the example configuration (each “block” is a list element).

Note there can be only one default configuration in a given project tree named .procman.yaml, however, you can override the default name via the environment variable PROCMAN_CFG=path/to/procman_othername.yaml. Additional config file guidance includes:

scripts_path:: the path to the script containing directory, ie, proc_dir, which can be relative, absolute, or null, depending on where the script directory is
scripts:: at least one process block with proc_enable: true should be present (under scripts)
proc_label:: is the process label for the script (see log display below)
proc_name:: the actual (file)name of the script
proc_dir:: the directory name where the script lives
proc_runner:: the name of the script interpreter, eg, python or ruby, or null if calling an executable directly
proc_enable:: enable/disable this process block
proc_opts:: any required script args (default is an empty list)

Install with pip

This package is not yet published on PyPI, thus use one of the following to install procman on any platform. Install from the main branch:

$ pip install https://github.com/sarnold/procman/archive/refs/heads/master.tar.gz

or use this command to install a specific release version:

$ pip install https://github.com/sarnold/procman/releases/download/0.1.0/procman-0.1.0.tar.gz

The full package provides the procman executable as well as a working demo with a reference configuration with defaults for all values.

Note

To run the example application, you need to first install redis via your system package manager.

If you’d rather work from the source repository, it supports the common idiom to install it on your system in a virtual env after cloning:

$ python -m venv env
$ source env/bin/activate
(env) $ pip install .[examples]
(env) $ procman --version
procman 0.1.1.dev16+g3b96476.d20230922
(env) $ deactivate

The alternative to python venv is the Tox test driver. If you have it installed already, clone this repository and try the following commands from the procman source directory.

To install the package with examples and run the checks:

$ tox -e py

To run pylint:

$ tox -e lint

To install in developer mode:

$ tox -e dev

To actually run the active configuration file for 30 seconds, run:

$ tox -e serv -- 30

Running the following command will install the package and then run the (built-in) example config via the --demo option for 10 seconds using the tox serv environment; note you can override the --demo option by providing the timeout value as shown above:

$ tox -e serv
serv: install_deps> python -I -m pip install 'pip>=21.1' 'setuptools_scm[toml]' '.[examples]'
serv: commands[0]> procman --countdown 10 --demo
02:15 system | redis started (pid=15356)
02:15 system | web started (pid=15355)
02:15 redis  | Using socket runtime dir: /tmp/redis-ipc
02:15 redis  | 15361:C 22 Sep 2023 14:02:15.793 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
02:15 redis  | 15361:C 22 Sep 2023 14:02:15.793 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=15361, just started
02:15 redis  | 15361:C 22 Sep 2023 14:02:15.793 # Configuration loaded
02:15 redis  | 15361:M 22 Sep 2023 14:02:15.794 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
02:15 redis  | 15361:M 22 Sep 2023 14:02:15.794 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
02:15 redis  | 15361:M 22 Sep 2023 14:02:15.794 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
02:15 redis  | 15361:M 22 Sep 2023 14:02:15.794 * monotonic clock: POSIX clock_gettime
02:15 redis  | 15361:M 22 Sep 2023 14:02:15.795 * Running mode=standalone, port=0.
02:15 redis  | 15361:M 22 Sep 2023 14:02:15.795 # Server initialized
02:15 redis  | 15361:M 22 Sep 2023 14:02:15.795 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
02:15 redis  | 15361:M 22 Sep 2023 14:02:15.796 * The server is now ready to accept connections at /tmp/redis-ipc/socket
02:15 web    |  * Serving Flask app 'app'
02:15 web    |  * Debug mode: on
02:15 web    | WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
02:15 web    |  * Running on http://localhost:8000
02:15 web    | Press CTRL+C to quit
02:15 web    |  * Restarting with stat
02:16 web    |  * Debugger is active!
02:16 web    |  * Debugger PIN: 112-588-591
02:25 system | sending SIGTERM to web (pid 15355)
02:25 system | sending SIGTERM to redis (pid 15356)
02:25 redis  | 15361:signal-handler (1695416545) Received SIGTERM scheduling shutdown...
02:25 system | web stopped (rc=0)
02:25 redis  | 15361:M 22 Sep 2023 14:02:25.853 # User requested shutdown...
02:25 redis  | 15361:M 22 Sep 2023 14:02:25.853 * Saving the final RDB snapshot before exiting.
02:25 redis  | 15361:M 22 Sep 2023 14:02:25.859 * DB saved on disk
02:25 redis  | 15361:M 22 Sep 2023 14:02:25.859 * Removing the pid file.
02:25 redis  | 15361:M 22 Sep 2023 14:02:25.859 * Removing the unix socket file.
02:25 redis  | 15361:M 22 Sep 2023 14:02:25.859 # Redis is now ready to exit, bye bye...
02:25 system | redis stopped (rc=-15)
  serv: OK (16.17=setup[5.88]+cmd[10.29] seconds)
  congratulations :) (16.22 seconds)

Note

After running the serv command, use the environment created by Tox just like any other Python virtual environment. As shown, the dev install mode of Pip allows you to edit the code and run it again while inside the virtual environment. By default Tox environments are created under .tox/ and named after the env argument (eg, dev).

Full list of additional tox commands:

tox -e dev pip “developer” install
tox -e serv will run the active configuration then stop (default: 10 sec)
tox -e style will run flake8 style checks
tox -e lint will run pylint (somewhat less permissive than PEP8/flake8 checks)
tox -e mypy will run mypy import and type checking
tox -e isort will run isort import checks
tox -e clean will remove all generated/temporary files

To build/lint the html docs, use the following tox commands:

tox -e docs build the documentation using sphinx and the api-doc plugin
tox -e docs-lint build the docs and run the sphinx link checking

To install the latest release, eg with your own tox.ini file in another project, use something like this:

$ pip install -U https://github.com/sarnold/procman/releases/download/0.1.0/procman-0.1.0-py3-none-any.whl

Making Changes & Contributing

We use the gitchangelog action to generate our changelog and GH Release page, as well as the gitchangelog message format to help it categorize/filter commits for a tidier changelog. Please use the appropriate ACTION modifiers in any Pull Requests.

This repo is also pre-commit enabled for various linting and format checks. The checks run automatically on commit and will fail the commit (if not clean) with some checks performing simple file corrections.

If other checks fail on commit, the failure display should explain the error types and line numbers. Note you must fix any fatal errors for the commit to succeed; some errors should be fixed automatically (use git status and git diff to review any changes).

See the following pages for more information on gitchangelog and pre-commit.

You will need to install pre-commit before contributing any changes; installing it using your system’s package manager is recommended, otherwise install with pip into your usual virtual environment using something like:

$ sudo emerge pre-commit  --or--
$ pip install pre-commit

then install it into the repo you just cloned:

$ git clone https://github.com/sarnold/procman
$ cd procman/
$ pre-commit install

It’s usually a good idea to update the hooks to the latest version:

pre-commit autoupdate

SBOM and license info

This project is now compliant the REUSE Specification Version 3.3, so the corresponding license information for all files can be found in the REUSE.toml configuration file with license text(s) in the LICENSES/ folder.

Related metadata can be (re)generated with the following tools and command examples.

reuse-tool - REUSE compliance linting and sdist (source files) SBOM generation
sbom4python - generate SBOM with full dependency chain

Commands

Use tox to create the environment and run the lint command:

$ tox -e reuse                      # --or--
$ tox -e reuse -- spdx > sbom.txt   # generate sdist files sbom

Note you can pass any of the other reuse commands after the -- above.

Use the above environment to generate the full SBOM in text format:

$ source .tox/reuse/bin/activate
$ sbom4python --system --use-pip -o <file_name>.txt

Be patient; the last command above may take several minutes. See the doc links above for more detailed information on the tools and specifications.