Overview

APM is Tom's Advanced Process Manager for Linux. One binary, copy and run — no config files required to get started. Complexity is APM's problem, not yours.

APM runs as a background daemon and manages worker processes. You interact with it through the apm CLI. The daemon auto-starts the first time you run any apm command.

Architecture

The daemon communicates with the CLI via an abstract Unix socket. Workers are child processes managed by the daemon. Each worker can have multiple parallel instances. The built-in reverse proxy routes incoming connections across instances using round-robin.

  CLI  ──(unix socket)──  Daemon  ──  Worker [4 instances]
                                  └──  Worker [1 instance]
                                  └──  GUI server (port 6789)

Philosophy

• Zero config to start — defaults are correct for 90% of cases
• CLI-first, config files for persistence and power users
• Every error message answers "what do I do now"
• Never crash on a bad optional field — warn and continue
• Linux only. Windows is not supported.

Installation

Run the install script as root. It downloads the right binary for your architecture, sets up the system group and log file, installs the init service, and starts the daemon.

# One-liner install
$ curl -fsSL https://processmanager.dev/install.sh | sudo bash

# Or download first, review, then run
$ curl -fsSL https://processmanager.dev/install.sh -o install.sh
$ sudo bash install.sh

# Verify
$ apm --version

The installer sets up:

• /usr/sbin/apm — binary
• /var/log/apm.log — daemon log (group-readable by apm)
• /etc/apm/apm.conf — default config (created if absent)
• apm OS group — add users with usermod -aG apm <user>
• Startup service (systemd, OpenRC, or SysV — auto-detected)

The daemon auto-loads /etc/apm/apm.conf at startup — no separate boot step required.

Run as a systemd service

The installer registers APM with your init system automatically. The service runs the daemon at boot, supervised by systemd.

# Check status
$ systemctl status apm

# View logs
$ journalctl -u apm -f

Uninstall

# Remove binary and config
$ sudo apm uninstall

# Remove everything including logs, group, service
$ sudo apm uninstall --purge

Command reference

All commands communicate with the running daemon. If no daemon is running, the CLI starts one — through apm.service where a systemd unit exists (it clears a stale failed state first and never spawns a daemon outside the unit), or as a background process on installs without one.

apm [command] [options]

The CLI does not require sudo. The daemon listens on an abstract Unix socket (@apm), which has no filesystem permissions — any local user can run apm list, apm reload, apm restart, apm info, etc. sudo is only needed for the installer (install.sh), apm upgrade and apm uninstall (which write to /usr/sbin, /etc/apm, and the init system), and for reading worker log files directly when those files are owned by root. The apm OS group exists solely to grant read access to /var/log/apm.log — it is not required to use the CLI.

Process commands

Inspection commands

Config commands

GUI & Monitor commands

Daemon commands

Run flags

Flags for apm run and apm update. Every config-file field (see Worker options) has a flag of the same name: a value flag takes a single dash (-name api, -instances 4, -server http://0.0.0.0:3000), a boolean flag takes two (--restart, --rolling, --trust_proxy) and is switched off with its --no_ form (--no_restart, --no_trust_proxy, --no_micro_dedupe_log). Flags are matched exactly: since v2.5.0 an unrecognised flag is refused with a suggestion and nothing is created or changed, so --instances 8 is an error, not a synonym. Arguments meant for your program go after -args (-args --inspect) so they are never mistaken for APM flags. Sub-block settings are flags too: -vanguard_rate_limit, -vanguard_ban_path, -vanguard_allow_ip …, -statsd_host / -statsd_prefix / -statsd_interval, and -crash_hook_url / -crash_hook_method / -crash_hook_debounce / -crash_hook_log_lines / -crash_hook_log_source / -crash_hook_secret. apm run --help and apm update --help print the full lists, generated from the parsers themselves.

apm update additionally accepts --no-restart, -exec; flags without a tick above are refused by apm update — put them in the conf file and apm reload.

# Start a worker from the CLI — no config file needed
$ apm run node server.js \
    -name        myapp \
    -instances   4 \
    -server      http://0.0.0.0:3000 \
    -watch       "*.js" \
    --restart                # restart on clean exit
    --rolling                # rolling restart mode

# Update a running worker's instance count without restarting
$ apm update myapp -instances 8 --no-restart

# Save it back to a conf file
$ apm saveconf myapp /etc/apm/apm.conf.d/myapp.conf

Signals

Any signal name can be used as a command to forward that signal to a worker's child processes — all instances, or a single one by index:

$ apm SIGHUP myworker        # send to all instances
$ apm SIGUSR1 myworker#2     # send to instance index 2 only

Config file

Config files define workers and daemon settings. They're loaded with apm load or apm reload. The system config path is /etc/apm/apm.conf.

Syntax

• Key-value pairs end with ;
• Blocks use { }
• Comments: # or // to end of line
• Strings: unquoted, or single/double/backtick quoted (quotes are stripped)
• Multiple values: comma-separated on one line, or repeat the key
• The : suffix on keys is optional
• include <glob>; inlines another file at parse position

# Simple worker
worker {
    name       myapp;
    exec       node;
    params     server.js;
    instances  4;
    restart    true;
    watch      *.js;
    server     http://0.0.0.0:3000;
}

Config hierarchy

APM's startup scripts call apm boot after the daemon starts, which loads /etc/apm/apm.conf. The main config is typically structured as:

1. /etc/apm/apm.conf — main config (daemon block + includes)
2. /etc/apm/apm.conf.d/*.conf — drop-in worker configs, sorted by filename

You can also load configs manually at any time with apm load <file> or do a live diff with apm reload <file>.

daemon {
    gui_port  6789;
}

# Load drop-in worker configs
include apm.conf.d/*.conf;

worker {
    name  portal;
    exec  node;
    params app.js;
}

Multiple values

# Comma-separated on one line
server  http://0.0.0.0:3000, ws://0.0.0.0:3001;

# Or repeat the key
ban_path  *.php;
ban_path  *wp-*;
ban_path  *.env;

Worker options

All options are available both as fields in a worker { } config block and as flags to apm run / apm update (see Run flags for the flag spelling rules).

Identity

Environment

Restart on clean exit

Restart on error exit

Shutdown

Startup dependencies

Health check

APM can probe a worker's health two ways. Pull mode — set health_check to a URL and APM sends periodic HTTP GETs (2xx/3xx = healthy). Push mode — set health_check to on and APM injects APM_HEALTH_URL into the child, which calls it to report in. Health status shows in apm list and the GUI.

Connection drain

Memory limit

Logging

Proxy / HTTP

TLS

File watcher

Rolling restart

Stats

Inter-worker IPC

Crash webhook (on_crash)

APM can POST a JSON payload (or send a GET request) to a URL of your choice whenever a child process crashes — i.e. exits with a non-zero code or is killed by a signal. Intentional stops (apm stop) are never reported.

worker {
    name  myapp;
    cmd   node server.js;

    on_crash {
        url         https://hooks.example.com/apm-crash;
        method      POST;           # POST (default) or GET
        debounce    10000;          # min ms between calls (floor: 5000)
        log_lines   20;             # tail lines to include in payload
        log_source  err;            # "err" (default) or "out"
        secret      mysecret;       # signs payload with HMAC-SHA256
    }
}

Request headers

POST payload

{
  "worker":         "myapp",
  "instance":       1,
  "exit_code":      1,
  "exit_signal":    "SIGKILL",   // omitted if process exited normally
  "runtime_ms":     4821,
  "error_restarts": 3,
  "timestamp":      "2025-06-01T12:00:00Z",
  "log":            "Error: cannot connect to DB\n..."  // omitted if log_lines = 0
}

Daemon config

Global APM settings live in a top-level daemon { } block. No daemon block = all defaults. Zero config still works.

daemon {
    gui_port      6789;        # web GUI port — 0 disables the GUI
    gui_bind      127.0.0.1;   # bind address — default is loopback
    gui_password  secret;      # GUI login password
    auto_reload   true;        # reload config files when they change on disk
    apm_network_token apmnet_xxxx;  # APM Network license token (pre-release)
    start_concurrency 8;         # workers started in parallel on load/boot (default: cores × 2)
}

# top-level keys — not inside daemon { }
telemetry       false;  # opt out of the anonymous usage ping
remote_logging  true;   # opt in: stream worker/child lifecycle events to the stats server

Top-level keys

Two keys sit at the top level of the config file, outside daemon { } and any worker { }:

APM Network pre-release

APM Network extends inter-worker IPC across machines. Daemons on different servers join one private mesh; a worker on server A can send / request / requestStream a worker on server B using the same IPC verbs — location is transparent. Each daemon holds a single persistent WebSocket to the relay at wss://apm.processmanager.dev; only daemons presenting the same license token can see each other (the token defines a private network).

Two settings enable it: apm_network_token in the daemon { } block (the license token), and the apm_network worker option (the mesh identifier a worker is reachable as). Existing send / request / requestStream to that channel then resolve both local listeners and remote workers sharing the token — no connector changes. If the relay is unreachable, IPC degrades to local-only (a request to a remote-only channel returns null on timeout, exactly as today).

daemon { apm_network_token apmnet_xxxx; }

worker {
    name         pricing;
    exec         node;
    params       pricing.js;
    listen       pricing;        # local IPC channel
    apm_network  pricing;        # reachable across the mesh as "pricing"
}

On any other server using the same token, a worker's unchanged connector code reaches it:

// resolves the remote "pricing" worker on server A over the mesh
const quote = await apm.request('pricing', { sku: 'A-17' });

Getting a token (pre-release)

A token is bound to your server's public IP and a maximum number of concurrent connections from that IP. The relay rejects connections with an unknown or inactive token, a disallowed IP, or once the per-IP connection cap is reached.

There is no self-service portal yet — during pre-release, tokens are issued manually:

1. Email [email protected] with your server's public IP and the number of daemons you expect to connect.
2. You'll receive a token of the form apmnet_….
3. Put it in the daemon { } block as apm_network_token apmnet_…; and reload.

Server types

APM's built-in proxy accepts connections and forwards them to worker instances via IPC. Specify servers with the server field. Multiple servers per worker are supported.

There is no https:// or wss:// scheme: set tls true on the worker and every listener it has becomes the TLS variant (see TLS).

worker {
    name    api;
    exec    node;
    params  api.js;

    # HTTP and WebSocket on separate ports
    server  http://0.0.0.0:3000;
    server  ws://0.0.0.0:3001;

    # Or combined on one line
    server  http://0.0.0.0:3000, ws://0.0.0.0:3001;
}

Client IP resolution

When behind a CDN or reverse proxy (e.g. nginx), enable trust_proxy so APM resolves the real client IP from X-Forwarded-For headers. This affects Vanguard rate limiting and ban decisions.

trust_proxy  true;

Micro-dedupe new in v2.7

Somewhere between a phone and your origin — Safari racing HTTP/3 against HTTP/2, iCloud Private Relay re-issuing on a stalled tunnel, a CDN edge retrying in parallel — a single navigation can arrive as two byte-identical requests a millisecond apart. A balancer that treats them as two requests hands one to each instance, and a page that creates state runs twice. APM treats a body-less GET/HEAD that matches one it forwarded within the last micro_dedupe milliseconds (default 3) as what it is: a bounce.

The bounce is decided on arrival but answered late. It is never forwarded to a child. It waits until the first copy's response is known and is answered with a copy of it — because somebody is waiting on it, and nobody knows which copy the racing hop is listening to. Dropping it would become a 502 within a millisecond, which beats the genuine 200 still hundreds of milliseconds away. The app sees one request; the client sees two identical answers. If the first copy fails with an APM-authored error, the bounce gets the same error; if it produces no response at all, the bounce is released and handled as a request of its own.

The signature is method, listener, path with query, resolved client IP and every request header except per-hop ones (cf-ray, x-request-id, traceparent, x-b3-*, x-forwarded-for, via, the worker's own trace_header, …) that legitimately differ between two copies of one request. Anything else differing — a cookie, a query value, the client IP — is a different request. POST, bodies and WebSocket upgrades are never touched.

worker {
    server            http://unix:/run/web.sock;
    micro_dedupe      3;        # ms; default 3, 0 = off
    micro_dedupe_log  true;     # one rate-limited line per hit
    micro_dedupe_max  4096;     # tracked signatures per worker
}

apm info <worker> shows the window, duplicates answered and near misses — the same signature arriving just outside the window, up to 10× it — so the default can be tuned from data. The log line carries the delta and both copies' cf-ray: the same ray means the edge retried, different rays mean the duplicate was born upstream of it. A change reloads live; nothing restarts. It catches copies that arrive inside the window — a retry seconds later is a request of its own — so a page that creates money-bearing state still has to be idempotent. This is defence in depth and, above all, visibility: every APM-hosted app now counts these.

WebSocket liveness & dead clients new in v2.12

A client that goes away cleanly — a tab closed, a process exited — is reported to the worker in milliseconds on every path. A client that vanishes — power cut, cable pulled, Wi-Fi gone, firmware wedged — sends nothing, and nothing anywhere can know it is gone until a probe goes unanswered. Detection is always probe interval + grace; the choice is only who probes, and what the probe can see through.

After ping_interval of silence APM sends a ping and waits ping_timeout for the pong; ping_miss unanswered pings in a row and the client is closed with code 1001 (ping timeout) and the worker receives connectionClosed, exactly as for any other dirty disconnect — the application needs no heartbeat logic of its own. Any frame from the client is proof of life (data, the pong, its own ping), so a connection that is talking is never pinged. A vanished client is reported ping_interval + ping_miss × ping_timeout after its last frame. Unset, the timing is what every earlier version had: a ping after keep_alive/2 of silence, closed at keep_alive.

worker {
    server         ws://unix:/run/fleet.sock;   # behind nginx or a CDN: only the ping can reach the device
    ping_interval  10000;   # ping after 10 s of silence …
    ping_timeout   2000;    # … wait 2 s for the pong, and between retries …
    ping_miss      2;       # … two unanswered and it is gone: 14 s, one probe per 10 s
}

Behind a reverse proxy or a CDN only the WebSocket ping reaches the device, so ping_interval is the one to set there; dead_timeout adds a kernel-level floor on a listener APM terminates itself. Cloudflare in particular never tells the origin about a dead client on its own (idle or with data in flight the worker learned nothing for two minutes), closes a WebSocket that carries no frames at all after about 125 s, and counts ping/pong as traffic. For a fleet of 10 000 fully idle devices ping_interval 10000 is about 1.7 Mbit/s at the server and 54 MB a month per device — cellular data plans, not the server, are what the interval is chosen for.

Per connection. The application can replace the policy for one session with the connector's session.setPing({ interval, timeout, miss }) (Python/Perl/Lua set_ping), so a fleet can give some devices faster detection than others; a field left out returns to the worker's setting and setPing() resets all three.

Latency, for free. Every ping carries its send time and the pong echoes it, so each cycle also measures the client's round trip. APM keeps the last sample and sixty one-minute buckets per connection; session.pingStats() returns last/lastAt/count and avg/min/max/n over the last 5 and 60 minutes together with the policy in force, and the worker's ws_rtt timing metric shows the fleet in the GUI and on StatsD.

Vanguard

Vanguard is APM's built-in request firewall. It runs before worker IPC — rejected connections never reach your app. Configure it with a vanguard { } sub-block inside a worker.

worker {
    name    api;
    exec    node;
    params  api.js;
    server  http://0.0.0.0:3000;

    vanguard {
        rate_limit    100;          # requests/sec per IP
        rate_burst    200;          # burst capacity
        ban_ttl       300000;       # auto-ban for 5 minutes
        ban_path      *.php, *wp-*, *.env, /.git*;
        ban_response  Forbidden;
    }
}

IP filtering

Path banning

Method filtering

vanguard {
    allow_method  GET;
    allow_method  POST;       # everything else → 405

    # or, blocklist style:
    ban_method    TRACE;
    ban_method    OPTIONS;
}

ban_path  *.php;          # ends-with  — block all .php requests
ban_path  *wp-*;          # contains   — block WordPress probes
ban_path  /.git*;         # starts-with — block .git exposure
ban_path  *.env;          # ends-with  — block .env file reads
ban_path  /admin/login;   # exact match — block a specific path

Rate limiting

Rate-limited requests receive 429 Too Many Requests. Path/IP bans return 403 Forbidden (or silent TCP RST).

Logging

vanguard {
    rate_limit   200;
    ban_path     *.php, *.env;

    log          off;       # silence per-event lines under sustained probing
    log_summary  60;        # one aggregate line per minute instead
}

CDN IP lists

APM's installer fetches Cloudflare's published egress IP ranges and writes them to /etc/apm/ips/ as ready-to-include partial configs. Include them inside a vanguard { } block to restrict direct access to CDN traffic only.

vanguard {
    # Only allow Cloudflare egress IPs (IPv4 + IPv6)
    include /etc/apm/ips/cloudflare-v4.part;
    include /etc/apm/ips/cloudflare-v6.part;

    rate_limit  500;
    ban_path    *.php, *wp-*, *.env, /.git*;
}

The IP lists are fetched by the installer and refreshed on every apm upgrade. To refresh them manually, re-run the installer:

$ curl -fsSL https://processmanager.dev/install.sh | sudo bash  # re-runs the installer, including the IP fetch

TLS

APM has first-class TLS support for all server types — HTTP, WebSocket, and TCP. Bring your own certificates, or let APM generate a self-signed pair when tls_cert / tls_key are omitted. TLS 1.2 is the minimum version. There is no https:// scheme: tls true turns every listener of the worker into its TLS variant.

worker {
    name      api;
    exec      node;
    params    api.js;
    server    http://0.0.0.0:443;   # becomes HTTPS with tls true

    tls       true;
    tls_cert  /etc/ssl/certs/myapp.crt;
    tls_key   /etc/ssl/private/myapp.key;
    # tls_ca for mutual TLS (client cert verification)
    tls_ca    /etc/ssl/certs/ca.crt;
}

File watcher

The file watcher monitors your source directory and triggers a worker restart when matching files change. Uses kernel file-watch events (inotify) — no polling.

worker {
    name          api;
    exec          node;
    params        server.js;
    path          /home/user/myapp;

    watch         *.js, *.json;        # watch .js and .json files
    watch_ignore  *node_modules*;      # ignore anything inside node_modules
    watch_delay   200;                  # 200ms debounce
}

Pattern syntax

Watch patterns use a simple glob-style syntax — no regex needed — and are tested against names, not paths. Four matching modes:

# Go source files, excluding generated code and vendor
watch         *.go;
watch_ignore  *_generated.go, *vendor*;

# JS/TS project — ignore build output and deps (directory names, not paths)
watch         *.js, *.ts, *.json;
watch_ignore  *node_modules*, dist, build;

# Python — any .py file anywhere under path
watch         *.py;
watch_ignore  *__pycache__*;

Directories moved into the tree are picked up too, so an atomic deploy (mv built/ live/) restarts the worker like an in-place edit does, and a directory moved aside leaves no stale watches behind.

Rolling restart

Rolling restarts cycle through instances one at a time, keeping the rest running to serve traffic. Zero downtime for multi-instance workers.

worker {
    instances     4;
    rolling       true;
    rolling_delay 1000;  # 1s between each instance restart
}

With session_persist true, open connections are migrated to a new instance before the old one is killed. Use session_wait to control how long APM waits for the new instance to become ready.

rolling          true;
rolling_delay    500;
session_persist  true;
session_wait     2000;  # wait up to 2s for new instance

Logger

APM has a built-in logger for each worker. Every line written to a child process's stdout or stderr is intercepted, prefixed with a timestamp and worker name, and written to the configured destination. Coloring is applied by APM before writing — use strip_ansi to remove it when logging to files.

Destinations

Prefix

Each log line is prefixed with the worker name (or a custom string). The prefix field supports the color syntax described below. APM automatically appends the instance number in multi-instance workers:

worker {
    name    api;
    exec    node;
    params  server.js;

    # cyan name, reset after — instance # is appended automatically
    prefix  ç51-api-serverçR-;

    log     /var/log/myapp/out.log;
    err_log /var/log/myapp/err.log;
}

For a worker with instances 3, the stdout prefix becomes api-server#1, api-server#2, api-server#3 — each in a distinct color so instances are visually distinct in the live GUI and in log files.

Timestamp format

The timestamp prepended to each line is controlled by log_time_format. The format string uses strftime-style tokens and supports color escapes. The default is ç214-%Y-%m-%d %Tç59-.%FçR- (orange date, dim fractional seconds).

Strftime tokens

# Default — orange date, dim microseconds
log_time_format  ç214-%Y-%m-%d %Tç59-.%FçR-;

# Compact — just HH:MM:SS in gray
log_time_format  ç59-%TçR-;

# No color — plain ISO timestamp
log_time_format  %Y-%m-%d %T;

Strip ANSI

Color syntax — çN-

APM uses a compact color escape based on the 256-color terminal palette. The ç character (U+00E7) acts as the escape marker. This syntax works in prefix, log_time_format, and anywhere APM renders text to the terminal or log files.

# Foreground only
prefix  ç82-myappçR-;      # bright green name
prefix  ç196-myappçR-;     # bright red name
prefix  ç214-myappçR-;     # orange name

# Foreground + background
prefix  ç15,88-ERRORçR-;    # white text on dark red background

# Bold foreground
prefix  ç51,0,1-myappçR-;   # bold cyan

Useful color reference

StatsD

APM can forward worker metrics to any StatsD-compatible endpoint (StatsD, Graphite, Datadog Agent, Telegraf) over UDP. Add a statsd { } block to a worker.

worker {
    name    api;
    exec    node;
    params  server.js;

    statsd {
        host      localhost:8125;   # StatsD UDP endpoint
        prefix    apm.api;          # metric namespace
        interval  1;                # flush interval, seconds
    }
}

System metrics

Forwarded automatically every interval, as gauges:

Custom metrics

Metrics emitted from worker code with apm.metric(name, value, type) are aggregated across all of the worker's instances and forwarded with the same prefix — counters summed (|c), timings averaged (|ms), gauges last-value-wins (|g).

One built-in: a worker with WebSocket clients emits a ws_rtt timing per answered liveness ping — the client's round trip in milliseconds, measured from the stamp the ping carries and the pong echoes (see WebSocket liveness). It is forwarded and shown in the GUI like any custom metric.

Web GUI

APM ships a built-in real-time web dashboard. Enable it with gui_port in the daemon config:

daemon {
    gui_port  6789;         # GUI port — omit or set 0 to disable
    gui_bind  127.0.0.1;   # default is 0.0.0.0 — set 127.0.0.1 to keep it local
}

When the daemon starts with the GUI enabled it prints the access URL:

$ apm start
GUI: http://127.0.0.1:6789/

Views

Server Info — latency

The Server Info page has a Latency section with two cards:

Disconnect behaviour

When the daemon stops, the WebSocket closes and the GUI immediately dims with a Session Ended overlay. Click Refresh Page to reconnect.

Dashboard

Each worker can expose a custom metric dashboard in the GUI. Define a dashboard { } block inside a worker config to create one. The dashboard is shown in the Dashboard tab when that worker is selected.

worker my-api {
    exec    node;
    params  server.js;
    server  http://127.0.0.1:3000;

    dashboard {
        name  My API;
        cols  6;
        rows  4;

        module {
            type  graph;
            id    1;
            name  Requests/sec;
            x 0; y 0; w 3; h 1;
        }
        module {
            type  gauge;
            id    2;
            name  CPU %;
            x 3; y 0; w 1; h 1;
            min 0; max 100; unit %;
        }
        module {
            type  counter;
            id    3;
            name  Total errors;
            x 4; y 0; w 1; h 1;
        }
        module {
            type  text;
            id    4;
            name  Last error;
            x 5; y 0; w 1; h 1;
        }
    }
}

Dashboard block fields

Module fields

Module types

Sending metrics from code

Use apm.setDashValue(id, value, color?) in the Node.js connector to push a value to a dashboard module. This is distinct from apm.metric(), which is for StatsD-style system metrics only.

const ApmModule = require('./apm_module.node.js')
const apm = new ApmModule(async (session) => { /* handle connections */ })

// Push a number to module id 1 (graph)
apm.setDashValue(1, requestsPerSecond)

// Push a number with a dynamic color
apm.setDashValue(2, cpuPercent, cpuPercent > 80 ? '#ff5a5a' : '#4f8cff')

// Push a string to a text module (id 4)
apm.setDashValue(4, lastErrorMessage)

// LED on/off (1 = on, 0 = off)
apm.setDashValue(5, isHealthy ? 1 : 0, isHealthy ? '#47d16c' : '#ff5a5a')

Counter vs gauge vs graph

Node.js connector

The Node.js connector ships in two equivalent forms — pick whichever matches your project:

• apm_module.node.js — CommonJS (require); Node 10+.
• apm_module.node.mjs — ES Modules (import); Node 14+, for "type":"module" packages.

Both files are line-for-line equivalent: same class, same API, same wire protocol. IPC happens over stdin/stdout using binary frames — no Unix sockets required in the child.

$ curl -fsSL https://processmanager.dev/connectors/apm_module.node.js -o apm_module.node.js

$ curl -fsSL https://processmanager.dev/connectors/apm_module.node.mjs -o apm_module.node.mjs

$ node apm_module.node.js -update
$ node apm_module.node.mjs -update

const ApmModule = require('./apm_module.node.js')
const apm = new ApmModule(async (session) => {
    // session.protocol  — 'http' | 'ws' | 'tcp' | 'udp'
    // session.method    — HTTP method
    // session.path      — full path + query
    // session.headers   — request headers
    // session.remoteIp  — real client IP (proxy-aware)
    // session.cookies   — parsed cookie map

    session.write('Hello World', {
        'content-type': 'text/plain',
        _status: 200
    })
    session.close()
})

If your worker doesn't handle sessions (e.g. a background job pushing dashboard metrics), pass an empty async function: new ApmModule(async () => {}).

Session API

Instance methods

Environment variables

APM injects the following into managed child processes:

WebSocket example

const ApmModule = require('./apm_module.node.js')
const apm = new ApmModule(async (session) => {
    if (session.protocol !== 'ws') {
        session.close(400)
        return
    }

    session.onData = (data, isBinary) => {
        // echo back
        session.write(data)
    }

    session.onClose = () => {
        console.log('disconnected', session.sessionId)
    }
})

PHP / Python / Perl / Lua connectors

Connectors for other languages follow the same pattern as Node.js: drop a single file into your project, require / include it, construct an instance with an onConnect callback, then run the event loop. All connectors implement the full APM IPC protocol over stdin/stdout — no extra dependencies beyond what's noted on the connectors page.

$ curl -fsSL https://processmanager.dev/connectors/apm_module.php   -o apm_module.php
$ curl -fsSL https://processmanager.dev/connectors/apm_module.py    -o apm_module.py
$ curl -fsSL https://processmanager.dev/connectors/ApmModule.pm     -o ApmModule.pm
$ curl -fsSL https://processmanager.dev/connectors/apm_module.lua   -o apm_module.lua

Each connector file also supports self-update — run it with -update to fetch the latest version from the server (e.g. php apm_module.php -update). See the connectors page for version info, MD5 checksums, and per-language update commands. All four expose the same setDashValue(id, value, color?) and metric(name, value, type?) methods as the Node.js connector — see the Dashboard section.

Each subsection below shows the same three worked patterns per language: a hello-world HTTP session, an IPC channel handler (the worker listens on a channel and replies to request()), and a stream initiator (the worker opens a stream to a peer and exchanges data). The wire protocol is identical across languages — anything you can do from Node.js works from these. For a worked end-to-end example with workers in two different languages talking to each other, see cross-language IPC walkthrough.

Python

Python 3.6+, stdlib only. The connector exposes ApmSession and IpcStream classes and uses snake_case method names. The session.write() argument can be bytes (sent as a binary frame) or str; write_binary() forces a binary frame. PHP has writeBinary(), Perl and Lua write_binary(), on sessions and on streams alike (connectors v3.1).

from apm_module import ApmModule

def on_connect(s):
    s.write(b'Hello from Python', {
        'content-type': 'text/plain',
        '_status':    200,
    })
    s.close()

apm = ApmModule(on_connect)
apm.run()

Pair with a worker block like worker { exec python3; params hello.py; path /opt/myapp; server :8080; }.

from apm_module import ApmModule

apm = ApmModule(lambda s: s.close())  # no HTTP sessions here

def on_channel(channel, data, reply):
    # `reply` is None for fire-and-forget send(); a callable for request()
    if reply is None:
        print(f'broadcast on {channel}: {data}')
        return
    reply({'pong': data.get('ping'), 'from': 'python'})

apm.on_channel = on_channel
apm.run()

Configure the listener in the worker block: listen "ping_service";. A peer (any language) can then call request("ping_service", {ping: 42}) and receive {pong: 42, from: 'python'}.

from apm_module import ApmModule

apm = ApmModule(lambda s: s.close())

def main():
    stream = apm.request_stream('iot', {'device': 'sensor-7'}, 5000)
    if stream is None:
        print('no peer accepted on \'iot\'')
        return
    stream.on_data  = lambda data, peer=None: print('in: ', data)
    stream.on_close = lambda: print('closed')
    stream.write(b'ready')

# Kick off the request after the connector loop is running
import threading; threading.Timer(0.1, main).start()
apm.run()

PHP

PHP 7.4+, no extra extensions. Methods use camelCase following PSR convention. Handlers are assigned to onChannel / onStream properties on the instance.

<?php
require_once __DIR__ . '/apm_module.php';

$apm = new ApmModule(function (ApmSession $s) {
    $s->write('Hello from PHP', [
        'content-type' => 'text/plain',
        '_status'    => 200,
    ]);
    $s->close();
});
$apm->run();

<?php
require_once __DIR__ . '/apm_module.php';

$apm = new ApmModule(function ($s) { $s->close(); });

$apm->onChannel = function ($channel, $data, $reply) {
    // $reply is null for fire-and-forget send(); callable for request()
    if ($reply === null) {
        error_log("broadcast on $channel");
        return;
    }
    $reply(['pong' => $data['ping'] ?? null, 'from' => 'php']);
};

$apm->run();

<?php
require_once __DIR__ . '/apm_module.php';

$apm = new ApmModule(function ($s) { $s->close(); });

// requestStream blocks until accepted/rejected/timeout; call after run() in
// a coroutine, or from a request handler. Returns IpcStream or null.
$apm->onChannel = function ($ch, $data, $reply) use ($apm) {
    if ($ch !== 'kick' || $reply === null) return;
    $stream = $apm->requestStream('iot', ['device' => 'sensor-7'], 5000);
    if ($stream === null) { $reply(['ok' => false]); return; }
    $stream->onData  = function ($data) { error_log("in: $data"); };
    $stream->onClose = function () { error_log('closed'); };
    $stream->write('ready');
    $reply(['ok' => true, 'stream' => $stream->id]);
};

$apm->run();

Perl

Perl 5.10+ with the JSON module (cpan install JSON or apt install libjson-perl). Object-oriented; handlers assigned via hash-element syntax ($apm->{on_channel} = sub { ... }).

use strict;
use warnings;
use ApmModule;

my $apm = ApmModule->new(sub {
    my ($s) = @_;
    $s->write('Hello from Perl', {
        'content-type' => 'text/plain',
        '_status'    => 200,
    });
    $s->close;
});
$apm->run;

use strict; use warnings;
use ApmModule;

my $apm = ApmModule->new(sub { $_[0]->close });

$apm->{on_channel} = sub {
    my ($channel, $data, $reply) = @_;
    if (!defined $reply) {
        warn "broadcast on $channel\n";
        return;
    }
    $reply->({ pong => $data->{ping}, from => 'perl' });
};

$apm->run;

use strict; use warnings;
use ApmModule;

my $apm = ApmModule->new(sub { $_[0]->close });

$apm->{on_channel} = sub {
    my ($ch, $data, $reply) = @_;
    return unless $ch eq 'kick' && $reply;
    my $stream = $apm->request_stream('iot', { device => 'sensor-7' }, 5000);
    if (!$stream) { $reply->({ ok => 0 }); return; }
    $stream->{on_data}  = sub { warn "in: $_[0]\n" };
    $stream->{on_close} = sub { warn "closed\n" };
    $stream->write('ready');
    $reply->({ ok => 1, stream => $stream->{id} });
};

$apm->run;

Lua

Lua 5.3+ with lua-cjson (apt install lua-cjson or luarocks install lua-cjson). Method calls use : syntax (apm:run()); callbacks use . assignment (apm.on_channel = …).

local ApmModule = require('apm_module')

local apm = ApmModule.new(function(s)
    s:write('Hello from Lua', {
        ['content-type'] = 'text/plain',
        ['_status']    = 200,
    })
    s:close()
end)
apm:run()

local ApmModule = require('apm_module')
local apm = ApmModule.new(function(s) s:close() end)

apm.on_channel = function(channel, data, reply)
    -- reply is nil for fire-and-forget send(); a function for request()
    if not reply then
        io.stderr:write('broadcast on ' .. channel .. '\n')
        return
    end
    reply({ pong = data.ping, from = 'lua' })
end

apm:run()

local ApmModule = require('apm_module')
local apm = ApmModule.new(function(s) s:close() end)

apm.on_channel = function(ch, data, reply)
    if ch ~= 'kick' or not reply then return end
    local stream = apm:request_stream('iot', { device = 'sensor-7' }, 5000)
    if not stream then reply({ ok = false }); return end
    stream.on_data  = function(d) io.stderr:write('in: ' .. d .. '\n') end
    stream.on_close = function()  io.stderr:write('closed\n') end
    stream:write('ready')
    reply({ ok = true, stream = stream.id })
end

apm:run()

Inter-worker IPC new in v2.0

Workers can now talk to each other through APM — across instances, across workers, and across languages. Two primitives: channels for stateless message passing, and streams for persistent bidirectional pipes. The daemon is the router; no extra sockets, no broker, no ports.

How it works

A worker declares a channel name with the listen config directive. The daemon maintains a registry mapping channel names → running workers. When any worker calls send(), request(), or requestStream(), the daemon routes the message to every running child of every worker listening on that channel. Workers never talk directly — all traffic flows over the existing stdin/stdout protocol that connectors already use, so no new dependencies and no new surface area.

            ┌────────────┐
            │ APM daemon │  ← channel registry & router
            └──┬──┬───┬───┘
               │  │   │
     ┌─────────┘  │   └─────────┐
     ▼            ▼             ▼
 ┌─────────┐ ┌─────────┐ ┌─────────┐
 │ sender  │ │ listener│ │ listener│
 └─────────┘ └─────────┘ └─────────┘

Why this matters

• Cross-language. A Python worker can request data from a Node.js worker and get a JSON reply. A PHP request handler can open a stream to a Go background worker. The connector API is identical across Node.js, Python, PHP, Perl, and Lua.
• Zero config. Add listen "channel_name"; to a worker block. That's it. The daemon rebuilds the registry automatically on every worker start/stop.
• Backwards compatible. Workers without listen behave exactly as before. The protocol is additive — old connectors are unaffected.
• Crash-safe. When any child dies, the daemon cleans up its streams and cancels its pending requests automatically. Writes to closed streams are silently dropped (no-scream policy).

Channels — fire-and-forget & request/reply

Channels are lightweight stateless messaging. A sender emits a message to a named channel; the daemon broadcasts it to every child of every worker listening on that channel.

send() — fire-and-forget

Broadcasts a message with no response expected. Returns immediately. If nobody is listening, the message is silently dropped.

// Push a device status update to everyone listening on "devices"
apm.send('devices', { deviceId: 'sensor-5', online: true })

// Worker config must contain:   listen "devices";
apm.onChannel = (channel, data) => {
  console.log('got', channel, data)
}

The sender is never echoed to itself, even if it also listens on the same channel. This keeps broadcasts loop-free.

request() — request / first-reply-wins

Sends a message and waits for the first reply from any listening child. Late replies are silently dropped. Returns null on timeout or when no listener is running.

// Ask any iot worker for the status of device sensor-5
const status = await apm.request('iot', { query: 'status', device: 'sensor-5' }, 2000)
if (status === null) { /* nobody answered in time */ }

apm.onChannel = (channel, data, reply) => {
  if (data.query === 'status') {
    reply({ temp: 42, online: true })
  }
}

Timeout priority: per-call timeout > worker ipc_timeout > default 500ms.

Streams — persistent bidirectional pipes

Streams are long-lived connections between workers. They follow a mediated star topology: one worker opens the stream (the mediator), others can attach as peers.

• Mediator writes fan out to all attached peers.
• Peer writes go to the mediator only, tagged with the peer ID so the mediator knows who spoke.

Think of it as a group trip organiser: the organiser can speak to everyone, but each participant only talks back to the organiser. This makes streams a natural fit for fan-out notification, shared live consoles, device control sessions, and worker-to-worker RPC sessions where one side coordinates many.

Typical flow

1. Mediator calls requestStream('channel', header). All listeners see a stream request.
2. One or more listeners call stream.accept(). Others can stream.reject() (or just ignore — the timeout handles "nobody accepted").
3. As soon as the first peer accepts, the mediator's Promise resolves with the stream object.
4. Both sides call stream.write(data) and handle stream.onData. Either side can call stream.close().
5. When the last peer detaches, the stream auto-closes. When any child dies, its streams are cleaned up automatically.

Peer bookkeeping: a peer's stream.peerID is the id the mediator sees its writes tagged with; the mediator gets stream.onAttach = ({ id, header }) => {} and stream.onDetach = ({ id }) => {} as peers come and go (on_attach / on_detach in Python, Perl and Lua). Binary payloads: pass a Buffer / bytes to write(), or use writeBinary() / write_binary().

// Open a live console to whichever iot worker owns device sensor-5
const stream = await apm.requestStream('iot_console', { device: 'sensor-5' }, 5000)
if (!stream) { /* no peer accepted within timeout */ return }

stream.onData = (chunk, peer) => { console.log('from', peer, chunk.toString()) }
stream.onClose = () => { console.log('console closed') }
stream.write('help\n')

apm.onStream = (stream) => {
  // header contains whatever the mediator sent
  if (stream.header.device !== myDeviceId) { stream.reject(); return }

  stream.accept({ name: 'sensor-5' })
  stream.onData = (chunk) => { runCommand(chunk.toString()) }
  stream.onClose = () => { /* cleanup */ }
}

No-scream policy

Writes to a closed stream are silently dropped — no error, no crash. The onClose callback still fires normally. This prevents race conditions when both sides close simultaneously, which is very common in real traffic.

Config & connector API

Worker config fields

# IoT WebSocket server — listens for control commands
worker {
    name          iot_ws;
    exec          node;
    params        iot_server.js;
    server        ws://0.0.0.0:9100;
    instances     4;
    listen        "iot_control";
    ipc_timeout   1000;
}

# Web control panel — sends commands to IoT workers
worker {
    name          web_panel;
    exec          node;
    params        panel.js;
    server        http://0.0.0.0:8080;
    instances     2;
}

Cross-language API

Every connector exposes the same primitives. Method names are adapted to each language's conventions.

Node.js request and requestStream return Promises. All other languages block internally (pumping their event loop where applicable) until a reply arrives or the timeout fires, so they can be used from straight-line code without async plumbing.

Cross-language IPC walkthrough

This section is a worked end-to-end example with workers in two different languages talking through APM. Same wire protocol on both sides, no extra plumbing — the daemon is the router.

Scenario: Node.js HTTP front-end calls a Python pricing service

An HTTP request hits the Node.js worker; the handler asks a separately-running Python worker for the current price of a product via an IPC request; the Node.js worker returns the answer to the client. Each side runs as its own APM worker (so each can scale, restart, and rolling-deploy independently).

worker {
    name     web;
    exec     node;
    params   front.js;
    path     /opt/shop;
    server   :8080;
    instances 4;
    # no `listen` here — this worker is a *caller*, not a service
}

worker {
    name     pricing;
    exec     python3;
    params   pricing.py;
    path     /opt/shop;
    listen   pricing;        # become the receiver for `pricing` channel
    ipc_timeout 500;          # default reply window in ms
    instances 2;
}

The listen "pricing" directive on the Python worker registers its instances as receivers for the pricing channel. When the Node.js side calls request('pricing', …), APM picks one running Python instance and routes the message; the first reply wins.

from apm_module import ApmModule

PRICES = {'sku-1': 9.99, 'sku-2': 14.50, 'sku-3': 99.00}

apm = ApmModule(lambda s: s.close())  # no HTTP, IPC only

def on_channel(channel, data, reply):
    # channel == 'pricing'; data is whatever the caller sent
    sku = data.get('sku')
    price = PRICES.get(sku)
    if reply is not None:
        reply({'sku': sku, 'price': price, 'currency': 'GBP'})

apm.on_channel = on_channel
apm.run()

const ApmModule = require('./apm_module.node.js')

const apm = new ApmModule(async (session) => {
    const sku = session.query_object.sku || 'sku-1'

    // Cross-language IPC: send to the Python `pricing` service and await reply.
    const reply = await apm.request('pricing', { sku }, 500)

    if (!reply || reply.price == null) {
        session.write(JSON.stringify({ error: 'unknown sku' }), {
            _status: 404, 'content-type': 'application/json',
        })
    } else {
        session.write(JSON.stringify(reply), {
            'content-type': 'application/json',
        })
    }
    session.close()
})

$ apm load /etc/apm/apm.conf.d/shop.conf
$ curl 'http://localhost:8080/?sku=sku-2'
{"sku":"sku-2","price":14.5,"currency":"GBP"}

What happened: the Node.js instance handling the request issued a request('pricing', …, 500). APM looked up registered listeners for the pricing channel, picked one Python instance (round-robin across the 2 instances), and delivered the message. The Python instance's on_channel fired with a non-null reply, which it invoked with a dict; APM serialised it back to the Node.js caller as the resolved value of the await. Total round-trip is one stdin/stdout frame per direction — no sockets opened, no JSON-over-HTTP, no broker.

Scaling notes

• Pick-one routing for request: if you scale pricing to N instances, APM picks one per request. The first reply wins. Use this for stateless services where any instance can answer.
• Fan-out for send: apm.send('pricing', …) broadcasts to all listener instances and ignores replies. Use this for cache invalidation or background notifications.
• Streams for stateful pipes: if you need a long-lived connection between two specific workers (e.g. a Lua mediator coordinating between two PHP peers, or a Node.js dashboard collector pulling continuous data from a Python sampler), use request_stream / onStream instead — see the Streams section.
• Mix and match: the same Python service can be called from a Node.js front-end, a Perl batch job, and a PHP cron simultaneously — APM doesn't care what language the caller is. The wire format is identical.

The four other connectors (PHP / Perl / Lua / Node.js) can be substituted for either side of the example above with the per-language snippets from the connectors section — only the syntax changes, not the wiring.

Augur — startup-failure diagnostician new in v2.1

When a worker crashes inside the startup_grace window (default 2 s) on its first attempt, APM's augur module takes over. Augur does three things, in order:

1. Classifies the stderr by language. It recognises Node.js, Python, Perl, PHP, Ruby, Java, Go, and Bash failure patterns (MODULE_NOT_FOUND, ModuleNotFoundError, Can't locate ... in @INC, Class 'X' not found, LoadError, ClassNotFoundException, etc.) and emits a single-line hint telling the user what to do next.
2. Surfaces universal errno conditions: EADDRINUSE / "Address already in use", ECONNREFUSED / "Connection refused", EACCES / "Permission denied" — matched as both symbolic and English forms so they fire across every runtime.
3. Deep-scans the worker's dependency manifest and source to list every missing dependency in one go — so a freshly-migrated worker that's missing five npm packages tells you all five at once, instead of crash → install → restart → next crash → install → restart …

Languages covered

Reactive mode (default)

Augur runs automatically when a child exits with non-zero or a signal within the startup-grace window on its first attempt. No configuration required — the captured stderr is dumped to the requesting CLI, classifier hints follow, then the deep-scan adds the augur — also missing (N): … summary if a manifest scan found additional gaps.

Pre-flight mode (opt-in)

Set augur_full_scan true on a worker and augur runs the manifest + source scan before every fork — including watcher- and auto-restart-triggered ones. If anything is missing the launch is aborted with a clear message; the child binary never starts. This costs one or two cheap probe sub-processes per restart but eliminates the "child crashes, child restarts, crashes again" loop on broken environments.

$ apm restart Portal4
● Starting Portal4
 - Starting child #1
✗ Portal4#1 failed during startup (exit 1, 515 ms)
── child stderr ──────────────────────────────────────────────
Error: Cannot find module 'redis'
Require stack:
- /WEBZ/portal/v4/server.node.js
    ...
    code: 'MODULE_NOT_FOUND',
──────────────────────────────────────────────────────────────
  → missing node module redis — run npm install
  → augur — also missing (5): ws axios dotenv mysql2 sharp   → cd /WEBZ/portal/v4 && npm install ws axios dotenv mysql2 sharp
✗ Worker Portal4 failed to start
  → Log: /WEBZ/portal/v4/Portal4.log (apm log Portal4)

The deep-scan probes run as the worker's target user (same setuid + login-PATH mechanism as the worker launch itself), in the worker's path directory, with a 3-second timeout each. They never touch global system state and never write anything.

What's new

Release notes, newest first. The same text ships as the release's commit message and on the downloads page.

What's new in v2.12.0 — a vanished device is reported in seconds, through any proxy, per device

A WebSocket client that vanishes — power cut, cable pulled, firmware wedged — sends nothing. dead_timeout (v2.6) catches that with kernel probes, but only on a listener APM terminates itself; behind nginx, a unix socket or Cloudflare the TCP peer is the proxy, which is alive, so the worker heard nothing until the keep_alive deadline, 120 s, or until the application's own heartbeat gave up. That chain is exactly what a device fleet behind a CDN runs.

ping_interval / ping_timeout / ping_miss — the fleet's dead-device detector

worker {
          server         ws://unix:./fleet.sock;
          ping_interval  10000;  # ping after 10 s of silence …
          ping_timeout   2000;   # … wait 2 s for the pong (and between retries) …
          ping_miss      2;      # … two unanswered pings and the client is gone: 14 s
      }

After ping_interval of silence APM sends a ping and waits ping_timeout; ping_miss unanswered pings in a row and the client is closed with code 1001 (ping timeout) and the worker gets connectionClosed, exactly as for any dirty disconnect — no heartbeat code in the application. Any frame from the client is proof of life, so a talking connection is never pinged, and the per-connection ping goroutine is gone: the read deadline is the timer. Control frames are forwarded end to end by nginx and by Cloudflare, so the only requirement is a client that answers pings, which every browser and standard WebSocket library does. Unset, the timing is what every earlier version had (ping after keep_alive/2, closed at keep_alive), minus the pings a busy connection used to receive every 60 s regardless.

Measured on a real link cut: 7–9 s through Cloudflare → nginx → unix socket at ping_interval 5000, 3 s direct at 3000, 2 s with dead_timeout on top; on defaults still 103–119 s, and Cloudflare itself never reported a dead client. Also learned: Cloudflare closes a WebSocket with no frames at all after ~125 s and counts ping/pong as traffic.

Per connection, and latency for free — connectors v3.4.0

session.setPing({ interval, timeout, miss }) (set_ping in Python, Perl and Lua) replaces the policy for one session, so a fleet can give some devices faster detection than others. Every ping now carries its send time and the pong echoes it, so each cycle measures the client's round trip; session.pingStats() returns the last sample and avg/min/max/count over the last 5 and 60 minutes, kept per connection in the daemon, and a built-in ws_rtt timing metric per worker shows the fleet in the GUI and on StatsD. All six connectors move to v3.4.0; the conformance harness covers both calls.

Upgrade notes: nothing changes unless you set ping_interval. Device fleets behind a proxy or CDN: set ping_interval 10000; ping_timeout 2000; ping_miss 2; on the worker and drop the application-level heartbeat if it was only there for liveness. Details in the timestamped notes beside this file.

What's new in v2.11.0 — the front door, the GUI and the child pipe

Review batch 4, the last of the October review. The proxy learns which peers to believe about the client IP (trusted_proxies, real_ip_header) and reads forwarded lists from the right; ban_path can no longer be walked around with //, /./, a trailing slash, an encoded letter or a bad escape in the query; a flood of fresh IPs no longer refuses clients with a full rate table. The web GUI always asks for a password (a generated one is in the daemon log and apm gui when none is set), sessions expire, logins back off, and the two root config files merge per key instead of one replacing the other. apm stop no longer deadlocks on a child that IPCs during shutdown, a stray frame byte in log output no longer swallows the log, a stalled child answers 503 instead of holding every request. Local IPC streams are isolated per worker. The connectors move to v3.3.0: large bodies in milliseconds instead of seconds, ordered WebSocket handlers in Node, numeric _status everywhere. The config parser accepts a BOM, a bare key; and diamond includes; saveconf quotes what it writes and keeps dashboard { }; healconf flags a single-value key set twice.

Upgrade notes: set trusted_proxies on listeners behind a proxy; if you rely on the GUI without a password, read the generated one from the daemon log or sudo apm gui, or set gui_password. Details in the timestamped notes beside this file.

What's new in v2.10.0 — releases are signed

Review batch 3. Every file apm upgrade acts on — the release manifest, the installer, the connectors — now carries an ed25519 signature, and the public key is compiled into the binary: an upgrade refuses anything the maintainer did not sign, a downgrade is refused unless asked for, a specific version can be installed, and the last three releases stay published as rollback targets. apm verify checks a file by hand; the connectors' self-update uses it.

Nothing to configure. The upgrade to 2.10 is the last one that trusts TLS alone; every one after it is verified by the binary you already have.

Details in the timestamped note beside this file.

What's new in v2.9.0 — root never follows a worker's symlink; a stop stops the whole tree

Review batch 2. The daemon runs as root and, until now, opened every file in a worker's own directory — its log, env file, conf, TLS key, the manifests augur reads, the tail the crash webhook sends — the way the standard library does: following whatever symlink the directory's owner put there. It no longer does; the one place that is the logger itself is go-logger v1.2.0. Separately, stopping a worker now stops its whole process group (kill_mode, default group), so a sh -c wrapper or a cluster primary cannot leave orphans holding the port.

Nothing to configure. If an app must shut its own children down in order, set kill_mode process;. The pid file moves to /run/apm/apm.pid; the installer's unit gains a few kernel-surface protections that no app can notice.

Details in the timestamped notes beside this file.

What's new in v2.8.1 — a 2.7 CLI can verify the 2.8 daemon it just installed

The first production upgrade to 2.8.0 ended with a false daemon did not come back: the old CLI that drives an upgrade only knows the abstract socket name @apm, which 2.8 stopped answering. The daemon now answers it for root only — the one caller a 2.7 CLI has there — so an upgrade from 2.7 reports what actually happened. Nothing else changes.

Details in the timestamped note beside this file.

What's new in v2.8.0 — the daemon trusts nothing it did not create

A full security and reliability review of v2.7.2 (2026-10-07) ranked its findings into batches. This release is the first two, plus the tool that makes upgrading across them painless.

Upgrading — read this first

Two things this version reads differently can stop a worker from starting; apm upgrade now checks for both after installing, and apm healconf fixes them:

• A worker with no user line whose path is owned by root used to run

as root silently. It now refuses to start and says so. Add user root; if that is what you mean, or the user the app should run as.

• The CLI talks to the daemon over /run/apm/apm.sock, which only **root

and members of group apm** can open. The installer has created that group and added the installing user since v2.1; anyone else who ran apm list without sudo needs sudo usermod -aG apm <user> and a new login. Commands that change what the daemon runs need root, or a worker that runs as the caller; apm load, reload, saveconf, stopall and exit daemon need root.

sudo apm healconf          # explains each problem, asks before fixing
      sudo apm healconf --check  # report only (exit 1 if something would fail)

healconf also renames keys brought from PM2 or systemd, repairs a missing ;, fixes booleans the daemon reads as false, and converts a PM2 ecosystem file. See the manual's CLI chapter.

Review batch 0 — contained fixes

A request-line cap that a bare-CR stream bypassed (an unauthenticated OOM of the root daemon), a socket chmod by path that followed symlinks, a privilege drop the worker's own user could veto, an error-restart counter that err_grace never reset, log rotation that broke after any stop/start, 64 MiB mesh frames queued 256-deep per child. Connectors v3.1.1: four languages no longer exit on a throwing handler, PHP no longer sends an empty frame on invalid UTF-8, no connector installs a truncated self-update.

Review batch 1 — the control socket knows who is calling

The abstract socket name @apm had no owner and no mode; every supervised worker could drive the root daemon. Now a file in a root-owned directory, peer credentials on every connection, a per-command table of who may do what, and a CLI exit status that finally says whether the command happened.

apm log

Follows a rotation into the new file instead of sitting silent on the renamed one; --no-follow prints the tail and exits, for scripts.

Details in the timestamped notes beside this file.

What's new in v2.7.2 — a restart tells every WebSocket client; a response can set several cookies

Two daemon behaviours that each cost a session a day, found on the same day by two of them.

A restart tells every WebSocket client

A browser kiosk on a WebSocket app froze after every restart of its worker, and only a browser restart brought it back. The daemon was keeping every client's WebSocket open across the child's exit with nothing behind it: no close frame, no TCP close, the client's pings answered by APM itself so no idle timeout ever fired, and every message the client sent written to the next process under a session id it had never been given. Present since the from-scratch server in v0.2.0.

When a child exits — crash, apm restart, watcher restart, stop, daemon shutdown — every WebSocket and TCP connection bound to it is now closed at once. A WebSocket client receives close code 1012 Service Restart and can reconnect; a TCP client sees the socket close; an in-flight HTTP request gets the 503 it always got. Nothing the client sends after the exit can reach the replacement process. The child's log says Closed N client connection(s) — ws close 1012 Service Restart. A session_persist session whose session_wait expires is closed the same way; sessions that are handed to the new child carry on as before.

A Node echo worker shaped like the incident's (instances 1, rolling false, drain_timeout 15000, watch *.js), one raw WebSocket client:

Also documented: drain_timeout waits its full length on a ws:// worker, because open WebSockets never finish on their own.

A response can set several cookies

A response header value may now be an array: one header line per element, in order — the way to send more than one Set-Cookie. It is what every connector already sent when the app passed a list; the daemon dropped it without a word, as it dropped a CR/LF-joined value (that one rightly). Numbers and bools are accepted too, keys come out sorted, and every dropped header is now logged: dropped response header Set-Cookie — CR/LF in value — several values go in an array, one per element.

Details in the timestamped notes beside this file.

What's new in v2.7.1 — an upgrade is down for the swap, not the download

apm upgrade used to stop the daemon first and then download, install, refresh the IP lists and register the service — every slow step inside the window with no daemon and no workers. Watched end to end on 2026-09-23, that window was 30 seconds on a 10-worker box, and only 0.3 s of it was the download. The rest was a systemd deadlock that every clean shutdown has been paying. Both are fixed here.

The order

The installer now does all the slow work while the old daemon keeps serving: download the binary, group and log file and config, the Cloudflare IP lists, init-system detection with the unit written, daemon-reload and enable. Then, as one block: replace the binary, stop the old daemon, start the new one. apm upgrade itself no longer stops anything; the installer owns the switch. Installer revision r9.

The 30 seconds

systemd runs a unit's ExecStop whenever the main process leaves — also when it leaves by itself, as it does after apm shutdown or the old apm upgrade. Ours is apm exit daemon. Finding no daemon, that CLI did what every other command does and started one — through systemctl start apm, from inside the stop job. The start queued behind the stop, the stop waited for the CLI, and TimeoutStopSec=30 was the only way out, leaving the unit in failed. The "Detecting init system..." pause was the installer waiting behind that job.

apm exit daemon is now the one command that never starts a daemon: with none running it prints Daemon is not running. and exits 0. A daemon that exits on its own leaves the unit inactive at once. The installer also puts the new binary in place before stopping the old daemon, so whatever ExecStop systemd runs from that moment is the fixed CLI; and when the stop job is already stuck — the first upgrade from 2.7.0 or older, whose old apm upgrade pre-stops the daemon — it queues a stop and kills the stuck control process instead of waiting 30 s behind it.

Measured

Same box, same download, daemon socket down to up:

Also: the installer's daemon matching is anchored to the exact …/apm --daemon command line; the old loose pkill -f "apm --daemon" also matched a bash -c "$(curl … install.sh)" shell and would have taken the installer down with the daemon. Details in the timestamped note beside this file.

What's new in v2.7.0 — two copies of one request are one request

A single tap on a phone reached the origin as two byte-identical GETs, 1 ms apart, on two HTTP/2 streams — something between Safari, iCloud Private Relay and the CDN raced the request and kept the first answer. APM balanced the copies across two instances, the app ran a state-creating page twice, and a paid order was destroyed. Traced 2026-09-22; the app is fixed; this is the infrastructure answer.

micro_dedupe — a same-request bounce filter, on by default

worker {
          server        http://unix:/run/web.sock;
          micro_dedupe  3;             # ms; default 3, 0 = off
      }

A body-less GET/HEAD whose signature — method, listener, path+query, client IP and every non-per-hop request header — matches one that arrived inside the window is a network bounce, not a second request. It is never forwarded: it parks until the first copy's response is known and is answered with a copy of it. Dropping it would be worse than forwarding it — a fast 502 beats a slow 200 in a race — so the duplicate always gets a real response. Decided before instance selection, so the two copies can never land on two instances.

Per-hop headers (cf-ray, x-request-id, traceparent, x-forwarded-for, the worker's trace_header, ...) are excluded from the signature; anything else differing means a different request. POST, bodies and WebSocket upgrades are never touched. micro_dedupe_log false; silences the rate-limited hit line, micro_dedupe_max caps the per-worker table (full = pass through), and apm info shows duplicates answered plus near misses (same signature just outside the window), so the window can be tuned from data.

The full story, the signature rules and the limits are in the timestamped note beside this file. Verified with go build/vet/test including the race detector against a live listener with a delayed fake child.

Docs are generated from one source — the code

A code-vs-docs audit found the manual, llms-full.txt, llms.txt and the skill file disagreeing with the binary on about twenty points, because the same reference material was typed by hand in four places. From this release every configuration key, default, flag, reload behaviour and CLI usage line comes from one Go table (modules/docs) whose tests re-derive the key sets from the parsers themselves, and the website build renders it into the manual and the skill and derives llms-full.txt from the rendered manual. Release notes on the site come straight from CHANGES/<version>/index.md. The manual's option tables gained a Reload column (live / restart) and a full apm run flag table. Details in the timestamped note beside this file.

What's new in v2.6.0 — a dead device is reported in seconds, not minutes

A force-powered-off WS device used to sit "online" for two minutes — the worker heard nothing until the keep_alive idle deadline fired. Raw tcp:// was worse: minutes on an idle death, 15+ of them if data was in flight. On a device dashboard that reads as a bug, because it is one.

dead_timeout — kernel-level dead-peer detection

worker {
          server        ws://0.0.0.0:9100;
          dead_timeout  10000;     # report a vanished client within ~10 s
      }

Off by default. When set, every accepted ws/tcp client socket gets tuned TCP keepalive (catches a peer that dies while the line is idle) plus TCP_USER_TIMEOUT (catches one that dies with data in flight — which Linux otherwise retransmits at for 15+ minutes). Probes are empty segments the client's kernel answers: no library support, no app wakeup, works the same for a browser and a microcontroller, and none are sent while real data flows.

Measured, real link-down against a live daemon: idle death → worker onClose in 8.2 s (was 120 s); death mid-push → 10.5 s (was 15+ min); dirty FIN → 3 ms. Slow links are safe by construction — only total silence for the whole window triggers, never slowness, so a wifi client behind three walls ACKs its way to safety at any throughput. Reload applies it live to new connections without touching children.

Every dirty WS disconnect leaked a socket — since v1.x

The WS "connection lost" path (read error, EOF without a close frame, idle timeout — exactly the dead-device paths) notified the child and cleaned up the session, but never closed the socket. One leaked fd per dirty disconnect for ever, CLOSE_WAIT pile-up on bare-FIN closes, and a permanently wedged handler goroutine when the child had unACKed data in flight (handleTCP had the same wait-before-close ordering flaw). The connection-side cleanup now closes the socket on every path. Verified: zero CLOSE_WAIT, daemon fd count returns to baseline.

Full stories with the raw-Node baseline measurements are in the timestamped notes beside this file. Verified with go build/vet/test, socket options read back via getsockopt, and an end-to-end daemon + connector + virtual-switch rig with real link-down.

What's new in v2.5.0 — nothing is dropped in silence

APM knew things and did not say them. An unrecognised flag was warned about and ignored, or ignored outright. A worker a restart was about to delete went unmentioned until somebody noticed it missing. The reason a worker failed to start was produced in full and then printed to a CLI that had already exited.

This release closes the last of the field-reported findings from the shared apm/findings note (items 4, 6, 7 and 8), plus two more found while fixing them. Verified with go build/vet/test -race and on a live KVM/systemd box.

The default --help hid the commands with the worst consequences

remove, unload, load and grep needed --full. A session went looking for a way to delete a stopped worker, read apm --help, concluded there was no such command, and wrote it up as a finding — apm remove had existed all along.

That is the cheap version of the mistake. v2.4.1 fixed the expensive one: upgrade was missing from the same list, which is exactly why somebody typed apm upgrade --help to find out what it did, and it did it.

The help is now split by the cost of not knowing a command rather than by how often it is typed — missing list wastes a minute, missing remove leaves litter for ever, missing upgrade stops every worker on the machine. The short list ends by saying what it is hiding.

apm run --help and apm update --help now print the flags each accepts, generated from the parsers themselves.

Unknown flags are refused

$ apm run node app.js -restart false
      ✗ Unknown flag -restart for apm run
        → Did you mean --restart?
        → Run apm run --help for the flag list. Nothing was created.

apm run used to print ⚠ Unknown flag … ignored and create the worker anyway. apm update had no such branch at all: apm update w -instances 2 -instnaces 5 applied the first flag, dropped the typo without a word and restarted the worker, so it looked exactly like success. Both now refuse, name the flag, suggest the near miss and exit non-zero. So does apm list, which validated nothing.

A flag meant for the program rather than for APM is told where it belongs: use -args --inspect.

⚠ Breaking: a command line that was silently having a flag dropped now fails instead of half-working. That is the point, but it will surface scripts that were never doing what they read as doing.

A daemon restart no longer deletes workers in silence

A worker registered with apm run, or loaded from a conf file that /etc/apm/apm.conf does not include, exists only in the running daemon's memory. Stopping the daemon does not stop it — it deletes it. Afterwards it is not listed as stopped; it is simply absent.

⚠ 2 worker(s) exist only in the running daemon and will NOT come back:
          nova_stats  /srv/nova/apm_stats.conf
          adhoc_probe (apm run — no config file)
        → Re-load with: apm load /srv/nova/apm_stats.conf
        → To make them survive, include them from /etc/apm/apm.conf.d

Printed on every route that stops the daemon — apm exit daemon (which is what systemctl stop apm runs, so it reaches the journal), apm shutdown before its confirmation and under --silent too, and apm upgrade before its confirmation. Two new ways to ask ahead of time:

apm ephemeral          # what a restart would lose
      apm list --origin      # where every worker's config came from

Persisting and auto-restoring these was considered and deliberately not done: it would resurrect one-off apm run workers and ad-hoc loads nobody meant to keep.

A failed start tells you why

$ apm run node probe.js -name probe
      ✗ probe#1 failed during startup (exit 1, 149 ms)
      ── child stderr ──────────────────────────────────────────────
      Error: Cannot find module 'redis'
      ──────────────────────────────────────────────────────────────
        → missing node module redis — run npm install (or check NODE_PATH)
      ✗ Worker probe failed to start
        → Log: /srv/app/probe.log (apm log probe)

APM was already producing all of this — from a detached goroutine that lost the race with the CLI disconnecting, so nobody ever saw it. ✗ failed to start — check logs was the whole output, and the log it named was empty. The diagnosis now goes to the code already holding the start open, so it prints while the CLI is attached and before the verdict line, which now names the log file.

A boot with a crashing worker measured 15.3 s before and after.

apm log no longer tails an empty file for ever: an empty log on a worker that is not running says so and exits.

restart false; warned that it had been ignored

The config layer emits --no_restart for an explicit restart false; and no parser understood it, so every first load of such a config printed ⚠ Unknown flag --no_restart ignored — reporting a dropped setting that had not been dropped. Fixed, with a test asserting every negation the config layer can emit is one the parser accepts.

Both flag lists are re-derived from the parsers by a test that parses the source, so a case added to either switch without the list following is a build failure. That test is what found this.

Full detail in 2026-08-25T1700-cli-discoverability-and-silent-drops.md.

What's new in v2.4.1 — parallel worker start, and a CLI that does what it says

Two independent things: config loads stop paying a startup delay per worker, and three CLI defects that could cost you an outage are closed. Verified with go build/vet/test -race and on a live KVM/systemd box.

Workers start in parallel

Every worker's first start is deliberately held open for startup_grace milliseconds (default 2000) so a process that dies on launch is reported as a failure rather than a success. That wait used to be paid once per worker, because config loads walked worker {} blocks one at a time — so most of a boot was spent idle:

● Starting worker-a   → ✔ (2.06s)
      ● Starting worker-b   → ✔ (2.07s)     ← nothing to do with worker-a
      ● Starting worker-c   → ✔ (2.07s)

A 13-worker box spent 26 seconds booting. A 30-worker one spent 129 seconds.

Blocks now start concurrently, in dependency order, with per-worker output held and printed as one block so nothing interleaves:

▸ Starting 30 workers in parallel (8 at a time)
      ...
      ▸ All workers loaded — 29 running, 1 not running in 13.3s (slowest: slow-app 7.1s)

The same 30-worker fleet: 129 s → 13 s. An apm reload that restarted 17 workers: 35 s → 8 s. Naming the slowest worker matters once everything runs at once — the total is the slowest one.

New daemon { start_concurrency N; } caps how many start together. The default is CPU cores × 2 (never below 4), which is about the CPU cost of processes actually booting — the startup_grace wait itself is idle. 0 is unlimited; 1 restores the old one-at-a-time walk.

depends_on now actually orders the fleet

A dependency declared later in the same config file could never be satisfied during a load — the loader had not reached it yet — so the dependent worker waited out the full depends_timeout (30 s by default) and then started anyway, in the wrong order:

⏳ Waiting for dependency database...
         ... 30 seconds ...
      ⚠ Dependency database not ready after 30000ms, starting anyway
      ● Starting database

depends_on only worked if you happened to declare dependencies before their dependents. Blocks are now grouped into dependency waves, so a worker's dependencies are genuinely running before it starts, whatever order the file is in — and the poll never fires. A three-worker chain declared in reverse cost 60 seconds of pure stall before this.

--help no longer runs the command

apm upgrade --help, typed to find out what upgrade would do, ran the upgrade — stopping the daemon and every worker on the box. The flag loop only recognised the flags it knew, so "no flags I recognise" silently meant "proceed".

• -h / --help is honoured by every command, in any argument position,

and prints real usage for that command before anything happens.

• An unrecognised flag is now an error, not something to skip past.
• apm upgrade asks first, naming what it will stop. --yes skips the prompt;

when stdin is not a terminal it refuses rather than proceeding silently, so a scripted upgrade has to opt in.

• upgrade and uninstall appear in apm --help — leaving upgrade out of

the default help is why people reached for apm upgrade --help.

• An unknown command is rejected before the CLI does anything with a side

effect. apm status used to print "Starting daemon" and then "Unknown command status". status is now a real alias for list.

The CLI respects the systemd unit that owns the daemon

With the daemon stopped, the next apm command started one as a plain child process rather than through apm.service. That daemon held the socket, so systemd could never bind it, gave up after five attempts, and left the unit failed while a healthy daemon ran outside it — one that systemctl could neither see nor stop, and systemctl restart apm could not recover.

Where an apm.service unit exists, the CLI now starts the daemon through systemd (clearing a stale failed state first, which otherwise refuses start). It never falls back to spawning a child: if it cannot start the unit it says so and tells you what to run. Installs without a unit file are unchanged.

apm upgrade also verifies the daemon came back afterwards instead of assuming the installer restarted it, and reports how many workers were running before.

apm restart re-reads the config file

restart used to bounce the processes without consulting the config at all. But env_file is read fresh on every spawn — so a restart did pick up .env edits, while env lines in the conf file were silently ignored. Half a configuration change applying from one command, which reported success either way.

A plain apm restart <worker> now re-reads the worker's block, applies what changed (including in-place work a process bounce would miss — log destinations, watcher patterns, Vanguard rules, statsd, proxy listeners), and then restarts.

• --no-reload keeps the old behaviour.
• A worker started with apm run has no config file, and says so.
• A config file that will not parse, or no longer declares the worker, is

reported and the worker restarts on its loaded config — never left down because a file was mid-edit.

• apm restart --force <worker> works with the flag in either position; it

previously looked up a worker literally named --force.

Docs

start_concurrency, depends_on wave ordering and restart's new behaviour are documented. Four docs/19_cli.md entries were wrong and are corrected: apm check parses a config file (it does not verify the daemon is running), apm boot loads /etc/apm/apm.conf (it does not install a systemd service), apm update changes a worker's settings (it does not update connectors), and apm restart --force targets one worker rather than force-restarting all of them.

What's new in v2.4.0 — review hardening, connector rewrite, memory_limit enforcement

A large correctness + security pass across the daemon, the six language connectors, the GUI, and the docs, from the 2026-08-23 full-source review (APM_REVIEW_2026-08-23.md). Every change was verified with go build/vet/test -race, a new six-language connector conformance harness, and a live upgrade rehearsal on a KVM/systemd box.

Daemon — crash, restart & lifecycle

• Signal/OOM child deaths now count as error exits. A SIGSEGV/SIGKILL/

OOM-kill left exitCode==0, so restart_err never fired and restart miscounted the death as a clean restart. Externally-sent SIGTERM now counts as an error too; APM's own stops are excluded.

• Log rotation fixed for err_log == log (the default). The shared-path

branch nil'd the error logger without closing it, so the live log ended up renamed to .2 and rotation silently died. Rotation also no longer races a child's stdout writer into a panic that wedged the child.

memory_limit — actually enforced (was silently inert on systemd)

Limits are now written to a cgroup v2 leaf inside the daemon's delegated cgroup; the old top-level /sys/fs/cgroup/apm/… was refused by systemd, so memory_limit did nothing (a 128 MB-limited worker reached 945 MB). A configured-but-unenforceable limit now logs a loud per-worker error instead of failing open. The installer now sets Delegate=yes on the systemd unit — existing installs need a re-install (or a matching drop-in) for enforcement.

Proxy / Vanguard

• Idle-based write deadline on the HTTP response path — a stalled client no

longer leaks the goroutine/fd/session forever, while a slow-but-progressing large download is never truncated.

• ban_ip now applies to the real client IP behind trust_proxy (was only

the raw peer); allow_ip stays connection-scoped by design.

• Per-IP rate-limit table bounded (was unbounded behind a proxy → memory

exhaustion); response-header CRLF filter (response-splitting); TLS MinVersion pinned to 1.2.

Config — load and reload now agree

Absent trust_proxy/user/log paths resolved differently at boot vs. reload, so the first reload after boot spuriously full-restarted workers and flipped trust_proxy. All three reconciled. apm saveconf quotes values that previously broke the parse or were silently split; apm convert comma-joins params (space-join crash-looped converted apps); apm stop; apm load no longer doubles argv/env/server binds.

APM Network (mesh)

The relay read loop no longer blocks on a slow child's stdin (one hung child used to stall the whole mesh). Inbound request/stream floods are capped, and stream data/teardown is checked against the sender's identity so a token member can't inject into or tear down streams it never joined.

Connectors v3.1.0 (all six)

• Python/Perl/Lua no longer stall until 64 KB of input accumulates — a

normal small request was never processed until the daemon's 504. Fixed to partial/exact-length reads.

• Lua no longer exits the host app on require (the arg guard misfired).
• request()/request_stream() are safe to call from callbacks in all

blocking connectors (deadlock/reentrancy fixed).

• Binary WS frames are typed binary; the daemon stamps HTTP body dataType

from Content-Type so Node stops corrupting binary uploads.

• close() uses the protocol default when no code is given (WS 1000, not an

invalid 200); atomic self-update; Node crash handlers no longer force-exit apps that install their own.

• A conformance harness (connectors/conformance/) drives all six under a

stub daemon and asserts identical wire behaviour.

GUI

Stored XSS via worker name fixed; dashboard module name/value/colour escaped; LED colour set via the CSSOM (an HTML escaper can't neutralise CSS).

Other

telemetry opt-out honours off/0/no/disabled (not only false); augur no longer misclassifies bundle as Node, caps manifest reads, and filters probe output; toolchain pinned to clear 6 reachable stdlib CVEs; crypto/rand for IDs that gate sessions/IPC. Watcher docs corrected (watch takes file-name patterns, never paths). The watcher now picks up directories moved into the tree — an atomic deploy (mv built/ live/) restarts the worker like an in-place edit did, and a directory moved aside no longer leaves stale watches behind.

What's new in v2.3.0 — v2.3.0

Reliability release: a source-level audit (adversarially verified) and a field-reported outage drove a large batch of concurrency, lifecycle, and untrusted-input fixes. None change configured behaviour — they remove crashes, hangs, leaks, and silent data loss.

Stale-listener self-heal. A worker's unix server: socket could reach a state where it showed LISTEN but refused every fresh connection, with apm list still green and only restart --force recovering it. The listener lifecycle is now serialised on a per-worker mutex, socket-file unlink is ownership-checked (APM no longer lets Go's unlink-on-close delete a socket a newer listener bound), and a watchdog probes every unix listener every 10s and rebinds a wedged one in place — children untouched — emitting a server_rebind event. Watch-triggered restart bursts are coalesced so a flurry of saves no longer churns through dozens of back-to-back restart cycles.

Deadlocks removed. Two daemon-wide hangs that presented as "green but serves nothing": the child-exit handler joined its stdout reader while holding the same mutex that reader's IPC dispatch needs, and a circular depends_on held a worker's mutex across its dependency wait. Both are fixed.

Lifecycle and leak fixes. Single-owner connection accounting (no more negative activeConns breaking drain); rolling restarts no longer abandon an unrelated sibling that crashes mid-cycle; sync.Once teardown for statsd and vanguard (no double-close panic); health-probe, GUI syscollector, inotify-fd, and apm log goroutine/fd leaks closed; IPC cleanup scoped to the exiting child (was wiping every in-flight request), request/stream ids namespaced per child, and child-supplied timeouts clamped.

Untrusted-input hardening. Bounds on HTTP body/headers (plus a body-read deadline), WebSocket fragmented messages, the child 0x05 stdout frame length, and the local-command / mesh / GUI WebSocket buffers — each was an unbounded allocation from a length field. The stderr reader no longer dies on long lines, and a slow client gets a clean truncated transfer instead of a silently corrupted stream. ban_path now also matches the percent-decoded target.

Config / saveconf / convert. apm reload <file> only stops workers owned by that file (no longer sweeps other files' or ad-hoc workers); saveconf serialises every reloadable field, preserves sibling worker blocks, and round-trips quoted values; apm_network reloads apply; apm convert emits a recognised memory_limit and no longer mangles date layouts; the config parser no longer hangs on a trailing unterminated quote.

What's new in v2.2.3 — v2.2.3

Security hardening from an external source-level audit (APM-002/003/007): the admin GUI now binds 127.0.0.1 by default and refuses a non-loopback bind when no gui_password is set; worker user-switching fails closed instead of silently running as root; and the metrics collector snapshot is deep-copied so the StatsD flush and GUI dashboard can no longer race the live collector maps into a fatal concurrent-map daemon crash.

Config reload now picks up new include files: apm reload with no argument re-reads the root config files and re-expands their include directives, so a file freshly dropped into an included directory (e.g. apm.conf.d/) is loaded without naming it, restarting the daemon, or any hardcoded drop-in path.

What's new in v2.2.2 — v2.2.2

Resilient APM Network auto-reconnect: reconnect-storm guard (backoff resets only after a stable session), jittered backoff (no thundering herd on relay restart), panic-proof reconnect loop, client keepalive ping with faster dead-link detection, and clean mesh-state teardown on every disconnect (with automatic re-announce on reconnect).

Daemon-wide panic hardening: every goroutine touching external input (CLI socket, proxied traffic, child stdout IPC, GUI, mesh frames, config reloads) now runs behind a recover boundary that logs the panic with its stack — one bad frame can no longer kill the daemon. WS frame parser gets a full 64-bit length decode and a 64 MiB per-frame cap. apm exit daemon now reports a clean shutdown to telemetry.

What's new in v2.2.1 — v2.2.1

Surface APM Network connection state in the CLI: apm list shows a ● connected to APM network status line and apm monitor shows a MESH line in its header (relay link state + remote-channel count). Complements the per-worker apm_network identifier already shown by apm info.

What's new in v2.2.0 — v2.2.0

APM Network — cross-machine inter-worker IPC (pre-release). Daemons sharing a license token form a private mesh via a persistent WebSocket to wss://apm.processmanager.dev; a worker tagged apm_network <id> is reachable by send/request/requestStream from workers on other machines using the same verbs — transparent, no connector changes, with local-only fallback when the relay is down. Tokens (public-IP + max-connections bound) are issued by email during pre-release ([email protected]).

Also: remote_logging true (opt-in) streams worker/child lifecycle events to the stats server for diagnosing unexpected worker/child churn.

What's new in v2.0.4 — APM — Next Release

Bug Fixes

Logger reload now applies live, no process restart

Changing log / err_log / prefix / syslog in a worker's config and running apm reload previously triggered a full process restart, and the restarted child kept writing to the old log file because the in-memory logger instances were never recreated. Workarounds required apm shutdown + apm boot.

Fix: Logger fields are now tracked separately from the runtime signature. A logger-only change is applied via Worker.RebuildLoggers() — the old workerLogInst and per-child logInst / errLogInst are closed and recreated from the new fields, with no impact on the running child processes. A change that includes both logger and runtime fields rebuilds loggers first, then restarts cleanly with the new destinations already in place.

What's new in v2.0.3 — APM v1.1.1 — Changes

Improvements

Config reload now syncs all worker parameters

apm reload previously only updated exec, path, params, and instances when a worker was already running. All other config fields (watcher patterns, restart settings, rolling delay, kill timeout, TLS, session, proxy flags, etc.) were silently ignored — the old values stayed active until the worker was fully stopped and restarted.

All fields are now applied live before the rolling restart. The file watcher is additionally closed and reopened if watch, watch_ignore, or watch_delay changed, so the new patterns take effect immediately without a full stop/start.

Bug Fixes

Orphaned processes on rapid file changes (watcher restart race)

When multiple file change events fired in quick succession, the debounce timer could fire a second WatcherRestart goroutine while the first was still running. Both goroutines would wait on c.wait.Wait(), both wake when the child exits, and both call c.Start() — spawning two processes for the same child slot. The second start overwrites c.cmd, orphaning the first process. With two instances this accumulates two extra processes per double-trigger.

Fix: Added watcherRestartMu sync.Mutex to Worker — WatcherRestart locks it on entry, so concurrent watcher-triggered restarts are serialised instead of racing. Added a StatusRunning guard at the top of Children.Start() as defence-in-depth against any concurrent call site.

watch_ignore patterns silently dropped when more than one pattern given

Config values with multiple comma-separated entries (e.g. watch_ignore web_*, node_modules) are stored as []string by the config parser. addWorker used GetString which returns "" for []string values — so watch_ignore was never passed to the worker and all .js files triggered restarts, including those in excluded directories.

Fix: addWorker now falls back to GetStrings + comma-join when GetString returns empty, restoring the comma-separated string that parseWatchPatterns expects.

What's new in v2.0.2 — APM v1.1.1 — Changes

Improvements

Config reload now syncs all worker parameters

apm reload previously only updated exec, path, params, and instances when a worker was already running. All other config fields (watcher patterns, restart settings, rolling delay, kill timeout, TLS, session, proxy flags, etc.) were silently ignored — the old values stayed active until the worker was fully stopped and restarted.

All fields are now applied live before the rolling restart. The file watcher is additionally closed and reopened if watch, watch_ignore, or watch_delay changed, so the new patterns take effect immediately without a full stop/start.

Bug Fixes

Orphaned processes on rapid file changes (watcher restart race)

When multiple file change events fired in quick succession, the debounce timer could fire a second WatcherRestart goroutine while the first was still running. Both goroutines would wait on c.wait.Wait(), both wake when the child exits, and both call c.Start() — spawning two processes for the same child slot. The second start overwrites c.cmd, orphaning the first process. With two instances this accumulates two extra processes per double-trigger.

Fix: Added watcherRestartMu sync.Mutex to Worker — WatcherRestart locks it on entry, so concurrent watcher-triggered restarts are serialised instead of racing. Added a StatusRunning guard at the top of Children.Start() as defence-in-depth against any concurrent call site.

watch_ignore patterns silently dropped when more than one pattern given

Config values with multiple comma-separated entries (e.g. watch_ignore web_*, node_modules) are stored as []string by the config parser. addWorker used GetString which returns "" for []string values — so watch_ignore was never passed to the worker and all .js files triggered restarts, including those in excluded directories.

Fix: addWorker now falls back to GetStrings + comma-join when GetString returns empty, restoring the comma-separated string that parseWatchPatterns expects.

What's new in v2.0.1 — APM v1.1.1 — Changes

Improvements

Config reload now syncs all worker parameters

apm reload previously only updated exec, path, params, and instances when a worker was already running. All other config fields (watcher patterns, restart settings, rolling delay, kill timeout, TLS, session, proxy flags, etc.) were silently ignored — the old values stayed active until the worker was fully stopped and restarted.

All fields are now applied live before the rolling restart. The file watcher is additionally closed and reopened if watch, watch_ignore, or watch_delay changed, so the new patterns take effect immediately without a full stop/start.

Bug Fixes

Orphaned processes on rapid file changes (watcher restart race)

When multiple file change events fired in quick succession, the debounce timer could fire a second WatcherRestart goroutine while the first was still running. Both goroutines would wait on c.wait.Wait(), both wake when the child exits, and both call c.Start() — spawning two processes for the same child slot. The second start overwrites c.cmd, orphaning the first process. With two instances this accumulates two extra processes per double-trigger.

Fix: Added watcherRestartMu sync.Mutex to Worker — WatcherRestart locks it on entry, so concurrent watcher-triggered restarts are serialised instead of racing. Added a StatusRunning guard at the top of Children.Start() as defence-in-depth against any concurrent call site.

watch_ignore patterns silently dropped when more than one pattern given

Config values with multiple comma-separated entries (e.g. watch_ignore web_*, node_modules) are stored as []string by the config parser. addWorker used GetString which returns "" for []string values — so watch_ignore was never passed to the worker and all .js files triggered restarts, including those in excluded directories.

Fix: addWorker now falls back to GetStrings + comma-join when GetString returns empty, restoring the comma-separated string that parseWatchPatterns expects.

Earlier releases

• v2.1.3 — apm check falsely warned "exec not found" for valid configs; StartWorker argv-parser was dropping a long list of config flags
• v2.1.2 — apm CLI occasionally hung after the command had finished; Docs: clarify that apm CLI does not need sudo; params values starting with - were silently dropped
• v2.1.1 — apm convert — PM2 ecosystem → APM worker block
• v2.1.0 — apm reload <workerName> — reload one worker from its origin file; Bug fix — reload silently dropped per-line timestamps; Augur — startup-failure diagnostician
• v2.0.10 — Startup UX hardening
• v2.0.9 — Logger goes silent after Rebuild/Rotate close-recreate cycle; CLI: apm list / apm info sometimes hangs after printing; apm upgrade leaves old daemon running on old version
• v2.0.8 — AI skill file + home-page download section; Manual / llms / SKILLS synced to the Go source (audit); Installer: symlink into /usr/local/bin when /usr/sbin is not in user PATH; Node.js ESM connector; ESM connector — docs and LLM-facing sync; Reload / restart correctness pass; Info accuracy, env masking, include-aware saveconf, action-map reload; Deferred cleanup: clear-on-reload, include-aware watch_conf, saveconf safety; GUI: doubled wgap tbody before late-added workers; Manual: per-language connector examples + cross-language IPC walkthrough
• v2.0.7 — apm reload -all — reload every loaded conf file in one shot; Vanguard: ban_path_ttl — auto-ban IPs that hit banned paths; Vanguard config no longer ignored on reload; Vanguard: ban_path_ci — case-insensitive ban_path matching
• v2.0.6 — Vanguard: HTTP method filtering
• v2.0.5 — Vanguard: log toggle and periodic event-count summary