Overview

APM is Tom's Advanced Process Manager for Linux. One binary, copy and run — no config files required to get started. Complexity is APM's problem, not yours.

APM runs as a background daemon and manages worker processes. You interact with it through the apm CLI. The daemon auto-starts the first time you run any apm command.

Architecture

The daemon communicates with the CLI via an abstract Unix socket. Workers are child processes managed by the daemon. Each worker can have multiple parallel instances. The built-in reverse proxy routes incoming connections across instances using round-robin.

  CLI  ──(unix socket)──  Daemon  ──  Worker [4 instances]
                                  └──  Worker [1 instance]
                                  └──  GUI server (port 6789)

Philosophy

• Zero config to start — defaults are correct for 90% of cases
• CLI-first, config files for persistence and power users
• Every error message answers "what do I do now"
• Never crash on a bad optional field — warn and continue
• Linux only. Windows is not supported.

Installation

Run the install script as root. It downloads the right binary for your architecture, sets up the system group and log file, installs the init service, and starts the daemon.

# One-liner install
$ curl -fsSL https://processmanager.dev/install.sh | sudo bash

# Or download first, review, then run
$ curl -fsSL https://processmanager.dev/install.sh -o install.sh
$ sudo bash install.sh

# Verify
$ apm --version

The installer sets up:

• /usr/sbin/apm — binary
• /var/log/apm.log — daemon log (group-readable by apm)
• /etc/apm/apm.conf — default config (created if absent)
• apm OS group — add users with usermod -aG apm <user>
• Startup service (systemd, OpenRC, or SysV — auto-detected)

The daemon auto-loads /etc/apm/apm.conf at startup — no separate boot step required.

Run as a systemd service

The installer registers APM with your init system automatically. The service runs the daemon at boot, supervised by systemd.

# Check status
$ systemctl status apm

# View logs
$ journalctl -u apm -f

Uninstall

# Remove binary and config
$ sudo apm uninstall

# Remove everything including logs, group, service
$ sudo apm uninstall --purge

Command reference

All commands communicate with the running daemon. If no daemon is running, APM starts one automatically.

apm [command] [options]

The CLI does not require sudo. The daemon listens on an abstract Unix socket (@apm), which has no filesystem permissions — any local user can run apm list, apm reload, apm restart, apm info, etc. sudo is only needed for apm install / apm uninstall (which write to /usr/sbin, /etc/apm, and the init system), and for reading worker log files directly when those files are owned by root. The apm OS group exists solely to grant read access to /var/log/apm.log — it is not required to use the CLI.

Process commands

Inspection commands

Config commands

GUI & Monitor commands

Daemon commands

Run flags

Flags for apm run and apm update. The same options are available as config file fields (see Worker options).

# Start a worker from the CLI — no config file needed
$ apm run node server.js \
    --name        myapp \
    --instances   4 \
    --server      http://0.0.0.0:3000 \
    --watch       "*.js" \
    --restart                # restart on clean exit
    --rolling                # rolling restart mode

# Update a running worker's instance count without restarting
$ apm update myapp --instances 8 --no-restart

# Save it back to a conf file
$ apm saveconf myapp /etc/apm/apm.conf.d/myapp.conf

Signals

Any signal name can be used as a command to forward that signal to a worker's child processes — all instances, or a single one by index:

$ apm SIGHUP myworker        # send to all instances
$ apm SIGUSR1 myworker#2     # send to instance index 2 only

Config file

Config files define workers and daemon settings. They're loaded with apm load or apm reload. The system config path is /etc/apm/apm.conf.

Syntax

• Key-value pairs end with ;
• Blocks use { }
• Comments: # or // to end of line
• Strings: unquoted, or single/double/backtick quoted (quotes are stripped)
• Multiple values: comma-separated on one line, or repeat the key
• The : suffix on keys is optional
• include <glob>; inlines another file at parse position

# Simple worker
worker {
    name       myapp;
    exec       node;
    params     server.js;
    instances  4;
    restart    true;
    watch      *.js;
    server     http://0.0.0.0:3000;
}

Config hierarchy

APM's startup scripts call apm boot after the daemon starts, which loads /etc/apm/apm.conf. The main config is typically structured as:

1. /etc/apm/apm.conf — main config (daemon block + includes)
2. /etc/apm/apm.conf.d/*.conf — drop-in worker configs, sorted by filename

You can also load configs manually at any time with apm load <file> or do a live diff with apm reload <file>.

daemon {
    gui_port  6789;
}

# Load drop-in worker configs
include apm.conf.d/*.conf;

worker {
    name  portal;
    exec  node;
    params app.js;
}

Multiple values

# Comma-separated on one line
server  http://0.0.0.0:3000, ws://0.0.0.0:3001;

# Or repeat the key
ban_path  *.php;
ban_path  *wp-*;
ban_path  *.env;

Worker options

All options are available both as CLI flags to apm start and as fields in a worker { } config block.

Identity

Environment

Restart on clean exit

Restart on error exit

Shutdown

Startup dependencies

Health check

APM can probe a worker's health two ways. Pull mode — set health_check to a URL and APM sends periodic HTTP GETs (2xx/3xx = healthy). Push mode — set health_check to on and APM injects APM_HEALTH_URL into the child, which calls it to report in. Health status shows in apm list and the GUI.

Connection drain

Memory limit

Logging

Proxy / HTTP

File watcher

Rolling restart

Stats

Inter-worker IPC

Crash webhook (on_crash)

APM can POST a JSON payload (or send a GET request) to a URL of your choice whenever a child process crashes — i.e. exits with a non-zero code or is killed by a signal. Intentional stops (apm stop) are never reported.

worker {
    name  myapp;
    cmd   node server.js;

    on_crash {
        url         https://hooks.example.com/apm-crash;
        method      POST;           # POST (default) or GET
        debounce    10000;          # min ms between calls (floor: 5000)
        log_lines   20;             # tail lines to include in payload
        log_source  err;            # "err" (default) or "out"
        secret      mysecret;       # signs payload with HMAC-SHA256
    }
}

Request headers

POST payload

{
  "worker":         "myapp",
  "instance":       1,
  "exit_code":      1,
  "exit_signal":    "SIGKILL",   // omitted if process exited normally
  "runtime_ms":     4821,
  "error_restarts": 3,
  "timestamp":      "2025-06-01T12:00:00Z",
  "log":            "Error: cannot connect to DB\n..."  // omitted if log_lines = 0
}

Daemon config

Global APM settings live in a top-level daemon { } block. No daemon block = all defaults. Zero config still works.

daemon {
    gui_port      6789;        # web GUI port — 0 disables the GUI
    gui_bind      127.0.0.1;   # bind address — default is 0.0.0.0 (all interfaces)
    gui_password  secret;      # GUI login password
    auto_reload   true;        # reload config files when they change on disk
}

# telemetry is a top-level key — not inside daemon { }
telemetry  false;   # opt out of the anonymous usage ping

Server types

APM's built-in proxy accepts connections and forwards them to worker instances via IPC. Specify servers with the server field. Multiple servers per worker are supported.

worker {
    name    api;
    exec    node;
    params  api.js;

    # HTTP and WebSocket on separate ports
    server  http://0.0.0.0:3000;
    server  ws://0.0.0.0:3001;

    # Or combined on one line
    server  http://0.0.0.0:3000, ws://0.0.0.0:3001;
}

Client IP resolution

When behind a CDN or reverse proxy (e.g. nginx), enable trust_proxy so APM resolves the real client IP from X-Forwarded-For headers. This affects Vanguard rate limiting and ban decisions.

trust_proxy  true;

Vanguard

Vanguard is APM's built-in request firewall. It runs before worker IPC — rejected connections never reach your app. Configure it with a vanguard { } sub-block inside a worker.

worker {
    name    api;
    exec    node;
    params  api.js;
    server  http://0.0.0.0:3000;

    vanguard {
        rate_limit    100;          # requests/sec per IP
        rate_burst    200;          # burst capacity
        ban_ttl       300000;       # auto-ban for 5 minutes
        ban_path      *.php, *wp-*, *.env, /.git*;
        ban_response  Forbidden;
    }
}

IP filtering

Path banning

Method filtering

vanguard {
    allow_method  GET;
    allow_method  POST;       # everything else → 405

    # or, blocklist style:
    ban_method    TRACE;
    ban_method    OPTIONS;
}

ban_path  *.php;          # ends-with  — block all .php requests
ban_path  *wp-*;          # contains   — block WordPress probes
ban_path  /.git*;         # starts-with — block .git exposure
ban_path  *.env;          # ends-with  — block .env file reads
ban_path  /admin/login;   # exact match — block a specific path

Rate limiting

Rate-limited requests receive 429 Too Many Requests. Path/IP bans return 403 Forbidden (or silent TCP RST).

Logging

vanguard {
    rate_limit   200;
    ban_path     *.php, *.env;

    log          off;       # silence per-event lines under sustained probing
    log_summary  60;        # one aggregate line per minute instead
}

CDN IP lists

APM's installer fetches Cloudflare's published egress IP ranges and writes them to /etc/apm/ips/ as ready-to-include partial configs. Include them inside a vanguard { } block to restrict direct access to CDN traffic only.

vanguard {
    # Only allow Cloudflare egress IPs (IPv4 + IPv6)
    include /etc/apm/ips/cloudflare-v4.part;
    include /etc/apm/ips/cloudflare-v6.part;

    rate_limit  500;
    ban_path    *.php, *wp-*, *.env, /.git*;
}

The IP lists are re-fetched automatically on every apm install or upgrade. To refresh them manually:

$ sudo apm install  # re-runs the full installer, including IP fetch

TLS

APM has first-class TLS support for all server types — HTTP, WebSocket, and TCP. Bring your own certificates.

worker {
    name      api;
    exec      node;
    params    api.js;
    server    https://0.0.0.0:443;

    tls       true;
    tls_cert  /etc/ssl/certs/myapp.crt;
    tls_key   /etc/ssl/private/myapp.key;
    # tls_ca for mutual TLS (client cert verification)
    tls_ca    /etc/ssl/certs/ca.crt;
}

File watcher

The file watcher monitors your source directory and triggers a worker restart when matching files change. Uses kernel file-watch events (inotify) — no polling.

worker {
    name          api;
    exec          node;
    params        server.js;
    path          /home/user/myapp;

    watch         *.js, *.json;        # watch .js and .json files
    watch_ignore  *node_modules*;      # ignore anything inside node_modules
    watch_delay   200;                  # 200ms debounce
}

Pattern syntax

Watch patterns use a simple glob-style syntax — no regex needed. Four matching modes:

# Go source files, excluding generated code and vendor
watch         *.go;
watch_ignore  *_generated.go, *vendor*;

# JS/TS project — watch src/, ignore build output and deps
watch         *.js, *.ts, *.json;
watch_ignore  *node_modules*, *dist/*;

# Python — any .py file anywhere under path
watch         *.py;
watch_ignore  *__pycache__*;

Rolling restart

Rolling restarts cycle through instances one at a time, keeping the rest running to serve traffic. Zero downtime for multi-instance workers.

worker {
    instances     4;
    rolling       true;
    rolling_delay 1000;  # 1s between each instance restart
}

With session_persist true, open connections are migrated to a new instance before the old one is killed. Use session_wait to control how long APM waits for the new instance to become ready.

rolling          true;
rolling_delay    500;
session_persist  true;
session_wait     2000;  # wait up to 2s for new instance

Logger

APM has a built-in logger for each worker. Every line written to a child process's stdout or stderr is intercepted, prefixed with a timestamp and worker name, and written to the configured destination. Coloring is applied by APM before writing — use strip_ansi to remove it when logging to files.

Destinations

Prefix

Each log line is prefixed with the worker name (or a custom string). The prefix field supports the color syntax described below. APM automatically appends the instance number in multi-instance workers:

worker {
    name    api;
    exec    node;
    params  server.js;

    # cyan name, reset after — instance # is appended automatically
    prefix  ç51-api-serverçR-;

    log     /var/log/myapp/out.log;
    err_log /var/log/myapp/err.log;
}

For a worker with instances 3, the stdout prefix becomes api-server#1, api-server#2, api-server#3 — each in a distinct color so instances are visually distinct in the live GUI and in log files.

Timestamp format

The timestamp prepended to each line is controlled by log_time_format. The format string uses strftime-style tokens and supports color escapes. The default is ç214-%Y-%m-%d %Tç59-.%FçR- (orange date, dim fractional seconds).

Strftime tokens

# Default — orange date, dim microseconds
log_time_format  ç214-%Y-%m-%d %Tç59-.%FçR-;

# Compact — just HH:MM:SS in gray
log_time_format  ç59-%TçR-;

# No color — plain ISO timestamp
log_time_format  %Y-%m-%d %T;

Strip ANSI

Color syntax — çN-

APM uses a compact color escape based on the 256-color terminal palette. The ç character (U+00E7) acts as the escape marker. This syntax works in prefix, log_time_format, and anywhere APM renders text to the terminal or log files.

# Foreground only
prefix  ç82-myappçR-;      # bright green name
prefix  ç196-myappçR-;     # bright red name
prefix  ç214-myappçR-;     # orange name

# Foreground + background
prefix  ç15,88-ERRORçR-;    # white text on dark red background

# Bold foreground
prefix  ç51,0,1-myappçR-;   # bold cyan

Useful color reference

StatsD

APM can forward worker metrics to any StatsD-compatible endpoint (StatsD, Graphite, Datadog Agent, Telegraf) over UDP. Add a statsd { } block to a worker.

worker {
    name    api;
    exec    node;
    params  server.js;

    statsd {
        host      localhost:8125;   # StatsD UDP endpoint
        prefix    apm.api;          # metric namespace
        interval  1;                # flush interval, seconds
    }
}

System metrics

Forwarded automatically every interval, as gauges:

Custom metrics

Metrics emitted from worker code with apm.metric(name, value, type) are aggregated across all of the worker's instances and forwarded with the same prefix — counters summed (|c), timings averaged (|ms), gauges last-value-wins (|g).

Web GUI

APM ships a built-in real-time web dashboard. Enable it with gui_port in the daemon config:

daemon {
    gui_port  6789;         # GUI port — omit or set 0 to disable
    gui_bind  127.0.0.1;   # default is 0.0.0.0 — set 127.0.0.1 to keep it local
}

When the daemon starts with the GUI enabled it prints the access URL:

$ apm start
GUI: http://127.0.0.1:6789/

Views

Server Info — latency

The Server Info page has a Latency section with two cards:

Disconnect behaviour

When the daemon stops, the WebSocket closes and the GUI immediately dims with a Session Ended overlay. Click Refresh Page to reconnect.

Dashboard

Each worker can expose a custom metric dashboard in the GUI. Define a dashboard { } block inside a worker config to create one. The dashboard is shown in the Dashboard tab when that worker is selected.

worker my-api {
    exec    node;
    params  server.js;
    server  http://127.0.0.1:3000;

    dashboard {
        name  My API;
        cols  6;
        rows  4;

        module {
            type  graph;
            id    1;
            name  Requests/sec;
            x 0; y 0; w 3; h 1;
        }
        module {
            type  gauge;
            id    2;
            name  CPU %;
            x 3; y 0; w 1; h 1;
            min 0; max 100; unit %;
        }
        module {
            type  counter;
            id    3;
            name  Total errors;
            x 4; y 0; w 1; h 1;
        }
        module {
            type  text;
            id    4;
            name  Last error;
            x 5; y 0; w 1; h 1;
        }
    }
}

Dashboard block fields

Module fields

Module types

Sending metrics from code

Use apm.setDashValue(id, value, color?) in the Node.js connector to push a value to a dashboard module. This is distinct from apm.metric(), which is for StatsD-style system metrics only.

const ApmModule = require('./apm_module.node.js')
const apm = new ApmModule(async (session) => { /* handle connections */ })

// Push a number to module id 1 (graph)
apm.setDashValue(1, requestsPerSecond)

// Push a number with a dynamic color
apm.setDashValue(2, cpuPercent, cpuPercent > 80 ? '#ff5a5a' : '#4f8cff')

// Push a string to a text module (id 4)
apm.setDashValue(4, lastErrorMessage)

// LED on/off (1 = on, 0 = off)
apm.setDashValue(5, isHealthy ? 1 : 0, isHealthy ? '#47d16c' : '#ff5a5a')

Counter vs gauge vs graph

Node.js connector

The Node.js connector ships in two equivalent forms — pick whichever matches your project:

• apm_module.node.js — CommonJS (require); Node 10+.
• apm_module.node.mjs — ES Modules (import); Node 14+, for "type":"module" packages.

Both files are line-for-line equivalent: same class, same API, same wire protocol. IPC happens over stdin/stdout using binary frames — no Unix sockets required in the child.

$ curl -fsSL https://processmanager.dev/connectors/apm_module.node.js -o apm_module.node.js

$ curl -fsSL https://processmanager.dev/connectors/apm_module.node.mjs -o apm_module.node.mjs

$ node apm_module.node.js -update
$ node apm_module.node.mjs -update

const ApmModule = require('./apm_module.node.js')
const apm = new ApmModule(async (session) => {
    // session.protocol  — 'http' | 'ws' | 'tcp'
    // session.method    — HTTP method
    // session.path      — full path + query
    // session.headers   — request headers
    // session.remoteIp  — real client IP (proxy-aware)
    // session.cookies   — parsed cookie map

    session.write('Hello World', {
        'content-type': 'text/plain',
        'x-status': '200'
    })
    session.close()
})

If your worker doesn't handle sessions (e.g. a background job pushing dashboard metrics), pass an empty async function: new ApmModule(async () => {}).

Session API

Instance methods

Environment variables

APM injects the following into managed child processes:

WebSocket example

const ApmModule = require('./apm_module.node.js')
const apm = new ApmModule(async (session) => {
    if (session.protocol !== 'ws') {
        session.close(400)
        return
    }

    session.onData = (data, isBinary) => {
        // echo back
        session.write(data)
    }

    session.onClose = () => {
        console.log('disconnected', session.sessionId)
    }
})

PHP / Python / Perl / Lua connectors

Connectors for other languages follow the same pattern as Node.js: drop a single file into your project, require / include it, construct an instance with an onConnect callback, then run the event loop. All connectors implement the full APM IPC protocol over stdin/stdout — no extra dependencies beyond what's noted on the connectors page.

$ curl -fsSL https://processmanager.dev/connectors/apm_module.php   -o apm_module.php
$ curl -fsSL https://processmanager.dev/connectors/apm_module.py    -o apm_module.py
$ curl -fsSL https://processmanager.dev/connectors/ApmModule.pm     -o ApmModule.pm
$ curl -fsSL https://processmanager.dev/connectors/apm_module.lua   -o apm_module.lua

Each connector file also supports self-update — run it with -update to fetch the latest version from the server (e.g. php apm_module.php -update). See the connectors page for version info, MD5 checksums, and per-language update commands. All four expose the same setDashValue(id, value, color?) and metric(name, value, type?) methods as the Node.js connector — see the Dashboard section.

Each subsection below shows the same three worked patterns per language: a hello-world HTTP session, an IPC channel handler (the worker listens on a channel and replies to request()), and a stream initiator (the worker opens a stream to a peer and exchanges data). The wire protocol is identical across languages — anything you can do from Node.js works from these. For a worked end-to-end example with workers in two different languages talking to each other, see cross-language IPC walkthrough.

Python

Python 3.6+, stdlib only. The connector exposes ApmSession and IpcStream classes and uses snake_case method names. The session.write() argument can be bytes or str.

from apm_module import ApmModule

def on_connect(s):
    s.write(b'Hello from Python', {
        'content-type': 'text/plain',
        'x-status':    '200',
    })
    s.close()

apm = ApmModule(on_connect)
apm.run()

Pair with a worker block like worker { exec python3; params hello.py; path /opt/myapp; server :8080; }.

from apm_module import ApmModule

apm = ApmModule(lambda s: s.close())  # no HTTP sessions here

def on_channel(channel, data, reply):
    # `reply` is None for fire-and-forget send(); a callable for request()
    if reply is None:
        print(f'broadcast on {channel}: {data}')
        return
    reply({'pong': data.get('ping'), 'from': 'python'})

apm.on_channel = on_channel
apm.run()

Configure the listener in the worker block: listen "ping_service";. A peer (any language) can then call request("ping_service", {ping: 42}) and receive {pong: 42, from: 'python'}.

from apm_module import ApmModule

apm = ApmModule(lambda s: s.close())

def main():
    stream = apm.request_stream('iot', {'device': 'sensor-7'}, 5000)
    if stream is None:
        print('no peer accepted on \'iot\'')
        return
    stream.on_data  = lambda data, peer=None: print('in: ', data)
    stream.on_close = lambda: print('closed')
    stream.write(b'ready')

# Kick off the request after the connector loop is running
import threading; threading.Timer(0.1, main).start()
apm.run()

PHP

PHP 7.4+, no extra extensions. Methods use camelCase following PSR convention. Handlers are assigned to onChannel / onStream properties on the instance.

<?php
require_once __DIR__ . '/apm_module.php';

$apm = new ApmModule(function (ApmSession $s) {
    $s->write('Hello from PHP', [
        'content-type' => 'text/plain',
        'x-status'    => '200',
    ]);
    $s->close();
});
$apm->run();

<?php
require_once __DIR__ . '/apm_module.php';

$apm = new ApmModule(function ($s) { $s->close(); });

$apm->onChannel = function ($channel, $data, $reply) {
    // $reply is null for fire-and-forget send(); callable for request()
    if ($reply === null) {
        error_log("broadcast on $channel");
        return;
    }
    $reply(['pong' => $data['ping'] ?? null, 'from' => 'php']);
};

$apm->run();

<?php
require_once __DIR__ . '/apm_module.php';

$apm = new ApmModule(function ($s) { $s->close(); });

// requestStream blocks until accepted/rejected/timeout; call after run() in
// a coroutine, or from a request handler. Returns IpcStream or null.
$apm->onChannel = function ($ch, $data, $reply) use ($apm) {
    if ($ch !== 'kick' || $reply === null) return;
    $stream = $apm->requestStream('iot', ['device' => 'sensor-7'], 5000);
    if ($stream === null) { $reply(['ok' => false]); return; }
    $stream->onData  = function ($data) { error_log("in: $data"); };
    $stream->onClose = function () { error_log('closed'); };
    $stream->write('ready');
    $reply(['ok' => true, 'stream' => $stream->id]);
};

$apm->run();

Perl

Perl 5.10+ with the JSON module (cpan install JSON or apt install libjson-perl). Object-oriented; handlers assigned via hash-element syntax ($apm->{on_channel} = sub { ... }).

use strict;
use warnings;
use ApmModule;

my $apm = ApmModule->new(sub {
    my ($s) = @_;
    $s->write('Hello from Perl', {
        'content-type' => 'text/plain',
        'x-status'    => '200',
    });
    $s->close;
});
$apm->run;

use strict; use warnings;
use ApmModule;

my $apm = ApmModule->new(sub { $_[0]->close });

$apm->{on_channel} = sub {
    my ($channel, $data, $reply) = @_;
    if (!defined $reply) {
        warn "broadcast on $channel\n";
        return;
    }
    $reply->({ pong => $data->{ping}, from => 'perl' });
};

$apm->run;

use strict; use warnings;
use ApmModule;

my $apm = ApmModule->new(sub { $_[0]->close });

$apm->{on_channel} = sub {
    my ($ch, $data, $reply) = @_;
    return unless $ch eq 'kick' && $reply;
    my $stream = $apm->request_stream('iot', { device => 'sensor-7' }, 5000);
    if (!$stream) { $reply->({ ok => 0 }); return; }
    $stream->{on_data}  = sub { warn "in: $_[0]\n" };
    $stream->{on_close} = sub { warn "closed\n" };
    $stream->write('ready');
    $reply->({ ok => 1, stream => $stream->{id} });
};

$apm->run;

Lua

Lua 5.3+ with lua-cjson (apt install lua-cjson or luarocks install lua-cjson). Method calls use : syntax (apm:run()); callbacks use . assignment (apm.on_channel = …).

local ApmModule = require('apm_module')

local apm = ApmModule.new(function(s)
    s:write('Hello from Lua', {
        ['content-type'] = 'text/plain',
        ['x-status']    = '200',
    })
    s:close()
end)
apm:run()

local ApmModule = require('apm_module')
local apm = ApmModule.new(function(s) s:close() end)

apm.on_channel = function(channel, data, reply)
    -- reply is nil for fire-and-forget send(); a function for request()
    if not reply then
        io.stderr:write('broadcast on ' .. channel .. '\n')
        return
    end
    reply({ pong = data.ping, from = 'lua' })
end

apm:run()

local ApmModule = require('apm_module')
local apm = ApmModule.new(function(s) s:close() end)

apm.on_channel = function(ch, data, reply)
    if ch ~= 'kick' or not reply then return end
    local stream = apm:request_stream('iot', { device = 'sensor-7' }, 5000)
    if not stream then reply({ ok = false }); return end
    stream.on_data  = function(d) io.stderr:write('in: ' .. d .. '\n') end
    stream.on_close = function()  io.stderr:write('closed\n') end
    stream:write('ready')
    reply({ ok = true, stream = stream.id })
end

apm:run()

Inter-worker IPC new in v2.0

Workers can now talk to each other through APM — across instances, across workers, and across languages. Two primitives: channels for stateless message passing, and streams for persistent bidirectional pipes. The daemon is the router; no extra sockets, no broker, no ports.

How it works

A worker declares a channel name with the listen config directive. The daemon maintains a registry mapping channel names → running workers. When any worker calls send(), request(), or requestStream(), the daemon routes the message to every running child of every worker listening on that channel. Workers never talk directly — all traffic flows over the existing stdin/stdout protocol that connectors already use, so no new dependencies and no new surface area.

            ┌────────────┐
            │ APM daemon │  ← channel registry & router
            └──┬──┬───┬───┘
               │  │   │
     ┌─────────┘  │   └─────────┐
     ▼            ▼             ▼
 ┌─────────┐ ┌─────────┐ ┌─────────┐
 │ sender  │ │ listener│ │ listener│
 └─────────┘ └─────────┘ └─────────┘

Why this matters

• Cross-language. A Python worker can request data from a Node.js worker and get a JSON reply. A PHP request handler can open a stream to a Go background worker. The connector API is identical across Node.js, Python, PHP, Perl, and Lua.
• Zero config. Add listen "channel_name"; to a worker block. That's it. The daemon rebuilds the registry automatically on every worker start/stop.
• Backwards compatible. Workers without listen behave exactly as before. The protocol is additive — old connectors are unaffected.
• Crash-safe. When any child dies, the daemon cleans up its streams and cancels its pending requests automatically. Writes to closed streams are silently dropped (no-scream policy).

Channels — fire-and-forget & request/reply

Channels are lightweight stateless messaging. A sender emits a message to a named channel; the daemon broadcasts it to every child of every worker listening on that channel.

send() — fire-and-forget

Broadcasts a message with no response expected. Returns immediately. If nobody is listening, the message is silently dropped.

// Push a device status update to everyone listening on "devices"
apm.send('devices', { deviceId: 'sensor-5', online: true })

// Worker config must contain:   listen "devices";
apm.onChannel = (channel, data) => {
  console.log('got', channel, data)
}

The sender is never echoed to itself, even if it also listens on the same channel. This keeps broadcasts loop-free.

request() — request / first-reply-wins

Sends a message and waits for the first reply from any listening child. Late replies are silently dropped. Returns null on timeout or when no listener is running.

// Ask any iot worker for the status of device sensor-5
const status = await apm.request('iot', { query: 'status', device: 'sensor-5' }, 2000)
if (status === null) { /* nobody answered in time */ }

apm.onChannel = (channel, data, reply) => {
  if (data.query === 'status') {
    reply({ temp: 42, online: true })
  }
}

Timeout priority: per-call timeout > worker ipc_timeout > default 500ms.

Streams — persistent bidirectional pipes

Streams are long-lived connections between workers. They follow a mediated star topology: one worker opens the stream (the mediator), others can attach as peers.

• Mediator writes fan out to all attached peers.
• Peer writes go to the mediator only, tagged with the peer ID so the mediator knows who spoke.

Think of it as a group trip organiser: the organiser can speak to everyone, but each participant only talks back to the organiser. This makes streams a natural fit for fan-out notification, shared live consoles, device control sessions, and worker-to-worker RPC sessions where one side coordinates many.

Typical flow

1. Mediator calls requestStream('channel', header). All listeners see a stream request.
2. One or more listeners call stream.accept(). Others can stream.reject() (or just ignore — the timeout handles "nobody accepted").
3. As soon as the first peer accepts, the mediator's Promise resolves with the stream object.
4. Both sides call stream.write(data) and handle stream.onData. Either side can call stream.close().
5. When the last peer detaches, the stream auto-closes. When any child dies, its streams are cleaned up automatically.

// Open a live console to whichever iot worker owns device sensor-5
const stream = await apm.requestStream('iot_console', { device: 'sensor-5' }, 5000)
if (!stream) { /* no peer accepted within timeout */ return }

stream.onData = (chunk, peer) => { console.log('from', peer, chunk.toString()) }
stream.onClose = () => { console.log('console closed') }
stream.write('help\n')

apm.onStream = (stream) => {
  // header contains whatever the mediator sent
  if (stream.header.device !== myDeviceId) { stream.reject(); return }

  stream.accept({ name: 'sensor-5' })
  stream.onData = (chunk) => { runCommand(chunk.toString()) }
  stream.onClose = () => { /* cleanup */ }
}

No-scream policy

Writes to a closed stream are silently dropped — no error, no crash. The onClose callback still fires normally. This prevents race conditions when both sides close simultaneously, which is very common in real traffic.

Config & connector API

Worker config fields

# IoT WebSocket server — listens for control commands
worker {
    name          iot_ws;
    exec          node;
    params        iot_server.js;
    server        ws://0.0.0.0:9100;
    instances     4;
    listen        "iot_control";
    ipc_timeout   1000;
}

# Web control panel — sends commands to IoT workers
worker {
    name          web_panel;
    exec          node;
    params        panel.js;
    server        http://0.0.0.0:8080;
    instances     2;
}

Cross-language API

Every connector exposes the same primitives. Method names are adapted to each language's conventions.

Node.js request and requestStream return Promises. All other languages block internally (pumping their event loop where applicable) until a reply arrives or the timeout fires, so they can be used from straight-line code without async plumbing.

Cross-language IPC walkthrough

This section is a worked end-to-end example with workers in two different languages talking through APM. Same wire protocol on both sides, no extra plumbing — the daemon is the router.

Scenario: Node.js HTTP front-end calls a Python pricing service

An HTTP request hits the Node.js worker; the handler asks a separately-running Python worker for the current price of a product via an IPC request; the Node.js worker returns the answer to the client. Each side runs as its own APM worker (so each can scale, restart, and rolling-deploy independently).

worker {
    name     web;
    exec     node;
    params   front.js;
    path     /opt/shop;
    server   :8080;
    instances 4;
    # no `listen` here — this worker is a *caller*, not a service
}

worker {
    name     pricing;
    exec     python3;
    params   pricing.py;
    path     /opt/shop;
    listen   pricing;        # become the receiver for `pricing` channel
    ipc_timeout 500;          # default reply window in ms
    instances 2;
}

The listen "pricing" directive on the Python worker registers its instances as receivers for the pricing channel. When the Node.js side calls request('pricing', …), APM picks one running Python instance and routes the message; the first reply wins.

from apm_module import ApmModule

PRICES = {'sku-1': 9.99, 'sku-2': 14.50, 'sku-3': 99.00}

apm = ApmModule(lambda s: s.close())  # no HTTP, IPC only

def on_channel(channel, data, reply):
    # channel == 'pricing'; data is whatever the caller sent
    sku = data.get('sku')
    price = PRICES.get(sku)
    if reply is not None:
        reply({'sku': sku, 'price': price, 'currency': 'GBP'})

apm.on_channel = on_channel
apm.run()

const ApmModule = require('./apm_module.node.js')

const apm = new ApmModule(async (session) => {
    const sku = session.query_object.sku || 'sku-1'

    // Cross-language IPC: send to the Python `pricing` service and await reply.
    const reply = await apm.request('pricing', { sku }, 500)

    if (!reply || reply.price == null) {
        session.write(JSON.stringify({ error: 'unknown sku' }), {
            'x-status': '404', 'content-type': 'application/json',
        })
    } else {
        session.write(JSON.stringify(reply), {
            'content-type': 'application/json',
        })
    }
    session.close()
})

$ apm load /etc/apm/apm.conf.d/shop.conf
$ curl 'http://localhost:8080/?sku=sku-2'
{"sku":"sku-2","price":14.5,"currency":"GBP"}

What happened: the Node.js instance handling the request issued a request('pricing', …, 500). APM looked up registered listeners for the pricing channel, picked one Python instance (round-robin across the 2 instances), and delivered the message. The Python instance's on_channel fired with a non-null reply, which it invoked with a dict; APM serialised it back to the Node.js caller as the resolved value of the await. Total round-trip is one stdin/stdout frame per direction — no sockets opened, no JSON-over-HTTP, no broker.

Scaling notes

• Pick-one routing for request: if you scale pricing to N instances, APM picks one per request. The first reply wins. Use this for stateless services where any instance can answer.
• Fan-out for send: apm.send('pricing', …) broadcasts to all listener instances and ignores replies. Use this for cache invalidation or background notifications.
• Streams for stateful pipes: if you need a long-lived connection between two specific workers (e.g. a Lua mediator coordinating between two PHP peers, or a Node.js dashboard collector pulling continuous data from a Python sampler), use request_stream / onStream instead — see the Streams section.
• Mix and match: the same Python service can be called from a Node.js front-end, a Perl batch job, and a PHP cron simultaneously — APM doesn't care what language the caller is. The wire format is identical.

The four other connectors (PHP / Perl / Lua / Node.js) can be substituted for either side of the example above with the per-language snippets from the connectors section — only the syntax changes, not the wiring.

Augur — startup-failure diagnostician new in v2.1

When a worker crashes inside the startup_grace window (default 2 s) on its first attempt, APM's augur module takes over. Augur does three things, in order:

1. Classifies the stderr by language. It recognises Node.js, Python, Perl, PHP, Ruby, Java, Go, and Bash failure patterns (MODULE_NOT_FOUND, ModuleNotFoundError, Can't locate ... in @INC, Class 'X' not found, LoadError, ClassNotFoundException, etc.) and emits a single-line hint telling the user what to do next.
2. Surfaces universal errno conditions: EADDRINUSE / "Address already in use", ECONNREFUSED / "Connection refused", EACCES / "Permission denied" — matched as both symbolic and English forms so they fire across every runtime.
3. Deep-scans the worker's dependency manifest and source to list every missing dependency in one go — so a freshly-migrated worker that's missing five npm packages tells you all five at once, instead of crash → install → restart → next crash → install → restart …

Languages covered

Reactive mode (default)

Augur runs automatically when a child exits with non-zero or a signal within the startup-grace window on its first attempt. No configuration required — the captured stderr is dumped to the requesting CLI, classifier hints follow, then the deep-scan adds the augur — also missing (N): … summary if a manifest scan found additional gaps.

Pre-flight mode (opt-in)

Set augur_full_scan true on a worker and augur runs the manifest + source scan before every fork — including watcher- and auto-restart-triggered ones. If anything is missing the launch is aborted with a clear message; the child binary never starts. This costs one or two cheap probe sub-processes per restart but eliminates the "child crashes, child restarts, crashes again" loop on broken environments.

$ apm restart Portal4
● Starting Portal4
 - Starting child #1
✗ Portal4#1 failed during startup (exit 1, 515 ms)
── child stderr ──────────────────────────────────────────────
Error: Cannot find module 'redis'
Require stack:
- /WEBZ/portal/v4/server.node.js
    ...
    code: 'MODULE_NOT_FOUND',
──────────────────────────────────────────────────────────────
  → missing node module redis — run npm install
  → augur — also missing (5): ws axios dotenv mysql2 sharp   → cd /WEBZ/portal/v4 && npm install ws axios dotenv mysql2 sharp
✗ Worker Portal4 failed to start — check logs

The deep-scan probes run as the worker's target user (same setuid + login-PATH mechanism as the worker launch itself), in the worker's path directory, with a 3-second timeout each. They never touch global system state and never write anything.

What's new in v2.1.0

Major — Augur startup-failure diagnostician

APM now reads worker crash output and tells you what to fix. When a child exits inside the startup-grace window on its first attempt, augur classifies the stderr by language (Node.js, Python, Perl, PHP, Ruby, Java, Go, Bash), surfaces POSIX errno conditions in plain English, and scans the worker's package.json / requirements.txt / composer.json / Gemfile.lock — or the source itself — to list every missing dependency in a single line. The migration scenario that previously took five restart cycles ("install redis, restart, install ws, restart, install mqtt, …") now takes one. See the augur section.

Opt-in pre-flight mode (augur_full_scan true) runs the scan before every fork, so the worker binary doesn't even attempt to start when the environment is broken.

Startup UX hardening (carried from v2.0.10)

Pre-flight validation of worker.path and the resolved exec binary; failed launches no longer stick in ◆ starting in apm list; apm restart with no arguments no longer crashes the daemon; first-8 KB of child stderr is captured and dumped to the requesting CLI when a fast-exit happens. New startup_grace knob controls the synchronous wait.

Reload-by-worker-name

apm reload <workerName> reloads just that worker from its origin conf file, without touching siblings declared in the same file. Useful when a multi-worker config holds several blocks and only one was edited.

Reload preserves log_time_format

A reload no longer silently wipes the per-line timestamp prefix when the conf block doesn't carry an explicit log_time_format — the daemon default is restored, matching the boot-time behaviour.

What's new in v2.0.0

Major — Inter-worker IPC

Workers can now communicate with each other through the daemon using two new primitives: channels (fire-and-forget and request/reply) and streams (persistent bidirectional pipes with mediated star topology). All routing happens inside the APM daemon over the existing stdin/stdout frame protocol — no new sockets, no new dependencies, no broker to run.

The feature is configured per worker with listen "channel_name"; and an optional ipc_timeout. All five official connectors (Node.js, Python, PHP, Perl, Lua) bumped to v3.0.0 with identical cross-language APIs. See the Inter-worker IPC section for the full guide.

Bug fix — listen / ipc_timeout dropped on initial config load

On first-time worker creation from a config file, addWorker translated config fields to CLI flags through workerParamMap, which was missing the two new IPC fields. Reloads picked them up correctly (they go through a separate path), but a cold boot never did. Both fields are now registered in the param map, so IPC works on first boot exactly as it does after a reload.

Improvement — apm info shows IPC config

apm info <worker> now displays an IPC section listing the configured listen channel and ipc_timeout (when set). This makes it easy to verify from the CLI that a worker is actually registered as a listener in the running daemon.

From v1.3.0 — still relevant

apm reload applies all worker fields live before the rolling restart, including watcher patterns, restart settings, TLS, session, and proxy flags. The file watcher is reopened in place when watch / watch_ignore / watch_delay change. watch_ignore with multiple patterns no longer silently dropped. Rapid file changes can no longer spawn duplicate WatcherRestart goroutines.