24 — Health System

Overview

The Health System is PyroRadius's internal operations dashboard. It provides a real-time and historical view of server health, queue worker status, RADIUS connectivity, NAS device status, cron job management, and recent artisan command execution logs. It is the primary tool for diagnosing production issues without needing SSH access.

---

What the Health System Monitors

1. Server Statistics (sys_stats_history)

Recorded by the RecordSysStats artisan command on a scheduled interval (every 5 minutes via the DB-driven scheduler). Stores time-series snapshots of:

Column	Description
cpu_percent	CPU utilization as a percentage (0–100)
ram_percent	RAM usage as a percentage of total RAM
disk_percent	Disk usage on the root partition
recorded_at	Timestamp of the reading

The Health/Index.jsx page renders these as line charts (Recharts) showing the last 24 hours or configurable window. Alerts are highlighted if any value exceeds thresholds (CPU > 85%, RAM > 90%, disk > 80%).

2. Queue Worker Status

PyroRadius uses Redis as the queue driver with multiple named queues managed by Supervisor. The health page polls worker status by checking:

• Supervisor process state via shell command (supervisorctl status)
• Number of jobs pending in each Redis queue (LLEN on the queue key)
• Failed job count from the failed_jobs table

Workers monitored:

Worker Name	Queue	Purpose
pyroradius-worker-default	default	General background jobs
pyroradius-worker-radius	radius	RADIUS sync operations
pyroradius-worker-whatsapp	whatsapp	WhatsApp message delivery
pyroradius-worker-notifications	notifications	In-app notification delivery

3. RADIUS Connectivity

The health page checks FreeRADIUS availability by attempting a test authentication (using the NAS shared secret from settings). A successful response means RADIUS is accepting packets. A timeout or reject is flagged as a connectivity failure. The check is passive — it does not consume a real session.

4. NAS Device Status

Each NAS device (MikroTik/OLT) has a last-seen timestamp updated whenever PollTraffic, PollNasPorts, or PollOltOnts communicates with it successfully. The health page shows:

• NAS name and IP
• Last successful poll timestamp
• Port count (from nas_port_readings)
• Reachability status (green/red badge based on age of last-seen)

A NAS is considered unreachable if its last successful poll is older than 10 minutes.

5. Cron Job Status

Every scheduled command is stored in the cron_jobs table. The health page lists all cron jobs with:

• Command class name
• Cron expression (human-readable)
• Enabled/disabled toggle
• Last run timestamp
• Last run result (success/failure)
• Log file path link

6. Recent Command Runs

The health page shows a log of the 50 most recent artisan command executions, including:

• Command name
• Triggered at timestamp
• Duration (ms)
• Exit status (0 = success, non-zero = failure)
• Truncated output or link to log file

---

HealthController

File: app/Http/Controllers/HealthController.php

Responsibilities:

• index() — Returns the Health/Index.jsx page with initial props (worker status, recent stats, NAS summary, cron job list, recent runs)
• sysStats() — API endpoint returning sys_stats_history for the chart (last N hours, defaults to 24)
• workerStatus() — Calls supervisorctl status via Process class, parses output, returns structured array
• queueDepths() — Uses Redis LLEN to return pending job counts per queue
• failedJobs() — Returns count and latest 10 failed jobs from failed_jobs table
• nasStatus() — Returns all NAS devices with last-seen age and port count
• radiusCheck() — Performs test RADIUS auth, returns up/down
• recentRuns() — Returns last 50 rows from command_runs table (or similar log table)

Authorization: Only super_admin or users with health.view permission can access any health route.

---

CronJobController

File: app/Http/Controllers/CronJobController.php

Provides full CRUD for the DB-driven scheduler. All actions are restricted to super_admin.

Method	Route	Description
index()	GET /cron-jobs	List all cron jobs
store()	POST /cron-jobs	Create a new cron job
update()	PUT /cron-jobs/{id}	Edit command, expression, flags
destroy()	DELETE /cron-jobs/{id}	Delete a cron job permanently
toggle()	PATCH /cron-jobs/{id}/toggle	Enable or disable without editing

Validation rules on store and update:

command      — required, string, must exist in a whitelist of known command classes
expression   — required, valid cron expression (5-part or Laravel macro like @hourly)
enabled      — boolean
without_overlap — boolean
log_file     — nullable, string, path relative to storage/logs/
run_in_background — boolean

The controller does not execute commands directly. It only manages the cron_jobs table. The scheduler reads this table on every scheduler tick and registers matching commands dynamically.

---

cron_jobs Table Schema

CREATE TABLE cron_jobs (
    id              BIGSERIAL PRIMARY KEY,
    command         VARCHAR(255) NOT NULL,        -- e.g. App\Console\Commands\RecordSysStats
    expression      VARCHAR(100) NOT NULL,        -- e.g. */5 * * * * or @hourly
    enabled         BOOLEAN NOT NULL DEFAULT true,
    without_overlap BOOLEAN NOT NULL DEFAULT true,
    log_file        VARCHAR(255) NULL,            -- e.g. scheduler/record-sys-stats.log
    run_in_background BOOLEAN NOT NULL DEFAULT false,
    last_run_at     TIMESTAMP NULL,
    last_run_status VARCHAR(20) NULL,             -- success | failed | running
    created_at      TIMESTAMP,
    updated_at      TIMESTAMP
);

The routes/console.php Schedule facade reads all rows where enabled = true at boot time and registers them as scheduled commands with the appropriate options (withoutOverlapping, runInBackground, appendOutputTo).

---

How to Add / Edit / Disable a Cron Job from the UI

Adding a New Cron Job

1. Go to Health → Cron Jobs in the sidebar.
2. Click Add Job.
3. Select the command class from the dropdown (only whitelisted commands appear).
4. Enter the cron expression (e.g., */5 * * * * for every 5 minutes, 0 * * * * for hourly).
5. Set Without Overlap to true if the command should not run if a previous instance is still running (recommended for most commands).
6. Optionally set a Log File path under storage/logs/. The scheduler will append command output here.
7. Click Save. The job takes effect on the next scheduler tick (within 1 minute, since the system cron runs php artisan schedule:run every minute).

Editing a Cron Job

1. Click the pencil icon next to the job in the list.
2. Modify the expression, overlap setting, or log file.
3. Save. The scheduler will pick up the change on the next tick.

Disabling a Cron Job

1. Use the toggle switch in the cron job row.
2. The job is immediately disabled — the scheduler skips it without deleting the record.
3. Re-enable by toggling again.

Deleting a Cron Job

Click the trash icon and confirm. This permanently removes the row. Use disable instead of delete if you may want to re-enable later.

---

sys_stats_history Table

CREATE TABLE sys_stats_history (
    id           BIGSERIAL PRIMARY KEY,
    cpu_percent  DECIMAL(5,2) NOT NULL,
    ram_percent  DECIMAL(5,2) NOT NULL,
    disk_percent DECIMAL(5,2) NOT NULL,
    recorded_at  TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_sys_stats_recorded_at ON sys_stats_history (recorded_at DESC);

Old records are pruned by the PruneSysStats command (or similar) to keep only the last 7 days. The index on recorded_at makes time-range queries fast even with millions of rows.

---

nas_port_readings Table

Populated by PollNasPorts. Schema:

CREATE TABLE nas_port_readings (
    id          BIGSERIAL PRIMARY KEY,
    nas_id      BIGINT NOT NULL REFERENCES nas(id),
    port_name   VARCHAR(100) NOT NULL,
    status      VARCHAR(20) NOT NULL,   -- up | down | unknown
    speed       INTEGER NULL,           -- Mbps
    polled_at   TIMESTAMP NOT NULL DEFAULT NOW()
);

The Health page aggregates port readings per NAS to show total ports, up count, and down count.

---

Health/Index.jsx Page Sections

The Health page is organized into collapsible card sections:

1. Server Stats — CPU / RAM / Disk gauges (current) and line charts (24h history). Recharts LineChart with ResponsiveContainer.
2. Queue Workers — Table showing each Supervisor worker: name, status (RUNNING / STOPPED / FATAL), uptime, pending jobs in queue, failed job count. Red badge if any worker is not RUNNING.
3. RADIUS Status — Single green/red indicator with last check timestamp and latency (ms).
4. NAS Devices — Cards for each NAS: name, IP, last-seen, reachability badge, port up/down summary.
5. Cron Jobs — Full CRUD table with enable/disable toggle per row. Inline edit via modal.
6. Recent Command Runs — Scrollable table of the last 50 runs with exit status badges and duration.

---

RecordSysStats Command

Class: App\Console\Commands\RecordSysStats

Signature: health:record-sys-stats

What it does:

1. Reads /proc/stat (Linux) for CPU usage delta since last poll.
2. Reads /proc/meminfo for RAM total and available.
3. Uses disk_free_space('/') and disk_total_space('/') for disk.
4. Inserts one row into sys_stats_history.

Typical schedule: every 5 minutes (*/5 * * * *).

---

Checking if Queue Workers Are Running

Via Health UI

The Health page worker section shows each worker's Supervisor status. A green badge means RUNNING. Any other state (STOPPED, FATAL, EXITED) is shown in red with the last exit code.

Via SSH

sudo supervisorctl status

Expected output:

pyroradius-worker-default      RUNNING   pid 12345, uptime 2 days, 4:32:01
pyroradius-worker-radius       RUNNING   pid 12346, uptime 2 days, 4:32:01
pyroradius-worker-whatsapp     RUNNING   pid 12347, uptime 2 days, 4:32:01
pyroradius-worker-notifications RUNNING  pid 12348, uptime 2 days, 4:32:01

Check failed jobs

cd /var/www/pyroradius
php artisan queue:failed

---

What to Do If a Worker Is Down

Step 1 — Restart via Supervisor

sudo supervisorctl restart pyroradius-worker-default
# or restart all
sudo supervisorctl restart all

Step 2 — Check Supervisor logs

sudo tail -100 /var/log/supervisor/pyroradius-worker-default.log

Look for OOM kills, PHP fatal errors, or uncaught exceptions that caused the worker to exit.

Step 3 — Check Laravel logs

sudo tail -200 /var/www/pyroradius/storage/logs/laravel.log

Step 4 — Check failed jobs

php artisan queue:failed
php artisan queue:retry all   # retry all failed jobs
# or retry specific job
php artisan queue:retry <uuid>

Step 5 — After a code deploy, always restart workers

Queue workers cache class definitions in memory. After deploying PHP changes, run:

sudo supervisorctl restart all

Or use the deploy sequence:

php artisan optimize:clear && php artisan optimize && systemctl restart php8.3-fpm
sudo supervisorctl restart all

---

Common Health Issues and Diagnosis

Symptom	Likely Cause	Fix
WhatsApp messages not sending	whatsapp worker STOPPED	supervisorctl restart pyroradius-worker-whatsapp
RADIUS sync not applying	radius worker FATAL	Check worker log, restart, retry failed jobs
Disk > 90%	Audit logs or logs not pruned	Run PruneAuditLogs, check /var/www/pyroradius/storage/logs/
CPU spike in chart	Bulk sync or import running	Check recent command runs, wait for completion
Cron job not running	enabled=false or expression wrong	Edit in Cron Jobs UI, verify expression
NAS shows unreachable	Network issue or SNMP/API down	SSH to NAS, check connectivity from production server