26 — Background Jobs¶

Overview¶

PyroRadius uses Laravel's queue system (Redis driver) for all asynchronous operations: WhatsApp message delivery, RADIUS synchronization for bulk operations, and campaign broadcasting. Long-running artisan commands handle scheduled maintenance tasks. This document covers the queue architecture, job classes, full command reference, DB-driven scheduler, and operational procedures.

Queue Architecture¶

Redis Driver¶

Laravel queues use Redis as the backend. The Redis connection is defined in config/database.php under the redis key, and config/queue.php sets QUEUE_CONNECTION=redis.

Redis queue keys follow the pattern queues:{queue_name}. Jobs are serialized to JSON and stored in Redis lists. Failed jobs are moved to the failed_jobs PostgreSQL table.

Named Queues¶

Queue Name	Purpose	Priority
`default`	General-purpose jobs, fallback	Normal
`radius`	RADIUS sync operations (SyncCustomerRadiusJob)	High
`whatsapp`	WhatsApp message delivery	Normal
`notifications`	In-app notifications	Low

Using named queues allows Supervisor workers to be allocated by priority. The radius queue worker can be given higher concurrency to keep RADIUS changes snappy.

Supervisor Configuration¶

Supervisor manages the queue worker processes and ensures they restart on failure. Configuration files are stored in /etc/supervisor/conf.d/.

Example worker configuration (/etc/supervisor/conf.d/pyroradius-worker.conf):

[program:pyroradius-worker-default]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/pyroradius/artisan queue:work redis --queue=default --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/log/supervisor/pyroradius-worker-default.log
stopwaitsecs=3600

[program:pyroradius-worker-radius]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/pyroradius/artisan queue:work redis --queue=radius --sleep=1 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=3
redirect_stderr=true
stdout_logfile=/var/log/supervisor/pyroradius-worker-radius.log
stopwaitsecs=3600

[program:pyroradius-worker-whatsapp]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/pyroradius/artisan queue:work redis --queue=whatsapp --sleep=3 --tries=2 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=1
redirect_stderr=true
stdout_logfile=/var/log/supervisor/pyroradius-worker-whatsapp.log
stopwaitsecs=3600

[program:pyroradius-worker-notifications]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/pyroradius/artisan queue:work redis --queue=notifications --sleep=5 --tries=3 --max-time=3600
autostart=true
autorestart=true
user=www-data
numprocs=1
redirect_stderr=true
stdout_logfile=/var/log/supervisor/pyroradius-worker-notifications.log
stopwaitsecs=3600

After editing Supervisor config:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start all

Queue Job Classes¶

SendWhatsAppNotificationJob¶

File: app/Jobs/SendWhatsAppNotificationJob.php

Queue: whatsapp

Payload:

[
    'to'       => '923001234567',     // E.164 mobile number
    'message'  => 'Your account...',  // Rendered template string
    'customer_id' => 42,              // Optional, for logging
]

Behavior:

Reads whatsapp_driver from settings.
If driver is log: writes to storage/logs/whatsapp.log.
If driver is openwa: POSTs to whatsapp_api_url/api/sendText with bearer token and payload.
On 4xx/5xx from OpenWA: throws exception, Laravel retries up to tries=2 times with exponential backoff.
On permanent failure: job is moved to failed_jobs table.

Retry policy: 2 attempts, backoff 60 seconds.

SendCampaignJob¶

File: app/Jobs/SendCampaignJob.php

Queue: whatsapp

Payload:

[
    'campaign_id' => 15,
    'batch_index' => 3,   // which chunk of recipients this job handles
]

Behavior:

Loads the campaign record (message template, filter criteria).
Loads the batch of recipient customers (pagination by batch_index).
For each recipient, dispatches a SendWhatsAppNotificationJob.
Updates campaign progress counter in the campaigns table.
Large campaigns are split into batches of 50 recipients per job to avoid timeout.

Rate limiting: A Redis rate limiter (campaign:{campaign_id}) is used to space out dispatches and avoid overwhelming the WhatsApp API.

SyncCustomerRadiusJob¶

File: app/Jobs/SyncCustomerRadiusJob.php

Queue: radius

Payload:

[
    'customer_id' => 123,
    'action'      => 'upsert',  // upsert | delete
]

Behavior:

Loads the customer with their active package and NAS.
Action upsert: writes/updates the customer's entry in radcheck, radreply, radusergroup tables. Sets bandwidth attributes based on package speed limits.
Action delete: removes all RADIUS entries for the customer (on expiry, archive, or deletion).
Updates customers.radius_synced_at timestamp.
On failure: retries 3 times. If all fail, logs to failed_jobs.

Important: This job directly writes to the FreeRADIUS database (radcheck, radreply, radusergroup). It uses the configured RADIUS DB connection (radius connection in config/database.php).

Artisan Command Reference¶

All commands live in app/Console/Commands/. They are registered in app/Console/Kernel.php.

#	Class Name	Artisan Signature	Description	Typical Schedule	Tables Affected
1	`BackupRun`	`backup:run`	Dumps PostgreSQL to a timestamped `.sql.gz` file and uploads to configured storage	Daily 02:00	None (reads all)
2	`CheckExpiry`	`customers:check-expiry`	Finds customers past expiry + grace period, sets `status=expired`, removes from `radcheck`/`radreply`/`radusergroup`	Every 30 min	customers, radcheck, radreply, radusergroup
3	`CloseGhostSessions`	`radius:close-ghost-sessions`	Finds `radacct` records with no `acctstoptime` older than threshold, inserts synthetic stop records	Every 15 min	radacct
4	`GenerateMonthlyInvoices`	`invoices:generate-monthly`	Creates invoices for all active customers who don't have an invoice for the current month	1st of month 00:05	invoices
5	`HealOverpaidInvoices`	`invoices:heal-overpaid`	Finds invoices where `paid_amount > amount`, corrects by adjusting `paid_amount` to `amount` and crediting excess to customer balance	Daily 01:00	invoices, customers
6	`KickExpiredOnline`	`radius:kick-expired-online`	Disconnects customers whose status is `expired` but who still have an active RADIUS session (sends PoD packet to NAS)	Every 10 min	radacct
7	`PollNasPorts`	`nas:poll-ports`	Reads port table from each NAS via SNMP/RouterOS API, stores in `nas_port_readings`	Every 5 min	nas_port_readings
8	`PollOltOnts`	`olt:poll-onts`	Reads ONT list from OLT device, updates `ont_cache` table with serial, status, rx_power	Every 5 min	ont_cache
9	`PollSignal`	`olt:poll-signal`	Reads `rx_power` from OLT for all ONTs and updates `ont_cache`	Every 5 min	ont_cache
10	`PollTraffic`	`nas:poll-traffic`	Reads bandwidth usage (in/out bytes) from NAS interfaces, stores in `traffic_readings`, caches per-NAS summary in Redis	Every 5 min	traffic_readings
11	`PruneAuditLogs`	`audit:prune`	Deletes audit log entries older than 30 days from `audit_logs`	Daily 03:00	audit_logs
12	`PruneNasHealth`	`nas:prune-health`	Removes old NAS health check records beyond retention window	Daily 03:15	nas_health_checks
13	`PruneNasPorts`	`nas:prune-ports`	Removes old port readings beyond retention window	Daily 03:30	nas_port_readings
14	`PruneRadpostauth`	`radius:prune-radpostauth`	Deletes old records from FreeRADIUS `radpostauth` table	Daily 02:30	radpostauth
15	`PruneTrafficReadings`	`nas:prune-traffic`	Removes traffic readings older than 7 days	Daily 03:45	traffic_readings
16	`RecordSysStats`	`health:record-sys-stats`	Reads CPU/RAM/disk and inserts into `sys_stats_history`	Every 5 min	sys_stats_history
17	`SyncAllRadius`	`radius:sync-all`	Bulk re-syncs RADIUS entries for all active customers (full rebuild)	Manual / Weekly Sunday 04:00	radcheck, radreply, radusergroup
18	`SyncBypassIps`	`nas:sync-bypass-ips`	Syncs `website_bypasses` table IP ranges to MikroTik firewall address lists	Every 15 min	website_bypasses
19	`SyncOntAssignments`	`olt:sync-ont-assignments`	Updates customer records with ONT serial + rx_power from `ont_cache`; only updates rows where data has changed (IS DISTINCT FROM guard)	Every 10 min	customers, ont_cache
20	`UpdateMacVendors`	`network:update-mac-vendors`	Downloads latest MAC OUI database and updates `mac_vendors` lookup table	Weekly Monday 05:00	mac_vendors

DB-Driven Scheduler¶

How It Works¶

PyroRadius does not hard-code all schedules in app/Console/Kernel.php. Instead, routes/console.php (or the Kernel's schedule() method) reads the cron_jobs table at every scheduler tick and dynamically registers enabled commands.

Conceptual implementation in app/Console/Kernel.php:

protected function schedule(Schedule $schedule): void
{
    $jobs = \App\Models\CronJob::where('enabled', true)->get();

    foreach ($jobs as $job) {
        $command = $schedule->command($job->command);

        $command->cron($job->expression);

        if ($job->without_overlap) {
            $command->withoutOverlapping();
        }

        if ($job->run_in_background) {
            $command->runInBackground();
        }

        if ($job->log_file) {
            $command->appendOutputTo(storage_path('logs/' . $job->log_file));
        }
    }
}

The system cron on the production server runs every minute:

* * * * * www-data php /var/www/pyroradius/artisan schedule:run >> /dev/null 2>&1

This means any change made to the cron_jobs table via the UI takes effect within 60 seconds.

withoutOverlapping¶

When without_overlap = true, Laravel acquires a cache lock before running the command. If the lock is already held (the previous run is still in progress), the scheduler skips this tick silently. This prevents double-runs for slow commands like SyncAllRadius or BackupRun.

Manually Triggering a Command¶

To run a command immediately outside the scheduler, SSH to the server and execute:

cd /var/www/pyroradius

# Run with default settings
php artisan customers:check-expiry

# Run with verbose output
php artisan radius:sync-all -v

# Run and tail the output
php artisan olt:sync-ont-assignments 2>&1 | tail -50

Commands can also be triggered from the Health UI (if a "Run Now" button is implemented) which fires a POST to CronJobController::runNow() which dispatches the command via Artisan::call() in a queued job to avoid blocking the HTTP response.

Checking Command Logs¶

Each cron job can have a log_file configured (e.g., scheduler/check-expiry.log). Laravel appends command output (stdout + stderr) to this file.

To tail a log:

tail -f /var/www/pyroradius/storage/logs/scheduler/check-expiry.log

For commands without a log_file, output goes to /dev/null by default. To debug such a command, run it manually in the terminal as shown above.

The Health UI displays the last N lines of configured log files in the Recent Command Runs section via an API endpoint that reads the file.

Queue Worker Restart Procedure After Deploy¶

Queue workers cache PHP class definitions in memory. After deploying any PHP file change, workers must be restarted to pick up the new code.

After PHP Changes (controllers, jobs, commands, models):¶

# 1. Clear and rebuild the cache
php artisan optimize:clear
php artisan optimize

# 2. Restart PHP-FPM
systemctl restart php8.3-fpm

# 3. Restart all Supervisor workers
sudo supervisorctl restart all

After JS-only Changes (React components, Vite assets):¶

# 1. Build assets
npm run build

# 2. Restart PHP-FPM (for view cache busting)
systemctl restart php8.3-fpm

# Workers do not need restart for JS-only changes

Graceful Restart (zero-downtime)¶

If you need a graceful restart (workers finish their current job before stopping):

php artisan queue:restart

This sends a signal via the cache. Workers will finish their current job and then exit. Supervisor auto-restarts them. All queued jobs remain in Redis — no jobs are lost.

Checking and Retrying Failed Jobs¶

# List failed jobs
php artisan queue:failed

# Retry a specific failed job by UUID
php artisan queue:retry <uuid>

# Retry all failed jobs
php artisan queue:retry all

# Delete a failed job
php artisan queue:forget <uuid>

# Clear all failed jobs
php artisan queue:flush

The failed_jobs table in PostgreSQL stores:

Column	Description
`uuid`	Unique job identifier
`connection`	Queue connection (redis)
`queue`	Queue name
`payload`	Full serialized job JSON
`exception`	Full stack trace of the failure
`failed_at`	Timestamp

When investigating a failed job, the exception column contains the full PHP stack trace. Most common causes: OpenWA API timeout (WhatsApp), RADIUS DB connection timeout (radius jobs), OLT API changed response format (poll jobs).