32 — Troubleshooting¶

Quick Diagnostic Reference¶

Before diving into specific issues, collect this information first:

# Check for recent errors
tail -n 100 /var/www/pyroradius/storage/logs/laravel.log

# Check PHP-FPM health
systemctl status php8.3-fpm

# Check Nginx
systemctl status nginx

# Check Redis
redis-cli ping

# Check queue workers
supervisorctl status all

# Check PostgreSQL
systemctl status postgresql

# Check RADIUS tables reachable
psql -U pyroradius -d radius -c "SELECT COUNT(*) FROM radcheck;"

AUTHENTICATION¶

Symptoms: Login page loads but submitting credentials fails, session error, or redirected back to login after successful login.

Causes and fixes:

1. Redis is down (session driver issue)

redis-cli ping
# If no response or "Connection refused":
systemctl restart redis
systemctl status redis

PyroRadius uses Redis as the session driver. If Redis is down, no session can be stored and login fails silently.

2. PHP-FPM not running

systemctl status php8.3-fpm
# If stopped:
systemctl restart php8.3-fpm

3. Laravel session cache corrupt

cd /var/www/pyroradius
php artisan cache:clear
php artisan session:clear   # if this command exists, otherwise:
# Clear redis sessions manually:
redis-cli FLUSHDB   # WARNING: clears all redis data, including queues — use carefully

A safer approach is to clear only the session prefix:

redis-cli KEYS "laravel_session:*" | xargs redis-cli DEL

4. APP_KEY changed

If APP_KEY in .env was changed, all existing encrypted sessions are invalid. Users must log in again. This is expected behavior and not a bug.

5. Wrong credentials

Confirm the user exists and the password is correct:

cd /var/www/pyroradius
php artisan tinker
>>> \App\Models\User::where('email', 'admin@pyronet.com.pk')->first()
# Check the record exists and is_active = true

Session Expires Too Fast¶

Symptom: Users are logged out after a short period of inactivity.

Fix: Increase SESSION_LIFETIME in .env:

nano /var/www/pyroradius/.env
# Change SESSION_LIFETIME=120 to SESSION_LIFETIME=480  (minutes)

php artisan optimize:clear
php artisan optimize
systemctl restart php8.3-fpm

BILLING¶

Expiry Date Not Updating After Save (FIXED 2026-07-13)¶

Historical issue: When editing a customer's expiry date, the save appeared to succeed but the expiry date reverted. Root cause was SyncOntAssignments running concurrently, acquiring a lock on the customers table row, causing a lock timeout (PostgreSQL error 55P03), and then the update silently failing or being retried with stale data.

Fix applied 2026-07-13: - SyncOntAssignments now uses IS DISTINCT FROM guard — it only writes when the value actually changed, reducing lock duration - Functional indexes added to ont_assignments to speed up the sync query - Retry loop added in CustomerController@update to handle transient lock timeouts

If this symptom recurs:

# Check for lock waits in PostgreSQL
psql -U pyroradius -d pyroradius -c "
  SELECT pid, wait_event_type, wait_event, state, query
  FROM pg_stat_activity
  WHERE wait_event_type = 'Lock'
  ORDER BY query_start;
"

# Check SyncOntAssignments is not running too frequently
# Check routes/console.php for the cron schedule
grep -n "SyncOntAssignments" /var/www/pyroradius/routes/console.php

Invoice Duplicate Issue¶

Symptom: Two invoices exist for the same customer and same billing period.

Expected behavior: Invoice generation is idempotent — calling generateInvoice for the same customer and period must always return the same invoice, never create a duplicate.

Check the guard:

psql -U pyroradius -d pyroradius -c "
  SELECT customer_id, period, COUNT(*) as cnt
  FROM invoices
  GROUP BY customer_id, period
  HAVING COUNT(*) > 1
  ORDER BY cnt DESC
  LIMIT 20;
"

If duplicates exist, check BillingService@generateInvoice — the idempotency guard uses firstOrCreate on (customer_id, period). If a unique constraint is missing from the database, add it:

ALTER TABLE invoices ADD CONSTRAINT invoices_customer_period_unique UNIQUE (customer_id, period);

Payment Not Applying to Invoice¶

Symptom: Payment is recorded but the invoice still shows as unpaid.

Diagnostic:

cd /var/www/pyroradius
php artisan tinker
>>> $customer = \App\Models\Customer::find(<id>);
>>> $customer->invoices()->where('paid', false)->orderBy('created_at')->get(['id','period','amount','paid']);
>>> $customer->payments()->latest()->first();

Common causes:

Payment amount less than invoice amount — partial payment does not mark invoice as paid. Check if partial payment logic is intended.
Invoice period mismatch — payment applied to wrong period. Check the period field on both payment and invoice.
Queue job failed — if payment triggers a queue job to apply to invoices, the job may have failed silently. Check:

tail -n 100 /var/www/pyroradius/storage/logs/laravel.log | grep -i "payment\|invoice\|billing"
supervisorctl status all  # workers must be running

RADIUS¶

Customer Can Authenticate but No Speed Limit Applied¶

Symptom: Customer connects to PPPoE but gets unrestricted bandwidth (no shaping), or a wrong speed profile is applied.

Cause: radgroupreply table is missing the speed attributes for the customer's group, or radusergroup entry is absent.

Diagnose:

psql -U pyroradius -d radius -c "
  SELECT * FROM radusergroup WHERE username = 'pppoe_username_here';
"
psql -U pyroradius -d radius -c "
  SELECT * FROM radgroupreply WHERE groupname = '<group_from_above>';
"

Fix: Run the full RADIUS sync command:

cd /var/www/pyroradius
php artisan sync:all-radius

This rewrites all radcheck, radreply, radgroupreply, and radusergroup entries for all active customers.

Symptom: Customer's router shows "Authentication failed" or "Access denied."

Diagnose:

# Check if username exists in radcheck
psql -U pyroradius -d radius -c "
  SELECT * FROM radcheck WHERE username = 'pppoe_username_here';
"

Expected: Two rows — one for Auth-Type := Accept (or similar) and one for Cleartext-Password.

If missing:

cd /var/www/pyroradius
php artisan tinker
>>> $customer = \App\Models\Customer::where('pppoe_username', 'pppoe_username_here')->first();
>>> app(\App\Services\RadiusService::class)->syncCustomer($customer);

If row exists but password is wrong: PPPoE password must be stored as plain text in radcheck. Never hash it. Check:

psql -U pyroradius -d pyroradius -c "
  SELECT pppoe_username, pppoe_password FROM customers WHERE pppoe_username = 'pppoe_username_here';
"

Session Stuck as Online (Ghost Session)¶

Symptom: Customer disconnected physically but still shows as "online" in the system.

Fix: Run the ghost session cleanup command:

cd /var/www/pyroradius
php artisan radius:close-ghost-sessions

Or manually remove from radacct:

psql -U pyroradius -d radius -c "
  UPDATE radacct
  SET acctstoptime = NOW(), acctterminatecause = 'Admin-Reset'
  WHERE username = 'pppoe_username_here' AND acctstoptime IS NULL;
"

MIKROTIK¶

Live Traffic Not Showing (Dashboard or Customer Detail)¶

Symptom: The live RX/TX traffic widget shows "N/A", 0, or does not update.

Causes:

1. RouterOS API credentials wrong

# Check .env
grep -E "MIKROTIK|NAS" /var/www/pyroradius/.env

# Test API connection via tinker
cd /var/www/pyroradius
php artisan tinker
>>> app(\App\Services\MikroTikService::class)->testConnection();

2. RouterOS API port blocked

MikroTik API uses port 8728 (plain) or 8729 (SSL). Check firewall rules on both VPS and MikroTik:

# From VPS, test connectivity:
nc -zv <mikrotik_ip> 8728

3. SNMP fallback not working

If RouterOS API fails, the system should fall back to SNMP. Check SNMP community string:

grep SNMP /var/www/pyroradius/.env

4. NAS record misconfigured

The NAS entry in the database must have correct api_user and api_password. Check in the admin UI: Settings → NAS Devices.

Customer Disconnect Not Working (MikroTik)¶

Symptom: Clicking "Disconnect" in the customer profile does not actually disconnect the PPPoE session on the router.

Diagnose:

cd /var/www/pyroradius
php artisan tinker
>>> app(\App\Services\MikroTikService::class)->disconnectPppoe('pppoe_username_here');

If this throws an exception, the issue is API connectivity. Check:

MikroTik API is enabled on the router (IP → Services → api should be enabled)
The API user has sufficient permissions on RouterOS
The VPS IP is not blocked by MikroTik firewall
The api_password stored in NAS table is correct

OLT¶

rx_power (Signal Level) Not Updating¶

Symptom: Customer's ONT signal level (rx_power) shows old value or NULL.

Cause: SyncOntAssignments cron is not running, or PollOltOnts is disabled.

Check:

# Check if scheduled commands are running
cd /var/www/pyroradius
php artisan tinker
>>> \Illuminate\Support\Facades\DB::table('scheduled_tasks_log')
...    ->orderBy('created_at', 'desc')
...    ->limit(10)
...    ->get(['command', 'ran_at', 'status']);

Or check cron/schedule directly:

# If using system cron:
crontab -l

# The scheduler entry should exist:
# * * * * * cd /var/www/pyroradius && php artisan schedule:run >> /dev/null 2>&1

Manual poll:

php artisan sync:ont-assignments
php artisan poll:olt-onts

WHATSAPP¶

Messages Not Sending¶

Symptom: WhatsApp messages (recharge confirmation, expiry reminder, etc.) are not delivered.

Diagnose in order:

1. WhatsApp is disabled in settings

psql -U pyroradius -d pyroradius -c "
  SELECT value FROM settings WHERE key = 'whatsapp_enabled';
"

Must be true. Change via admin UI: Settings → WhatsApp.

2. WhatsApp session not connected

psql -U pyroradius -d pyroradius -c "
  SELECT value FROM settings WHERE key = 'whatsapp_session_status';
"

Must be ready. If not, reconnect the WhatsApp session via admin UI.

3. LOG_CHANNEL is set to null or stack in testing

Ensure .env does not have WHATSAPP_ENABLED=false or LOG_CHANNEL=null in production.

4. Queue worker not running

WhatsApp messages are dispatched via queue. Workers must be running:

supervisorctl status all

5. Check logs for WhatsApp errors

grep -i "whatsapp\|WA" /var/www/pyroradius/storage/logs/laravel.log | tail -50

QUEUE¶

Jobs Not Processing (Queue Stuck)¶

Symptom: Actions complete on screen but background work (RADIUS sync, WhatsApp, invoice generation) does not happen.

Diagnose:

# Check workers
supervisorctl status all

# Check Redis is alive
redis-cli ping

# Check queue length
redis-cli LLEN queues:default

# Run a worker manually with verbose output to see what's failing
cd /var/www/pyroradius
php artisan queue:work --verbose --tries=1

Fix:

# Restart Supervisor workers
supervisorctl restart all

# If workers keep dying, check the worker log
tail -n 100 /var/www/pyroradius/storage/logs/worker.log

# Check for failed jobs
php artisan queue:failed
# Retry failed jobs:
php artisan queue:retry all
# Flush failed jobs (discard them):
php artisan queue:flush

DATABASE¶

Lock Timeout (55P03)¶

Symptom: Error in laravel.log: ERROR: canceling statement due to lock timeout or PostgreSQL error code 55P03.

Cause: Two processes are writing to the same row simultaneously. Most commonly seen when SyncOntAssignments runs while a user edits a customer.

Short-term fix: Retry the operation. The CustomerController already has a retry loop for this.

Long-term fix: - Reduce SyncOntAssignments concurrency (ensure it does not run overlapping instances) - Add IS DISTINCT FROM guards to skip unnecessary writes - Add functional indexes on commonly locked tables

Diagnostic query:

psql -U pyroradius -d pyroradius -c "
  SELECT pid, usename, state, wait_event_type, wait_event, left(query, 80) as query_snippet
  FROM pg_stat_activity
  WHERE state != 'idle'
  ORDER BY query_start;
"

Deadlock (40P01)¶

Symptom: ERROR: deadlock detected, PostgreSQL error code 40P01.

Cause: Two concurrent transactions each hold a lock that the other needs.

Fix: Retry logic is implemented in BillingService. If a deadlock is detected, the service retries the transaction up to 3 times.

If deadlocks are frequent, review the transaction order in BillingService — all transactions should acquire locks in the same order (e.g., always lock customers before invoices, never the reverse).

PERFORMANCE¶

Dashboard Loading Slowly¶

Symptom: Dashboard takes more than 3 seconds to load.

Causes:

1. Redis online count cache expired or missing

The "Online Now" count is cached in Redis. If the cache is cold and the MikroTik query is slow, the page blocks.

redis-cli KEYS "online_count*"
# If empty, the cache is cold. The next page load will regenerate it.

# Check RouterOsService timeout setting in .env
grep ROUTER_OS_TIMEOUT /var/www/pyroradius/.env

2. MikroTik API timeout

RouterOsService queries MikroTik for the online count. If MikroTik is unreachable, the service should time out gracefully and return a cached value or 0.

3. PostgreSQL slow query

Enable query logging temporarily:

psql -U pyroradius -d pyroradius -c "
  SELECT query, calls, total_time/calls as avg_ms, total_time
  FROM pg_stat_statements
  ORDER BY total_time DESC
  LIMIT 20;
"

Customer List Loading Slowly¶

Symptom: Customers index page takes more than 2 seconds.

Check indexes:

psql -U pyroradius -d pyroradius -c "
  SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'customers' ORDER BY indexname;
"

Critical indexes that must exist:

-- dealer_id + status composite index (most common filter combination)
CREATE INDEX IF NOT EXISTS idx_customers_dealer_id_status ON customers (dealer_id, status);

-- sub_dealer_id index
CREATE INDEX IF NOT EXISTS idx_customers_sub_dealer_id ON customers (sub_dealer_id);

-- expiry_date index (for expiring soon queries)
CREATE INDEX IF NOT EXISTS idx_customers_expiry_date ON customers (expiry_date);

-- pppoe_username index (for RADIUS lookup)
CREATE INDEX IF NOT EXISTS idx_customers_pppoe_username ON customers (pppoe_username);

Diagnostic Commands Reference¶

Laravel Log¶

# Live tail
tail -f /var/www/pyroradius/storage/logs/laravel.log

# Last 100 lines
tail -n 100 /var/www/pyroradius/storage/logs/laravel.log

# Filter by level
grep "ERROR\|CRITICAL" /var/www/pyroradius/storage/logs/laravel.log | tail -50

# Filter by keyword
grep -i "billing\|radius\|whatsapp" /var/www/pyroradius/storage/logs/laravel.log | tail -50

Queue Status¶

# Workers
supervisorctl status all

# Queue length in Redis
redis-cli LLEN queues:default
redis-cli LLEN queues:high

# Failed jobs
cd /var/www/pyroradius && php artisan queue:failed

# Run one job manually
cd /var/www/pyroradius && php artisan queue:work --once --verbose

RADIUS Tables¶

# Count entries
psql -U pyroradius -d radius -c "SELECT 'radcheck', COUNT(*) FROM radcheck UNION ALL SELECT 'radreply', COUNT(*) FROM radreply UNION ALL SELECT 'radusergroup', COUNT(*) FROM radusergroup UNION ALL SELECT 'radgroupreply', COUNT(*) FROM radgroupreply;"

# Look up a specific customer
psql -U pyroradius -d radius -c "SELECT * FROM radcheck WHERE username = 'pppoe_username_here';"
psql -U pyroradius -d radius -c "SELECT * FROM radusergroup WHERE username = 'pppoe_username_here';"

# Active sessions
psql -U pyroradius -d radius -c "SELECT username, nasipaddress, acctstarttime FROM radacct WHERE acctstoptime IS NULL ORDER BY acctstarttime DESC LIMIT 20;"

PostgreSQL Diagnostics¶

# Active connections
psql -U pyroradius -d pyroradius -c "SELECT count(*) FROM pg_stat_activity WHERE state != 'idle';"

# Blocking queries
psql -U pyroradius -d pyroradius -c "
  SELECT blocked_locks.pid AS blocked_pid,
         blocking_locks.pid AS blocking_pid,
         blocked_activity.query AS blocked_statement,
         blocking_activity.query AS blocking_statement
  FROM pg_catalog.pg_locks blocked_locks
  JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
  JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype
    AND blocking_locks.pid != blocked_locks.pid
    AND blocking_locks.granted
  JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
  WHERE NOT blocked_locks.granted;
"

# Table sizes
psql -U pyroradius -d pyroradius -c "
  SELECT relname AS table, pg_size_pretty(pg_total_relation_size(relid)) AS size
  FROM pg_catalog.pg_statio_user_tables
  ORDER BY pg_total_relation_size(relid) DESC;
"

Tinker (Laravel REPL)¶

cd /var/www/pyroradius
php artisan tinker

# Example: find a customer
>>> \App\Models\Customer::where('pppoe_username', 'testuser')->first()

# Example: check settings
>>> \App\Models\Setting::where('key', 'whatsapp_enabled')->value('value')

# Example: manually trigger RADIUS sync for one customer
>>> $c = \App\Models\Customer::find(1); app(\App\Services\RadiusService::class)->syncCustomer($c);

# Example: check a user's permissions
>>> \App\Models\User::where('email', 'dealer@example.com')->value('permissions')

32 — Troubleshooting¶

Quick Diagnostic Reference¶

AUTHENTICATION¶

Login Not Working (page shows error or infinite redirect)¶

Session Expires Too Fast¶

BILLING¶

Expiry Date Not Updating After Save (FIXED 2026-07-13)¶

Invoice Duplicate Issue¶

Payment Not Applying to Invoice¶

RADIUS¶

Customer Can Authenticate but No Speed Limit Applied¶

Customer Cannot Authenticate (PPPoE Login Fails)¶

Session Stuck as Online (Ghost Session)¶

MIKROTIK¶

Live Traffic Not Showing (Dashboard or Customer Detail)¶

Customer Disconnect Not Working (MikroTik)¶

OLT¶

rx_power (Signal Level) Not Updating¶

WHATSAPP¶

Messages Not Sending¶

QUEUE¶

Jobs Not Processing (Queue Stuck)¶

DATABASE¶

Lock Timeout (55P03)¶

Deadlock (40P01)¶

PERFORMANCE¶

Dashboard Loading Slowly¶

Customer List Loading Slowly¶

Diagnostic Commands Reference¶

Laravel Log¶

Queue Status¶

RADIUS Tables¶

PostgreSQL Diagnostics¶

Tinker (Laravel REPL)¶