Files
goclaw/cmd
Luan Vu 50871eaaae fix(cron): prevent scheduler loop from blocking when a job hangs (#820)
* fix(cron): prevent scheduler loop from blocking when a job hangs

The cron scheduler's runLoop calls checkAndRunDueJobs() every second,
which previously used wg.Wait() to block until ALL claimed jobs complete.
If any single job hung (LLM provider timeout, agent loop stuck, network
issue), wg.Wait() would block indefinitely, preventing the scheduler
from ever checking for new due jobs — effectively killing all cron
scheduling until a container restart.

Changes:
- Remove wg.Wait() from both PG and SQLite cron schedulers — jobs now
  run as independent goroutines that don't block the check loop
- Add panic recovery to PG runLoop (safeCheckAndRunDueJobs wrapper)
  and per-job goroutines, matching the existing safego.Recover pattern
  in the SQLite scheduler
- Add 10-minute context timeout to the cron job handler so a hung
  agent run is cancelled instead of blocking forever
- Use select with context.Done() in the handler to respect the timeout
  when waiting for the scheduler outcome
- Invalidate PG job cache per-job on completion instead of after the
  (now-removed) batch wait

The SQLite scheduler already had safego.Recover on job goroutines but
still used wg.Wait() — this commit removes that blocking wait as well.

* fix(cron): make job timeout configurable + add SQLite runLoop panic recovery

- Add `cron.job_timeout` config field (Go duration string, default "10m")
  so operators can tune the per-job timeout for complex agent workloads
  without code changes
- Add `safeCheckJobs` panic recovery wrapper to SQLite cron runLoop,
  matching the PG scheduler's `safeCheckAndRunDueJobs` for consistency
- Use dynamic timeout string in error message for better diagnostics

* fix: remove unused "time" import from gateway_cron.go

* fix(cron): apply same fixes to SQLite DB scheduler (sqlitestore)

The SQLite DB-backed scheduler (used by desktop edition with SQLite
backend) had the exact same wg.Wait() blocking issue and missing
panic recovery as the PG scheduler. Apply identical fixes:

- Remove wg.Wait() — jobs run as independent goroutines
- Add safeCheckAndRunDueJobs panic recovery wrapper for runLoop
- Add per-job panic recovery and cache invalidation

---------

Co-authored-by: Luvu182 <208665161+Luvu182@users.noreply.github.com>
2026-04-10 19:50:49 +07:00
..