When multiple scheduled tasks or database backups run concurrently on
the same server, they compete for the same SSH multiplexed connection
socket, causing race conditions and SSH exit code 255 errors.
This fix adds a `disableMultiplexing` parameter to bypass SSH
multiplexing for jobs that may run concurrently:
- Add `disableMultiplexing` param to `generateSshCommand()`
- Add `disableMultiplexing` param to `instant_remote_process()`
- Update `ScheduledTaskJob` to use `disableMultiplexing: true`
- Update `DatabaseBackupJob` to use `disableMultiplexing: true`
- Add debug logging to track execution without multiplexing
- Add unit tests for the new parameter
Each backup and scheduled task now gets an isolated SSH connection,
preventing contention on the shared multiplexed socket.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Allows user-configured backup timeouts > 3600 to be respected. Previously, the SSH process used a hardcoded 3600 second timeout regardless of the job timeout setting. Now the timeout is passed through to instant_remote_process() for all backup operations.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add retry configuration to CoolifyTask (3 tries, 600s timeout)
- Add retry configuration to ScheduledTaskJob (3 tries, configurable timeout)
- Add retry configuration to DatabaseBackupJob (2 tries)
- Implement exponential backoff for all jobs (30s, 60s, 120s intervals)
- Add failed() handlers with comprehensive error logging to scheduled-errors channel
- Add execution tracking: started_at, retry_count, duration (decimal), error_details
- Add configurable timeout field to scheduled tasks (60-3600s, default 300s)
- Update UI to include timeout configuration in task creation/editing forms
- Increase ScheduledJobManager lock expiration from 60s to 90s for high-load environments
- Implement safe queue cleanup with restart vs runtime modes
- Restart mode: aggressive cleanup (marks all processing jobs as failed)
- Runtime mode: conservative cleanup (only marks jobs >12h as failed, skips deployments)
- Add cleanup:redis --restart flag for system startup
- Integrate cleanup into Dev.php init() for development environment
- Increase scheduled-errors log retention from 7 to 14 days
- Create comprehensive test suite (unit and feature tests)
- Add TESTING_GUIDE.md with manual testing instructions
Fixes issues with jobs failing after single attempt and "attempted too many times" errors
Changed backup_log_uuid property to nullable and removed eager initialization in constructor. This allows the ID to be generated when actually needed rather than upfront.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Introduced a new notification class, BackupSuccessWithS3Warning, to alert users when local backups succeed but S3 uploads fail.
- Updated DatabaseBackupJob to track local backup success and handle S3 upload errors, improving error reporting and user notifications.
- Modified ScheduledDatabaseBackupExecution model to include a new s3_uploaded boolean field for tracking S3 upload status.
- Adjusted views and validation logic to reflect changes in backup execution status and S3 handling.
- Added tests to ensure the new s3_uploaded column is correctly implemented and validated.
- Refactored the DatabaseBackupJob to replace instances of backup->uuid with backup_log_uuid for consistency in container naming.
- Ensured that all related Docker commands and processes reference the updated backup_log_uuid, improving clarity and reducing potential errors during backup operations.
- Refactored the DatabaseBackupJob to generate a unique UUID for each database backup execution within the loop, improving clarity and ensuring uniqueness for each backup attempt.
- Removed redundant UUID generation logic from the initial part of the handle method.
- Enhanced the DatabaseBackupJob to generate a unique UUID for each backup execution attempt.
- Added logic to retry UUID generation up to three times if a duplicate is detected, ensuring uniqueness and preventing execution conflicts.
Andras Bacsai