mirror of
https://github.com/tiennm99/coolify.git
synced 2026-04-17 17:21:04 +00:00
feat: add validation for YAML parsing, integer parameters, and Docker Compose custom fields
This commit adds comprehensive validation improvements and DRY principles for handling Coolify's custom Docker Compose extensions. ## Changes ### 1. Created Reusable stripCoolifyCustomFields() Function - Added shared helper in bootstrap/helpers/docker.php - Removes all Coolify custom fields (exclude_from_hc, content, isDirectory, is_directory) - Handles both long syntax (arrays) and short syntax (strings) for volumes - Well-documented with comprehensive docblock - Follows DRY principle for consistent field stripping ### 2. Fixed Docker Compose Modal Validation - Updated validateComposeFile() to use stripCoolifyCustomFields() - Now removes ALL custom fields before Docker validation (previously only removed content) - Fixes validation errors when using templates with custom fields (e.g., traccar.yaml) - Users can now validate compose files with Coolify extensions in UI ### 3. Enhanced YAML Validation in CalculatesExcludedStatus - Added proper exception handling with ParseException vs generic Exception - Added structure validation (checks if parsed result and services are arrays) - Comprehensive logging with context (error message, line number, snippet) - Maintains safe fallback behavior (returns empty collection on error) ### 4. Added Integer Validation to ContainerStatusAggregator - Validates maxRestartCount parameter in both aggregateFromStrings() and aggregateFromContainers() - Corrects negative values to 0 with warning log - Logs warnings for suspiciously high values (> 1000) - Prevents logic errors in crash loop detection ### 5. Comprehensive Unit Tests - tests/Unit/StripCoolifyCustomFieldsTest.php (NEW) - 9 tests, 43 assertions - tests/Unit/ContainerStatusAggregatorTest.php - Added 6 tests for integer validation - tests/Unit/ExcludeFromHealthCheckTest.php - Added 4 tests for YAML validation - All tests passing with proper Log facade mocking ### 6. Documentation - Added comprehensive Docker Compose extensions documentation to .ai/core/deployment-architecture.md - Documents all custom fields: exclude_from_hc, content, isDirectory/is_directory - Includes examples, use cases, implementation details, and test references - Updated .ai/README.md with navigation links to new documentation ## Benefits - Better UX: Users can validate compose files with custom fields - Better Debugging: Comprehensive logging for errors - Better Code Quality: DRY principle with reusable validation - Better Reliability: Prevents logic errors from invalid parameters - Better Maintainability: Easy to add new custom fields in future 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Andras Bacsai
@@ -303,3 +303,286 @@ Routes: [routes/api.php](mdc:routes/api.php)
|
||||
- **External database** connections
|
||||
- **Third-party monitoring** tools
|
||||
- **Custom notification** channels
|
||||
|
||||
---
|
||||
|
||||
## Coolify Docker Compose Extensions
|
||||
|
||||
Coolify extends standard Docker Compose with custom fields (often called "magic fields") that provide Coolify-specific functionality. These extensions are processed during deployment and stripped before sending the final compose file to Docker, maintaining full compatibility with Docker's compose specification.
|
||||
|
||||
### Overview
|
||||
|
||||
**Why Custom Fields?**
|
||||
- Enable Coolify-specific features without breaking Docker Compose compatibility
|
||||
- Simplify configuration by embedding content directly in compose files
|
||||
- Allow fine-grained control over health check monitoring
|
||||
- Reduce external file dependencies
|
||||
|
||||
**Processing Flow:**
|
||||
1. User defines compose file with custom fields
|
||||
2. Coolify parses and processes custom fields (creates files, stores settings)
|
||||
3. Custom fields are stripped from final compose sent to Docker
|
||||
4. Docker receives standard, valid compose file
|
||||
|
||||
### Service-Level Extensions
|
||||
|
||||
#### `exclude_from_hc`
|
||||
|
||||
**Type:** Boolean
|
||||
**Default:** `false`
|
||||
**Purpose:** Exclude specific services from health check monitoring while still showing their status
|
||||
|
||||
**Example Usage:**
|
||||
```yaml
|
||||
services:
|
||||
watchtower:
|
||||
image: containrrr/watchtower
|
||||
exclude_from_hc: true # Don't monitor this service's health
|
||||
|
||||
backup:
|
||||
image: postgres:16
|
||||
exclude_from_hc: true # Backup containers don't need monitoring
|
||||
restart: always
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Container status is still calculated from Docker state (running, exited, etc.)
|
||||
- Status displays with `:excluded` suffix (e.g., `running:healthy:excluded`)
|
||||
- UI shows "Monitoring Disabled" indicator
|
||||
- Functionally equivalent to `restart: no` for health check purposes
|
||||
- See [Container Status with All Excluded](application-architecture.md#container-status-when-all-containers-excluded) for detailed status handling
|
||||
|
||||
**Use Cases:**
|
||||
- Sidecar containers (watchtower, log collectors)
|
||||
- Backup/maintenance containers
|
||||
- One-time initialization containers
|
||||
- Containers that intentionally restart frequently
|
||||
|
||||
**Implementation:**
|
||||
- Parsed: `bootstrap/helpers/parsers.php`
|
||||
- Status logic: `app/Traits/CalculatesExcludedStatus.php`
|
||||
- Validation: `tests/Unit/ExcludeFromHealthCheckTest.php`
|
||||
|
||||
### Volume-Level Extensions
|
||||
|
||||
Volume extensions only work with **long syntax** (array/object format), not short syntax (string format).
|
||||
|
||||
#### `content`
|
||||
|
||||
**Type:** String (supports multiline with `|` or `>`)
|
||||
**Purpose:** Embed file content directly in compose file for automatic creation during deployment
|
||||
|
||||
**Example Usage:**
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
image: node:20
|
||||
volumes:
|
||||
# Inline entrypoint script
|
||||
- type: bind
|
||||
source: ./entrypoint.sh
|
||||
target: /app/entrypoint.sh
|
||||
content: |
|
||||
#!/bin/sh
|
||||
set -e
|
||||
echo "Starting application..."
|
||||
npm run migrate
|
||||
exec "$@"
|
||||
|
||||
# Configuration file with environment variables
|
||||
- type: bind
|
||||
source: ./config.xml
|
||||
target: /etc/app/config.xml
|
||||
content: |
|
||||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<config>
|
||||
<database>
|
||||
<host>${DB_HOST}</host>
|
||||
<port>${DB_PORT}</port>
|
||||
</database>
|
||||
</config>
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Content is written to the host at `source` path before container starts
|
||||
- File is created with mode `644` (readable by all, writable by owner)
|
||||
- Environment variables in content are interpolated at deployment time
|
||||
- Content is stored in `LocalFileVolume` model (encrypted at rest)
|
||||
- Original `docker_compose_raw` retains content for editing
|
||||
|
||||
**Use Cases:**
|
||||
- Entrypoint scripts
|
||||
- Configuration files
|
||||
- Environment-specific settings
|
||||
- Small initialization scripts
|
||||
- Templates that require dynamic content
|
||||
|
||||
**Limitations:**
|
||||
- Not suitable for large files (use git repo or external storage instead)
|
||||
- Binary files not supported
|
||||
- Changes require redeployment
|
||||
|
||||
**Real-World Examples:**
|
||||
- `templates/compose/traccar.yaml` - XML configuration file
|
||||
- `templates/compose/supabase.yaml` - Multiple config files
|
||||
- `templates/compose/chaskiq.yaml` - Entrypoint script
|
||||
|
||||
**Implementation:**
|
||||
- Parsed: `bootstrap/helpers/parsers.php` (line 717)
|
||||
- Storage: `app/Models/LocalFileVolume.php`
|
||||
- Validation: `tests/Unit/StripCoolifyCustomFieldsTest.php`
|
||||
|
||||
#### `is_directory` / `isDirectory`
|
||||
|
||||
**Type:** Boolean
|
||||
**Default:** `true` (if neither `content` nor explicit flag provided)
|
||||
**Purpose:** Indicate whether bind mount source should be created as directory or file
|
||||
|
||||
**Example Usage:**
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
volumes:
|
||||
# Explicit file
|
||||
- type: bind
|
||||
source: ./config.json
|
||||
target: /app/config.json
|
||||
is_directory: false # Create as file
|
||||
|
||||
# Explicit directory
|
||||
- type: bind
|
||||
source: ./logs
|
||||
target: /var/log/app
|
||||
is_directory: true # Create as directory
|
||||
|
||||
# Auto-detected as file (has content)
|
||||
- type: bind
|
||||
source: ./script.sh
|
||||
target: /entrypoint.sh
|
||||
content: |
|
||||
#!/bin/sh
|
||||
echo "Hello"
|
||||
# is_directory: false implied by content presence
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- If `is_directory: true` → Creates directory with `mkdir -p`
|
||||
- If `is_directory: false` → Creates empty file with `touch`
|
||||
- If `content` provided → Implies `is_directory: false`
|
||||
- If neither specified → Defaults to `true` (directory)
|
||||
|
||||
**Naming Conventions:**
|
||||
- `is_directory` (snake_case) - **Preferred**, consistent with PHP/Laravel conventions
|
||||
- `isDirectory` (camelCase) - **Legacy support**, both work identically
|
||||
|
||||
**Use Cases:**
|
||||
- Disambiguating files vs directories when no content provided
|
||||
- Ensuring correct bind mount type for Docker
|
||||
- Pre-creating mount points before container starts
|
||||
|
||||
**Implementation:**
|
||||
- Parsed: `bootstrap/helpers/parsers.php` (line 718)
|
||||
- Storage: `app/Models/LocalFileVolume.php` (`is_directory` column)
|
||||
- Validation: `tests/Unit/StripCoolifyCustomFieldsTest.php`
|
||||
|
||||
### Custom Field Stripping
|
||||
|
||||
**Function:** `stripCoolifyCustomFields()` in `bootstrap/helpers/docker.php`
|
||||
|
||||
All custom fields are removed before the compose file is sent to Docker. This happens in two contexts:
|
||||
|
||||
**1. Validation (User-Triggered)**
|
||||
```php
|
||||
// In validateComposeFile() - Edit Docker Compose modal
|
||||
$yaml_compose = Yaml::parse($compose);
|
||||
$yaml_compose = stripCoolifyCustomFields($yaml_compose); // Strip custom fields
|
||||
// Send to docker compose config for validation
|
||||
```
|
||||
|
||||
**2. Deployment (Automatic)**
|
||||
```php
|
||||
// In Service::parse() - During deployment
|
||||
$docker_compose = parseCompose($docker_compose_raw);
|
||||
// Custom fields are processed and then stripped
|
||||
// Final compose sent to Docker has no custom fields
|
||||
```
|
||||
|
||||
**What Gets Stripped:**
|
||||
- Service-level: `exclude_from_hc`
|
||||
- Volume-level: `content`, `isDirectory`, `is_directory`
|
||||
|
||||
**What's Preserved:**
|
||||
- All standard Docker Compose fields
|
||||
- Environment variables
|
||||
- Standard volume definitions (after custom fields removed)
|
||||
|
||||
### Important Notes
|
||||
|
||||
#### Long vs Short Volume Syntax
|
||||
|
||||
**✅ Long Syntax (Works with Custom Fields):**
|
||||
```yaml
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ./data
|
||||
target: /app/data
|
||||
content: "Hello" # ✅ Custom fields work here
|
||||
```
|
||||
|
||||
**❌ Short Syntax (Custom Fields Ignored):**
|
||||
```yaml
|
||||
volumes:
|
||||
- "./data:/app/data" # ❌ Cannot add custom fields to strings
|
||||
```
|
||||
|
||||
#### Docker Compose Compatibility
|
||||
|
||||
Custom fields are **Coolify-specific** and won't work with standalone `docker compose` CLI:
|
||||
|
||||
```bash
|
||||
# ❌ Won't work - Docker doesn't recognize custom fields
|
||||
docker compose -f compose.yaml up
|
||||
|
||||
# ✅ Works - Use Coolify's deployment (strips custom fields first)
|
||||
# Deploy through Coolify UI or API
|
||||
```
|
||||
|
||||
#### Editing Custom Fields
|
||||
|
||||
When editing in "Edit Docker Compose" modal:
|
||||
- Custom fields are preserved in the editor
|
||||
- "Validate" button strips them temporarily for Docker validation
|
||||
- "Save" button preserves them in `docker_compose_raw`
|
||||
- They're processed again on next deployment
|
||||
|
||||
### Template Examples
|
||||
|
||||
See these templates for real-world usage:
|
||||
|
||||
**Service Exclusions:**
|
||||
- `templates/compose/budibase.yaml` - Excludes watchtower from monitoring
|
||||
- `templates/compose/pgbackweb.yaml` - Excludes backup service
|
||||
- `templates/compose/elasticsearch-with-kibana.yaml` - Excludes elasticsearch
|
||||
|
||||
**Inline Content:**
|
||||
- `templates/compose/traccar.yaml` - XML configuration (multiline)
|
||||
- `templates/compose/supabase.yaml` - Multiple config files
|
||||
- `templates/compose/searxng.yaml` - Settings file
|
||||
- `templates/compose/invoice-ninja.yaml` - Nginx config
|
||||
|
||||
**Directory Flags:**
|
||||
- `templates/compose/paperless.yaml` - Explicit directory creation
|
||||
|
||||
### Testing
|
||||
|
||||
**Unit Tests:**
|
||||
- `tests/Unit/StripCoolifyCustomFieldsTest.php` - Custom field stripping logic
|
||||
- `tests/Unit/ExcludeFromHealthCheckTest.php` - Health check exclusion behavior
|
||||
- `tests/Unit/ContainerStatusAggregatorTest.php` - Status aggregation with exclusions
|
||||
|
||||
**Test Coverage:**
|
||||
- ✅ All custom fields (exclude_from_hc, content, isDirectory, is_directory)
|
||||
- ✅ Multiline content (YAML `|` syntax)
|
||||
- ✅ Short vs long volume syntax
|
||||
- ✅ Field stripping without data loss
|
||||
- ✅ Standard Docker Compose field preservation
|
||||
|
||||
Reference in New Issue
Block a user