mirror of
https://github.com/tiennm99/composes.git
synced 2026-10-04 18:13:19 +00:00
feat: merge linux+docker alloy configs and source creds from env
Embed unified config.alloy via compose configs, combining node_exporter + journal (linux) and cadvisor + docker logs (docker) collectors into one container. Add remotecfg block for grafana fleet-management. Replace hardcoded credentials with sys.env() reads of nine shell variables (ALLOY_HOSTNAME, REMOTECFG_*, PROM_*, LOKI_*, GRAFANA_TOKEN).
This commit is contained in:
1 parent
f009b37ca0
commit
94c63452eb
2 files changed
+212
-137
No files matched your search
+32
-45
@@ -1,78 +1,65 @@
|
||||
# alloy-docker-compose
|
||||
|
||||
One-file [Grafana Alloy](https://grafana.com/docs/alloy/latest/) setup that ships host and container telemetry to Grafana Cloud — sized for the Free tier.
|
||||
One-file [Grafana Alloy](https://grafana.com/docs/alloy/latest/) setup that ships host (linux) and container (docker) telemetry to Grafana Cloud, plus pulls remote config from Grafana Fleet Management.
|
||||
|
||||
- **Single container, single file** — no `remotecfg`, no fleet-management, no split `alloy` + `alloy-docker` deployments.
|
||||
- **Env-driven** — six shell variables, no `.env` file.
|
||||
- **Least-privilege** — `cap_drop: [ALL]`, no `privileged`, no `pid: host`, UI not published.
|
||||
- **Free-tier safe** — `keep`-filter on metric names holds a typical 15-container host to ~1,600 of the 10,000 active-series cap.
|
||||
- **Single container** running both `node_exporter` (host) and `cadvisor` (containers) collectors.
|
||||
- **Config embedded inline** via Compose `configs:` — no sidecar `config.alloy` file on disk.
|
||||
- **Env-driven** — nine shell variables, no `.env` file.
|
||||
- **Remote config** via `remotecfg` block (Grafana Fleet Management).
|
||||
|
||||
## What it collects
|
||||
|
||||
| Source | Component | Notes |
|
||||
|---|---|---|
|
||||
| Host metrics | `prometheus.exporter.unix` | CPU, memory (+MemFree/Available), load, disk I/O + IOPS + saturation, filesystem (size/avail/readonly/inodes), network (bytes/errs/drop), uname, boot time |
|
||||
| Container metrics | `prometheus.exporter.cadvisor` | CPU, memory (usage + working_set), fs usage/limit, network, `last_seen` |
|
||||
| Host metrics | `prometheus.exporter.unix` | CPU, memory, load, disk I/O, filesystem, network, uname, boot time |
|
||||
| Container metrics | `prometheus.exporter.cadvisor` | CPU, memory, fs usage/limit, network, `last_seen` |
|
||||
| Container logs | `loki.source.docker` | all running containers, labeled with `container`, `stream`, `instance` |
|
||||
| System logs | `loki.source.journal` | systemd journal with `unit` + `level` labels |
|
||||
| System logs | `loki.source.journal` (via `journal_module`) | systemd journal with `unit`, `boot_id`, `transport`, `level` labels |
|
||||
| Remote config | `remotecfg` | polls Grafana Fleet Management every 60s |
|
||||
|
||||
60-second scrape interval. Alloy's own logs are *not* excluded — useful for debugging the agent itself.
|
||||
`keep`-filter on metric names trims the firehose down to the standard Grafana Cloud integration dashboards (node-exporter + docker).
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
export ALLOY_HOSTNAME=<host-name> # e.g. miti-jp
|
||||
export ALLOY_HOSTNAME=miti-jp # also used as Loki/Prometheus instance label
|
||||
export REMOTECFG_URL=https://fleet-management-prod-013.grafana.net
|
||||
export REMOTECFG_ID=miti-jp # fleet-management agent id
|
||||
export REMOTECFG_USER=1431677 # fleet-management user id
|
||||
export PROM_URL=https://prometheus-prod-XX-<region>.grafana.net/api/prom/push
|
||||
export PROM_USER=<prometheus-user-id>
|
||||
export LOKI_URL=https://logs-prod-XXX.grafana.net/loki/api/v1/push
|
||||
export LOKI_USER=<loki-user-id>
|
||||
export GRAFANA_TOKEN=glc_... # Cloud Access Policy, scopes: metrics:write + logs:write
|
||||
export GRAFANA_TOKEN=glc_... # one Cloud Access Policy token, scopes: metrics:write + logs:write + fleet-management:read
|
||||
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Find the `PROM_*` / `LOKI_*` values under Grafana Cloud → your stack → **Details** on the Prometheus and Loki data sources. Create one Cloud Access Policy with both `metrics:write` and `logs:write` scopes, then generate a token from it and reuse for both writes.
|
||||
Find the `PROM_*` / `LOKI_*` / `REMOTECFG_*` values under Grafana Cloud → your stack → **Details** on each data source / Fleet Management. The same token is reused for `remotecfg`, Prometheus, and Loki basic-auth.
|
||||
|
||||
Any unset required variable makes `docker compose up` fail fast with a clear error.
|
||||
Any unset required variable makes `docker compose up` fail fast.
|
||||
|
||||
## Multi-host
|
||||
|
||||
The same file works on every host — just export a different `ALLOY_HOSTNAME`. Each instance independently ships to the same Grafana Cloud stack; filter/compare in Grafana with `instance=~"..."`.
|
||||
Same compose file on every host — change `ALLOY_HOSTNAME` and `REMOTECFG_ID` per host. Filter in Grafana with `instance=~"..."`.
|
||||
|
||||
## UI access
|
||||
## Security note
|
||||
|
||||
The UI (port `12345`) is **not** published to the host. To reach it:
|
||||
Runs as `privileged: true` (matching the upstream Grafana Cloud docker integration). This is required for cadvisor to read cgroups via `/sys` and for `/dev/kmsg` access. If you need least-privilege, see the upstream Alloy docker integration docs and tighten capabilities.
|
||||
|
||||
```bash
|
||||
# from another container on the same Docker network:
|
||||
curl http://alloy:12345/graph
|
||||
## Mounts
|
||||
|
||||
# or expose temporarily for local debugging:
|
||||
docker compose exec alloy wget -qO- http://localhost:12345/-/ready
|
||||
```
|
||||
|
||||
## Security posture
|
||||
|
||||
- Runs as `user: "0:0"` (root) inside the container — required to read `/var/run/docker.sock` (0660 root:docker) and `/var/log/journal/*` (0640 root:systemd-journal).
|
||||
- All Linux capabilities dropped (`cap_drop: [ALL]`). Root without caps still reads files (DAC bypass) but cannot do privileged syscalls.
|
||||
- `no-new-privileges:true` on seccomp.
|
||||
- All host mounts are read-only. Only the `alloy-data` named volume is writable (WAL/positions).
|
||||
- The UI listens on `0.0.0.0:12345` *inside* the container network only; no `ports:` mapping to the host.
|
||||
|
||||
## Free-tier cardinality budget (Grafana Cloud)
|
||||
|
||||
- **Metrics**: 10,000 active series hard cap. This config emits ~1,500–1,800 per host (cAdvisor dominates at ~150/container). Two hosts ≈ 32% utilisation.
|
||||
- **Logs**: 50 GB/month ingestion. `batch_wait=5s`, `batch_size=1MiB` minimise push overhead. Typical home server stays well under.
|
||||
- **Retention**: 14 days (set by the Free plan, not configurable here).
|
||||
|
||||
If you add a third host or scrape extra applications, re-check cardinality before declaring victory.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`hostname: required` error from Compose** — you didn't `export ALLOY_HOSTNAME` in the current shell.
|
||||
- **No system logs in Loki** — host has no persistent journal. Enable with `sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal && sudo systemctl restart systemd-journald`.
|
||||
- **Empty panels on "Node Exporter Full" dashboard** — the `keep`-filter trims rarely-used metrics (Shmem, Slab, Active/Inactive memory breakdowns). Add names to the regex in `prometheus.relabel "filter"` to restore specific panels.
|
||||
- **Docker logs missing for recently-started containers** — discovery refreshes every 5s; wait a few seconds.
|
||||
| Mount | Why |
|
||||
|---|---|
|
||||
| `/proc:/rootproc:ro` | node-exporter cpu/mem/load |
|
||||
| `/sys:/sys:ro` | node-exporter + cadvisor cgroups |
|
||||
| `/:/rootfs:ro` | filesystem collector |
|
||||
| `/dev/disk/:/dev/disk:ro` | diskstats device labels |
|
||||
| `/var/run/docker.sock` | docker discovery + log streaming |
|
||||
| `/var/lib/docker:ro` | cadvisor container metadata |
|
||||
| `/var/log/journal:ro` | `loki.source.journal` |
|
||||
| `/dev/kmsg` (device) | cadvisor OOM detection |
|
||||
| `alloy-data` (named volume) | WAL + remotecfg cache |
|
||||
|
||||
## License
|
||||
|
||||
|
||||
+180
-92
@@ -1,6 +1,6 @@
|
||||
# Required shell env vars (export before `docker compose up`):
|
||||
# ALLOY_HOSTNAME PROM_URL PROM_USER LOKI_URL LOKI_USER GRAFANA_TOKEN
|
||||
# UI reachable only from inside the Docker network: http://alloy:12345
|
||||
# Required env vars (export before `docker compose up -d`):
|
||||
# ALLOY_HOSTNAME, REMOTECFG_URL, REMOTECFG_ID, REMOTECFG_USER,
|
||||
# PROM_URL, PROM_USER, LOKI_URL, LOKI_USER, GRAFANA_TOKEN
|
||||
|
||||
services:
|
||||
alloy:
|
||||
@@ -8,31 +8,33 @@ services:
|
||||
container_name: alloy
|
||||
restart: unless-stopped
|
||||
hostname: ${ALLOY_HOSTNAME:?required}
|
||||
user: "0:0" # root needed for docker.sock + journal files
|
||||
privileged: true
|
||||
environment:
|
||||
PROM_URL: ${PROM_URL:?required}
|
||||
PROM_USER: ${PROM_USER:?required}
|
||||
LOKI_URL: ${LOKI_URL:?required}
|
||||
LOKI_USER: ${LOKI_USER:?required}
|
||||
ALLOY_DEPLOY_MODE: docker
|
||||
REMOTECFG_URL: ${REMOTECFG_URL:?required}
|
||||
REMOTECFG_ID: ${REMOTECFG_ID:?required}
|
||||
REMOTECFG_USER: ${REMOTECFG_USER:?required}
|
||||
PROM_URL: ${PROM_URL:?required}
|
||||
PROM_USER: ${PROM_USER:?required}
|
||||
LOKI_URL: ${LOKI_URL:?required}
|
||||
LOKI_USER: ${LOKI_USER:?required}
|
||||
GRAFANA_TOKEN: ${GRAFANA_TOKEN:?required}
|
||||
security_opt: [ "no-new-privileges:true" ]
|
||||
cap_drop: [ ALL ]
|
||||
cap_add: [ DAC_OVERRIDE, SYSLOG ] # DAC_OVERRIDE: write alloy-owned /var/lib/alloy/data + read root-owned mounts. SYSLOG: read /dev/kmsg when kernel.dmesg_restrict=1.
|
||||
devices:
|
||||
- /dev/kmsg:/dev/kmsg:r # cadvisor OOM detection (needs device-cgroup allow, not just bind-mount)
|
||||
volumes:
|
||||
- alloy-data:/var/lib/alloy/data # remotecfg + storage state
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro # discovery.docker + loki.source.docker (streams log API)
|
||||
- /var/lib/docker:/var/lib/docker:ro # cadvisor container metadata
|
||||
- /proc:/host/proc:ro # node-exporter cpu/mem/load/net
|
||||
- /sys:/host/sys:ro # node-exporter + cadvisor cgroups
|
||||
- /:/rootfs:ro,rslave # filesystem collector (rslave picks up new mounts)
|
||||
- /var/log/journal:/var/log/journal:ro # loki.source.journal
|
||||
- /etc/machine-id:/etc/machine-id:ro # stable host id for journal reader
|
||||
- /run/udev/data:/run/udev/data:ro # node-exporter diskstats device labels (model/serial/WWN)
|
||||
- alloy-data:/var/lib/alloy/data
|
||||
- /proc:/rootproc:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- /sys:/sys:ro
|
||||
- /:/rootfs:ro
|
||||
- /dev/disk/:/dev/disk:ro
|
||||
- /var/lib/docker/:/var/lib/docker:ro
|
||||
- /var/log/journal:/var/log/journal:ro
|
||||
extra_hosts:
|
||||
- 'host.docker.internal:host-gateway'
|
||||
devices:
|
||||
- /dev/kmsg
|
||||
configs:
|
||||
- { source: alloy_config, target: /etc/alloy/config.alloy }
|
||||
command: [ run, --server.http.listen-addr=0.0.0.0:12345, --storage.path=/var/lib/alloy/data, /etc/alloy/config.alloy ]
|
||||
command: 'run --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy'
|
||||
|
||||
volumes:
|
||||
alloy-data:
|
||||
@@ -40,9 +42,21 @@ volumes:
|
||||
configs:
|
||||
alloy_config:
|
||||
content: |
|
||||
prometheus.remote_write "gc" {
|
||||
remotecfg {
|
||||
url = sys.env("REMOTECFG_URL")
|
||||
id = sys.env("REMOTECFG_ID")
|
||||
poll_frequency = "60s"
|
||||
|
||||
basic_auth {
|
||||
username = sys.env("REMOTECFG_USER")
|
||||
password = sys.env("GRAFANA_TOKEN")
|
||||
}
|
||||
}
|
||||
|
||||
prometheus.remote_write "metrics_service" {
|
||||
endpoint {
|
||||
url = sys.env("PROM_URL")
|
||||
|
||||
basic_auth {
|
||||
username = sys.env("PROM_USER")
|
||||
password = sys.env("GRAFANA_TOKEN")
|
||||
@@ -50,12 +64,10 @@ configs:
|
||||
}
|
||||
}
|
||||
|
||||
loki.write "gc" {
|
||||
external_labels = { instance = constants.hostname }
|
||||
loki.write "grafana_cloud_loki" {
|
||||
endpoint {
|
||||
url = sys.env("LOKI_URL")
|
||||
batch_wait = "5s"
|
||||
batch_size = "1MiB"
|
||||
url = sys.env("LOKI_URL")
|
||||
|
||||
basic_auth {
|
||||
username = sys.env("LOKI_USER")
|
||||
password = sys.env("GRAFANA_TOKEN")
|
||||
@@ -63,93 +75,169 @@ configs:
|
||||
}
|
||||
}
|
||||
|
||||
prometheus.exporter.unix "host" {
|
||||
rootfs_path = "/rootfs"
|
||||
procfs_path = "/host/proc"
|
||||
sysfs_path = "/host/sys"
|
||||
filesystem { mount_points_exclude = "^/(dev|proc|sys|run|var/lib/docker)($|/)" }
|
||||
netdev { device_exclude = "^(veth.*|docker.*|br-.*|lo)$" }
|
||||
}
|
||||
discovery.relabel "integrations_node_exporter" {
|
||||
targets = prometheus.exporter.unix.integrations_node_exporter.targets
|
||||
|
||||
prometheus.exporter.cadvisor "containers" {
|
||||
docker_only = true
|
||||
}
|
||||
|
||||
prometheus.scrape "metrics" {
|
||||
targets = array.concat(
|
||||
prometheus.exporter.unix.host.targets,
|
||||
prometheus.exporter.cadvisor.containers.targets,
|
||||
)
|
||||
forward_to = [prometheus.relabel.filter.receiver]
|
||||
scrape_interval = "60s"
|
||||
}
|
||||
|
||||
// Sets instance label + keeps only metrics used by standard dashboards.
|
||||
// keep-filter essential for the 10k active-series cap on Grafana Cloud Free.
|
||||
prometheus.relabel "filter" {
|
||||
forward_to = [prometheus.remote_write.gc.receiver]
|
||||
rule {
|
||||
target_label = "instance"
|
||||
replacement = constants.hostname
|
||||
}
|
||||
|
||||
rule {
|
||||
target_label = "job"
|
||||
replacement = "integrations/node_exporter"
|
||||
}
|
||||
}
|
||||
|
||||
prometheus.exporter.unix "integrations_node_exporter" {
|
||||
disable_collectors = ["ipvs", "btrfs", "infiniband", "xfs", "zfs"]
|
||||
|
||||
filesystem {
|
||||
fs_types_exclude = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
|
||||
mount_points_exclude = "^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)"
|
||||
mount_timeout = "5s"
|
||||
}
|
||||
|
||||
netclass {
|
||||
ignored_devices = "^(veth.*|cali.*|[a-f0-9]{15})$"
|
||||
}
|
||||
|
||||
netdev {
|
||||
device_exclude = "^(veth.*|cali.*|[a-f0-9]{15})$"
|
||||
}
|
||||
}
|
||||
|
||||
prometheus.scrape "integrations_node_exporter" {
|
||||
targets = discovery.relabel.integrations_node_exporter.output
|
||||
forward_to = [prometheus.relabel.integrations_node_exporter.receiver]
|
||||
}
|
||||
|
||||
prometheus.relabel "integrations_node_exporter" {
|
||||
forward_to = [prometheus.remote_write.metrics_service.receiver]
|
||||
|
||||
rule {
|
||||
source_labels = ["__name__"]
|
||||
regex = "up|node_arp_entries|node_boot_time_seconds|node_context_switches_total|node_cpu_seconds_total|node_disk_io_time_seconds_total|node_disk_io_time_weighted_seconds_total|node_disk_read_bytes_total|node_disk_read_time_seconds_total|node_disk_reads_completed_total|node_disk_write_time_seconds_total|node_disk_writes_completed_total|node_disk_written_bytes_total|node_filefd_allocated|node_filefd_maximum|node_filesystem_avail_bytes|node_filesystem_device_error|node_filesystem_files|node_filesystem_files_free|node_filesystem_readonly|node_filesystem_size_bytes|node_intr_total|node_load1|node_load15|node_load5|node_md_disks|node_md_disks_required|node_memory_Active_anon_bytes|node_memory_Active_bytes|node_memory_Active_file_bytes|node_memory_AnonHugePages_bytes|node_memory_AnonPages_bytes|node_memory_Bounce_bytes|node_memory_Buffers_bytes|node_memory_Cached_bytes|node_memory_CommitLimit_bytes|node_memory_Committed_AS_bytes|node_memory_DirectMap1G_bytes|node_memory_DirectMap2M_bytes|node_memory_DirectMap4k_bytes|node_memory_Dirty_bytes|node_memory_HugePages_Free|node_memory_HugePages_Rsvd|node_memory_HugePages_Surp|node_memory_HugePages_Total|node_memory_Hugepagesize_bytes|node_memory_Inactive_anon_bytes|node_memory_Inactive_bytes|node_memory_Inactive_file_bytes|node_memory_Mapped_bytes|node_memory_MemAvailable_bytes|node_memory_MemFree_bytes|node_memory_MemTotal_bytes|node_memory_SReclaimable_bytes|node_memory_SUnreclaim_bytes|node_memory_ShmemHugePages_bytes|node_memory_ShmemPmdMapped_bytes|node_memory_Shmem_bytes|node_memory_Slab_bytes|node_memory_SwapTotal_bytes|node_memory_VmallocChunk_bytes|node_memory_VmallocTotal_bytes|node_memory_VmallocUsed_bytes|node_memory_WritebackTmp_bytes|node_memory_Writeback_bytes|node_netstat_Icmp6_InErrors|node_netstat_Icmp6_InMsgs|node_netstat_Icmp6_OutMsgs|node_netstat_Icmp_InErrors|node_netstat_Icmp_InMsgs|node_netstat_Icmp_OutMsgs|node_netstat_IpExt_InOctets|node_netstat_IpExt_OutOctets|node_netstat_TcpExt_ListenDrops|node_netstat_TcpExt_ListenOverflows|node_netstat_TcpExt_TCPSynRetrans|node_netstat_Tcp_InErrs|node_netstat_Tcp_InSegs|node_netstat_Tcp_OutRsts|node_netstat_Tcp_OutSegs|node_netstat_Tcp_RetransSegs|node_netstat_Udp6_InDatagrams|node_netstat_Udp6_InErrors|node_netstat_Udp6_NoPorts|node_netstat_Udp6_OutDatagrams|node_netstat_Udp6_RcvbufErrors|node_netstat_Udp6_SndbufErrors|node_netstat_UdpLite_InErrors|node_netstat_Udp_InDatagrams|node_netstat_Udp_InErrors|node_netstat_Udp_NoPorts|node_netstat_Udp_OutDatagrams|node_netstat_Udp_RcvbufErrors|node_netstat_Udp_SndbufErrors|node_network_carrier|node_network_info|node_network_mtu_bytes|node_network_receive_bytes_total|node_network_receive_compressed_total|node_network_receive_drop_total|node_network_receive_errs_total|node_network_receive_fifo_total|node_network_receive_multicast_total|node_network_receive_packets_total|node_network_speed_bytes|node_network_transmit_bytes_total|node_network_transmit_compressed_total|node_network_transmit_drop_total|node_network_transmit_errs_total|node_network_transmit_fifo_total|node_network_transmit_multicast_total|node_network_transmit_packets_total|node_network_transmit_queue_length|node_network_up|node_nf_conntrack_entries|node_nf_conntrack_entries_limit|node_os_info|node_sockstat_FRAG6_inuse|node_sockstat_FRAG_inuse|node_sockstat_RAW6_inuse|node_sockstat_RAW_inuse|node_sockstat_TCP6_inuse|node_sockstat_TCP_alloc|node_sockstat_TCP_inuse|node_sockstat_TCP_mem|node_sockstat_TCP_mem_bytes|node_sockstat_TCP_orphan|node_sockstat_TCP_tw|node_sockstat_UDP6_inuse|node_sockstat_UDPLITE6_inuse|node_sockstat_UDPLITE_inuse|node_sockstat_UDP_inuse|node_sockstat_UDP_mem|node_sockstat_UDP_mem_bytes|node_sockstat_sockets_used|node_softnet_dropped_total|node_softnet_processed_total|node_softnet_times_squeezed_total|node_systemd_unit_state|node_textfile_scrape_error|node_time_zone_offset_seconds|node_timex_estimated_error_seconds|node_timex_maxerror_seconds|node_timex_offset_seconds|node_timex_sync_status|node_uname_info|node_vmstat_oom_kill|node_vmstat_pgfault|node_vmstat_pgmajfault|node_vmstat_pgpgin|node_vmstat_pgpgout|node_vmstat_pswpin|node_vmstat_pswpout|process_max_fds|process_open_fds"
|
||||
action = "keep"
|
||||
regex = "up|node_(cpu_seconds_total|load(1|5|15)|boot_time_seconds|uname_info|memory_(MemTotal|MemFree|MemAvailable|Cached|Buffers|SwapTotal|SwapFree)_bytes|filesystem_(size|avail)_bytes|filesystem_(files|files_free|readonly)|disk_(reads|writes)_completed_total|disk_(read|written)_bytes_total|disk_io_time(_weighted)?_seconds_total|network_(receive|transmit)_(bytes|errs|drop)_total)|container_(cpu_usage_seconds_total|memory_(usage|working_set)_bytes|fs_(usage|limit)_bytes|network_(receive|transmit)_bytes_total|last_seen)|machine_(memory_bytes|scrape_error)"
|
||||
}
|
||||
}
|
||||
|
||||
discovery.docker "containers" {
|
||||
host = "unix:///var/run/docker.sock"
|
||||
refresh_interval = "5s"
|
||||
}
|
||||
|
||||
discovery.relabel "container_logs" {
|
||||
targets = discovery.docker.containers.targets
|
||||
loki.relabel "integrations_node_exporter" {
|
||||
forward_to = [loki.write.grafana_cloud_loki.receiver]
|
||||
rule {
|
||||
source_labels = ["__meta_docker_container_name"]
|
||||
regex = "/(.*)"
|
||||
target_label = "container"
|
||||
target_label = "job"
|
||||
replacement = "integrations/node_exporter"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__meta_docker_container_log_stream"]
|
||||
target_label = "stream"
|
||||
target_label = "instance"
|
||||
replacement = constants.hostname
|
||||
}
|
||||
}
|
||||
|
||||
// Grafana Cloud Loki rejects entries older than 7 days (HTTP 400).
|
||||
// Long-running containers replay weeks of backlog on first start — drop
|
||||
// anything older than 6 days to stay safely under the cutoff.
|
||||
loki.process "drop_old" {
|
||||
forward_to = [loki.write.gc.receiver]
|
||||
stage.drop {
|
||||
older_than = "144h"
|
||||
drop_counter_reason = "entry_too_old"
|
||||
journal_module "integrations_node_exporter" {
|
||||
forward_to = [loki.relabel.integrations_node_exporter.receiver]
|
||||
}
|
||||
|
||||
//JOURNAL
|
||||
declare "journal_module" {
|
||||
argument "forward_to" {
|
||||
optional = false
|
||||
}
|
||||
|
||||
loki.source.journal "default" {
|
||||
max_age = "12h0m0s"
|
||||
forward_to = [loki.process.default.receiver]
|
||||
relabel_rules = loki.relabel.default.rules
|
||||
}
|
||||
|
||||
loki.relabel "default" {
|
||||
rule {
|
||||
source_labels = ["__journal__systemd_unit"]
|
||||
target_label = "unit"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__journal__boot_id"]
|
||||
target_label = "boot_id"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__journal__transport"]
|
||||
target_label = "transport"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__journal_priority_keyword"]
|
||||
target_label = "level"
|
||||
}
|
||||
forward_to = []
|
||||
}
|
||||
loki.process "default" {
|
||||
forward_to = argument.forward_to.value
|
||||
}
|
||||
}
|
||||
|
||||
loki.source.docker "containers" {
|
||||
host = "unix:///var/run/docker.sock"
|
||||
targets = discovery.relabel.container_logs.output
|
||||
forward_to = [loki.process.drop_old.receiver]
|
||||
prometheus.exporter.cadvisor "integrations_cadvisor" {
|
||||
docker_only = true
|
||||
}
|
||||
discovery.relabel "integrations_cadvisor" {
|
||||
targets = prometheus.exporter.cadvisor.integrations_cadvisor.targets
|
||||
|
||||
rule {
|
||||
target_label = "job"
|
||||
replacement = "integrations/docker"
|
||||
}
|
||||
|
||||
rule {
|
||||
target_label = "instance"
|
||||
replacement = constants.hostname
|
||||
}
|
||||
}
|
||||
|
||||
loki.relabel "journal" {
|
||||
forward_to = []
|
||||
prometheus.relabel "integrations_cadvisor" {
|
||||
forward_to = [prometheus.remote_write.metrics_service.receiver]
|
||||
|
||||
rule {
|
||||
source_labels = ["__journal__systemd_unit"]
|
||||
target_label = "unit"
|
||||
}
|
||||
rule {
|
||||
source_labels = ["__journal_priority_keyword"]
|
||||
target_label = "level"
|
||||
source_labels = ["__name__"]
|
||||
regex = "up|container_cpu_usage_seconds_total|container_fs_inodes_free|container_fs_inodes_total|container_fs_limit_bytes|container_fs_usage_bytes|container_last_seen|container_memory_usage_bytes|container_network_receive_bytes_total|container_network_tcp_usage_total|container_network_transmit_bytes_total|container_spec_memory_reservation_limit_bytes|machine_memory_bytes|machine_scrape_error"
|
||||
action = "keep"
|
||||
}
|
||||
}
|
||||
|
||||
loki.source.journal "system" {
|
||||
max_age = "12h"
|
||||
path = "/var/log/journal"
|
||||
forward_to = [loki.write.gc.receiver]
|
||||
relabel_rules = loki.relabel.journal.rules
|
||||
labels = { job = "systemd-journal" }
|
||||
prometheus.scrape "integrations_cadvisor" {
|
||||
targets = discovery.relabel.integrations_cadvisor.output
|
||||
forward_to = [prometheus.relabel.integrations_cadvisor.receiver]
|
||||
}
|
||||
|
||||
discovery.docker "logs_integrations_docker" {
|
||||
host = "unix:///var/run/docker.sock"
|
||||
refresh_interval = "5s"
|
||||
}
|
||||
discovery.relabel "logs_integrations_docker" {
|
||||
targets = []
|
||||
|
||||
rule {
|
||||
target_label = "job"
|
||||
replacement = "integrations/docker"
|
||||
}
|
||||
|
||||
rule {
|
||||
target_label = "instance"
|
||||
replacement = constants.hostname
|
||||
}
|
||||
|
||||
rule {
|
||||
source_labels = ["__meta_docker_container_name"]
|
||||
regex = "/(.*)"
|
||||
target_label = "container"
|
||||
}
|
||||
|
||||
rule {
|
||||
source_labels = ["__meta_docker_container_log_stream"]
|
||||
target_label = "stream"
|
||||
}
|
||||
}
|
||||
loki.source.docker "logs_integrations_docker" {
|
||||
host = "unix:///var/run/docker.sock"
|
||||
targets = discovery.docker.logs_integrations_docker.targets
|
||||
forward_to = [loki.write.grafana_cloud_loki.receiver]
|
||||
relabel_rules = discovery.relabel.logs_integrations_docker.rules
|
||||
refresh_interval = "5s"
|
||||
}
|
||||
Reference in new issue
Block a user