diff --git a/alloy/README.md b/alloy/README.md index af196dd..30da07e 100644 --- a/alloy/README.md +++ b/alloy/README.md @@ -1,78 +1,65 @@ # alloy-docker-compose -One-file [Grafana Alloy](https://grafana.com/docs/alloy/latest/) setup that ships host and container telemetry to Grafana Cloud — sized for the Free tier. +One-file [Grafana Alloy](https://grafana.com/docs/alloy/latest/) setup that ships host (linux) and container (docker) telemetry to Grafana Cloud, plus pulls remote config from Grafana Fleet Management. -- **Single container, single file** — no `remotecfg`, no fleet-management, no split `alloy` + `alloy-docker` deployments. -- **Env-driven** — six shell variables, no `.env` file. -- **Least-privilege** — `cap_drop: [ALL]`, no `privileged`, no `pid: host`, UI not published. -- **Free-tier safe** — `keep`-filter on metric names holds a typical 15-container host to ~1,600 of the 10,000 active-series cap. +- **Single container** running both `node_exporter` (host) and `cadvisor` (containers) collectors. +- **Config embedded inline** via Compose `configs:` — no sidecar `config.alloy` file on disk. +- **Env-driven** — nine shell variables, no `.env` file. +- **Remote config** via `remotecfg` block (Grafana Fleet Management). ## What it collects | Source | Component | Notes | |---|---|---| -| Host metrics | `prometheus.exporter.unix` | CPU, memory (+MemFree/Available), load, disk I/O + IOPS + saturation, filesystem (size/avail/readonly/inodes), network (bytes/errs/drop), uname, boot time | -| Container metrics | `prometheus.exporter.cadvisor` | CPU, memory (usage + working_set), fs usage/limit, network, `last_seen` | +| Host metrics | `prometheus.exporter.unix` | CPU, memory, load, disk I/O, filesystem, network, uname, boot time | +| Container metrics | `prometheus.exporter.cadvisor` | CPU, memory, fs usage/limit, network, `last_seen` | | Container logs | `loki.source.docker` | all running containers, labeled with `container`, `stream`, `instance` | -| System logs | `loki.source.journal` | systemd journal with `unit` + `level` labels | +| System logs | `loki.source.journal` (via `journal_module`) | systemd journal with `unit`, `boot_id`, `transport`, `level` labels | +| Remote config | `remotecfg` | polls Grafana Fleet Management every 60s | -60-second scrape interval. Alloy's own logs are *not* excluded — useful for debugging the agent itself. +`keep`-filter on metric names trims the firehose down to the standard Grafana Cloud integration dashboards (node-exporter + docker). ## Quick start ```bash -export ALLOY_HOSTNAME= # e.g. miti-jp +export ALLOY_HOSTNAME=miti-jp # also used as Loki/Prometheus instance label +export REMOTECFG_URL=https://fleet-management-prod-013.grafana.net +export REMOTECFG_ID=miti-jp # fleet-management agent id +export REMOTECFG_USER=1431677 # fleet-management user id export PROM_URL=https://prometheus-prod-XX-.grafana.net/api/prom/push export PROM_USER= export LOKI_URL=https://logs-prod-XXX.grafana.net/loki/api/v1/push export LOKI_USER= -export GRAFANA_TOKEN=glc_... # Cloud Access Policy, scopes: metrics:write + logs:write +export GRAFANA_TOKEN=glc_... # one Cloud Access Policy token, scopes: metrics:write + logs:write + fleet-management:read docker compose up -d ``` -Find the `PROM_*` / `LOKI_*` values under Grafana Cloud → your stack → **Details** on the Prometheus and Loki data sources. Create one Cloud Access Policy with both `metrics:write` and `logs:write` scopes, then generate a token from it and reuse for both writes. +Find the `PROM_*` / `LOKI_*` / `REMOTECFG_*` values under Grafana Cloud → your stack → **Details** on each data source / Fleet Management. The same token is reused for `remotecfg`, Prometheus, and Loki basic-auth. -Any unset required variable makes `docker compose up` fail fast with a clear error. +Any unset required variable makes `docker compose up` fail fast. ## Multi-host -The same file works on every host — just export a different `ALLOY_HOSTNAME`. Each instance independently ships to the same Grafana Cloud stack; filter/compare in Grafana with `instance=~"..."`. +Same compose file on every host — change `ALLOY_HOSTNAME` and `REMOTECFG_ID` per host. Filter in Grafana with `instance=~"..."`. -## UI access +## Security note -The UI (port `12345`) is **not** published to the host. To reach it: +Runs as `privileged: true` (matching the upstream Grafana Cloud docker integration). This is required for cadvisor to read cgroups via `/sys` and for `/dev/kmsg` access. If you need least-privilege, see the upstream Alloy docker integration docs and tighten capabilities. -```bash -# from another container on the same Docker network: -curl http://alloy:12345/graph +## Mounts -# or expose temporarily for local debugging: -docker compose exec alloy wget -qO- http://localhost:12345/-/ready -``` - -## Security posture - -- Runs as `user: "0:0"` (root) inside the container — required to read `/var/run/docker.sock` (0660 root:docker) and `/var/log/journal/*` (0640 root:systemd-journal). -- All Linux capabilities dropped (`cap_drop: [ALL]`). Root without caps still reads files (DAC bypass) but cannot do privileged syscalls. -- `no-new-privileges:true` on seccomp. -- All host mounts are read-only. Only the `alloy-data` named volume is writable (WAL/positions). -- The UI listens on `0.0.0.0:12345` *inside* the container network only; no `ports:` mapping to the host. - -## Free-tier cardinality budget (Grafana Cloud) - -- **Metrics**: 10,000 active series hard cap. This config emits ~1,500–1,800 per host (cAdvisor dominates at ~150/container). Two hosts ≈ 32% utilisation. -- **Logs**: 50 GB/month ingestion. `batch_wait=5s`, `batch_size=1MiB` minimise push overhead. Typical home server stays well under. -- **Retention**: 14 days (set by the Free plan, not configurable here). - -If you add a third host or scrape extra applications, re-check cardinality before declaring victory. - -## Troubleshooting - -- **`hostname: required` error from Compose** — you didn't `export ALLOY_HOSTNAME` in the current shell. -- **No system logs in Loki** — host has no persistent journal. Enable with `sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal && sudo systemctl restart systemd-journald`. -- **Empty panels on "Node Exporter Full" dashboard** — the `keep`-filter trims rarely-used metrics (Shmem, Slab, Active/Inactive memory breakdowns). Add names to the regex in `prometheus.relabel "filter"` to restore specific panels. -- **Docker logs missing for recently-started containers** — discovery refreshes every 5s; wait a few seconds. +| Mount | Why | +|---|---| +| `/proc:/rootproc:ro` | node-exporter cpu/mem/load | +| `/sys:/sys:ro` | node-exporter + cadvisor cgroups | +| `/:/rootfs:ro` | filesystem collector | +| `/dev/disk/:/dev/disk:ro` | diskstats device labels | +| `/var/run/docker.sock` | docker discovery + log streaming | +| `/var/lib/docker:ro` | cadvisor container metadata | +| `/var/log/journal:ro` | `loki.source.journal` | +| `/dev/kmsg` (device) | cadvisor OOM detection | +| `alloy-data` (named volume) | WAL + remotecfg cache | ## License diff --git a/alloy/docker-compose.yml b/alloy/docker-compose.yml index 6216094..e7a4a0c 100644 --- a/alloy/docker-compose.yml +++ b/alloy/docker-compose.yml @@ -1,6 +1,6 @@ -# Required shell env vars (export before `docker compose up`): -# ALLOY_HOSTNAME PROM_URL PROM_USER LOKI_URL LOKI_USER GRAFANA_TOKEN -# UI reachable only from inside the Docker network: http://alloy:12345 +# Required env vars (export before `docker compose up -d`): +# ALLOY_HOSTNAME, REMOTECFG_URL, REMOTECFG_ID, REMOTECFG_USER, +# PROM_URL, PROM_USER, LOKI_URL, LOKI_USER, GRAFANA_TOKEN services: alloy: @@ -8,31 +8,33 @@ services: container_name: alloy restart: unless-stopped hostname: ${ALLOY_HOSTNAME:?required} - user: "0:0" # root needed for docker.sock + journal files + privileged: true environment: - PROM_URL: ${PROM_URL:?required} - PROM_USER: ${PROM_USER:?required} - LOKI_URL: ${LOKI_URL:?required} - LOKI_USER: ${LOKI_USER:?required} + ALLOY_DEPLOY_MODE: docker + REMOTECFG_URL: ${REMOTECFG_URL:?required} + REMOTECFG_ID: ${REMOTECFG_ID:?required} + REMOTECFG_USER: ${REMOTECFG_USER:?required} + PROM_URL: ${PROM_URL:?required} + PROM_USER: ${PROM_USER:?required} + LOKI_URL: ${LOKI_URL:?required} + LOKI_USER: ${LOKI_USER:?required} GRAFANA_TOKEN: ${GRAFANA_TOKEN:?required} - security_opt: [ "no-new-privileges:true" ] - cap_drop: [ ALL ] - cap_add: [ DAC_OVERRIDE, SYSLOG ] # DAC_OVERRIDE: write alloy-owned /var/lib/alloy/data + read root-owned mounts. SYSLOG: read /dev/kmsg when kernel.dmesg_restrict=1. - devices: - - /dev/kmsg:/dev/kmsg:r # cadvisor OOM detection (needs device-cgroup allow, not just bind-mount) volumes: - - alloy-data:/var/lib/alloy/data # remotecfg + storage state - - /var/run/docker.sock:/var/run/docker.sock:ro # discovery.docker + loki.source.docker (streams log API) - - /var/lib/docker:/var/lib/docker:ro # cadvisor container metadata - - /proc:/host/proc:ro # node-exporter cpu/mem/load/net - - /sys:/host/sys:ro # node-exporter + cadvisor cgroups - - /:/rootfs:ro,rslave # filesystem collector (rslave picks up new mounts) - - /var/log/journal:/var/log/journal:ro # loki.source.journal - - /etc/machine-id:/etc/machine-id:ro # stable host id for journal reader - - /run/udev/data:/run/udev/data:ro # node-exporter diskstats device labels (model/serial/WWN) + - alloy-data:/var/lib/alloy/data + - /proc:/rootproc:ro + - /var/run/docker.sock:/var/run/docker.sock + - /sys:/sys:ro + - /:/rootfs:ro + - /dev/disk/:/dev/disk:ro + - /var/lib/docker/:/var/lib/docker:ro + - /var/log/journal:/var/log/journal:ro + extra_hosts: + - 'host.docker.internal:host-gateway' + devices: + - /dev/kmsg configs: - { source: alloy_config, target: /etc/alloy/config.alloy } - command: [ run, --server.http.listen-addr=0.0.0.0:12345, --storage.path=/var/lib/alloy/data, /etc/alloy/config.alloy ] + command: 'run --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy' volumes: alloy-data: @@ -40,9 +42,21 @@ volumes: configs: alloy_config: content: | - prometheus.remote_write "gc" { + remotecfg { + url = sys.env("REMOTECFG_URL") + id = sys.env("REMOTECFG_ID") + poll_frequency = "60s" + + basic_auth { + username = sys.env("REMOTECFG_USER") + password = sys.env("GRAFANA_TOKEN") + } + } + + prometheus.remote_write "metrics_service" { endpoint { url = sys.env("PROM_URL") + basic_auth { username = sys.env("PROM_USER") password = sys.env("GRAFANA_TOKEN") @@ -50,12 +64,10 @@ configs: } } - loki.write "gc" { - external_labels = { instance = constants.hostname } + loki.write "grafana_cloud_loki" { endpoint { - url = sys.env("LOKI_URL") - batch_wait = "5s" - batch_size = "1MiB" + url = sys.env("LOKI_URL") + basic_auth { username = sys.env("LOKI_USER") password = sys.env("GRAFANA_TOKEN") @@ -63,93 +75,169 @@ configs: } } - prometheus.exporter.unix "host" { - rootfs_path = "/rootfs" - procfs_path = "/host/proc" - sysfs_path = "/host/sys" - filesystem { mount_points_exclude = "^/(dev|proc|sys|run|var/lib/docker)($|/)" } - netdev { device_exclude = "^(veth.*|docker.*|br-.*|lo)$" } - } + discovery.relabel "integrations_node_exporter" { + targets = prometheus.exporter.unix.integrations_node_exporter.targets - prometheus.exporter.cadvisor "containers" { - docker_only = true - } - - prometheus.scrape "metrics" { - targets = array.concat( - prometheus.exporter.unix.host.targets, - prometheus.exporter.cadvisor.containers.targets, - ) - forward_to = [prometheus.relabel.filter.receiver] - scrape_interval = "60s" - } - - // Sets instance label + keeps only metrics used by standard dashboards. - // keep-filter essential for the 10k active-series cap on Grafana Cloud Free. - prometheus.relabel "filter" { - forward_to = [prometheus.remote_write.gc.receiver] rule { target_label = "instance" replacement = constants.hostname } + + rule { + target_label = "job" + replacement = "integrations/node_exporter" + } + } + + prometheus.exporter.unix "integrations_node_exporter" { + disable_collectors = ["ipvs", "btrfs", "infiniband", "xfs", "zfs"] + + filesystem { + fs_types_exclude = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$" + mount_points_exclude = "^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)" + mount_timeout = "5s" + } + + netclass { + ignored_devices = "^(veth.*|cali.*|[a-f0-9]{15})$" + } + + netdev { + device_exclude = "^(veth.*|cali.*|[a-f0-9]{15})$" + } + } + + prometheus.scrape "integrations_node_exporter" { + targets = discovery.relabel.integrations_node_exporter.output + forward_to = [prometheus.relabel.integrations_node_exporter.receiver] + } + + prometheus.relabel "integrations_node_exporter" { + forward_to = [prometheus.remote_write.metrics_service.receiver] + rule { source_labels = ["__name__"] + regex = "up|node_arp_entries|node_boot_time_seconds|node_context_switches_total|node_cpu_seconds_total|node_disk_io_time_seconds_total|node_disk_io_time_weighted_seconds_total|node_disk_read_bytes_total|node_disk_read_time_seconds_total|node_disk_reads_completed_total|node_disk_write_time_seconds_total|node_disk_writes_completed_total|node_disk_written_bytes_total|node_filefd_allocated|node_filefd_maximum|node_filesystem_avail_bytes|node_filesystem_device_error|node_filesystem_files|node_filesystem_files_free|node_filesystem_readonly|node_filesystem_size_bytes|node_intr_total|node_load1|node_load15|node_load5|node_md_disks|node_md_disks_required|node_memory_Active_anon_bytes|node_memory_Active_bytes|node_memory_Active_file_bytes|node_memory_AnonHugePages_bytes|node_memory_AnonPages_bytes|node_memory_Bounce_bytes|node_memory_Buffers_bytes|node_memory_Cached_bytes|node_memory_CommitLimit_bytes|node_memory_Committed_AS_bytes|node_memory_DirectMap1G_bytes|node_memory_DirectMap2M_bytes|node_memory_DirectMap4k_bytes|node_memory_Dirty_bytes|node_memory_HugePages_Free|node_memory_HugePages_Rsvd|node_memory_HugePages_Surp|node_memory_HugePages_Total|node_memory_Hugepagesize_bytes|node_memory_Inactive_anon_bytes|node_memory_Inactive_bytes|node_memory_Inactive_file_bytes|node_memory_Mapped_bytes|node_memory_MemAvailable_bytes|node_memory_MemFree_bytes|node_memory_MemTotal_bytes|node_memory_SReclaimable_bytes|node_memory_SUnreclaim_bytes|node_memory_ShmemHugePages_bytes|node_memory_ShmemPmdMapped_bytes|node_memory_Shmem_bytes|node_memory_Slab_bytes|node_memory_SwapTotal_bytes|node_memory_VmallocChunk_bytes|node_memory_VmallocTotal_bytes|node_memory_VmallocUsed_bytes|node_memory_WritebackTmp_bytes|node_memory_Writeback_bytes|node_netstat_Icmp6_InErrors|node_netstat_Icmp6_InMsgs|node_netstat_Icmp6_OutMsgs|node_netstat_Icmp_InErrors|node_netstat_Icmp_InMsgs|node_netstat_Icmp_OutMsgs|node_netstat_IpExt_InOctets|node_netstat_IpExt_OutOctets|node_netstat_TcpExt_ListenDrops|node_netstat_TcpExt_ListenOverflows|node_netstat_TcpExt_TCPSynRetrans|node_netstat_Tcp_InErrs|node_netstat_Tcp_InSegs|node_netstat_Tcp_OutRsts|node_netstat_Tcp_OutSegs|node_netstat_Tcp_RetransSegs|node_netstat_Udp6_InDatagrams|node_netstat_Udp6_InErrors|node_netstat_Udp6_NoPorts|node_netstat_Udp6_OutDatagrams|node_netstat_Udp6_RcvbufErrors|node_netstat_Udp6_SndbufErrors|node_netstat_UdpLite_InErrors|node_netstat_Udp_InDatagrams|node_netstat_Udp_InErrors|node_netstat_Udp_NoPorts|node_netstat_Udp_OutDatagrams|node_netstat_Udp_RcvbufErrors|node_netstat_Udp_SndbufErrors|node_network_carrier|node_network_info|node_network_mtu_bytes|node_network_receive_bytes_total|node_network_receive_compressed_total|node_network_receive_drop_total|node_network_receive_errs_total|node_network_receive_fifo_total|node_network_receive_multicast_total|node_network_receive_packets_total|node_network_speed_bytes|node_network_transmit_bytes_total|node_network_transmit_compressed_total|node_network_transmit_drop_total|node_network_transmit_errs_total|node_network_transmit_fifo_total|node_network_transmit_multicast_total|node_network_transmit_packets_total|node_network_transmit_queue_length|node_network_up|node_nf_conntrack_entries|node_nf_conntrack_entries_limit|node_os_info|node_sockstat_FRAG6_inuse|node_sockstat_FRAG_inuse|node_sockstat_RAW6_inuse|node_sockstat_RAW_inuse|node_sockstat_TCP6_inuse|node_sockstat_TCP_alloc|node_sockstat_TCP_inuse|node_sockstat_TCP_mem|node_sockstat_TCP_mem_bytes|node_sockstat_TCP_orphan|node_sockstat_TCP_tw|node_sockstat_UDP6_inuse|node_sockstat_UDPLITE6_inuse|node_sockstat_UDPLITE_inuse|node_sockstat_UDP_inuse|node_sockstat_UDP_mem|node_sockstat_UDP_mem_bytes|node_sockstat_sockets_used|node_softnet_dropped_total|node_softnet_processed_total|node_softnet_times_squeezed_total|node_systemd_unit_state|node_textfile_scrape_error|node_time_zone_offset_seconds|node_timex_estimated_error_seconds|node_timex_maxerror_seconds|node_timex_offset_seconds|node_timex_sync_status|node_uname_info|node_vmstat_oom_kill|node_vmstat_pgfault|node_vmstat_pgmajfault|node_vmstat_pgpgin|node_vmstat_pgpgout|node_vmstat_pswpin|node_vmstat_pswpout|process_max_fds|process_open_fds" action = "keep" - regex = "up|node_(cpu_seconds_total|load(1|5|15)|boot_time_seconds|uname_info|memory_(MemTotal|MemFree|MemAvailable|Cached|Buffers|SwapTotal|SwapFree)_bytes|filesystem_(size|avail)_bytes|filesystem_(files|files_free|readonly)|disk_(reads|writes)_completed_total|disk_(read|written)_bytes_total|disk_io_time(_weighted)?_seconds_total|network_(receive|transmit)_(bytes|errs|drop)_total)|container_(cpu_usage_seconds_total|memory_(usage|working_set)_bytes|fs_(usage|limit)_bytes|network_(receive|transmit)_bytes_total|last_seen)|machine_(memory_bytes|scrape_error)" } } - discovery.docker "containers" { - host = "unix:///var/run/docker.sock" - refresh_interval = "5s" - } - - discovery.relabel "container_logs" { - targets = discovery.docker.containers.targets + loki.relabel "integrations_node_exporter" { + forward_to = [loki.write.grafana_cloud_loki.receiver] rule { - source_labels = ["__meta_docker_container_name"] - regex = "/(.*)" - target_label = "container" + target_label = "job" + replacement = "integrations/node_exporter" } rule { - source_labels = ["__meta_docker_container_log_stream"] - target_label = "stream" + target_label = "instance" + replacement = constants.hostname } } - // Grafana Cloud Loki rejects entries older than 7 days (HTTP 400). - // Long-running containers replay weeks of backlog on first start — drop - // anything older than 6 days to stay safely under the cutoff. - loki.process "drop_old" { - forward_to = [loki.write.gc.receiver] - stage.drop { - older_than = "144h" - drop_counter_reason = "entry_too_old" + journal_module "integrations_node_exporter" { + forward_to = [loki.relabel.integrations_node_exporter.receiver] + } + + //JOURNAL + declare "journal_module" { + argument "forward_to" { + optional = false + } + + loki.source.journal "default" { + max_age = "12h0m0s" + forward_to = [loki.process.default.receiver] + relabel_rules = loki.relabel.default.rules + } + + loki.relabel "default" { + rule { + source_labels = ["__journal__systemd_unit"] + target_label = "unit" + } + rule { + source_labels = ["__journal__boot_id"] + target_label = "boot_id" + } + rule { + source_labels = ["__journal__transport"] + target_label = "transport" + } + rule { + source_labels = ["__journal_priority_keyword"] + target_label = "level" + } + forward_to = [] + } + loki.process "default" { + forward_to = argument.forward_to.value } } - loki.source.docker "containers" { - host = "unix:///var/run/docker.sock" - targets = discovery.relabel.container_logs.output - forward_to = [loki.process.drop_old.receiver] + prometheus.exporter.cadvisor "integrations_cadvisor" { + docker_only = true + } + discovery.relabel "integrations_cadvisor" { + targets = prometheus.exporter.cadvisor.integrations_cadvisor.targets + + rule { + target_label = "job" + replacement = "integrations/docker" + } + + rule { + target_label = "instance" + replacement = constants.hostname + } } - loki.relabel "journal" { - forward_to = [] + prometheus.relabel "integrations_cadvisor" { + forward_to = [prometheus.remote_write.metrics_service.receiver] + rule { - source_labels = ["__journal__systemd_unit"] - target_label = "unit" - } - rule { - source_labels = ["__journal_priority_keyword"] - target_label = "level" + source_labels = ["__name__"] + regex = "up|container_cpu_usage_seconds_total|container_fs_inodes_free|container_fs_inodes_total|container_fs_limit_bytes|container_fs_usage_bytes|container_last_seen|container_memory_usage_bytes|container_network_receive_bytes_total|container_network_tcp_usage_total|container_network_transmit_bytes_total|container_spec_memory_reservation_limit_bytes|machine_memory_bytes|machine_scrape_error" + action = "keep" } } - loki.source.journal "system" { - max_age = "12h" - path = "/var/log/journal" - forward_to = [loki.write.gc.receiver] - relabel_rules = loki.relabel.journal.rules - labels = { job = "systemd-journal" } + prometheus.scrape "integrations_cadvisor" { + targets = discovery.relabel.integrations_cadvisor.output + forward_to = [prometheus.relabel.integrations_cadvisor.receiver] + } + + discovery.docker "logs_integrations_docker" { + host = "unix:///var/run/docker.sock" + refresh_interval = "5s" + } + discovery.relabel "logs_integrations_docker" { + targets = [] + + rule { + target_label = "job" + replacement = "integrations/docker" + } + + rule { + target_label = "instance" + replacement = constants.hostname + } + + rule { + source_labels = ["__meta_docker_container_name"] + regex = "/(.*)" + target_label = "container" + } + + rule { + source_labels = ["__meta_docker_container_log_stream"] + target_label = "stream" + } + } + loki.source.docker "logs_integrations_docker" { + host = "unix:///var/run/docker.sock" + targets = discovery.docker.logs_integrations_docker.targets + forward_to = [loki.write.grafana_cloud_loki.receiver] + relabel_rules = discovery.relabel.logs_integrations_docker.rules + refresh_interval = "5s" }