Monitoring a Home Lab Exposed to the Internet

Why bother

Exposing services to the internet means anyone can now hit the home server, not just devices on the LAN. Most of that traffic is noise - scanners probing for .env files, bots trying default WordPress paths that don’t exist here, the usual internet background radiation. Some of it isn’t.

Without visibility, “someone is probing my server” and “nothing is happening” look identical from the couch. So the home lab also runs a small monitoring stack: logs and metrics from Caddy land in Grafana.

A slice of what actually hits the server

A few hours of real Caddy access log entries, straight from Loki - the DNS-over-HTTPS probe, the WordPress/phpMyAdmin path scans, the bot that burned through eight .php payloads against the same IP in under four seconds:

The stack

Everything runs as docker-compose projects on the home server itself - nothing sent off to a third party:

Getting logs into Loki

Caddy already writes structured JSON logs to a file and to stdout:

1
(logs) {
2
  log {
3
        output file /var/log/caddy/access.log {
4
            roll_size 10mb
5
            roll_keep 5
6
        }
7
        format json
8
    }
9

10
  log {
11
    output stdout
12
    format json
13
  }
14
}

Alloy is the piece that turns “a JSON blob in a container’s stdout” into “a searchable, labeled log line in Loki”. Its config is built around discovery.docker, which watches the Docker socket and finds the running containers, filtered down to the one named caddy:

1
discovery.relabel "caddy" {
2
  targets = discovery.docker.containers.targets
3

4
  rule {
5
    source_labels = ["__meta_docker_container_name"]
6
    regex         = "/caddy"
7
    action        = "keep"
8
  }
9
}
10

11
loki.source.docker "caddy" {
12
  host       = "unix:///var/run/docker.sock"
13
  targets    = discovery.relabel.caddy.output
14
  forward_to = [loki.process.caddy_json.receiver]
15
}

The actual parsing happens in a loki.process pipeline: pull fields out of the JSON, enrich the client IP with a GeoIP lookup, and turn a handful of those fields into Loki labels so they’re cheap to filter and group by later:

1
loki.process "caddy_json" {
2
  forward_to = [loki.write.default.receiver]
3

4
  stage.json {
5
    expressions = {
6
      request_uri  = "request.uri",
7
      status       = "status",
8
      duration     = "duration",
9
      client_ip    = "request.client_ip",
10
      user_agent   = "request.headers.User-Agent[0]",
11
    }
12
  }
13

14
  stage.geoip {
15
    source  = "client_ip"
16
    db      = "/etc/geoip/GeoLite2-City.mmdb"
17
    db_type = "city"
18
  }
19

20
  stage.labels {
21
    values = {
22
      status             = "status",
23
      client_ip          = "client_ip",
24
      geoip_country_name = "geoip_country_name",
25
      geoip_city_name    = "geoip_city_name",
26
    }
27
  }
28
}

That stage.geoip block is the one that makes “who is hitting my server, and from where” an actual question you can answer instead of a column of raw IP addresses. Alloy resolves every client_ip against a local MaxMind GeoLite2 city database (a flat .mmdb file mounted read-only, no external API calls per request) and attaches country/city/lat/lon as labels.

Authelia (the auth layer sitting in front of some subdomains) gets the same treatment through a second loki.process block, so failed/successful login attempts show up in Loki labeled by username and result, independent of the Caddy pipeline.

Metrics: Prometheus

Alongside logs, Prometheus pulls numeric time series on a 15 second interval:

1
scrape_configs:
2
  - job_name: prometheus
3
    static_configs:
4
      - targets: ["prometheus:9090"]
5

6
  - job_name: caddy
7
    static_configs:
8
      - targets: ["caddy:2019"]

Caddy exposes its own metrics on the admin API port (:2019) once metrics { per_host } is turned on in the global options block - request counts and latencies broken down per virtual host, so a spike on one subdomain doesn’t get averaged away by traffic on the others.

Logs answer “what happened, in detail, on this one request”. Metrics answer “is anything trending in the wrong direction, in aggregate”. Grafana sits on top of both, so a dashboard panel showing a spike in 403s can link straight into the matching Loki query to see exactly which requests caused it.

Grafana dashboards

This is where logs and metrics actually become useful day to day, rather than just sitting in storage.

The GeoIP labels attached back in the Alloy pipeline make a request geomap possible - a literal map of where traffic is actually coming from, which makes “why do I have thousands of requests from a country nobody in this household has ever been to” a visual, obvious pattern instead of something buried in a log line.

Wrapping up

None of these pieces are doing anything exotic on their own - Alloy parsing JSON, Loki storing logs, Prometheus scraping numbers. What matters is that they all read from the same two sources (Caddy’s access logs and its metrics endpoint), so a Grafana dashboard panel and a raw log line can always be cross-referenced for the same request. Exposing a home lab to the internet without this is running blind; this is the minimum to actually see what’s hitting it. Blocking the obviously malicious stuff automatically is a separate concern, worth its own post.