GoAccess 1.11 unter Debian/Ubuntu mit Nginx einrichten: anonymisierte IP-Ausgabe, Bots ausfiltern, HTML-Report erzeugen und Statistik per Basic Auth schützen.

Links for the day

Learn how to analyze Squid proxy logs in real time using GoAccess on Linux, with interactive dashboards for traffic monitoring and troubleshooting. The post How to Set Up Real-Time Squid Proxy Analytics with GoAccess on Linux appeared first on Linux Today.

手法はGoAccessを使った解析。 https://github.com/allinurl/goaccess 生のテキストを食わせると解析してくれるのだが、Synology DSMのログセンターはご丁寧にもSQLite3で格納してしまうのでそのまま読めないのだが、Synology DSMに入っているsqlite3コマンドでテキストに出してあげればよいのだ。 docker run --name...

In 2016, I removed Google Analytics from this blog to avoid being complicit in feeding the biggest machine for harvesting personal data. Instead, I relied on GoAccess to analyze my server logs.1 For the past couple of years, the statistics have made no sense, despite my attempts to filter bots: AI scrapers inflate the number of visitors to around 2,000 per day. Eventually, I settled on GoatCounter, an open-source, privacy-friendly web analytics platform. I replaced the JavaScript client to filter bots more aggressively and added a CSS fallback. To improve reliability, I implemented a local proxy running on each of the five web servers serving this blog. The rest of this post details how these pieces fit together and how I deploy them on NixOS. ❄️ * Why GoatCounter? * Custom JavaScript client * Local proxy * Deploying on NixOS * Deploying applications in containers * GoatCounter server * GoatCounter proxy * Backup of the SQLite database with Litestream # Why GoatCounter? GoatCounter does not collect personal data: instead of storing the reader’s IP address or relying on cookies, it creates a session identifier valid for 8 hours from the user agent and the IP address. Its feature set is modest but sufficient for a blog. If you want to look at the interface, GoatCounter’s author runs a public instance for his site. A hosted version lets you try it before running your own instance. With a single binary and an SQLite database, GoatCounter is one of the lightest self-hosted solutions. Privacy-friendly alternatives, in increasing order of complexity, include Umami, Plausible, and Rybbit. # Custom JavaScript client GoatCounter includes a small JavaScript client—2,189 bytes minified and gzipped. It ships some features I don’t use: a visitor counter, tracking clicks, configurable settings, etc. I replace it with this function to register a hit: const count = ({ event, title } = {}) => { const params = new URLSearchParams({ p: event || location.pathname, t: title || document.title, r: document.referrer, q: location.search, s: document.documentElement.clientWidth, e: !!event, rnd: Math.random().toString(36).slice(2, 7), }); fetch(`/count?${params}`, { keepalive: true }).catch(() => {}); }; To filter bots,2 I go the extra mile by requiring a user interaction—an idea I stole from Bear Blog. let sendHit = () => (sendHit = () => {}, count()); ["touchmove", "mousemove", "keydown", "pointerdown"].forEach((eventName) => document.addEventListener(eventName, sendHit, { once: true, passive: true, }), ); If a reader has disabled JavaScript in their browser, I record the hit using a CSS image. The `:hover` pseudo-class loads it only after an interaction, another trick stolen from Bear Blog. About 2% of my visitors fit into this bucket.3 <!DOCTYPE html> <html lang="en" class="nojs"> <head> <script> // The JavaScript code for this blog requires ES6 if ("noModule" in HTMLScriptElement.prototype) document.documentElement.classList.remove("nojs"); </script> </head> <body> <!-- ... --> <style> .nojs body:hover { border-width: 0; border-image: url('/count?p=/en/blog/2026-kpi-goodhart&t=Building...&r=NoJS&e=false'); } </style> </body> </html> Where GoAccess reported around 2,000 visitors a day, GoatCounter counts fewer than 200 humans.4 I assume AI scrapers use a low-effort approach: if the content is available without barriers, as on this blog, they don’t spawn a complex mechanized browser that could trigger a page view. Even crawlers running JavaScript, like Googlebot with its headless Chromium, do not interact with the page and never trigger the events I listen to. The interaction-based “proof of humanity” I use is likely to keep working. # Local proxy Five servers ~~across the world~~ in Europe and in North America serve the content of this website, but GoatCounter runs on only one of them. To avoid losing track of visitors when GoatCounter is down, I run a local proxy listening on the same `/count` endpoint. On each server, it stores the hits in memory with a buffer large enough to survive several days of downtime. It sends them in batches to the upstream backend using the /api/v0/count authenticated endpoint. I proposed the code for the proxy in pull request #909. GoatCounter’s maintainer declined to maintain so much code for such a niche use case. As a fellow open-source developer, I often hold the same position for my own projects: a one-time contributor effort may translate into a long-term maintainer commitment. I expose the endpoint for the proxy on the domain of this website to evade ad blockers. This sounds like I don’t respect the reader’s choice, but as GoatCounter is privacy-friendly, I find it acceptable. location = /count { access_log off; proxy_pass http://127.0.0.3:8087/count; proxy_pass_request_headers off; proxy_set_header Accept-Language $http_accept_language; proxy_set_header User-Agent $http_user_agent; proxy_set_header X-Real-Ip $remote_addr; } # Deploying on NixOS My web servers run NixOS, a declarative Linux distribution with built-in configuration management. I manage this small fleet with Colmena, a stateless deployment tool for NixOS. My configuration is available on GitHub. ## Deploying applications in containers For better isolation, each application runs inside an ephemeral lightweight container, powered by systemd-nspawn. Each container runs a stripped-down NixOS instance. A module wraps NixOS’s containers options to avoid repeating the same options for each application.5 The containers share their network namespace with the host: the additional isolation is not worth the increased complexity. For a smaller footprint, I also disable a few non-essential services. { config, lib, ... }: let cfg = config.luffy.containers; in { # User-configurable settings for our custom module options.luffy.containers = lib.mkOption { default = { }; description = "Ephemeral containers sharing the host network."; type = lib.types.attrsOf (lib.types.submodule { options = { config = lib.mkOption { type = lib.types.deferredModule; default = { }; description = "NixOS configuration of the container."; }; }; }); }; # Translate our options to NixOS containers config = { containers = lib.mapAttrs (name: container: { ephemeral = true; autoStart = true; privateNetwork = false; extraFlags = [ "--resolv-conf=replace-host" ]; config = { imports = [ container.config ]; networking.firewall.enable = false; system.stateVersion = config.system.stateVersion; systemd.services = { console-getty.enable = false; systemd-logind.enable = false; systemd-oomd.enable = false; }; }; }) cfg; }; } To configure a GoatCounter instance running in a container and listening on `127.0.0.4:8088`, we import the module6 and declare the container in the `config.luffy.containers` attribute set: { pkgs, config, ... }: { imports = [ ./modules/container.nix ]; config.luffy.containers.goatcounter = { config = { services.goatcounter = { enable = true; address = "127.0.0.4"; port = 8088; proxy = true; }; }; }; } As the containers are ephemeral, we need to keep persistent data in directories on the host. We add a `mounts` option and ask NixOS’s containers to expose the configured directories through the `bindMounts` option. { config, lib, ... }: let cfg = config.luffy.containers; in { options.luffy.containers = lib.mkOption { type = lib.types.attrsOf (lib.types.submodule { options = { mounts = lib.mkOption { type = lib.types.listOf lib.types.str; default = [ ]; description = "Host directories mounted read-write at the same place."; }; }; }); }; config = { containers = lib.mapAttrs (name: container: { bindMounts = lib.genAttrs container.mounts (path: { hostPath = path; isReadOnly = false; }); }) cfg; }; } For example, to persist GoatCounter’s database in the `/var/db/goatcounter` directory on the host, we add the directory to the `mounts` option and alter the service definition to tell GoatCounter where the database is. { config, ... }: let databaseDirectory = "/var/db/goatcounter"; in { config.luffy.containers.goatcounter = { mounts = [ databaseDirectory ]; config = { services.goatcounter = { extraArgs = [ "-db=sqlite+${databaseDirectory}/db.sqlite" ]; }; }; }; } A container may also need some secrets. Colmena can upload secrets without storing them in the Nix store. We add a `keys` option to our containers. It takes an attribute set mapping secret names to the commands to populate them. Then, the module declares the required secrets to Colmena in the `deployment.keys` option, makes the container depend on the presence of the secrets, and exposes them to the container. { config, lib, ... }: let cfg = config.luffy.containers; in { options.luffy.containers = lib.mkOption { type = lib.types.attrsOf (lib.types.submodule { options = { keys = lib.mkOption { type = lib.types.attrsOf (lib.types.listOf lib.types.str); default = { }; description = "Secrets, as a command to run locally. They are mounted in /etc."; }; }; }); }; config = { # Colmena uploads each secret in `/var/keys` and make them available # to the group "keys". deployment.keys = lib.concatMapAttrs (_: container: lib.mapAttrs (_: keyCommand: { inherit keyCommand; group = "keys"; permissions = "0640"; destDir = "/var/keys"; }) container.keys) cfg; # The container can only start if the required secrets are available. systemd.services = lib.mapAttrs' (name: container: let units = map (key: "${key}-key.service") (lib.attrNames container.keys); in lib.nameValuePair "container@${name}" { requires = units; after = units; }) cfg; # Mount each secret inside the container. containers = lib.mapAttrs (name: container: { bindMounts = lib.mapAttrs' (key: _: lib.nameValuePair "/etc/${key}" { hostPath = "/var/keys/${key}"; isReadOnly = true; }) container.keys; }) cfg; }; } For example, GoatCounter needs credentials to download the GeoIP database. I provide a local command to fetch the secret from my password manager and expose it inside the container through the `/etc/goatcounter.env` environment file. { pkgs, config, ... }: let keyCommand = variable: [ "${pkgs.runtimeShell}" "-c" "pass show personal/nixops/secrets | grep '^${variable}='" ]; in { config.luffy.containers.goatcounter = { keys."goatcounter.env" = keyCommand "GOATCOUNTER_GEODB"; config = { systemd.services.goatcounter.serviceConfig = { EnvironmentFile = "/etc/goatcounter.env"; SupplementaryGroups = [ "keys" ]; }; }; }; } ## GoatCounter server Nixpkgs already packages GoatCounter. By overriding the `src` and `vendorHash` attributes, I reuse its definition for my custom version with the proxy: { goatcounter, fetchFromGitHub }: goatcounter.overrideAttrs (_: { src = fetchFromGitHub { owner = "vincentbernat"; repo = "goatcounter"; rev = "feature/proxy"; hash = "sha256-dJRlQlFu3tjcEgabT1LEbyFrasJlhmYu4L/T7EkoNcY="; }; vendorHash = "sha256-c9Q5OrbZR+q6pD3SgPPWe8JUzcZco1AVUKGaV61k5DE="; }) I wrote a NixOS module to encapsulate GoatCounter: the container definition, the service definition, and the secrets. The module accepts the following options: `package`, `serve.enable`, `serve.listenAddress`, `serve.port`, and `serve.databaseFile`. I already detailed the container configuration in the previous section. In the end, I chose not to reuse the GoatCounter module from NixOS: it’s small, so it’s better to insulate my module from unexpected future changes. { config, pkgs, lib, ... }: let cfg = config.luffy.goatcounter; databaseDirectory = builtins.dirOf cfg.serve.databaseFile; chown = "${pkgs.coreutils}/bin/chown -R"; in { config.luffy.containers.goatcounter = { config.systemd.services.goatcounter = { description = "GoatCounter Web Analytics"; wantedBy = [ "multi-user.target" ]; serviceConfig = { EnvironmentFile = "/etc/goatcounter.env"; SupplementaryGroups = [ "keys" ]; DynamicUser = true; Restart = "always"; ExecStart = lib.escapeShellArgs [ (lib.getExe cfg.package) "serve" "-listen=${cfg.serve.listenAddress}:${toString cfg.serve.port}" "-tls=none" "-db=sqlite+${cfg.serve.databaseFile}" "-automigrate" ]; # Transfer database ownership to dynamically assigned user "goatcounter". ExecStartPre = "+${chown} goatcounter:goatcounter ${databaseDirectory}"; ReadWritePaths = databaseDirectory; }; }; }; } The following snippet configures GoatCounter to listen on `127.0.0.4:8088`: { luffy.goatcounter = { serve = { enable = true; listenAddress = "127.0.0.4"; port = 8088; }; }; } The last step is to configure nginx to expose GoatCounter on the Internet. I disable the `/count` endpoint as the local proxy handles it. { config, ... }: let cfg = config.luffy.goatcounter.serve; in { services.nginx.virtualHosts."goatcounter.luffy.cx" = { forceSSL = true; locations = { "/" = { proxyPass = "http://${cfg.listenAddress}:${toString cfg.port}"; }; "= /count".extraConfig = '' return 404; ''; }; }; } ## GoatCounter proxy The same NixOS module configures the local proxy, with the following options: `proxy.enable`, `proxy.listenAddress`, `proxy.port`, and `proxy.site`—the site receiving the batches of page views. The local proxy has no persistent data, but it needs the API key to authenticate to the main GoatCounter instance: its container uses the `keys` option but not the `mounts` option. { config, pkgs, lib, ... }: let cfg = config.luffy.goatcounter; keyCommand = _: [ "…" ]; in { config.luffy.containers.goatcounter-proxy = { keys."goatcounter-proxy.env" = keyCommand "GOATCOUNTER_API_KEY"; config.systemd.services.goatcounter = { description = "GoatCounter Proxy."; wantedBy = [ "multi-user.target" ]; serviceConfig = { EnvironmentFile = "/etc/goatcounter-proxy.env"; SupplementaryGroups = [ "keys" ]; DynamicUser = true; Restart = "always"; ExecStart = lib.escapeShellArgs [ (lib.getExe cfg.package) "proxy" "-site=${cfg.proxy.site}" "-listen=${cfg.proxy.listenAddress}:${toString cfg.proxy.port}" "-ratelimit=10/1" # 10 requests per second per IP ]; }; }; }; } For each server, I enable the local proxy with the following snippet. The nginx configuration shown earlier exposes the `/count` endpoint under the same domain as my blog. { luffy.goatcounter = { proxy = { enable = true; site = "goatcounter.luffy.cx"; listenAddress = "127.0.0.3"; port = 8087; }; }; } ## Backup of the SQLite database with Litestream Litestream is a streaming replication tool for SQLite databases. It compresses the changes committed to the write-ahead log (WAL) next to the database and sends them to a remote destination. I encapsulate its configuration in a NixOS module, which takes an attribute set `databases` mapping a name to the path of the database to back up. Litestream also runs in a container. I mount the databases to replicate, as well as the secrets to push the backups to a Hetzner storage box using SFTP: { config, pkgs, lib, ... }: let cfg = config.luffy.litestream; databaseDirs = lib.unique (map builtins.dirOf (builtins.attrValues cfg.databases)); in { config = lib.mkIf (cfg.databases != { }) { luffy.containers.litestream = { mounts = databaseDirs; keys."litestream.env" = [ "${pkgs.runtimeShell}" "-c" "pass show personal/nixops/secrets | grep '^SQLITE_BACKUP_'" ]; }; }; } Inside the container, I configure Litestream through NixOS’s `services.litestream` options: * full snapshots every day, kept for 15 days, * three levels of compaction for transaction files: 5 minutes, 30 minutes, and 3 hours, * auto-recovery,7 * replica stored in a directory matching the host name, and * credentials read from `/etc/litestream.env` and exposed through variable expansion. { config, pkgs, lib, ... }: let cfg = config.luffy.litestream; in { config.luffy.containers.litestream = { config = { # The databases belong to dynamically allocated users, whose UID is # not known here, so Litestream runs as root. systemd.services.litestream.serviceConfig = { User = lib.mkForce "root"; Group = lib.mkForce "root"; }; # Use NixOS service. services.litestream = { enable = true; environmentFile = "/etc/litestream.env"; settings = { auto-recover = true; snapshot = { interval = "24h"; retention = "360h"; }; levels = [ { interval = "5m"; } { interval = "30m"; } { interval = "3h"; } ]; dbs = lib.mapAttrsToList (name: path: { inherit path; replica = { type = "sftp"; host = "\${SQLITE_BACKUP_HOST}"; user = "\${SQLITE_BACKUP_USER}"; password = "\${SQLITE_BACKUP_PASSWORD}"; host-key = "\${SQLITE_BACKUP_HOSTKEY}"; path = "${config.networking.hostName}/${name}"; }; }) cfg.databases; }; }; }; }; } To back up GoatCounter’s database, I declare a `goatcounter` attribute in `luffy.litestream.databases`, set to the database path: { config, ... }: let cfg = config.luffy.goatcounter.serve; in { luffy.litestream.databases.goatcounter = cfg.databaseFile; } On the SFTP server, we can inspect Litestream’s work, with the compacted transactions and the full snapshots: ❯ ls web02/goatcounter/ltx web02/goatcounter/ltx/0 web02/goatcounter/ltx/1 web02/goatcounter/ltx/2 web02/goatcounter/ltx/3 web02/goatcounter/ltx/9 ❯ ls -lh web02/goatcounter/ltx/1 29.1K Sep 5 01:25 0000000000003f2a-0000000000003f2b.ltx 72.4K Sep 5 02:03 0000000000003f2c-0000000000003f2d.ltx 63.3K Sep 5 02:24 0000000000003f2e-0000000000003f2f.ltx […] ❯ ls -lh web02/goatcounter/ltx/9 8.5M Sep 5 02:00 0000000000000001-0000000000003f2b.ltx 8.5M Sep 6 02:03 0000000000000001-0000000000004008.ltx 8.6M Sep 7 02:03 0000000000000001-00000000000043a8.ltx […] We can restore the database from the backup with a few shell commands. First, we stop the containers. Then, we move the damaged database away, invoke `litestream restore` from the right environment, and restart the containers.8 # systemctl stop container@goatcounter container@litestream # mv /var/db/goatcounter/db.sqlite{,.old} # ( . /etc/nixos-containers/litestream.conf ; > set -a ; . /var/keys/litestream.env ; set +a ; > $SYSTEM_PATH/sw/bin/litestream \ > restore -config $SYSTEM_PATH/etc/litestream.yml /var/db/goatcounter/db.sqlite) # ls -lh /var/db/goatcounter/db.sqlite -rw-r--r-- 1 root root 20M Sep 20 07:33 /var/db/goatcounter/db.sqlite # systemctl start container@goatcounter container@litestream * * * Ten years after removing Google Analytics, JavaScript-based analytics is back on this blog, but without storing cookies or IP addresses, and without involving a third party. I still write for myself first, notably because it lets me dig into a topic and refer back to it years later. But knowing a bit more about my fellow human readers is a nice bonus, even the ones disabling JavaScript. 🐐 * * * 1. Nginx scrambles IP addresses before storing them, thanks to the ipscrub nginx module. ↩ 2. GoatCounter already filters some bots based on the user agent or the IP address. But AI scrapers lie about their user agent and hide behind residential proxies. ↩ 3. Without JavaScript, I cannot send the real referrer. I insert “NoJS” instead. ↩ 4. I am missing the humans reading the RSS feed. I am not comfortable adding a tracking pixel, and bots are likely to fetch it, compromising the statistics. I’ll live without counting these readers. ↩ 5. A NixOS module is a function receiving the configuration of the whole system as `config` and returning three attributes: * `imports` adds other modules to import, * `options` declares user-configurable settings, each with a type and a default value, and * `config` sets values for options declared by any module, like `containers` from NixOS. If a module does not need to declare options, you can return the `config` attribute set directly. When a module does not require any argument, you can define it as an attribute set instead. ↩ 6. Most of the time, you don’t need an explicit import. NixOS automatically imports the modules shipped with Nixpkgs. For my own modules, some machinery also imports them automatically. ↩ 7. Litestream warns against using the auto-recover option as it can cause data loss. But I don’t properly monitor my servers and I prefer uninterrupted backups to a slight chance of losing the last few records. ↩ 8. In my case, the process is slow: around 20 minutes for a 20 MiB database. You can test by restoring to a copy with the `-o` option, but you still need to stop the Litestream container. ↩ *[WAL]: Write-ahead log *[SFTP]: Secure File Transfer Protocol

重くて複雑なGA4から卒業！タグ設置によるページ遅延をゼロにし、Core Web Vitalsを改善するNginxログ解析ツール「GoAccess」の導入ガイド。RAMディスクログを活用し、サーバーやSSDへの負荷ゼロで爆速リアルタイム解析を実現する方法を解説します。

Google Analytics（GA4）を導入したものの、「JavaScriptタグの読み込みでページ表示が遅くなる」「レポートが重くて使いにくい」「Core Web Vitalsのスコアを少しでも上げたい」とお悩みで […]

GoAccess : Analyser les logs de votre serveur web en temps réelLes fichiers de logs générés par les serveurs web contiennent des statistiques cruciales concernant la fréquentation des sites. Cependant, ces fichiers, comme le access.log, accumulent quotidiennement des centaines, voire des milliers de

Docker Compose and Nginx configuration for applications deployment on my personal server: Flask applications, Ghost, GoAccess, Joplin Server, PHP applications, Podfetch, Sandstorm, an onion service...

GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.