DEV Community: Sergej Voronko

Systemd-nspawn Homelab Guide: Containers Without Docker

Sergej Voronko — Wed, 04 Nov 2026 23:42:02 +0000

Systemd-nspawn is a namespace containerization tool built directly into systemd, allowing you to run fully isolated operating systems on your homelab without installing Docker or KVM. Most modern Linux distributions ship with it natively. Instead of running a single process behind a heavy daemon and overlay filesystem, systemd-nspawn boots a complete initialization tree directly on your host's network and storage. This guide covers how to deploy Debian 12 and Alpine Linux services using systemd-nspawn, the configurations required, and how this built-in tool compares to standard Docker setups.

Systemd-nspawn vs Docker for Homelab Servers

When evaluating systemd-nspawn vs docker, the architecture dictates how you manage your homelab services. Docker and Podman are application containers. They package a single daemon, like Nginx or PostgreSQL, into immutable layers. The Docker engine manages NAT routing, volume mounts, and logs.

Systemd-nspawn provides system containers. It operates identically to a Proxmox LXC container or a lightweight KVM virtual machine. You give it a root directory containing a full Linux filesystem, and it boots the OS inside a namespace. The container has its own PID 1, system logs, and user space.

This architectural difference changes administration. With Docker, you pull a pre-configured image and pass environment variables. With systemd-nspawn, you drop into a shell, use a package manager to install your software, and configure it using standard Linux configuration files. You get the isolation of a container with the predictable, persistent filesystem of a bare-metal server.

Storage handling presents another critical difference. Docker depends on overlayfs to stack read-only image layers with a read-write upper directory. While efficient for deploying hundreds of ephemeral web servers, overlayfs adds complexity for stateful homelab services. Persistent data in Docker requires mapping volumes or bind-mounting host paths to avoid data loss on recreation.

Systemd-nspawn sidesteps overlayfs completely. It treats a standard host directory as the literal root filesystem. If you write a file to /etc/nginx.conf inside the container, it appears instantly as a standard file on your host's disk. If your host uses a Btrfs filesystem, machinectl automatically utilizes Btrfs subvolumes for zero-copy cloning and instant snapshots.

Resource consumption also varies. Docker requires running the dockerd and containerd background daemons at all times, consuming RAM to manage the environment. Systemd-nspawn requires zero background management daemons on a modern Linux host. It is a native C binary that configures the kernel's control groups (cgroups) and namespaces, launches the guest init system, and exits the management path.

Creating a Systemd-nspawn Debian Container

To create a Debian environment, you need the base systemd container utilities and debootstrap, which constructs the basic Debian filesystem hierarchy.

sudo apt update
sudo apt install systemd-container debootstrap bridge-utils

Next, bootstrap a fresh Debian 12 (Bookworm) environment directly into the default machines directory at /var/lib/machines. Including the dbus package is a documented requirement. Without it, you cannot spawn a shell session later.

sudo debootstrap --include=systemd,dbus bookworm /var/lib/machines/debian-web

Once the filesystem finishes building, you manage the container using machinectl. Start the container and open a root shell to configure your service:

sudo machinectl start debian-web
sudo machinectl shell debian-web

Inside the shell, you are running a fully isolated Debian system. You can run apt install nginx, modify /etc/nginx/nginx.conf, and use systemctl enable nginx exactly as you would on a standalone VPS. To make the container automatically boot when your homelab server restarts, enable it on the host:

sudo systemctl enable systemd-nspawn@debian-web

Since systemd-nspawn containers are managed as standard systemd services on the host, you can apply resource limits using native systemd commands instead of Docker's specific runtime flags. If you want to limit the debian-web container to 512MB of RAM and one CPU core, you apply the limit on the host via standard cgroups:

sudo systemctl set-property systemd-nspawn@debian-web.service MemoryMax=512M CPUQuota=100%

Systemd writes these limits to a drop-in file under /etc/systemd/system.control/, and the kernel enforces them instantly.

Alpine Linux Containers with Systemd-nspawn

Alpine Linux remains popular for homelabs due to its minimal footprint. You can pull an Alpine environment using machinectl pull-tar. The official Alpine minirootfs tarballs integrate directly.

sudo machinectl pull-tar https://dl-cdn.alpinelinux.org/alpine/v3.24/releases/x86_64/alpine-minirootfs-3.24.3-x86_64.tar.gz alpine-svc

Here is a common gotcha for Alpine containers. Alpine does not use systemd; it uses OpenRC. Furthermore, the standard minirootfs tarball lacks a functioning init system by default. If you run machinectl start alpine-svc immediately, the boot process fails because it cannot locate /sbin/init.

Before booting the container, you must bypass the standard startup to manually install OpenRC. Use the systemd-nspawn command interactively:

sudo systemd-nspawn -D /var/lib/machines/alpine-svc /bin/sh

Inside the chroot environment, install OpenRC and register the basic boot services:

apk add openrc
rc-update add sysinit
rc-update add bootmisc
rc-update add root
exit

Now you can start the Alpine container via sudo machinectl start alpine-svc. Systemd-nspawn successfully executes the OpenRC initialization sequence.

Another Alpine limitation is shell access. In a Debian container, you can type machinectl shell debian-web to get a secure login prompt. This mechanism relies on dbus and systemd's login services running inside the guest. Because Alpine runs OpenRC and lacks these systemd-specific integration services, machinectl shell fails, throwing an error about a missing DBus connection.

To administer an Alpine systemd-nspawn container after it boots, you have two options. You can use the raw nspawn command to execute a shell directly:

sudo systemd-nspawn -D /var/lib/machines/alpine-svc /bin/sh

Alternatively, you can install an SSH server inside the Alpine container during your initial setup:

apk add openssh
rc-update add sshd

Once SSH runs, you manage the container exactly like any other remote Linux host on your network, decoupling your administration workflow from machinectl.

Network Configuration and Storage Mounts

Network behavior in systemd-nspawn frequently confuses administrators because the defaults change based on how you start the container.

If you run systemd-nspawn -D /var/lib/machines/debian-web directly from the command line, the container shares the host network namespace. However, when you use machinectl start debian-web, it uses the systemd-nspawn@.service template, which hardcodes the --network-veth flag. This creates a virtual ethernet pair, placing the container in an isolated private network.

To map ports from the host to your isolated container, similar to Docker's -p 8080:80 flag, you create an .nspawn configuration file. Create a new directory and file at /etc/systemd/nspawn/debian-web.nspawn:

[Exec]
Boot=yes

[Network]
VirtualEthernet=yes
Port=tcp:8080:80

Restart the container with machinectl restart debian-web. The host forwards traffic on port 8080 directly to port 80 inside the container.

If you prefer the container to obtain its own IP address directly from your homelab router, bind it to a host bridge interface instead. Assuming you have a bridge named br0 on your host:

[Network]
VirtualEthernet=yes
Bridge=br0

This configuration makes the container operate identically to a KVM virtual machine on your local network, receiving its own DHCP lease without host NAT overhead.

Homelab containers often need access to large storage pools, such as a media directory or a database backup folder. In Docker, you declare volume binds in a Compose file. With systemd-nspawn, you use the [Files] section of the .nspawn file.

To mount a host directory at /mnt/homelab-media into the container at /var/media in read-only mode, modify the /etc/systemd/nspawn/debian-web.nspawn file:

[Exec]
Boot=yes

[Files]
BindReadOnly=/mnt/homelab-media:/var/media

[Network]
VirtualEthernet=yes
Port=tcp:8080:80

These declarative configuration files completely replace Docker Compose for system containers. You define network behavior, port forwards, storage mounts, and privilege drops in one place, and systemd applies them securely during every boot sequence.

What's Next

FAQ

What is systemd-nspawn?

Systemd-nspawn is a native Linux utility that spawns namespace containers. It operates as a virtual machine manager but uses lightweight containerization, booting a full init system.

How is systemd-nspawn different from Docker?

Docker isolates single processes and relies heavily on image layers and a management daemon. Systemd-nspawn behaves more like a traditional Linux server, running a full OS tree and init system without requiring a daemon.

Does systemd-nspawn work with non-systemd Linux distributions?

Yes. While optimized for systemd guests like Debian, it can run Alpine Linux containers. You must manually install and configure an init system like OpenRC first.

Originally published at linuxcore.dev.

Fail2ban vs CrowdSec: Homelab IPS Battle

Sergej Voronko — Fri, 30 Oct 2026 23:42:02 +0000

When evaluating fail2ban vs crowdsec for a homelab IPS, the choice replaces a local regex log scanner with a distributed consensus engine. While Fail2ban parses text logs and directly appends blocking rules to your firewall, CrowdSec separates the workload into a log parser, a Local API, and distinct remediation components (commonly known as CrowdSec bouncers). For a homelab administrator, the decision comes down to whether the immediate simplicity of a single Python script outweighs the performance and shared threat intelligence of a decoupled architecture.

How Fail2ban Processes Logs and Iptables Rules

Fail2ban is the default intrusion prevention system for most self-hosted servers. The service is a Python daemon that runs regular expressions against system log files. When an IP address matches a failure condition too many times within a specific findtime window, Fail2ban triggers an action. That action usually involves appending an IP block to an iptables or nftables chain on the local machine.

The implementation requires zero external dependencies. You define a filter file containing a regex, link it to a jail in jail.local, specify a ban duration, and start the service.

The limitations appear under load. Regular expression evaluation is notoriously CPU-intensive. If a homelab exposes a public application that suffers an aggressive brute-force attack or a rapid vulnerability scan, the sudden influx of log entries forces the Python daemon to constantly evaluate complex regex rules against a fast-moving stream of text. Server CPU usage frequently spikes during these events.

Relying on vanilla iptables rules to drop connections from thousands of IP addresses forces the Linux kernel to perform linear searches. Unless Fail2ban is specifically configured to use an ipset action, a large ban list severely degrades packet forwarding performance. A large list of banned IPs means every valid incoming packet must be checked against those sequential firewall rules before reaching its destination.

Fail2ban also stores its active bans in a local SQLite database located at /var/lib/fail2ban/fail2ban.sqlite3. If the dbpurgeage parameter is not tuned correctly, this database grows massive over time and slows down service restarts.

CrowdSec Architecture and Bouncer Design

CrowdSec is written in Go and fundamentally changes the intrusion prevention model. Instead of one monolithic script handling parsing, state management, and blocking, the architecture is split into discrete services.

The core agent uses YAML configurations and Grok patterns to parse logs. Grok patterns execute much faster than raw regex evaluations in Python, reducing CPU overhead during high-volume attacks.

When the agent detects malicious behavior based on defined scenarios, it does not touch the host firewall directly. Instead, it registers a decision with the Local API (LAPI), which stores the event in a local SQLite or PostgreSQL database.

Remediation components (the CrowdSec bouncers) query this LAPI. The bouncer is responsible for enforcing the ban. If you install the host firewall bouncer, it creates an ipset or nftables set and adds the malicious IP address to it for fast lookups. Because the components communicate over an HTTP API, you can run the parsing agent on an internal web server and run the bouncer on your perimeter router.

This architecture includes a community threat intelligence feature. The LAPI periodically sends anonymous data about banned IPs to a Central API. In return, your homelab receives a curated blocklist of IPs that have attacked other CrowdSec users. Your server drops packets from known malicious actors before they ever attempt a connection.

Deploying CrowdSec via Docker Compose

Deploying Fail2ban in Docker often requires elevated privileges and messy host file mounts. CrowdSec is designed with containers in mind. The agent runs as a standard unprivileged container, and you configure your reverse proxy to share its log directory.

Below is a minimal Docker Compose snippet based on the current official CrowdSec documentation. The configuration mounts the host log directory, exposes the Local API on port 8080, and installs common collections.

services:
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: crowdsec
    environment:
      - COLLECTIONS=crowdsecurity/linux crowdsecurity/traefik crowdsecurity/http-cve
      - GID=1000
    volumes:
      - crowdsec-db:/var/lib/crowdsec/data/
      - crowdsec-config:/etc/crowdsec/
      - /var/log/auth.log:/var/log/auth.log:ro
      - /path/to/proxy/logs:/var/log/proxy:ro
    ports:
      - "127.0.0.1:8080:8080"
    restart: unless-stopped

volumes:
  crowdsec-db:
  crowdsec-config:

The COLLECTIONS environment variable automatically downloads parsers and scenarios from the CrowdSec Hub. A collection is a bundled set of parsing rules and detection thresholds for a specific service. The Local API binds to 127.0.0.1 so that host-level bouncers can query it securely.

Connecting CrowdSec Bouncers to the LAPI

Running the CrowdSec agent alone only populates a database with decisions; it does not block anything. You must install a bouncer to apply the remediations.

Bouncers come in two primary categories: Infrastructure-level (Layer 3/4) and Application-level (Layer 7).
The infrastructure bouncer (crowdsec-firewall-bouncer) runs directly on the host operating system and issues kernel-level drops.
The application bouncers integrate directly into reverse proxies like Traefik, Caddy, or Nginx. Instead of dropping the packet at the kernel, a Layer 7 bouncer can return a 403 Forbidden HTTP error or serve a CAPTCHA challenge to suspicious IPs.

To connect a bouncer, you must generate an API token using the cscli command line tool inside the CrowdSec container:

docker exec crowdsec cscli bouncers add homelab-firewall

The command outputs an alphanumeric token. You paste this token into the bouncer's configuration file and restart the bouncer service.

Homelab Gotchas: WAL Mode and Docker Networking

The distributed design introduces management complexity. Moving from Fail2ban to CrowdSec requires learning a new taxonomy of agents, scenarios, parsers, and bouncers. You will rely heavily on the cscli tool to inspect decisions and manage collections.

The SQLite database lock issue is a documented gotcha for high-traffic homelabs. By default, the CrowdSec container initializes a standard SQLite database. When multiple logs are parsed rapidly and the bouncer pulls updates simultaneously, the database can throw a "database is locked" error in the container logs. The fix requires manually adding use_wal: true to the db_config section of /etc/crowdsec/config.yaml to enable Write-Ahead Logging, followed by a container restart.

Docker networking creates another frequent issue with bouncers. If you run a bouncer in a separate Docker container on a custom bridge network, it communicates with the LAPI via a dynamic container IP address. If the bouncer container updates or restarts, Docker assigns it a new internal IP. The CrowdSec LAPI sees a new connection and leaves the old bouncer registered. Over time, stale bouncer registrations accumulate. Administrators must implement static container IPs or prune inactive bouncers periodically using the cscli bouncers prune command.

Conversely, Fail2ban requires almost no maintenance once the jail is dialed in. If you have a single virtual private server or a single home server running SSH, Fail2ban is highly effective. You install the package, enable the sshd jail, and the configuration survives system reboots and package updates without demanding attention.

When to Stick With Fail2ban vs CrowdSec

A homelab running a simple WireGuard endpoint and SSH does not require CrowdSec. Fail2ban provides immediate protection with minimal setup. The CPU penalty of regex parsing is irrelevant when your public attack surface is restricted to a single port and traffic volume is low.

Homelabs hosting multiple public-facing web applications behind a reverse proxy gain significant advantages from CrowdSec. The Go-based log parsing handles heavy traffic without spiking processor usage. The community threat intelligence preemptively drops known malicious scanners, reducing the sheer volume of traffic hitting your applications. Most importantly, the bouncers allow you to decouple detection from enforcement, parsing proxy logs safely inside an isolated container while executing the ban via the host network stack.

What's Next

FAQ

What is the main difference between Fail2ban and CrowdSec?

Fail2ban uses local regex log parsing to update iptables directly. CrowdSec uses a decoupled architecture with Go-based grok parsing and a distributed threat intelligence network.

Can Fail2ban and CrowdSec run together?

Running both on the same logs creates duplicate firewall rules and race conditions. Pick one intrusion prevention system per service.

Do CrowdSec bouncers cost money?

The core CrowdSec engine and standard bouncers are free and open-source. The project limits API pulls for enterprise volume, but standard homelab usage remains free.

Originally published at linuxcore.dev.

Astro MD vs MDX: Content Collection Build Performance

Sergej Voronko — Wed, 28 Oct 2026 23:42:02 +0000

Standard Markdown (.md) and MDX (.mdx) process content drastically differently within Astro content collections. The build pipeline parses standard Markdown as raw text data, requiring minimal memory and scaling linearly. The build process compiles MDX files into full JSX components, forcing the Vite bundler to process every .mdx file as a discrete JavaScript module. This architectural difference means MDX processing introduces significant parser overhead and increases build times on low-resource homelab CI runners. Use standard Markdown for static text and reserve MDX strictly for files requiring embedded interactive components.

Astro Markdown vs MDX: Parser Architecture

Standard .md files in Astro bypass the JavaScript module graph entirely. When the content collections API processes a .md file, the parser reads the frontmatter for validation against your defined Zod schema. The standard remark and rehype plugins then transform the body text into HTML. The default Astro pipeline relies on a Rust-powered processor that significantly improves build times. The content remains isolated from the application's core JavaScript bundle.

MDX skips lightweight text processing. When you install the @astrojs/mdx package, the compiler evaluates every .mdx file as a functional component. The parser transforms your Markdown syntax into JSX, injects imports for any Astro, Vue, or React components referenced in the file, and hands the resulting code to Vite. The pipeline then adds these modules to the overall dependency graph.

This fundamental shift alters how the server builds the site. A basic Markdown file maps directly to static HTML output. An MDX file requires the build system to resolve imports, manage potential hydration directives, and pass the output through the Vite build pipeline before generating that same static HTML.

Build Speeds and Parser Overhead

If you run builds on a minimal Proxmox virtual machine or a cheap VPS, the performance delta between .md and .mdx becomes a hard limit. A content collection containing 1,000 standard Markdown files compiles in under a few seconds. Converting those same 1,000 files to MDX pushes the build time into the double digits and routinely triggers out-of-memory kills on runners with under 2GB of RAM.

The performance drop stems from the compilation pipeline. When the build pipeline encounters an MDX file, it does not just render text. The @mdx-js/mdx compiler parses the document into a Markdown Abstract Syntax Tree (AST), applies any configured remark plugins, and converts the output to an HTML AST. The compiler then applies rehype plugins and maps the resulting structure into raw JSX syntax. Finally, the process passes this JSX to the internal bundler to transpile into standard JavaScript. This entire chain executes for every single MDX file. Processing 2,000 blog posts requires 2,000 distinct JavaScript module compilations.

When a homelab CI runner hits the default Node.js heap limit during an Astro build, the pipeline crashes. You can temporarily bypass the crash by forcing Node to use more memory, but this is a patch, not a fix.

# Expanding the heap size for heavy MDX builds
NODE_OPTIONS="--max-old-space-size=4096" npm run build

The proper fix is limiting MDX usage to files that actually require it.

Component Architecture and Hydration

Standard Markdown restricts you to standard HTML tags. You can apply custom styling via CSS, but you cannot embed stateful client-side code directly into the prose.

Many developers assume they need MDX just to add custom CSS classes to alerts or styled callouts. You do not need MDX for basic structural changes. You can write or import custom remark and rehype plugins for standard Markdown that automatically inject HTML div wrappers or CSS classes around standard blockquotes. Standard Markdown in Astro also utilizes Shiki for syntax highlighting by default, meaning you do not need MDX just to render formatted code blocks.

MDX exists specifically to import UI components directly into the content body. This is necessary when an article requires a functional, interactive element, like a live search bar, a dynamic data table fetched from an API, or a React component tracking client-side state.

---
title: "Server Status"
pubDate: 2026-10-27
---

import PingTest from '../../components/PingTest.jsx';

The current server latency is listed below. Click the button to refresh the ping.

<PingTest client:load target="192.168.1.100" />

Because Astro uses an Islands architecture, the MDX file itself still renders to static HTML. Client-side JavaScript only ships to the browser if you explicitly append a client directive like client:load or client:visible to the embedded component. Without the directive, the component renders strictly on the server.

Implementing MDX in Astro Content Collections

If your project requires embedded islands, you must explicitly enable MDX integration and configure your content loader to recognize the extension.

Install the official integration using the Astro CLI:

npx astro add mdx

Optimize the @astrojs/mdx integration in your configuration file. The optimize flag instructs the internal rehype plugin to generate a smaller abstract syntax tree, which cuts down build times and reduces memory consumption for large collections. If you do not require standard Markdown configuration settings to apply to your MDX files, setting extendMarkdownConfig to false ignores inherited project-level Markdown options entirely.

// astro.config.mjs
import { defineConfig } from 'astro/config';
import mdx from '@astrojs/mdx';

export default defineConfig({
  integrations: [
    mdx({
      optimize: true,
      extendMarkdownConfig: false
    })
  ]
});

Next, configure your content collection loader to target both file types. Astro uses the glob loader for local file fetching. You can mix .md and .mdx files in the same directory by defining a multi-extension pattern. The [^_] regex ensures Astro ignores files prefixed with an underscore.

// src/content/config.ts
import { defineCollection, z } from 'astro:content';
import { glob } from 'astro/loaders';

const docsCollection = defineCollection({
  loader: glob({ pattern: '**/[^_]*.{md,mdx}', base: './src/content/docs' }),
  schema: z.object({
    title: z.string(),
    pubDate: z.date(),
    draft: z.boolean().default(false)
  })
});

export const collections = { docs: docsCollection };

When rendering the collection in a dynamic route, the render() function parses both file types automatically.

---
// src/pages/docs/[...slug].astro
import { getCollection, render } from 'astro:content';

export async function getStaticPaths() {
  const docs = await getCollection('docs');
  return docs.map(entry => ({
    params: { slug: entry.id },
    props: { entry },
  }));
}

const { entry } = Astro.props;
const { Content } = await render(entry);
---

<html>
  <head>
    <title>{entry.data.title}</title>
  </head>
  <body>
    <article>
      <Content />
    </article>
  </body>
</html>

When to Choose Raw Astro Markdown

Stick to .md files if your content consists only of headings, paragraphs, standard code blocks, and images. The native Astro content collections API handles all frontmatter validation and basic HTML rendering automatically.

Standard Markdown formatting keeps your content portable. If you migrate away from Astro in the future, raw Markdown transfers cleanly to Hugo, Jekyll, or any other static site generator. MDX files littered with framework-specific component imports require manual clean-up and regex replacements to become standard text again.

Integrating a headless CMS like Decap or Ghost is much simpler with standard Markdown. Remote content management systems typically export plain text or JSON. Forcing remote content into an MDX pipeline requires custom loaders and complex hydration strategies that break the native Astro content flow.

Use .mdx exclusively when a specific document requires interactivity that standard Markdown cannot provide. You can safely mix .md and .mdx within the same Astro content collection. Keep the bulk of your site as fast-compiling Markdown, and selectively upgrade individual files to MDX when the content demands custom React, Vue, or Svelte components. This split approach maintains fast build times on limited CI runners while preserving the ability to embed complex UI elements.

What's Next

FAQ

Can I mix Markdown and MDX files in the same Astro content collection?

Yes. The glob loader accepts multiple extensions. Configure the pattern to '*/[^_].{md,mdx}' to process both file types in a single collection.

Does MDX ship JavaScript to the browser by default?

No. MDX compiles to HTML at build time. Client-side JavaScript only ships if you append a client directive like client:load to the embedded framework component.

Why are my Astro MDX builds failing with out-of-memory errors?

Vite bundles every MDX file as a discrete JavaScript module. Large MDX collections overwhelm CI runners with limited RAM. Enable the optimize flag in the MDX integration or increase runner memory.

Originally published at linuxcore.dev.

Expose Share Links on Private Homelab Apps With Caddy

Sergej Voronko — Fri, 23 Oct 2026 22:42:02 +0000

Homelab administrators often lock down self-hosted applications using an identity provider like Authelia or Authentik. This creates a hard authentication barrier before a visitor reaches the application backend. Strict zero-trust access blocks brute-force attacks and vulnerability exploits, but it breaks a common workflow: sharing a public link with users who do not have an SSO account. You can solve this by configuring Caddy to bypass the authentication gateway for specific URLs, exposing only the exact paths required for the share links to function.

Caddy Forward Auth Mechanics

Caddy provides the forward_auth directive to handle external authentication. It proxies a clone of the incoming HTTP request to an authentication gateway. If the gateway returns a successful 2xx status code, Caddy copies the authorized user headers and passes the request to your application. If the gateway returns any other status code, Caddy forwards the gateway's response back to the client. This usually triggers an HTTP 302 redirect to the SSO login portal.

To protect an entire application, a basic Caddyfile block looks like this:

nextcloud.example.com {
    forward_auth authelia:9091 {
        uri /api/verify?rd=https://auth.example.com
        copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
    }
    reverse_proxy nextcloud:80
}

This configuration rejects all unauthenticated traffic. No public share links will load under this block because the authentication gateway intercepts every request.

Configuring Homelab Reverse Proxy Public Share Links

To allow external users to access specific files without a login, you must split the Caddy configuration. Caddy uses named matchers and the handle directive to apply rules selectively based on the request path.

For Nextcloud, public shares are generated under the /s/ path. Bypassing this single path is not enough. The user will see a broken HTML page without styling. Nextcloud requires CSS, JavaScript, and image assets located in other directories to render the share page correctly.

This Caddyfile snippet bypasses authentication for the required paths and protects the rest of the application:

nextcloud.example.com {
    # Match the share path and necessary asset directories
    @public path /s/* /index.php/s/* /apps/* /core/* /dist/* /img/* /js/* /css/* /avatar/* /svg/*

    # Allow public access to the matched paths
    handle @public {
        reverse_proxy nextcloud:80
    }

    # Protect everything else with Authelia
    handle {
        forward_auth authelia:9091 {
            uri /api/verify?rd=https://auth.example.com
            copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
        }
        reverse_proxy nextcloud:80
    }
}

This setup routes any request matching the @public rules directly to Nextcloud. All other requests fall through to the second handle block and hit the forward_auth directive.

You apply the same logic if you use Authentik instead of Authelia. The URI endpoint and the copied headers differ, but the Caddy handle structure remains identical:

    # Protect everything else with Authentik
    handle {
        forward_auth authentik:9000 {
            uri /outpost.goauthentik.io/auth/caddy
            copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid
        }
        reverse_proxy nextcloud:80
    }

The API Trap with Immich

Not all applications handle public shares using simple static HTML paths. Modern platforms often operate as single-page applications that rely heavily on backend APIs.

When you create a shared link in Immich, the frontend loads via the /share/ path. The frontend then makes asynchronous calls to the /api/ endpoints to fetch the actual photo data. If you bypass just the /share/ path, the user gets a blank interface. The initial page loads, but Authelia blocks the subsequent API requests.

Whitelist the entire /api/ path and you expose the administrative endpoints, nullifying the security benefits of the reverse proxy. To securely expose Immich share links via Caddy, you must whitelist the exact API routes required for shared links. These specific routes change across major updates.

Because maintaining a precise list of API paths requires constant intervention, administrators often deploy the community-maintained Immich Public Proxy. This runs as a separate, stateless Docker container that exclusively handles shared links. It strips metadata and blocks unauthorized paths automatically, removing the need to modify your Caddy forward auth rules for the main Immich instance.

Caddy Protect Self Hosted Apps and Mobile Clients

Many administrators implement Caddy protect self hosted apps setups to block automated scanners, but this approach breaks native mobile clients. Mobile applications for Nextcloud, Immich, and Jellyfin expect direct access to the application API. When Caddy intercepts a mobile app's API request and returns an Authelia HTML login redirect, the mobile app fails to parse the response. The application crashes or hangs. It will not display the SSO login portal.

You have three options to handle native mobile applications:

Bypass the API paths completely. This reduces the security benefit of the identity provider because scanners can hit the application's login form directly.
Require a VPN like Tailscale or WireGuard on mobile devices. The application stays fully internal, and you only use the public domain for web access.
Use application-specific bypass headers.

Some mobile clients allow you to inject custom HTTP headers into your requests. You can configure Caddy to bypass forward_auth if a specific secret header is present.

immich.example.com {
    @mobile_bypass header Immich-Skip-Authelia "SuperSecretString123"

    handle @mobile_bypass {
        reverse_proxy immich:2283
    }

    handle {
        forward_auth authelia:9091 {
            uri /api/verify?rd=https://auth.example.com
            copy_headers Remote-User
        }
        reverse_proxy immich:2283
    }
}

This configuration acts as a static API key. If the header is present, the request bypasses Authelia. You must use TLS (HTTPS) on your public domain to encrypt the header in transit and prevent interception.

Preventing Header Spoofing Vulnerabilities

When Caddy validates a request through the authentication gateway, it copies identity headers to the backend application. The application reads these headers to log the user in automatically.

You must ensure external users cannot spoof these headers to gain administrative access on the bypassed public routes. Caddy normally clears incoming headers before passing them to the application if configured properly, but explicit bypass blocks require manual intervention.

In your Caddyfile, ensure the reverse_proxy directive does not implicitly trust incoming identity headers from the outside. If an attacker visits a bypassed path like /s/xyz and manually includes a Remote-User: admin HTTP header, Caddy must strip it. The forward_auth directive automatically overwrites these headers on a successful login, but for paths in the @public block, no overwrite occurs.

You must manually strip the headers from unauthenticated requests to prevent privilege escalation:

handle @public {
    request_header -Remote-User
    request_header -Remote-Groups
    request_header -Remote-Name
    request_header -Remote-Email
    reverse_proxy nextcloud:80
}

This directive removes the headers before the request reaches Nextcloud.

The backend application also needs verification. Nextcloud requires the trusted_proxies array in its config.php file to validate where the headers come from.

'trusted_proxies' => ['192.168.1.100'],

Set this to the IP address of your Caddy server or Docker network gateway. If you leave trusted_proxies empty or misconfigured, Nextcloud may accept spoofed headers from untrusted sources, rendering your SSO setup useless.

What's Next

FAQ

Does forward_auth break native mobile apps?

Yes. Mobile apps expect direct API access and fail when redirected to a login portal. You must either bypass the API paths, use custom headers, or rely on a VPN for mobile access.

Do I need to bypass CSS and JS paths for share links?

Usually, yes. Nextcloud share pages rely on assets in the /apps/ and /core/ directories. Bypassing only the /s/ path results in a broken layout.

Can I use Authentik instead of Authelia?

Yes. Caddy's forward_auth directive works with any authentication gateway that supports the forward authentication specification.

Originally published at linuxcore.dev.

Amazon ECS Gateway: Cheap Ingress for AWS Homelabs

Sergej Voronko — Thu, 22 Oct 2026 04:46:56 +0000

Amazon ECS Gateway solves the most expensive part of running hybrid homelab workloads on AWS: the idle Application Load Balancer (ALB). Instead of permanently running a dedicated ALB at $16.42 per month base fee (in us-east-1) at the time of writing just to wait for failover traffic, this managed ingress resource dynamically provisions your load balancer only when a container service connects. When your homelab is healthy and your cloud tasks scale to zero, ECS removes the ALB, completely eliminating the standby cost.

Before October 2026, exposing an Amazon Elastic Container Service (ECS) cluster cheaply required painful trade-offs. You either ran AWS Fargate tasks with public IP addresses, which change on every deployment and force you to script your own Route 53 DNS updates, or you provisioned an ALB. An ALB provides static endpoints, AWS Certificate Manager (ACM) TLS certificates, and clean host-based routing, but AWS bills a base hourly rate regardless of traffic volume. For a homelab admin keeping a secondary Nextcloud or Bitwarden instance in AWS for disaster recovery, paying nearly $200 a year for an idle load balancer defeats the purpose of cheap cloud storage.

Amazon ECS Gateway completely abstracts the load balancer away from individual services. It operates as an account-level infrastructure manager. You define your ingress standards just once, specifying your VPC subnets, security groups, and listener rules.

When you create or update an ECS service, you opt in by adding a gateway connector parameter. ECS then automatically provisions the required resources directly in your account. If it is the first service connecting, ECS creates an ALB. It generates a unique DNS hostname, requests an ACM certificate, and configures the target groups automatically.

Sharing ALBs for Cheap ECS Ingress

Running multiple lightweight containers in the cloud usually causes ALB sprawl. Standard configurations require either multiple ALBs or manual manipulation of listener rules to share one. ECS Gateway handles shared hosting natively. A single gateway supports up to 25 services sharing the same underlying ALB using host-based routing. You pay only for the actual infrastructure provisioned in your account. Whether you expose one homelab dashboard or twenty separate microservices, your base AWS bill remains tied to a single load balancer.

Once you connect a 26th service, ECS automatically provisions a second ALB to handle the capacity, immediately doubling your base infrastructure cost. Keep your total active service count under the limit if budget optimization is your primary goal.

Before ECS Gateway, sharing an ALB meant manually creating Route 53 DNS records for each new service and mapping them to the ALB canonical name. ECS Gateway handles the DNS provisioning automatically. Each service connecting to the gateway receives a unique DNS hostname instantly. You can point your homelab CNAME records directly at these dynamically generated endpoints. When the gateway tears down the ALB during scale-to-zero operations, your CNAME remains intact, ready to route traffic the moment the ALB spins back up.

Scale-to-Zero AWS Cloud Fallback

The defining feature for hybrid homelabs is automated lifecycle management. The official documentation confirms that ECS removes the underlying ALB when no services remain connected to the gateway.

If your local server loses power, you scale your standby ECS services from zero to one. ECS registers the connected service, provisions the ALB, requests the ACM certificate, and begins routing traffic. Your disaster recovery site comes online with secure ingress automatically configured. Once your local Proxmox host is restored and you scale the AWS services back to zero, ECS tears down the ALB. Your monthly AWS bill drops back to standard object storage and idle database rates.

Creating an Amazon ECS Gateway

Setting up an ECS Gateway requires an IAM role granting ECS permission to provision ALBs, security groups, and certificates on your behalf. Once the IAM role is ready, you create the gateway using the AWS CLI.

aws ecs create-gateway \
  --name homelab-fallback \
  --infrastructure-role-arn arn:aws:iam::123456789012:role/ECSGatewayRole \
  --configuration '{ "HostSharedAlb": { "networkConfiguration": { "subnets": [ "subnet-0abcd1234efgh5678" ] } } }'

The --configuration flag accepts a JSON object defining your infrastructure limits. By specifying HostSharedAlb, you instruct the gateway to use standard Application Load Balancers configured for host-based routing. The networkConfiguration block accepts your VPC subnets, dictating exactly where the load balancer interfaces are placed. If your ECS tasks sit in private subnets, ensure you select public subnets here so the ALB can route external internet traffic into your private tasks.

Specific configuration settings are immutable. Your gateway name, traffic routing scheme, VPC, subnets, and listener port cannot be modified after creation. Modifying your subnets later requires creating an entirely new gateway and migrating your active services over.

ECS awsvpc Network Mode Requirements

When defining your ECS task, you must specify the awsvpc network mode. ECS Gateway relies on this specific networking mode to enforce security group rules. ECS automatically generates a service security group that permits inbound traffic exclusively from the ALB. This completely isolates your containers from direct internet access, even if placed in a public subnet.

This requirement requires no changes if you use AWS Fargate, as awsvpc is the default and only supported mode. If you run your containers on self-managed EC2 instances to save money, you must ensure your task definitions explicitly declare the awsvpc mode.

AWS WAF and Security Integration

Publicly exposing internal services carries risk, especially during an automated failover where you might not be actively monitoring logs. Previously, attaching an AWS WAF required manually linking it to an existing ALB. ECS Gateway accepts a WAF Web ACL directly in the core gateway configuration.

When ECS provisions the underlying ALB, it automatically attaches the firewall rules. You can define IP allowlists limiting access strictly to your home static IP or your WireGuard exit node. If your homelab fails over to AWS, the WAF rules immediately protect the exposed containers from automated scraping and exploitation attempts.

ECS Gateway IAM Role Requirements

Because ECS Gateway automates infrastructure creation, it depends on an infrastructure IAM role. ECS assumes this role to generate the ACM certificates, deploy the ALB, and configure the necessary security groups. You must create a dedicated role with trust policies allowing the ecs.amazonaws.com service principal to assume it.

Never reuse your standard ECS execution task roles for this purpose. The infrastructure role requires extensive permissions for Elastic Load Balancing, ACM, and Amazon EC2 security group management. Keeping this role strictly scoped prevents an attacker inside a compromised container from provisioning arbitrary AWS networking infrastructure.

Monitoring ECS Gateway Costs

While the load balancer provisioning happens automatically in the background, the resulting ALB exists directly in your AWS account. You retain full visibility into the network traffic. Standard ALB metrics automatically populate in Amazon CloudWatch. You can inspect target health, request latency, and HTTP 5xx error rates exactly as you would with a manually provisioned ALB.

If a container crashes, standard Elastic Load Balancing target health checks mark the task unhealthy and stop routing traffic. Because the resources live visibly in your account, they also show up in AWS Cost Explorer, allowing you to track the exact hourly spend of your ALB while your disaster recovery site remains active.

What's Next

FAQ

What is Amazon ECS Gateway?

It is an account-level resource that automates ingress for ECS services by dynamically provisioning Application Load Balancers, TLS certificates, and DNS records on demand.

Does Amazon ECS Gateway cost extra?

No. The gateway itself is free. You only pay for the underlying infrastructure (like ALBs) while it is actively provisioned for connected services.

Can I use ECS Gateway for Fargate tasks?

Yes. ECS Gateway supports the awsvpc network mode, which is required for AWS Fargate.

Originally published at linuxcore.dev.