Fix your failed uploads in Nextcloud or Immich due to CrowdSec

A failed upload in my self-hosted stack could be due to multiple reasons : Cloudflare, Caddy, CrowdSec or the service configuration. Let's go through each potential issue one-by-one to resolve these failed upload notifications !

Reason #1 : Cloudflare WAF

Cloudflare has an upload limit of 100MB if you use their proxy service.

Disabling the proxy service means your IP address is resolved directly by clients. The clients are able to see your real IP instead of Cloudflare's. Without the proxy service, you're not able to protect your website using their WAF. Indeed, Cloudflare needs the proxy to be able to intercept requests identified as malicious.

I have disabled Cloudflare proxy on the services requiring large uploads (Nextcloud, Immich...). I suggest you don't do this if you don't have a local WAF already setup.

Reason #2 : Caddy max request body size

Caddy is able to retrict the maximum body size of a request. You can find more about it bellow. However, I don't have such configuration in my Caddyfile. By default Caddy does not limit the maximum size of a request. If you're using another proxy like Nginx, I suggest you check the documentation.

Caddy - The Ultimate Server with Automatic HTTPS

Caddy is a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go

Caddy Web ServerCaddy Web Server

Reason #3: CrowdSec

I am running CrowdSec with the AppSec engine.

Looking into the documentation, you'll see there is a maximum body size check.

Before the request body is handed over to the rules engine, the Application Security Component reads it into memory itself. To protect the engine from oversized requests, the body is bounded by a maximum size (defaults to 10MB).

Hooks | CrowdSec

The Application Security Component lets you hook into different stages to change behavior at runtime.

CrowdSec

Now that we have identified the issue, we just have to create a whitelist to disable the body inspection on some URLs. This is a bit brutal because CrowdSec can be configured to only analyze the first 10MB and then let the request pass. See SetBodySizeExceededAction("partial").

The advantages of completly disabling the body inspection on the upload URL are the potential performance gains as the request does not have to go through CrowdSec processing. For me, it's also easier to understand.

Add the custom AppSec configuration to your appsec.yaml :

appsec_configs:
  - crowdsecurity/appsec-default
  - custom/upload-ignore-body-size
labels:
  type: appsec
listen_addr: 0.0.0.0:7422
source: appsec

Create your custom configuration file with the domains and URL you want to exclude :

name: custom/upload-ignore-body-size
pre_eval:
 - filter: req.Host == "immich.domain.fr" && req.URL.Path startsWith "/api/assets"
   apply:
    - DisableBodyInspection()
 - filter: req.Host == "nextcloud.domain.fr" && req.URL.Path startsWith "/remote.php/dav/uploads"
   apply:
    - DisableBodyInspection()

Now your upload with pass through Cloudflare, your proxy and CrowdSec. If an upload fails it's super likely to be your web app being misconfigured. I did not have to change either Nextcloud (with heavy client) or Immich configuration to make the uploads work properly.