TLS / HTTPS in Self-Hosting

Problem

I have a bunch of services running on my VPS. Most of them are exposed to the Internet. For those, the authentication I rely on is Cloudflare Zero Trust. To log in to the services that are behind this authentication, one will be asked to log in with their email (you can decide what email addresses is allowed) and one-time password, or through a OIDC I also self-host.

Self-hosted applications | Cloudflare Zero Trust docs

Cloudflare Access allows you to securely publish internal tools and applications to the Internet by providing an authentication layer between the end user and your origin server. You can use signals from your existing identity providers (IdPs), device posture providers, and other rules to control who can access your application.

Cloudflare Docs

However, I have services that cannot be exposed to the broader Internet at all. For example, AdGuard Home.

GitHub - AdguardTeam/AdGuardHome: Network-wide ads & trackers blocking DNS server

Network-wide ads & trackers blocking DNS server. Contribute to AdguardTeam/AdGuardHome development by creating an account on GitHub.

GitHubAdguardTeam

As a DNS server, it does not come with authentication mechanism at all. If you decide to host it to the Internet, you basically cannot stop anyone from accessing it. Zero Trust also won't work because your clients won't know how to get pass by the Cloudflare's authentication by themselves, so it will be blocking all traffic from reaching AdGuard Home.

In this case, you have to serve this kind of applications in a LAN, and use tools like VPN to connect to the LAN when you want to use the applications. So, nobody can even access the applications without connecting to your LAN because the IPs will be some private ones.

However, this poses a new problem. If you just access those apps with raw IP addresses or have some internal DNS setup, you can only connect to those with HTTP. Most browsers yell at your or warn you about the danger of websites that don't support HTTPS.

In one sentence, how to get HTTPS / SSL working for our internal services that are running on a local network such as your home lab's?

Get to know about HTTPS

I won't be writing about the basic concepts of HTTPS / SSL and how that works, as other people have already done great jobs of explaining these. I recommend the following resources.

This video above explains how SSL works, how to get a free SSL from Let's Encrypt, how to use certbot to make the process easier.

This video demonstrates how to use Nginx Proxy Manager to let it easily deal with Let's Encrypt certificates issuing and renewal. Also, it talks about how to get SSL certificates for services that you don't intend to expose to the Internet (i.e. your home lab).

DNS Challenge

Read this doc for more about different challenge types, and specifically, the DNS Challenge. This, compared to HTTP Challenge, does not require your server to have ports reachable from Internet.

Challenge Types

When you get a certificate from Let’s Encrypt, our servers validate that you control the domain names in that certificate using “challenges,” as defined by the ACME standard. Most of the time, this validation is handled automatically by your ACME client, but if you need to make some more complex configuration decisions, it’s useful to know more about them. If you’re unsure, go with your client’s defaults or with HTTP-01.

Let's Encrypt

Most of the time, this challenge needs API token from your DNS provider because the process of challenge requires changing of some TXT records of the DNS of your hostname.

Here comes the interesting part. Due to the reason that different DNS providers have different APIs, the popular reverse proxies might or might not have this DNS Challenge functionality built in.

Nginx Proxy Manager

It is built in, so you just need to go to the UI and choose the DNS provider in the drop-down menu.

Traefik

Traefik Docker DNS Challenge Documentation - Traefik

Learn how to create a certificate with the Let’s Encrypt DNS challenge to use HTTPS on a Service exposed with Traefik Proxy. Read the technical documentation.

logo

It's also built-in.

Caddy

How to use DNS provider modules in Caddy 2

Caddy 2 uses a new and improved DNS provider interface for solving the ACME DNS challenge. All you have to do is plug the service provider(s) you need into your build, then add the DNS challenge to your configuration! Getting a DNS provider plugin How you choose to get a custom Caddy build is up to you; we’ll describe two common methods here. Method 1: Go to the Caddy download page. Find your DNS provider in the list of modules (dns.providers.*) and select it. Download your custom Caddy bu…

Caddy Communitymatt

However, for Caddy, it's not the same. Different providers show as different modules, and you need to pick the ones you need and have a custom build of Caddy.

How to with Coolify?

It will be different depending on which proxy you use, Traefik or Caddy.

Traefik

How to create wildcard SSL certificates with Traefik in Coolify

A guide to configure wildcard subdomain redirects (with Traefik wildcard certificates) in Coolify.

Coolify

Thankfully, there is an official doc for this.

Caddy

This is where things become interesting. As described above, Caddy has DNS Challenge providers as modules. At the time of this writing, there is no way to specify which variant of Caddy you want in Coolify. So we have to figure something out on our own.

After some trying, this is what I figured out.

Create a new certbot project

Certbot with Cloudflare DNS using DNS-01 challenge in Docker

Let’s Encrypt will issue you free SSL certificates, but you have to verify you control the domain, before they issue the certificates. The 2 major ways of …

no d inrogers open-menu closeme

I followed this tutorial to create a new Docker container with certbot/dns-cloudflare image. Change this to your own DNS provider's variant.

Mount the generated certificates to Caddy

In the above tutorial we have a line in volume that says this:

- ./letsencrypt:/etc/letsencrypt

The container spits out the certificates in a subfolder of /etc/letsencrypt. So, we have to map this to a file system location that is also visible to the Caddy proxy.

What I do is:

- '/data/coolify/proxy/caddy/data/caddy/certificates/certbot-ycsh/letsencrypt:/etc/letsencrypt'

If you install Coolify in the official way, you should have the Coolify configs and data under /data/coolify as well. And, if you are using Caddy as the proxy of your choice, you can find out that /data/coolify/proxy/caddy/data is mapped to /data in the proxy.

And I found out that, for certificates under /data/caddy/certificates Caddy knows to pick them up automatically, without you having to make any modification to other services that you deployed.

Conclusion

There you have it! Now you can enjoy your private services with the benefit of HTTPS.