DON'T Expose Internal Applications To The Internet! Restrict Access NOW!
Vložit
- čas přidán 26. 06. 2024
- The majority of Traefik tutorials all have the same problem, they expose all services routed through Traefik to the Internet. This is bad from a security perspective and increases the attack surface for your homelab. It's not a good idea to externalise Portainer, PiHole, Vaultwarden etc...
In this video I provide 3 options to restrict access to applications so you can specify exactly which services you want to expose and to which clients. Let's secure Traefik now!
Traefik Docker Files:
github.com/JamesTurland/JimsG...
Recommended Hardware: github.com/JamesTurland/JimsG...
Discord: / discord
Twitter: / jimsgarage_
Reddit: / jims-garage
GitHub: github.com/JamesTurland/JimsG...
00:00 - Introduction to the Traefik 'Security Problem'
01:29 - Demonstrating the 'Problem'
03:23 - Option 1: Cloudflare Proxy
05:00 - Option 2: Multiple Traefik Entrypoints
14:44 - Option 3: Traefik WhiteList
19:20 - Bonus Option: Deploy Two Traefik Instances
20:30 - Outro - Věda a technologie
Once again, another amazing video. Already waiting for the next one.
Thanks 👍
@@Jims-Garage By the way, what's the best way to expose bitwarden to the internet so we can use it thru the android APP ? Whitelistening isn't the best option as my ISP does not reserve an IP for mobile phones. Right now I have a VPN but it takes too steps to do when I need to open bitwarden...
@@joelfrojmowicz I use bitwarden on my mobile and I don't expose it. It should cache your passwords on the device (thus doesn't need to connect).
Thanks for the demo and info, have a great day
Thanks, Chris. You too!
Very nice hints in this video. I'm improving a lot of my homelab based on your videos. It would be nice if you could do a video explaining with more detail how to create firewall rules in Sophos-XG and how to apply IPS and any other improved security mechanisms.
Thank you, do you mean this one? czcams.com/video/tRBe1EbKyoU/video.html
Excellent information! One addition thing I have done is restrict the entry of my NAT rules for ports 80 and 443 to the Cloudflare ASN in Opnsense. It doesn't even appear to be an open port any longer.
Thanks, that's a good additional step.
Did you have to find out all of the IP ranges used by the Cloudflare ASN and then apply these ranges to your firewall rules or does Opensense let you just specify the Cloudflare ASN ? I tried the former approach on pfSense (trying to find all of the IP ranges and then creating a rule for these IP addresses) but must have missed some because it didn't work for me. Would love to be able to just specify Cloudflare ASN.
@@luckbeforeleapi did this by creating an alias from a pair of urls (for v4 and v6) cloudflare provides that contain a list of all their current ips in use for dns proxy, then setting the source of my https port forward rule to that alias
Godsent! Could not have come at a better moment in time for me. I just hope I can manage the complexity. I love the idea of docker-compose, but I've already let blood for a decent amount of time with the docker + docker compose documentation. And man if you already only half know what you're doing docker / compose / volumes /networking sure ain't gonna help that. The worst part is: it's difficult to diagnose. Is it fw rules? Is it vlan? Is it DNS? (it's always DNS) etc. I love the tinkering, but it to be a harsh mistress .. and hard on free time and money on top.
Haha yes, homelabbing isn't easy, and it takes a lot of time... There's tons of useful stuff to learn though that can be very useful outside of the lab. If you get stuck jump in the Discord, plenty of people to help you.
Another thing you can do is find out your cellular provider's IP address ranges (e.g. find one public IP address used by your phone and then use Hurricane Electric BGP Tools to find ASN and ranges), and apply these as Allow rules on your firewall. Then, when you are away from your house, only access your home network from your laptop when it is tethered to your phone's Internet connection. You will have limited your attack surface a lot with just that step. I also use a VPN with this so my firewall will only accept the VPN connection when it is coming from an IP address used by my cellular provider.
Thanks for highlighting this point. It should be the most overlooked security flaw for most homelabbers. Updating my traefik config tonight!
Great, hop into Discord if you're not sure.
If you only use external access for yourself and maybe a few family members you should look at a Cloudflare tunnel
I'd recommend everyone use logically seperated traefik instances for internal and external. Yes it does require a bit more configuration but it's going to be the most secure.
I run the two instance setup. I have my docker compose set up so I use environment variables for configuration, then there is a merge feature where I have most of the config in a section that is then referenced and the internal/external portions only specify their unique elements (such as the IP whitelist).
Additionally, on the external point, I have an additional docker constraint tag, so by default enabling items only registers them on the internal instance, but i can add an additional label to list them on the external provider as well.
My setup is is all about doing as much of the work in the traefik configuration itself as possible, then the services themselves only have to specify a bare minimum, typically only a hostname and to enable (i have exposedByDefault set to false). I even have them sharing a cert store by mounting the same file, but only giving the external instance read-only access to it. Now THAT is something I wish I had a better way of managing, especially as most acme providers can run into conflict if you request multiple certs at the same time, and I don't want any internal subdomains explicitly listed on my publicly served cert.
That's a good setup and makes sense. I'm going to revisit Traefik soon for V3.
@@Jims-Garage Thanks for the awesome videos I am learning so much! I wanted to setup the 2 instance way as well, but I wanted to use the CF tunnel, does it mean that for 1 domain you need to setup 2 different cloudflare tunnels? 1 per traefik instance?
Great video, though I would use different ports for the external, something like 8080 and 8443
Thanks! Yes, just choose whatever works for your setup (I already had those in use).
great video! So informative. Is there anything wrong with have all traffic (both internal and external) routed through the external entrypoint?
Nothing wrong per se, just that there are likely services that you don't want to be exposed.
Thank you, great video, Jim!
There is one thing, I realized. When setting the sourcerange of ipwhitelist to my local network (eg. 192.168.0.0/24), it is for some reason not considered. Going on the Service in the Browser I get an "Forbidden". In trafik-log I saw that traefik "only sees" my WAN-IP, which is dynamic. Is there a workaround to constantly check the WAN-IP and alter the rule on the fly? Or is there another hint to make that work?
Still learning along the way, mostly because of your videos! So, if you would like to expose only certain apps to certain users/ip-adresses, would it not be safer to just use something like Twiingate or OpenZiti? So you would not have to open any port whatsoever? Or would that be too strict and hamper usability?
That's always the trade-off. For apps you only want certain people to access that would be a valid approach, but for a public website or app that's clearly not a viable option.
Its so great! Thinking about switching from nginx proxy to full on traefik like you..
One thing is holding me back is the thought of having managment interfaces/services accessable on the untrusted vlan.
My internal services are on the same docker host as the traefik instance so puting those service on untrusted is a bit of a no go.
for traefik to be able to access the gui's from different ip's would need to use a lot more "complicated" firewall rules.
How did you do it or still doing it?
You can disable the dashboard, plus it's read only anyway, and there's nothing particularly sensitive. You can use separate entrypoints for internal and external services, and use whitelists to prevent access.
@@Jims-Garage well sometimes i need to watch the video a bit further Xd
Does this vulnerability also apply to NGINX proxy manager? Eg port 443 coming from WAN routed to port 443 on nginx proxy manager which then redirects it to jellyfin for example?
super tutorial. thanks Jim.
i try to do. but it isn't work. have i do crowdsec (your tutorial video) first? than after traefik-secure tutorial video?
thanks
Yes, to add crowdsec you need to update Traefik. The config files are in the same folder.
I think the biggest flaws is in the volume mountpoint, mounting the Docker socket directly is not recommended, even the traefik documentation state you should not do that in prod. If somehow an external user get access to your traefik instance even through CloudFlare, they will have root access to your Docker, unless you are running traefik as nonroot
That's an important point and something I will cover later. I mention it in my podman and kubernetes video.
@@red_dautI used the same, then exposed it through TCP, also my Docker socket proxy is in another Docker network, with option internal=true and masquerade=false, and then my traefik are in Frontend network and socket network, container that need to be exposed are in the Frontend network
You're missing what I think is the best option. Just run 2 Traefiks. One on an internal-only IP, and one that's exposed to the outside world running your public projects. You can (and should) even put them on their own vlans.
Thanks, I do mention running two Traefiks as an option.
What about using Keycloak or something similar and having all applications OAuth2/OIDC enabled? All apps without a valid login session will redirect to the Keycloak login page...
hey great video!
Qustion, out of curiosity, if you are not exposing your password manager (vaultwarden - I use it as well), how do you you it on your phone or when your are not connected to your local network?
Vaultwarden caches your passwords on the phone, you shouldn't need to connect to use it. If ever I did (creating a new password etc) I would just use my VPN to remote home (I have a few videos on VPNs).
@@Jims-Garage tnx for the idea! never though about it!
hey again@@Jims-Garage just letting you know that i'm following your videos and creating a new homelab using them, so thanks!
Also, I have 2 questions. 1. for jellyfin, when trying to connect if from within my network, it required me to add :444 at the end of the URL, otherwise it says "not found".
2) do I need to add a domain name fot it also in pihole
a suggestion for a follow-up video, how do i use the domain I bought in cloudflare and exposing services?(for some reason, i'm struggling with it nad none of the videeos i wathc helped) tbnx again!
Great videos 👍 i just have a question about jellyfin. Will cloudflare not shut it down after time. Would love to stream my music through my jellyfin app on my mobile remotely. Is there so many mb you have?
No quoted limit AFAIK. I'm streamed through it successfully with movies, suspect audio wouldn't be a problem.
Would really love a video about caddy.
Thanks, it's on the list
Especially with Docker! :D @@Jims-Garage
Excellent video! I feel really dumb right now because I'm re-watching the video to find out why I couldn't get this setup to work on my end then I tried it a couple of days ago, and at the 5-minute mark I see why... I forgot to route ports 81 and 444 to Traefik in the docker-compose file. >_
Glad you managed to sort it
@@Jims-Garage I got one step closer... But when I try to http to an external service, it redirects to :444 in the browser (fails), direct to https works. And services on the internal http and https just plain don't connect even though Traefik says everything is fine. Really frustrating, but I'll come back to it after a bit of rest.
@@SnorreSelmer read up on hairpin NAT, that's what you need.
@@Jims-Garage Supposedly my UDM Pro has that automatically, it just doesn’t seem to work. But I’ll keep digging!
ive foolllowed your steps but after adding the ports ans the external parts i cant long in to the traefik dashboard i get 404 page not found error
traefik and crowdsec how do i reserve proxy outside docker
I'm having an issue where using the http-external tag works fine (sending traffic through port 81). As soon as I add the https-external label, the site is no longer reachable.
Port 81 and 444 are routed correctly.
Has anyone else run into this or have any advise?
I like to keep my admin interfaces on my admin VLAN/subnet. It's surprising how few tutorials cover this, and how buried in documentation it is. Even Debian is doing its best to make sure the ListenAddress option of sshd does not work:(
Anyway, I wonder if i can use some similar approach to limit Docker stuff to particular subnets internally.
You can use macvlan to put containers on different subnets. I do this for DMZ containers.
Why not add a whitelist to the cloudflare tunnel as well?
Tha ks very muchbfor the video, like someone commented have been trying to read ducumentation to understa d but is a task. Have two questions , so is tgos the way to protect the traefik dashboard? Also how can point traefik to another docker host for other applications? What i would like to do is have one docker with just traefik and crowdsec for enamplle and host the other apps on another docker host. What are youbthoughts on this eg which security apps beed to be on the same host as traefik and which ones dont. Thanks for any feedback
Thank you. You can restrict access to the dashboard this way. For none Docker service routing you need to create an external service reference, you do this in the traefik.yaml file. What you want to do with 2 hosts is possible but I'd do it all on the same host for simplicity and better Traefik integration. Multiple hosts is better suited for Docker Swarm or better yet, Kubernetes.
@Jims-Garage ok noted and thanks for the. Feedback, have to take a stab back at Crowdsec to see if I get that working
@@Jr-hv1ct check my recent config update on GitHub, the old one might have caused the issue.
@Jims-Garage ok thanks for that info.
Please keep in mind, that only applications which have 'Routers' in traefik are exposed... Like your SQL, Redis, Elastic, etc containers is probably not exposed to the public... I have two middlewares that I reference in my labels, public@file and private@file depending on if I want it exposed to the public or not, then in that middleware is the crowdsec bouncer and whitelist depending on which middleware
Could you please explain a bit more under what circumstances this security issue exists? And which local services are accessible? Do you mean services that are considered only for local usage and still have a config within traefik or how how does the infrastructure look when vulnerable?
Say for example you're running PiHole, Proxmox, vaultwarden, or anything else solely for internal use and it's routed through the proxy. If you also have something like Plex, or anything else designed for internet access that shares the same entrypoint, then it's accessible from the internet if you do not have some of the countermeasures I state in the video, provided I have your IP address.
Test it out from your mobile or over a VPN. Create a hosts record as I showed in the video.
@@Jims-Garage Thanks for the clarification. So by "routed though the proxy" it means that if I for example just configure Plex in traefik and just use Bitwarden locally with it's own ip address and port there should be no way to connect to it from outside? Because it isn't routed through the proxy in that case in my understanding.
@@jT-dj9sj correct, this is only for things using Traefik with a default config
What I done, open remote port for once, get certbot do its things (filling my acme.Json) after that close the port, is I am doing wrong?
Use DNS challenge, you don't need any open ports 👍
If I may:
Does JC21 Reverse Proxy have the same issue?
I'm not familiar with that one.
Thank for the tutorial though I don't know if I am missing something. I appreciate you hairpin NAT explanation but I'm unable to access my (https-external) service on my LAN. I have NAT reflection enabled on my OPNSense router but get a 404. My DNS overrides still list the internal IP's of all services. Is this correct?
My NAT rule was configured on the WAN interface only. I added the LAN interface to the NAT rule and bingo, LAN access to (http(s)-external) traefik services. Leaving this comment here for anybody else experiencing the same issue.
You need a nat rule that translates port 80 and 443 to your external port numbers. E.g., internal network -> translate port 80 to 81
I solve this issue with Authelia, only specified services are allowed either by bypass, one_factor or two_factor.
Yes, a default deny rule in Authelia is also a good option if you're using it. Authentik can also do a similar thing.
What about using cloudflare tunnel? Do i have the same issues when only defining some public hostnames? Is it also possible to access to internal apps, which have in my case slso another subdomain as my external apps?
No, Cloudflare Tunnels shouldn't create the same issue. They create a whole other nightmare with zero privacy though...
@@Jims-Garage What are the other issues, except the case, that you have to trust cloudflare?
You don't need to guess a subdomain, It's really easy to get all subdomains just by knowing the main domain name. I would do this to yours, as a test to prove it to you but here in the UK without your explicit permission to do so it would be illegal under computer misuse laws.
Great video. I use option 3 with the whitelists and restrict access for my internal apps to my local subnet and my vpn subnet ranges.
Im currently trying to work out how i can have it give a 404 rather than a 403. I also want to create a custom error page. Would be great if you could explain and show something with ref to this
Thanks, yes. I've used custom error pages before, but I agree it would be much better if you could generate a 404. I'll consider it for a follow up video.
Why not Authelia/Authentik to add 2FA to your endpoints and use it as the middleware and make them have to go through multifactor authentication in order to get to the service?
That is an option, but a better one is not to expose it if you don't have to. Always reduce the attack surface.
I think you could also use MAC VLAN and have a separate ip for external, no ports needed.
Thanks, how do you mean? Two IPs for Traefik?
Why would you want to poke holes in your firewall? Just run a cloudflare tunnel and get rid of that proxy and firewall openings. Those are serious attack vectors.
I don't want Cloudflare to see all of my traffic, it's a privacy nightmare. Plus, you're putting all of your trust into a single vendor... Better to run everything though a firewall, even if you're using a tunnel IMO.
Can't you make IP rules lists within Cloudflare (free)? Am I missing something?
You could, but it's still placing full trust into Cloudflare. These options enhance that by adding additional layers, especially with IPS/IDS on a firewall.
Considering method 3, is it possible to a hacker spoof its IP and pass as a whitelisted ip? My gut feeling is that option 2 is safer, but as I said, I’m not an expert, just curious hahaha
It's always possible but I find it very unlikely anyone would bother for a homelab. Technically it would require serious compromises of underlying infrastructure.
Don’t you want your vaultwarden accessible to outside? What happens when you’re out and need to update a password? My partner won’t deal with VPNs
I use a VPN, but also, existing passwords are saved on the phone. The only issue you have is you cannot add new ones until you're back on the home network (hence the VPN).
thanks. pls make a video how to access your services only via vpn (h. how to config firewall and how to config headscale for it.
Please check my recent videos, I have two videos on Headscale. One if you are able to port forward, and one using a VPS if you cannot.
If I proxy through Cloudflare, or use their Tunnels service, can they see the data that's being transmitted in plain text? Can the decrypt the data on their side?
Proxy, no. Tunnel, yes (if it's just HTTPS).
@@Jims-Garage So if I use 'davs', 'ftps' or any other protocol, they can't? Also, is there a way to prevent this from happening (when using Tunnel)?
@@kshitijkadlag anything that is end to end encrypted, no.
@@Jims-Garage Okay cool, thanks! :)
When you’re using an SSL Cert from Cloudflare then theirs are technically capable of reading your traffic. This is the case when you use the Proxy or Tunnels, only when you’re the only one who is holding the Private key then only you can decrypt the content, but when using the cloudflare services they are holding the private key so it’s possible that they decrypt your traffic and they do it to protect you from DDoS etc.
Sure but why proxy your internal apps at all? Most guys suggest running a proxy docker network for traefik. Ideally you have separate networks for your dbs internal apps and one for the socket proxy. Then your internal apps only sit in the internal where there is no traefik endpoint. Meanwhile because these are all custom networks you can still call apps by their container name.
Most apps now want valid SSL certificates for web interfaces etc and not using them is difficult. Furthermore, some apps I want both internal and external access to (e.g. Jellyfin)
The NSA just has to be all up inside Cloudflare.
All companies need to abide by the law of local jurisdictions, even VPNs...
OPNsense, and you're good to go
How so, what specifically are you using?
While whitelisting seems to be the right thing, it doesn't seem to work. If I whitelist my LAN addresses and try to open the site, it says forbidden, and the log says that is not allowed to access. If I try to use the depth of ipstrategy, the log says that the IP is empty. Does anyone know a solution?
same problem
Thanks for the good video. Option 4 would be not do a port forwarding at all and use eg. Cloudflare tunnels to expose selected services instead.
True, provided that you fix the issue with Cloudflare Tunnels czcams.com/video/1n9lCYCLUYI/video.htmlsi=ZLhG173lomp9C0SF
Good luck doing that for Jellyfin/Plex without getting banned.
@@dustojnikhummer I've done it, albeit I'm a very light user. Not sure if there are specific rules on it?
@@Jims-Garage Cloudflare doesn't allow video streaming through Tunnel or Proxy
@@dustojnikhummer thanks for confirming
I dont understand how to access internal services. Can someone explain?
I assume you mean using the proxy with a URL? Check out my Traefik and PiHole videos, that should give you everything you need.
@@Jims-Garage but without pi-hole it is impossible to access local services? I precisely followed the tutorial btw and the external network is working except internal. For example i routed the traefik dash internally and cant access it.
Where is your traefik full tutorial?
Look for the thumbnail that pops up. czcams.com/video/XH9XgiVM_z4/video.html
@@Jims-Garage thanks a lot 👍
Wouldn’t it be easier to setup ip whitelists?
Potentially, depends on the intended audience. If it's meant to be publicly accessible then that strategy doesn't work, you'd want additional layers of scanning and segmentation.
@@Jims-Garage For me, I have some apps exposed to the internet and some others like vaultwarden are only accessible via whitelist. For those, I just use WireGuard to access these apps. Seems easier
I just can't help it, but this is how this video reads to me: 'Are you using this proxy to expose services to the internet? Well, guess what, there is a significant security flaw, and you are probably not aware of it. You have exposed services to the internet.'
And yes, I know there might be people who simply don't understand the concepts and just slap in a Traefik instance in the middle of their stack because they read one tutorial and want this specific thing accessible from the internet. But who are these people who 'have a stack' and then 'accidentally' expose it to the internet as a whole? I just cannot fathom the cross-section between these two groups of people who seemingly have knowledge of running a server with a stack but then don't realize what the Traefik instance they just added to the stack does. But again, I might be biased as I have an IT background.
No, not quite right. The issue is that by routing everything through Traefik, by default all applications are accessible if you know the IP address. That includes things you only want available locally (e.g., vaultwarden).
How is possible that people thinks is a good practice to expose services to the exterior
Just don't do it. If you want to access internal apps externally, come into your home network with your own VPN. It's the only safe way.
I agree, but if you're not careful and you use a single Traefik for both then you can access internal apps via the web.
@@Jims-Garage 100% and the warning is much appreciated for those who may have otherwise overlooked this very dodgy situation.
I just naturally realized this about my network around the same time this video was posted and came to the same 2 traefik conclusions. Ended up going with the IPwhitelist middleware
Great 😃
will you update this on opnsense? 👀
Hopefully 🤞