I run a small server with Proxmox, and I’m wondering what are your opinions on running Docker in separate LXC containers vs. running a specific VM for all Docker containers?
I started with LXC containers because I was more familiar with installing services the classic Linux way. I later added a VM specifically for running Docker containers. I’m thinking if I should continue this strategy and just add some more resources to the docker VM.
On one hand, backups seem to be easier with individual LXCs (I’ve had situations where I tried to update a Docker container but the new container broke the existing configuration and found it easiest just to restore the entire VM from backup). On the otherhand, it seems like more overhead to install Docker in each individual LXC.
Regardless of VM or LXC, I would only install docker once. There’s generally no need to create multiple docker VMs/LXCs on the same host. Unless you have a specific reason; like isolating outside traffic by creating a docker setup for only public services.
Backups are the same with VM or LXC on Proxmox.
The main advantages of LXC that I can think of:
- Slightly less resource overhead, but not much (debian minimal or alpine VM is pretty lightweight already).
- Ability to pass-through directories from the host.
- Ability to pass-through hardware acceleration from a GPU, without passing through the entire GPU.
- Ability to change CPU cores or RAM while it’s running.
Also LXC shares the host filesystem to there is less concern with corruption due to power loss.
What’s the purpose of running container in a container? Why not install docker on your host machine?
If you do that, Docker is stuck on that host. If it’s in an LXC it can move to another host. Plus, backing up and snapshotting are easier IMO.
Snapshotting in docker is as easy as
docker commit. After that you can back it up withdocker save. Then move to another host, but not without downtime.However normally you need to backup/move only volumes attached to containers. If that’s not the way how you like to organize your services, you likely don’t need docker.
Dockers ‘take-over-system’ style of network management will interfere with proxmox networking.
Well, I don’t use proxmox, however docker coexists with libvirt and other virtualization systems. If there are overlapping networks that docker ant proxmox attempt to manage, they are configurable.
I don’t use proxmox, but it works absolutely fine for me on my regular Linux system, which has a firewall, some background services, etc. Could you be more specific on the issues you’re running into?
Also, I only really expose two services on my host:
- Caddy - handles all TLS and proxies to all other services in the internal docker network
- Jellyfin - my crappy smart TV doesn’t seem to be able to handle Jellyfin + TLS for some reason, it causes the app to lock up
Everything else just connects through an internal-only docker network.
If you’re getting conflicts, I’m guessing you’ve configured things oddly, because by default, docker creates its own virtual interface to explicitly not interfere with anything else on the host.
A couple posts down explains it, docker completely steamrolls networking when you install it. https://forum.proxmox.com/threads/running-docker-on-the-proxmox-host-not-in-vm-ct.147580/
The other reason is if it’s on the host you can’t back it up using proxmox backup server with the rest of the VMs/CTs
I don’t use proxmox, so I guess I don’t understand the appeal. I don’t see any reason to backup a container or a VM, I just backup configs and data. Backing up a VM makes sense if you have a bunch of customizations, but that’s pretty much the entire point of docker, you quarantine your customizations to your configs so it’s completely reproducible if you have the configs and data.
Ease of use mostly, one click to restore everything including the OS is nice. Can also easily move them to other hosts for HA or maintenance.
Not everything runs in docker too, so it’s extra useful for those VMs.
That’s fair.
That said, I can’t think of anything I’d want to run that doesn’t work in docker, except maybe pf? But I’d probably put that on a dedicated machine anyway. Pretty much everything else runs on Linux or has a completely viable Linux alternative, so I could easily built a docker image for it.
Honestly, I never really thought of installing Docker directly on Proxmox. I guess that might be a simpler solution, to run Dockers directly, but I kind of like to keep the hypervisor more stripped down.
Honestly you can do either.
LXC
-
shares host kernel (theoretically lighter weight)
-
less isolation from host (less secure)
-
devices are passed via device files
-
less flexible due to dependence on host
-
no live transfers
-
filesystem shared with host
virtualization
-
has own kernel and filesystem
-
supports live transfers
-
hardware pass though is done at the device level
-
more flexible due to independent kernel
-
more overhead
-
You can also create a single LXC for Docker and run multiple Docker containers on it. The VM argument is for security as it’s harder to escalate to the host from a VM than from an LXC.
I can’t say much to docker in LXC as I’m not using it, I vaguely remember some limitation I’ve read of but if it works fine for you those don’t seem to apply.
A VM has more overhead than an LXC, but with several LXCs maybe a single VM wins on overhead.
I currently have most Docker containers in one VM and am thinking about splitting it, the main reason is that 2 deployments have way larger volumes than the rest. This leads to the snapshots of the VM being very large as well and if I would need to restore from snapshots for a “small” application, it would take super long because of the large ones.
A single VM may be a bit easier on maintenance than several LXCs.
If you don’t have a specific reason to switch, I would not.
You could create a fresh container, install docker, and create a new template image from it. This way the overhead of installing disapears. The overhead in resource usage for each docker installation would remain the same as before.
As mentioned in another reply, you could run several container in one lxc. For example with docker compose or podman. Since I have no experience with podman but with docker compose, docker compose is pretty simple.
But all in all, I prefer to install everything “bare metal” in lxc containers. The main reason is, I don’t want to mess around with the extra layer of configurating ports etc.
I personally like lxc’s over vms for my home lab and i run a dedicated lxc for docker and one running a single node k8s.
Is your server a dedicated server, or a VPS? Because if it’s a VPS, you’re probably already running in a VM.
Adding a VM might provide more security, especially if you aren’t an expert in LXC security configuration. It will add overhead. Running Docker inside Docker provides nothing but more overhead and unnecessary complexity to your setup.
Also, because it isn’t clear to me from your post: LXC and Docker are two ways of doing the same thing, using the same Kernel capabilities. Docker was, in fact, written in top of LXC. The only real difference is the container format. Saying “running Docker on LXC” is like saying “running Docker on Docker,” or “running Docker on Podman,” or “running LXC on Docker”. All you’re doing is nesting container implementations. As opposed to VMs, which do not just use Linux namespace capabilities, and which emulate an entirely different computer.
LXC, Podman, and Docker use the underlying OS kernel and resources. VMs create new, virtual hardware (necessarily sharing the same hardware architecture, but nothing else from the host) and run their own kernels.
Saying “Docker VM” is therefore confusing. Containers - LXC, Podman, or Docker - don’t create VMs. They partition and segregate off resources from the host, but they do not provide a virtual machine. You can not run OpenBSD in a Docker container on Linux; you can run OpenBSD in a VM on Linux.
I think they mean a VM running docker
It’s a dedicated server (a small Dell micro-pc). Thanks for the comment, I understand the logic, I was approaching it more from an end-user perspective of what’s easier to work with. Which given my skill set are LXC containers. I have a VM on top of Proxmox specifically for Docker :-)
