I have many services running on my server and about half of them use postgres. As long as I installed them manually I would always create a new database and reuse the same postgres instance for each service, which seems to me quite logical. The least amount of overhead, fast boot, etc.
But since I started to use docker, most of the docker-compose files come with their own instance of postgres. Until now I just let them do it and were running a couple of instances of postgres. But it’s kind of getting rediciolous how many postgres instances I run on one server.
Do you guys run several dockerized instances of postgres or do you rewrite the docker compose files to give access to your one central postgres instance? And are there usually any problems with that like version incompatibilities, etc.?
You must log in or # to comment.
I have a single big Postgres instance, shared among immich, paperless, lldap, grafana, and others. I only use the provided docker-compose as inspiration and do my own thing. It’s nicer to back up a single database (plus additional volumes, but still).
I only use the provided docker-compose as inspiration and do my own thing
This is the correct way to look at it. Most applications that provide a docker compose do so as a convenience to get started quickly. It’s not necessarily what you should run.
It is recommended to run postgres for each service though since they may have completely different needs / configurations for the queries to be optimal. For self hosting Lemmy and matrix would be the big concerns here.
It is recommended to run postgres for each service
Absolute sentences like this are rarely true. Sometimes it does make sense and sometimes it doesn’t. One database is often quite capable of supporting the needs of many applications. And sometimes you need to fine-tune things for a specific application.
Say what you want it’s a recommendation and it’s documented in quite a few deployment methods. The only benefit of centralizing it is if you are managing postres without other tools since it’d be a pain in the butt. You’ll still run into apps that doesn’t run on later versions and others that require later versions though.
An example of a very popular one:
How many databases should be hosted in a single PostgreSQL instance?
Our recommendation is to dedicate a single PostgreSQL cluster (intended as primary and multiple standby servers) to a single database, entirely managed by a single microservice application. However, by leveraging the “postgres” superuser, it is possible to create as many users and databases as desired (subject to the available resources).
The reason for this recommendation lies in the Cloud Native concept, based on microservices. In a pure microservice architecture, the microservice itself should own the data it manages exclusively. These could be flat files, queues, key-value stores, or, in our case, a PostgreSQL relational database containing both structured and unstructured data. The general idea is that only the microservice can access the database, including schema management and migrations.
CloudNativePG has been designed to work this way out of the box, by default creating an application user and an application database owned by the aforementioned application user.
Reserving a PostgreSQL instance to a single microservice owned database, enhances:
resource management: in PostgreSQL, CPU, and memory constrained resources are generally handled at the instance level, not the database level, making it easier to integrate it with Kubernetes resource management policies at the pod level physical continuous backup and Point-In-Time-Recovery (PITR): given that PostgreSQL handles continuous backup and recovery at the instance level, having one database per instance simplifies PITR operations, differentiates retention policy management, and increases data protection of backups application updates: enable each application to decide their update policies without impacting other databases owned by different applications database updates: each application can decide which PostgreSQL version to use, and independently, when to upgrade to a different major version of PostgreSQL and at what conditions (e.g., cutover time)You’re talking about a microservices architecture running in a kubernetes cluster? FFS… 🙄
That’s a ridiculous recommendation for a home-gamer. It’s all up to how you want to manage dependencies, backups, performance, etc. If one is happy to have a single instance then there’s nothing wrong with that. If one wants multiple instances for other reasons that’s fine too. There are pros and cons to each approach. Your “I saw somebody recommend it on the internets” notwithstanding.
It’s the one I’m using but it’s not just running in a cluster. Even some applications recommend running separately like matrix. You can’t run everything on the same.versiom all the time anyways.
You can’t run everything on the same.versiom all the time anyways.
Unless you’re doing something very specific with the database - yes you can. Most applications are fine with pretty generic SQL. For those that have specific requirements, well then give them their own instance. Or use that version for the ones that don’t much care…
I use the provided databases in the docker-compose file, since some services require a specific version and I’m too lazy to investigate whether it works on my existing service or not.
Not so long ago I had the same question myself, and I ended up setting 1 Postgress instance and 1 MySQL instance for all services to share. In the long run, I had so many version and settings incompatibilities across services that moved back to one DB per service that is tuned specifically for it. Also, I add a backup app to all my docker compose files that have a DB in it. This way, backups happen periodically and automatically.
Which db backup app do you use if you don’t mind me asking?
You don’t need a db backup app… bind mount the data to a location then just stop the container and have borg take the backups. You can do this with all your containers.
/docker/postgres
/docker/postgres/data
/docker/postgres/compose.yml
And do that with every container. Easy as fuck to backup and restore them.
I keep each service separate as far as DBs, if something breaks or get a major upgrade I don’t have to worry about other containers.
If I have several backends that more or less depend on each other anyway (for example: Lemmy + pict-rs), then I will create separate databases for them within a single postgres - reason being, if something bad happens to the database for one of them, then it affects the other one as well anyway, so there isn’t much to gain from isolating the databases.
Conversely, for completely unrelated services, I will always set up separate postgres instances, for full isolation.
I used to, but now I just have one big one (still in Docker) that’s sized and tuned to handle all of my applications.
I’ve only had one version compatibility issue, but that was because I was on pgSQL 13 and the updated version of one application needed 15. Upgrading that didn’t affect any other applications. If it had, I would have just broken that one application out to its own stack-local Postgres.
I have a separate network for postgres. Every service which needs a DB is attached to it. I use a single postgres container with several DBs and finetuned with PGTune.
The most important thing as always: proper backups ☝🏻
The only service with extra DB container is Immich, since it uses a custom variant and I am too lazy to modify the existing container 😁
I run Proxmox with a few nodes, and each of my services are (usually) dockerized, each running in a Proxmox Linux container.
As I like to keep things segregated as much as possible, I really only have one shared Postgres, for the stuff I don’t really care about (ie. if it goes down, I honestly don’t care about the services it takes with it, or the time it’ll take me to get them back).
My main Postgres instances are below - there’s probably others, but these are the ones I backup religiously, and test the backups frequently.
- RADIUS database: for wireless auth
- paperless-ngx: document management indexing & data
- Immich: because Immich has a very specific set of Postgres requirements
- Shared: 2 x Sonarr, 3 x Radarr, 1 x Lidarr, a few others
In theorey lots of people recommend having everything in a single docker-compose file for easier transfer and separation, though I have so much running, that it’s grouped by purpose. One of those is data storage. So I have a single server with all the databases (as far as compatibility goes). I would like to some day have a highly available postgres cluster with automatic failover and failback. But that needs a lot of testing and I’m no postgres admin, so also a lot of time to research how to do it properly.
Asked the same question a little while ago. See https://swg-empire.de/post/625121 for more opinions.
I ended up putting Nextcloud and Lemmy on one service. No problems so far, resource usage and performance are great. But I’ve not been running it for very long.
deleted by creator
My database instances downtime is only when the server itself is rebooting. Never had a single downtime in 20+ years beside that.
You’ve never had to run migrations that lock tables or rebuild an index in two decades?
Why would that have blocked all my databases at once? That would affect the same database I was migrating, not the others.
Yes, it would cause downtime for the one being migrated - right? Or does that not count as downtime?
Yes it counts indeed… But in that case the service is down while its migrated so the fact the database is also down does it count?
I mean, it’s a self hosted home service, not your bank ATM network…
This is one of the annoying issues with docker, or better, on how docker is abused in production.
The single instance/multiple databases is the correct way to go, docker approach mess up with that.
Rewriting docker files is always a possibility but honestly defies the reason why docker is used by self hosters.
Also beware that some devs will shunt you out of support if you do, specially the apps that ships docker files by default.
Go bare metal if possible, that way you have full control. Do docker for testing up stuff quickly and be flexible at cost of accepting how stuff is packaged by upstream
The official Postgres Docker image is geared towards single database per instance for several reasons - security through isolation and the ability to run different versions easily on the same server chief among them. The performance overhead is negligible.
And about devs not supporting custom installation methods, I’m more inclined to think it’s for lack of time to support every individual native setup and just responding to tickets about their official one (which also is why Docker exists in the first place).
