We have been rolling out CENTOS8 in our lower environments for testing, we use a dedicated vmware virtual server with centos8 minimal install, we only apply hardening techniques to these systems other than the main application, which is pg11 here. These systems use a LVM mounted ext4 filesystem for the data directory.<\/p>\n
\r\n\/dev\/mapper\/vg01-data1\t\t\t\t\/u02\/data1\t\text4 defaults, nofail\t\t0 2\r\n<\/pre>\nRecently on 3 of the new PG VMS after reboot we noticed that PG did not start, this also seemed intermittent, even though we have enabled the systemd service to start on reboots. So I checked the pg startup log and did not find too much about the issue. So I checked \/var\/log\/messages and found the issue. <\/p>\n
\r\npostgresql-11-check-db-dir[1038]: \"\/u02\/data1\/pg\/data11\" is missing or empty.\r\n<\/pre>\nI checked the systemd service file and saw that out of the box postgres had the following:<\/p>\n
\r\n[Unit]\r\nDescription=PostgreSQL 11 database server\r\nDocumentation=https:\/\/www.postgresql.org\/docs\/11\/static\/\r\nAfter=syslog.target\r\nAfter=network.target\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n<\/pre>\nAfter=Syslog.target This is a special target unit in systemd and is the standardized name to pull in a syslog implementation.<\/p><\/blockquote>\n
After=network.target has very little meaning during start-up. It only indicates that the network management stack is up after it has been reached. Whether any network interfaces are already configured when it is reached is undefined.<\/p><\/blockquote>\n
WantedBy=multi-user.target normally defines a system state where all network services are started up and the system will accept logins, but a local GUI is not started. This is the typical default system state for server systems, which might be rack-mounted headless systems in a remote server room.<\/p><\/blockquote>\n
Those options above will not ensure that all filesystems in fstab are mounted before postgres starts. So what we were seeing was a classic race condition where postgres started before the data directory was mounted. As I previously mentioned we use a custom PGDATA location. So after some research I found my option that fixed this. You will need to edit the pg11 service and add the following, then reload systemd and reboot and all should work. You can find your LVM mount by running the following:<\/p>\n
\r\n[root@server ~]# systemctl list-units --type=mount\r\nUNIT LOAD ACTIVE SUB DESCRIPTION \r\n-.mount loaded active mounted Root Mount \r\nboot-efi.mount loaded active mounted \/boot\/efi \r\nboot.mount loaded active mounted \/boot \r\ndev-hugepages.mount loaded active mounted Huge Pages File System \r\ndev-mqueue.mount loaded active mounted POSIX Message Queue File System \r\nrun-user-1328029883.mount loaded active mounted \/run\/user\/1328029883 \r\nsys-fs-fuse-connections.mount loaded active mounted FUSE Control File System \r\nsys-kernel-config.mount loaded active mounted Kernel Configuration File System\r\nsys-kernel-debug.mount loaded active mounted Kernel Debug File System \r\nu02-data1.mount loaded active mounted \/u02\/data1 \r\n\r\nLOAD = Reflects whether the unit definition was properly loaded.\r\nACTIVE = The high-level unit activation state, i.e. generalization of SUB.\r\nSUB = The low-level unit activation state, values depend on unit type.\r\n\r\n10 loaded units listed. Pass --all to see loaded but inactive units, too.\r\nTo show all installed unit files use 'systemctl list-unit-files'.\r\n<\/pre>\nYou can see my u02-data1.mount in the output, so edit and add the override file with the following, if you have multiple mounts, you can add them as well.
\nEdit with: systemctl edit postgresql-11<\/p>\n\r\n[Unit]\r\nAfter=local-fs.target u02-data1.mount\r\n\r\n[Service]\r\nEnvironment=PGDATA=\/u02\/data1\/pg\/data11\r\n<\/pre>\nReload the daemon with: systemctl daemon-reload<\/p>\n
After=local-fs.target systemd-fstab-generator(3) automatically adds dependencies of type Before= to all mount units that refer to local mount points for this target unit. In addition, it adds dependencies of type Wants= to this target unit for those mounts listed in \/etc\/fstab that have the auto mount option set.<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"
We have been rolling out CENTOS8 in our lower environments for testing, we use a dedicated vmware virtual server with centos8 minimal install, we only apply hardening techniques to these systems other than the main application, which is pg11 here. These systems use a LVM mounted ext4 filesystem for the data directory. \/dev\/mapper\/vg01-data1 \/u02\/data1 ext4 […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[89,51,91,88,90],"class_list":["post-858","post","type-post","status-publish","format-standard","hentry","category-general-code","tag-centos8","tag-postgres","tag-postgresql-11-check-db-dir-is-missing-or-empty","tag-postgresql11","tag-systemd"],"_links":{"self":[{"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/posts\/858","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jasonralph.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=858"}],"version-history":[{"count":12,"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/posts\/858\/revisions"}],"predecessor-version":[{"id":865,"href":"https:\/\/jasonralph.org\/index.php?rest_route=\/wp\/v2\/posts\/858\/revisions\/865"}],"wp:attachment":[{"href":"https:\/\/jasonralph.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=858"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jasonralph.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=858"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jasonralph.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=858"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}