MariaDB 10.1 CentOS 7 Fails to Start Using New Disk or Data Location
If you changed your data location for mysql / mariadb, using a new disk and get an error like this after a start or restart:
# systemctl start mariadb Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
Here’s the extended status:
# systemctl status mariadb ● mariadb.service - MariaDB 10.1 database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2016-03-29 11:49:49 CDT; 3s ago Process: 5233 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS) Process: 5219 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=exited, status=1/FAILURE) Process: 5185 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS) Process: 5162 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS) Main PID: 5219 (code=exited, status=1/FAILURE) Status: "MariaDB server is down" Mar 29 11:49:48 db1 systemd[1]: Starting MariaDB 10.1 database server... Mar 29 11:49:49 db1 mysqld[5219]: 2016-03-29 11:49:49 140524370049152 [Note] /usr/libexec/mysqld (mysqld 10.1.12-MariaDB) starting as process 5219 ... Mar 29 11:49:49 db1 mysqld[5219]: 2016-03-29 11:49:49 140524370049152 [Warning] Can't create test file /var/lib/mysql/db1.lower-test Mar 29 11:49:49 db1 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE Mar 29 11:49:49 db1 systemd[1]: Failed to start MariaDB 10.1 database server. Mar 29 11:49:49 db1 systemd[1]: Unit mariadb.service entered failed state. Mar 29 11:49:49 db1 systemd[1]: mariadb.service failed
It more than likely has everything to do with SELinux running. To check, do this:
# getenforce Enforcing
Check to see if this is the issue and temporarily allow us to allow access:
root@db1 /etc/my.cnf.d# semanage permissive -a mysqld_t
Start MariaDB:
root@db1 /etc/my.cnf.d# systemctl start mariadb
Looks good, let’s check the status:
root@db1 /etc/my.cnf.d# systemctl -l status mariadb ● mariadb.service - MariaDB 10.1 database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2016-03-29 11:52:07 CDT; 13s ago Process: 5233 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS) Process: 5493 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS) Process: 5430 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS) Process: 5407 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS) Main PID: 5464 (mysqld) Status: "Taking your SQL requests now..." CGroup: /system.slice/mariadb.service └─5464 /usr/libexec/mysqld --basedir=/usr Mar 29 11:52:06 db1 systemd[1]: Starting MariaDB 10.1 database server... Mar 29 11:52:07 db1 mysqld[5464]: 2016-03-29 11:52:07 140638769318016 [Note] /usr/libexec/mysqld (mysqld 10.1.12-MariaDB) starting as process 5464 ... Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: The datadir located at /var/lib/mysql needs to be upgraded using 'mysql_upgrade' tool. This can be done using the following steps: Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: 1. Back-up your data before with 'mysql_upgrade' Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: 2. Start the database daemon using 'service mariadb start' Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: 3. Run 'mysql_upgrade' with a database user that has sufficient privileges Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: Read more about 'mysql_upgrade' usage at: Mar 29 11:52:07 db1 mysql-check-upgrade[5493]: https://mariadb.com/kb/en/mariadb/documentation/sql-commands/table-commands/mysql_upgrade/ Mar 29 11:52:07 db1 systemd[1]: Started MariaDB 10.1 database server.
SIDE NOTE: Looks like we need to upgrade our disk mounted at /var/lib/mysql now:
# mysql_upgrade -hlocalhost -uroot -pxxxxxxxxxxxxxx Phase 1/6: Checking and upgrading mysql database Processing databases mysql mysql.columns_priv OK mysql.db OK mysql.event OK mysql.func OK mysql.help_category OK mysql.help_keyword OK mysql.help_relation OK mysql.help_topic OK mysql.host OK mysql.ndb_binlog_index OK mysql.plugin OK mysql.proc OK mysql.procs_priv OK mysql.proxies_priv OK mysql.servers OK mysql.tables_priv OK mysql.time_zone OK mysql.time_zone_leap_second OK mysql.time_zone_name OK mysql.time_zone_transition OK mysql.time_zone_transition_type OK mysql.user OK Phase 2/6: Fixing views Phase 3/6: Running 'mysql_fix_privilege_tables' Phase 4/6: Fixing table and database names Phase 5/6: Checking and upgrading tables Processing databases information_schema performance_schema Phase 6/6: Running 'FLUSH PRIVILEGES' OK
Now, let’s permanently & persistently allow for SELinux to allow us to write to the new disk:
Start by temporarily putting mysqld_t into permissive mode (should already be there):
# semanage permissive -a mysqld_t
Now, we record what MariaDB is doing and to create a policy to allow that (but nothing else).
Switch SELinux to permissive mode and remove dontaudits from the policy:
# semodule -DB # semanage permissive -a mysqld_t
Start MariaDB, then use the generated audit log to create a policy:
# grep mysqld /var/log/audit/audit.log | audit2allow -M mariadb_local # semodule -i mariadb_local.pp
Let’s adjust the file system labelling so that SELinux knows this is MariaDB’s datadir. Check with:
# ls -ldaZ /datadir drwxr-xr-x. root root unconfined_u:object_r:var_t:s0
# semanage fcontext -a -t mysqld_db_t "/var/lib/mysql(/.*)?" # restorecon -Rv /var/lib/mysql
You can check what’s in the policy here:
# semanage fcontext -l -C SELinux fcontext type Context /var/lib/mysql(/.*)? all files system_u:object_r:mysqld_db_t:s0 SELinux Local fcontext Equivalence /opt/remi/php70/root = / /var/opt/remi/php70 = /var /etc/opt/remi/php70 = /etc
Now, remove the permissive mode for mysqld_t and restore dontaudits:
# semodule -B # semanage permissive -d mysqld_t