**Warning: Carry out these steps at your own risk, always ensure you have backups taken**
Example:
This is a 3 node Galera cluster with 1 primary node (node1).
node2 and node3 have failed due to insufficient memory.
When the both node2 and node3, are restarted, MariaDB fails to start with the following error msg:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
-- Unit mysql.service has begun starting up. Aug 15 11:10:14 node2 mysql[107053]: Starting MariaDB.190815 11:10:14 mysqld_safe Logging to '/var/lib/mysql/node2.err'. Aug 15 11:10:14 node2 mysql[107053]: 190815 11:10:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql Aug 15 11:10:20 node2 mysql[107053]: .../etc/rc.d/init.d/mysql: line 264: kill: (107088) - No such process Aug 15 11:10:20 node2 mysql[107053]: [FAILED] Aug 15 11:10:20 node2 systemd[1]: mysql.service: control process exited, code=exited status=1 Aug 15 11:10:20 node2 systemd[1]: Failed to start LSB: start and stop MariaDB. -- Subject: Unit mysql.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit mysql.service has failed. |
Check the status of the Primary node (node1):
|
1 2 3 4 5 6 7 |
MariaDB [(none)]> show status like 'wsrep_cluster_status'; +----------------------+-------------+ | Variable_name | Value | +----------------------+-------------+ | wsrep_cluster_status | non-Primary | +----------------------+-------------+ 1 row in set (0.09 sec) |
Check cluster size, in this example node1 is the only surviving node
|
1 2 3 4 5 6 7 |
MariaDB [(none)]> show status like 'wsrep_cluster_size'; +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | wsrep_cluster_size | 1 | +--------------------+-------+ 1 row in set (0.00 sec) |
Promote node 1 as Primary node
|
1 2 |
MariaDB [(none)]> SET GLOBAL wsrep_provider_options="pc.bootstrap=1"; Query OK, 0 rows affected (0.09 sec) |
node1 should now be Primary
|
1 2 3 4 5 6 7 |
MariaDB [(none)]> show status like 'wsrep_cluster_status'; +----------------------+---------+ | Variable_name | Value | +----------------------+---------+ | wsrep_cluster_status | Primary | +----------------------+---------+ 1 row in set (0.01 sec) |
On node2 and node3
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
[root@node2 mysql]# cat grastate.dat # GALERA saved state version: 2.1 uuid: f928c252-95af-11e9-82a5-36aa3d541d35 seqno: -1 safe_to_bootstrap: 0 [root@node2 mysql]# sed -ie 's/safe_to_bootstrap: 0/safe_to_bootstrap: 1/g' grastate.dat [root@node2 mysql]# cat grastate.dat # GALERA saved state version: 2.1 uuid: f928c252-95af-11e9-82a5-36aa3d541d35 seqno: -1 safe_to_bootstrap: 1 |
Start MariaDB again on node2 and node3 and check cluster size
|
1 2 3 4 5 6 7 |
MariaDB [(none)]> show status like 'wsrep_cluster_size'; +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | wsrep_cluster_size | 3 | +--------------------+-------+ 1 row in set (0.00 sec) |