After playing with my Kubernetes cluster for a while and trying out various WordPress and MySQL installations, I was not happy with the performance.
I had created a replicated MySQL installation with a master and 2 slaves using the Bitnami Helm chart and then set up a WordPress pod that used NFS mounted from internal Longhorn storage, the performance was terrible – it took up to 30 seconds just to load the initial page, subsequent pages were not much better,
I thought maybe it was the NFS storage on the WordPress pod that was causing the issue, but after swapping this out for dedicated local Longhorn storage, it made almost no difference.
I thought maybe it was just that the Lenovo M92 Tiny servers I was using were not up to the task, I highly doubted this as they should be more powerful than the VMs I was running on my Dell R220 ii server, so I installed the same website on the M93 Tiny that I was using to run Minecraft and Rancher under Docker and set up a MySQL and WordPress containers on there as well.
This time the website took less than a second to load the first page and all subsequent pages were just as fast, so the problem is not the hardware.
Next I decided to point the Kubernetes WordPress pod at the database on the Lenovo M93, and this was almost as fast as the local Docker based website, again it made little difference if I was using local disk or NFS disk from within the Kubernetes cluster.
The problem it would seem is that the replicated MySQL database is too slow running under Kubernetes for the website to be even vaguely performant.
Now since I am currently using 3 of the servers as master nodes for the ETCD database that seem to do next to nothing and K3s is happy to use an external MySQL database, I decided to rebuild the cluster, but this time using Percona as the database on those same 3 servers and then install K3s on the remaining 5 nodes, having them all as master/agent nodes.
I chose not to re-install from scratch as K3s uninstalls pretty cleanly if you use the uninstall script.
I updated/upgraded the OS, created a file system for /var/lib/mysql from the unused 75GB of OS disk and set about installing Percona according to their installation instructions.
After having installed on all 3 nodes and stopped the MySQL instances, I needed to modify the configuration file for my environment and also create a bunch of certificates for encrypting the inter node traffic, this is all detailed on their website.
I then needed to bootstrap the database, join the other nodes, stop the bootstrap node, and rejoin as a normal node so I would then have 3 nodes in my cluster.
Now I needed a load balancer to sit in front of the database and for this I decided to use HAProxy on my PFSense firewall.
I created a new Virtual IP (VIP) 192.168.99.70 on the LAN interface, then created a new backend that made use of all 3 nodes of the Percona cluster and port 3306 as the database port with Round Robin load balancing. The health check method used is HTTP GET and an advanced per server pass thru setting of ‘port 9200’.
The frontend listens using the VIP and port 3306 and type tcp and the Percona backend I just created as its default backend.
While HAProxy can check MySQL databases if you create a login on the database for it to check with, this will not validate that the servers are in sync with each other.
For this I made use of the mysqlchk and clusterchk scripts written by Olaf van Zandwijk and Raghavendra Prabhu from their github site.
I needed to install xinetd, as this was not part of my standard OS install and follow the instructions on the github site, although I had to make a few changes.
I put the clusterchk script in /usr/local/bin, so had to update the /etc/xinetd.d/mysqlchk and /etc/systemd/system/mysqlchk.service scripts accordingly.
As I am using Ubuntu, the default database configuration file is not /etc/my.cnf, but /etc/mysql/mysql.conf/mysqld.cnf, so that had to be updated in the DEFAULTS_EXTRA_FILE definition. There was also a typo on the MYSQL_USERNAME definition where the user name was not being set correctly and needed ‘:=’ removing.
Once I had proven that the script worked correctly locally and also via a browser on port 9200 for each of the database servers, I could enable the load balancer in HAProxy.
I can now access my Percona database cluster via port 3306 using the load balancer VIP of 192.168.99.70.
After trying to use this clustered database for my various web server environments I ultimately gave up on this solution.
Every time I imported the saved backup, at some point ruing the ‘recovery’ the active node would switch and the import would then fail.
All this effort and pain for something that was just too fragile to support what I was after.
I ultimately managed to persuade independent copies of MySQL and WordPress, one per website work with a sensible response time, so that is what I have stuck with moving forwards.
I have also moved from trying to load balance/reverse proxy from HAProxy, and instead use Traefik inside the K3s cluster.