{"id":1328,"date":"2025-08-11T12:05:22","date_gmt":"2025-08-11T11:05:22","guid":{"rendered":"https:\/\/blog.hslracing.com\/myhome\/?p=1328"},"modified":"2025-08-11T13:00:49","modified_gmt":"2025-08-11T12:00:49","slug":"my-next-cluster-installation-and-configuration","status":"publish","type":"post","link":"https:\/\/blog.hslracing.com\/myhome\/2025\/08\/my-next-cluster-installation-and-configuration.html","title":{"rendered":"My Next Cluster – Installation and Configuration"},"content":{"rendered":"

Some of the technologies that will be used on this installation:<\/p>\n

\"\"<\/a> \"\"<\/a> \"\"<\/a> \"\"<\/a> \u00a0\"\"<\/a> \"\"<\/a> \"\"<\/a> \u00a0\"\"<\/a><\/p>\n

Kubernetes<\/a> or K8s (K ‘8 letters’ S – pronounced ‘cates’) is a container orchestration system that automates the deployment, scaling and management of containerised applications<\/p>\n

K3s<\/a> (K ‘3 letters’ S) is a lightweight version of Kubernetes<\/a> designed for IoT<\/a> and Edge Computing<\/a> platforms<\/p>\n

Cluster #1 – Proxmox<\/h2>\n

Proxmox<\/a> Virtual Environment is a virtualisation platform for hosting virtual machines, much like having your own private cloud solution.<\/p>\n

Having built the physical servers, it was now time to install the operating systems and guest virtual machines.<\/p>\n

The initial install involved a little BIOS configuration, a BIOS update, reset the vPro<\/a> management utility and then PXE<\/a> booting off my NetBootXYZ<\/a> Docker<\/a> container to enable me to install Proxmox<\/a> VE on the 64GB SATA NVMe drive on each node<\/p>\n

With the base operating system installed I then updated the network configuration – the default is to have a network bridge configured on the primary interface, however I wanted the primary 1Gb\u00a0 interface to be purely for Proxmox<\/a> cluster and management traffic, and then use a bridge on one of the 10GB interfaces for the various outward facing Virtual Machines.<\/p>\n

Once the network interfaces had been configured, I joined the servers into a Proxmox cluster<\/a>.<\/p>\n

Current load on the Proxmox cluster:<\/a><\/p>\n

\"\"<\/a><\/p>\n

Cluster #2 – Ceph<\/h2>\n

Ceph<\/a> is a unified storage solution, providing object, block and file interfaces from a single cluster.<\/p>\n

I then installed Ceph<\/a> on each node and configured a Ceph<\/a> cluster\u00a0 with a private network on the 10GB interface, once I have some more cables, I will probably move this to a dedicated 10GB interface to keep it separate from the Virtual Machine data.<\/p>\n

Having installed Ceph<\/a>, I created a Manager, Monitor and MetaData service on each node as well as configuring an OSD for the 1TB NVMe drive on each node, this gave me 1.29TiB of replicated storage across the 4 nodes<\/p>\n

\"\"<\/a><\/p>\n

I then created a Ceph<\/a> Pool that I could use for backend storage for the Virtual Machines I was about to build.<\/p>\n

Infrastructure as code<\/h2>\n

The next part of the setup involved installing Terraform<\/a> on my iMac then setting up an API key and user within Proxmox<\/a> that Terraform<\/a> could interact with along with some provider and node configuration details.<\/p>\n

The first step was to make the build image available on each of the nodes, for this I chose to download the ‘Noble Numbat’ (Ubuntu<\/a> 24.04 LTS) cloud image to each Proxmox<\/a> cluster node.<\/p>\n

I could then start creating the configuration for my Rancher<\/a>\u00a0and K3s<\/a> Kubernetes<\/a> clusters.<\/p>\n

I opted to use the bpg\/Proxmox<\/a> Terraform<\/a> provider to create my virtual machines along with the integrations\/github<\/a> provider to install my ssh keys from GitHub<\/a>.<\/span><\/p>\n

I knew it was going to take a few goes to perfect the various Kubernetes<\/a> installations, this way I could stand up and tear down the entire environment in a matter of minutes – the 7 node Kubernetes<\/a> cluster only takes just over 2 minutes to create the virtual machines and about 30 seconds to tear them all down again ready for the next run.<\/p>\n

Cluster #3 – Rancher<\/h2>\n

Rancher<\/a> is a platform for managing and deploying Kubernetes<\/a> clusters, it lives in its own minimal Kubernetes<\/a> cluster.<\/p>\n

My previous Rancher<\/a> installation was a Docker<\/a> container running a single node instance of Rancher<\/a>, which, whilst it worked well for managing my last Kubernetes<\/a> cluster, I wanted something a bit more resilient for this time around.<\/p>\n

The Rancher<\/a> cluster is 3 servers, each with 2 cores 4GB or memory and 32GB of disk (supplied from the Ceph<\/a> pool), one assigned to each of the first 3\u00a0 Proxmox<\/a> cluster nodes<\/p>\n

With the base OS installed, the next step was installing K3s<\/a> via k3sup<\/a> (pronounced ketchup), one initial server, then two additional servers<\/p>\n

The latest version of k3sup<\/a> has a ‘plan’ option which if supplied with a json<\/a> file containing host names and IP addresses will create a ‘bootstrap’ script to install K3s<\/a> on all of the nodes<\/p>\n

k3sup plan \\\r\ndevices.json \\\r\n--user ubuntu \\\r\n--servers 3 \\\r\n--server-k3s-extra-args \"--disable servicelb\" \\\r\n--background > bootstrap.sh<\/pre>\n
After running the bootstrap script, it is then a matter of providing the minimal cluster configuration required to setup Rancher<\/a><\/p>\n
Configure Kube VIP<\/a> (virtual cluster IP and load balancer), Install Cert-Manager<\/a> via Helm<\/a>, install Rancher<\/a> via Helm<\/a> and finally expose the Rancher<\/a> deployment via the Kube VIP<\/a> load balancer.<\/p>\n
I have yet to script the configuration post K3s<\/a> cluster creation, but I have all of the steps saved for when I decide on the best automation technology to use<\/p>\n
Rancher<\/a> cluster nodes as seen from the Rancher<\/a> dashboard:<\/p>\n
\"\"<\/a><\/p>\n
Cluster #4 – Kubernetes<\/h2>\n
The K3s<\/a> Kubernetes<\/a> cluster is made up of 7 servers – 3 control nodes, each with 2 cores, 4GB or memory and 32GB of disk (supplied from the Ceph<\/a> pool) and 4 worker nodes, each with 4 cores 16GB or memory and 32GB of disk (supplied from the Ceph<\/a> pool)<\/p>\n
For the control nodes, one assigned to each of the first 3\u00a0 Proxmox<\/a> cluster nodes, for the worker nodes, one is assigned to each of the 4 Proxmox<\/a> cluster nodes.<\/p>\n
K3s<\/a> was once again installed via k3sup<\/a>, one initial server, two additional servers, then the four worker servers<\/p>\n
The rest of the initial configuration was very similar to the Rancher<\/a> cluster: configure Kube VIP<\/a>, install Cert-Manager<\/a> via Helm<\/a>, install Traefik<\/a> ingress manager via Helm<\/a>, then import it on the Rancher<\/a> Cluster for easier management.<\/p>\n
I then also installed Prometheus<\/a>\/Grafana<\/a> via Helm<\/a>, from Rancher<\/a> for monitoring<\/p>\n
Next I needed to setup the various Ceph<\/a> storage classes, one for CephFS<\/a> and one for Ceph RBD<\/a> via the Ceph CSI<\/a> driver, this required some setup on the Proxmox<\/a> cluster in order for the Ceph CSI<\/a> driver to be able to talk to the Ceph<\/a> cluster to provision storage as required.<\/p>\n
I then needed to create a couple of Cluster Issuers for Cert-Manager<\/a> so it could request and manage LetsEncrypt<\/a> site certificates for the various websites that the cluster will be hosting<\/p>\n
Traefik<\/a> also required some additional configuration to expose and secure the dashboard as well as setup a bunch of re-write middleware rules to ensure all inbound traffic uses the encrypted ‘https’ route to these websites.<\/p>\n
The post K3s<\/a> cluster configuration is again still manual at this point but again I have all of the steps saved for future automation.<\/p>\n
K3s<\/a>\u00a0cluster nodes as seen from the Rancher<\/a> dashboard:<\/p>\n
\"\"<\/a><\/p>\n
Application Deployments<\/h2>\n
Each application lives in its own Kubernetes<\/a> namespace for isolation purposes and is made up of a bunch of yaml<\/a> files which are all deployed at once using the kustomize<\/a> aspect of kubectl<\/a>, the command line interface for Kubernetes<\/a>.<\/p>\n
A typical Website install consists of the following:<\/p>\n