diff --git a/Kubernetes/K3S-Deploy/README.md b/Kubernetes/K3S-Deploy/README.md new file mode 100644 index 0000000..7d6df7b --- /dev/null +++ b/Kubernetes/K3S-Deploy/README.md @@ -0,0 +1,151 @@ +# K3S HA Deploy (k3sup + kube-vip + MetalLB) + +Deploys a highly-available k3s cluster over SSH with `k3sup`, a floating +control-plane VIP via **kube-vip**, and `type: LoadBalancer` support via +**MetalLB**. + +> **Following the YouTube tutorial?** This script has been modernized since +> the video — see [What changed](#what-changed-from-the-video) below. + +## What it builds + +- A 3-server (control-plane) + 2-agent (worker) k3s cluster with embedded etcd. +- **kube-vip** advertises a single virtual IP for the Kubernetes API across + all control-plane nodes (survives a node failure). +- **MetalLB** hands out real IPs to `type: LoadBalancer` services. + +### Why both kube-vip and MetalLB? + +They solve different problems. **kube-vip** provides the *control-plane* VIP +(one address for the API server, HA across masters). **MetalLB** provides +*service* load balancing (`type: LoadBalancer` for your apps). The older +kube-vip cloud-provider that overlapped MetalLB has been removed to avoid +duplicate LoadBalancer controllers. + +## Prerequisites + +- **Ubuntu/Debian** nodes (the script installs prerequisites with `apt`; on a + non-apt distro it exits with a clear message). +- **Passwordless sudo** for the SSH user on every node. Example cloud-init: + + ```yaml + #cloud-config + users: + - name: + sudo: ["ALL=(ALL) NOPASSWD:ALL"] + groups: [sudo] + ``` + +- An **SSH key pair** you can use to reach the nodes (the script distributes + the public key and never clobbers your `~/.ssh/config`). +- **5 nodes by default** (3 servers + 2 agents). The minimum for HA is 3 + servers (etcd needs a quorum); workers are optional. Adjust the node list in + the config block if you have fewer machines. +- Run the script from a **separate Ubuntu/Debian admin machine** (your + laptop/workstation) that can SSH to every node — not on one of the nodes + themselves. It installs `k3sup` and `kubectl` locally if missing. + +## Important caveats + +- **The VIP and the MetalLB range must be outside your DHCP pool**, and must + not overlap each other or any node IP. If your router hands out `vip` or an + address in `lbrange` via DHCP, you'll get intermittent, hard-to-debug + failures — reserve them on your router/DHCP server first. +- **Same subnet / L2 only.** kube-vip (ARP mode) and MetalLB (L2) both + advertise via ARP, so the VIP and LoadBalancer IPs must be on the **same + subnet/VLAN** as the nodes and the clients reaching them — they are not + routed across subnets. +- **`interface` must match your nodes' real NIC.** It's `eth0` on many + systems but often `ens18`/`enp0s3` on Proxmox and cloud images. Check with + `ip -o -4 route show default`. +- **Host keys are trusted on first use** (`accept-new`). If you re-image a + node, clear its stale entry first: `ssh-keygen -R `. +- This is a **homelab tutorial, not production-hardened** as-is (aggressive + leader election, no etcd backups, single L2 domain). + +## Usage + +1. Snapshot your VMs (so you can roll back — the script modifies every node). +2. Place your SSH **private** key at `~/` or `~/.ssh/`, + with the matching `.pub` beside it. `certName` is the key's *filename only* + — no path, no `.pub` suffix. Modern OpenSSH defaults to `id_ed25519`, so set + `certName` to match the key you actually have. +3. Edit the **"YOU SHOULD ONLY NEED TO EDIT THIS SECTION"** block in + `k3s.sh` — node IPs, `user`, `interface`, `vip`, `lbrange`, `certName`. + The example IPs (`192.168.3.x`, VIP `.50`, range `.60-.80`) and + `interface=eth0` are placeholders — change them for your network. +4. `chmod +x k3s.sh && ./k3s.sh` +5. Review the pre-flight summary and confirm. Grab a coffee. + +### After it finishes + +The kubeconfig is **merged** into `~/.kube/config` as context `k3s-ha` (set by +`context` in the config block), pointed at the VIP. Select it and check the +cluster: + +```bash +kubectl config use-context k3s-ha +kubectl get nodes -o wide +curl http:// # the script prints the assigned IP +``` + +Worker nodes are labeled `worker=true` and `longhorn=true` (the latter for the +companion Longhorn storage tutorial — harmless if you don't use it). + +If a node fails to join, reset just that node and re-run the script (it's +idempotent): `k3s-uninstall.sh` on a server, `k3s-agent-uninstall.sh` on a +worker. + +### Options (environment variables) + +| Variable | Purpose | +| ---------------- | ----------------------------------------------------- | +| `ASSUME_YES=1` | Skip the pre-flight `[y/N]` prompt (unattended runs). | +| `NO_COLOR=1` | Disable colored output. | +| `RAW_BASE=` | Where to fetch the sibling manifests (kube-vip, ipAddressPool, l2Advertisement); default: upstream `main`. Override for a fork/branch/local copy. | + +### Tracking the latest k3s instead of a pinned version + +In the config block set `k3sChannel="stable"` and leave `k3sVersion=""`. +The script then always installs the current stable k3s release. + +### Upgrading kube-vip + +The `kube-vip` manifest is generated from the pinned image (its env schema +changes between releases). To bump: + +```bash +docker run --rm ghcr.io/kube-vip/kube-vip: manifest daemonset \ + --interface eth0 --address 10.0.0.254 \ + --controlplane --arp --leaderElection --taint --inCluster > kube-vip +sed -i 's/value: eth0/value: REPLACE_INTERFACE/' kube-vip +sed -i 's/value: 10.0.0.254/value: REPLACE_VIP/' kube-vip +``` + +`--inCluster` is required on k3s: it makes kube-vip use the `kube-vip` +ServiceAccount (created by the RBAC the script applies) instead of the +kubeadm `/etc/kubernetes/admin.conf` kubeconfig, which does not exist on k3s. +Then update `KVVERSION` in `k3s.sh` to match. + +## What changed from the video + +- **Versions:** k3s `v1.35.6+k3s1`, kube-vip `v1.2.1`, MetalLB `v0.16.0`. +- **kube-vip** now runs on **all** control-plane nodes, and your kubeconfig + points at the **VIP** (not master1) — fixes `localhost:8080`/API errors. A + readiness check confirms the VIP is answering before the script continues. +- Removed the redundant kube-vip **cloud-provider**; MetalLB alone handles + `type: LoadBalancer`. +- **MetalLB** installs from a single native manifest (which creates its own + `metallb-system` namespace) — the old separate namespace apply and its + mismatched MetalLB versions are gone. +- **k3sup:** the join token is fetched once and reused across all nodes, and + `k3sup ready` waits for the cluster instead of a hand-rolled poll loop. +- **Safer SSH:** host keys are added via `ssh-keyscan`; the script no longer + overwrites `~/.ssh/config`. +- **Hardening:** `set -euo pipefail`, per-node time sync, `apt` prerequisite + install with a clear message on unsupported distros, arch-aware `kubectl`, + and it is safe to re-run. +- **Cleaner output:** non-blinking step-by-step logging (only the banner + still blinks — for old times' sake), a pre-flight config summary with a + `[y/N]` confirmation (`ASSUME_YES=1` to skip), and a final summary listing + the API VIP and the assigned LoadBalancer IP. diff --git a/Kubernetes/K3S-Deploy/ipAddressPool b/Kubernetes/K3S-Deploy/ipAddressPool index ffd58cc..64d8c29 100644 --- a/Kubernetes/K3S-Deploy/ipAddressPool +++ b/Kubernetes/K3S-Deploy/ipAddressPool @@ -5,4 +5,4 @@ metadata: namespace: metallb-system spec: addresses: - - $lbrange \ No newline at end of file + - REPLACE_LBRANGE diff --git a/Kubernetes/K3S-Deploy/k3s.sh b/Kubernetes/K3S-Deploy/k3s.sh old mode 100644 new mode 100755 index faae1d7..5e39324 --- a/Kubernetes/K3S-Deploy/k3s.sh +++ b/Kubernetes/K3S-Deploy/k3s.sh @@ -1,232 +1,404 @@ -#!/bin/bash +#!/usr/bin/env bash +# +# JimsGarage — K3S HA deploy (k3sup + kube-vip + MetalLB) +# Tutorial: https://youtube.com/@jims-garage +# See ./README.md for prerequisites, configuration, and what changed +# from the original video. +set -euo pipefail + +# ── Banner (kept from the original tutorial; blink preserved) ────────── echo -e " \033[33;5m __ _ _ ___ \033[0m" echo -e " \033[33;5m \ \(_)_ __ ___( )__ / _ \__ _ _ __ __ _ __ _ ___ \033[0m" echo -e " \033[33;5m \ \ | '_ \` _ \/ __| / /_\/ _\` | '__/ _\` |/ _\` |/ _ \ \033[0m" echo -e " \033[33;5m /\_/ / | | | | | \__ \ / /_\\ (_| | | | (_| | (_| | __/ \033[0m" echo -e " \033[33;5m \___/|_|_| |_| |_|___/ \____/\__,_|_| \__,_|\__, |\___| \033[0m" echo -e " \033[33;5m |___/ \033[0m" - echo -e " \033[36;5m _ _________ ___ _ _ _ \033[0m" echo -e " \033[36;5m | |/ |__ / __| |_ _|_ _ __| |_ __ _| | | \033[0m" echo -e " \033[36;5m | ' < |_ \__ \ | || ' \(_-| _/ _\` | | | \033[0m" echo -e " \033[36;5m |_|\_|___|___/ |___|_||_/__/\__\__,_|_|_| \033[0m" -echo -e " \033[36;5m \033[0m" echo -e " \033[32;5m https://youtube.com/@jims-garage \033[0m" -echo -e " \033[32;5m \033[0m" +# ── Output helpers (non-blinking; auto-disable when not a TTY) ────────── +if [[ -t 1 && -z "${NO_COLOR:-}" ]]; then + c_reset=$'\033[0m'; c_step=$'\033[1;36m'; c_info=$'\033[36m' + c_ok=$'\033[32m'; c_warn=$'\033[33m'; c_err=$'\033[31m' +else + c_reset=; c_step=; c_info=; c_ok=; c_warn=; c_err= +fi +step() { printf '\n%s━━━ %s ━━━%s\n' "$c_step" "$*" "$c_reset"; } +info() { printf ' %s•%s %s\n' "$c_info" "$c_reset" "$*"; } +ok() { printf ' %s✓%s %s\n' "$c_ok" "$c_reset" "$*"; } +warn() { printf ' %s!%s %s\n' "$c_warn" "$c_reset" "$*" >&2; } +err() { printf ' %s✗%s %s\n' "$c_err" "$c_reset" "$*" >&2; } +die() { err "$*"; exit 1; } ############################################# # YOU SHOULD ONLY NEED TO EDIT THIS SECTION # ############################################# -# Version of Kube-VIP to deploy -KVVERSION="v0.6.3" +# k3s version to install. To always track the latest stable release +# instead, set k3sChannel="stable" and leave k3sVersion empty (see README). +k3sVersion="v1.35.6+k3s1" +k3sChannel="" -# K3S Version -k3sVersion="v1.26.10+k3s2" +# kube-vip: must match the image tag baked into the ./kube-vip manifest. +# To bump, regenerate ./kube-vip (see README "Upgrading kube-vip"). +KVVERSION="v1.2.1" -# Set the IP addresses of the master and work nodes +# MetalLB version (used to build the manifest URL). +METALLB_VERSION="v0.16.0" + +# Node IP addresses master1=192.168.3.21 master2=192.168.3.22 master3=192.168.3.23 worker1=192.168.3.24 worker2=192.168.3.25 -# User of remote machines +# SSH user on the remote nodes user=ubuntu -# Interface used on remotes +# Network interface used on the remote nodes interface=eth0 -# Set the virtual IP address (VIP) +# Virtual IP (VIP) for the HA control plane vip=192.168.3.50 -# Array of master nodes -masters=($master2 $master3) - -# Array of worker nodes -workers=($worker1 $worker2) - -# Array of all -all=($master1 $master2 $master3 $worker1 $worker2) - -# Array of all minus master -allnomaster1=($master2 $master3 $worker1 $worker2) - -#Loadbalancer IP range +# MetalLB LoadBalancer address range lbrange=192.168.3.60-192.168.3.80 -#ssh certificate name variable +# SSH private key name (in ~/.ssh) used to reach the nodes certName=id_rsa -#ssh config file -config_file=~/.ssh/config +# kubeconfig context name to create locally +context=k3s-ha ############################################# # DO NOT EDIT BELOW # ############################################# -# For testing purposes - in case time is wrong due to VM snapshots -sudo timedatectl set-ntp off -sudo timedatectl set-ntp on -# Move SSH certs to ~/.ssh and change permissions -cp /home/$user/{$certName,$certName.pub} /home/$user/.ssh -chmod 600 /home/$user/.ssh/$certName -chmod 644 /home/$user/.ssh/$certName.pub +# Additional control-plane servers joined after master1 +masters=("$master2" "$master3") +# Agent (worker) nodes +workers=("$worker1" "$worker2") +# All control-plane nodes (kube-vip manifest is placed on each) +masters_all=("$master1" "${masters[@]}") +# Every node (for SSH prep loops) +all=("$master1" "${masters[@]}" "${workers[@]}") -# Install k3sup to local machine if not already present -if ! command -v k3sup version &> /dev/null -then - echo -e " \033[31;5mk3sup not found, installing\033[0m" - curl -sLS https://get.k3sup.dev | sh - sudo install k3sup /usr/local/bin/ +# Base URL for the sibling manifests (kube-vip, ipAddressPool, +# l2Advertisement). Override to test from a branch/fork/local copy, e.g. +# RAW_BASE="file://$HOME/JimsGarage" +RAW_BASE="${RAW_BASE:-https://raw.githubusercontent.com/JamesTurland/JimsGarage/main}" +manifest_base="$RAW_BASE/Kubernetes/K3S-Deploy" + +ssh_key="$HOME/.ssh/$certName" +ssh_opts=(-i "$ssh_key" -o StrictHostKeyChecking=accept-new) + +# k3s version selector: prefer an explicit version, else a channel. +if [[ -n "$k3sVersion" ]]; then + k3s_selector=(--k3s-version "$k3sVersion"); k3s_display="$k3sVersion" else - echo -e " \033[32;5mk3sup already installed\033[0m" + k3s_selector=(--k3s-channel "${k3sChannel:-stable}"); k3s_display="channel:${k3sChannel:-stable}" fi -# Install Kubectl if not already present -if ! command -v kubectl version &> /dev/null -then - echo -e " \033[31;5mKubectl not found, installing\033[0m" - curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" - sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl +# Control-plane k3s args: disable bundled traefik + servicelb (we use +# MetalLB), add the VIP to the API serving cert SAN, pin the flannel interface +# + node IP, taint as control-plane. Applied to EVERY server so master2/3 +# declare the VIP SAN too (not just master1). +# Explicit --disable is kept (not k3sup --no-extras) for tutorial clarity. +server_extra_args() { # $1 = node ip + printf '%s' "--disable traefik --disable servicelb --tls-san=$vip --flannel-iface=$interface --node-ip=$1 --node-taint node-role.kubernetes.io/control-plane=true:NoSchedule" +} + +# k3sup prints a promo tip ("Create clusters on Mac…/slicervm.com") on every +# install/join. Filter it out. Use sed, not grep -v: sed always exits 0, so +# pipefail still surfaces a real k3sup failure (grep would exit 1 on no match +# and spuriously abort under set -e). Only wraps the noisy display commands — +# never node-token (whose stdout is captured for the token). +k3sup_quiet() { command k3sup "$@" 2>&1 | sed '/Create clusters on Mac/d'; } + +# Apply a manifest, retrying on transient failure. MetalLB installs a +# validating webhook served by its controller; there is a brief window where +# the controller Deployment reports Available but the webhook endpoints/TLS +# are not serving yet, so the first apply can fail with "no endpoints +# available for service metallb-webhook-service". Retry to ride that out. +kubectl_apply_retry() { # $1 = manifest path or URL + local i + for i in $(seq 1 6); do + kubectl apply -f "$1" && return 0 + warn "kubectl apply $1 failed (attempt $i/6) — retrying in 5s" + sleep 5 + done + die "kubectl apply $1 failed after 6 attempts" +} + +# ── Pre-flight: show config, confirm before touching any node ────────── +step "Pre-flight · review configuration" +printf ' %-16s %s\n' \ + "k3s:" "$k3s_display" \ + "kube-vip:" "$KVVERSION" \ + "metallb:" "$METALLB_VERSION" \ + "control plane:" "${masters_all[*]}" \ + "workers:" "${workers[*]}" \ + "api VIP:" "$vip (interface $interface)" \ + "lb range:" "$lbrange" \ + "ssh user/key:" "$user / $ssh_key" \ + "kube context:" "$context" +if [[ "${ASSUME_YES:-}" == "1" ]]; then + info "ASSUME_YES=1 — proceeding without prompt" else - echo -e " \033[32;5mKubectl already installed\033[0m" + read -rp "$(printf '\n Proceed? [y/N] ')" reply || reply="" + [[ "$reply" =~ ^[Yy]$ ]] || die "Aborted." fi -# Check for SSH config file, create if needed, add/change Strict Host Key Checking (don't use in production!) - -if [ ! -f "$config_file" ]; then - # Create the file and add the line - echo "StrictHostKeyChecking no" > "$config_file" - # Set permissions to read and write only for the owner - chmod 600 "$config_file" - echo "File created and line added." +# ── Step 1/9 · Local tools (k3sup, kubectl) ──────────────────────────── +step "Step 1/9 · Local tools (k3sup, kubectl)" +if ! command -v k3sup &>/dev/null; then + info "Installing k3sup" + curl -sLS https://get.k3sup.dev | sh + # The installer leaves the binary in the CWD — named "k3sup", or + # "k3sup-" when run unprivileged (it can't self-install to + # /usr/local/bin). Install whichever it produced. + k3sup_bin="" + for f in k3sup k3sup-*; do + [[ -f "$f" ]] && { k3sup_bin="$f"; break; } + done + [[ -n "$k3sup_bin" ]] || die "k3sup installer produced no binary" + sudo install "$k3sup_bin" /usr/local/bin/k3sup + rm -f "$k3sup_bin" else - # Check if the line exists - if grep -q "^StrictHostKeyChecking" "$config_file"; then - # Check if the value is not "no" - if ! grep -q "^StrictHostKeyChecking no" "$config_file"; then - # Replace the existing line - sed -i 's/^StrictHostKeyChecking.*/StrictHostKeyChecking no/' "$config_file" - echo "Line updated." - else - echo "Line already set to 'no'." - fi - else - # Add the line to the end of the file - echo "StrictHostKeyChecking no" >> "$config_file" - echo "Line added." - fi + ok "k3sup present" +fi +if ! command -v kubectl &>/dev/null; then + info "Installing kubectl" + arch="$(dpkg --print-architecture 2>/dev/null || uname -m)" + case "$arch" in + amd64|x86_64) arch=amd64 ;; + arm64|aarch64) arch=arm64 ;; + *) die "Unsupported architecture for kubectl: $arch" ;; + esac + kver="$(curl -L -s https://dl.k8s.io/release/stable.txt)" + curl -LO "https://dl.k8s.io/release/${kver}/bin/linux/${arch}/kubectl" + sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl +else + ok "kubectl present" fi -#add ssh keys for all nodes +# ── Step 2/9 · SSH keys and known_hosts ──────────────────────────────── +step "Step 2/9 · SSH keys and known_hosts" +mkdir -p "$HOME/.ssh"; chmod 700 "$HOME/.ssh" +# If the key was dropped in $HOME (per the README), move it into ~/.ssh. +if [[ -f "$HOME/$certName" && ! -f "$ssh_key" ]]; then + info "Moving $certName into ~/.ssh" + cp "$HOME/$certName" "$ssh_key" + [[ -f "$HOME/$certName.pub" ]] && cp "$HOME/$certName.pub" "$ssh_key.pub" +fi +[[ -f "$ssh_key" ]] || die "SSH key not found: $ssh_key (see README prerequisites)" +chmod 600 "$ssh_key"; [[ -f "$ssh_key.pub" ]] && chmod 644 "$ssh_key.pub" +# Trust host keys without clobbering ~/.ssh/config (issue #62). for node in "${all[@]}"; do - ssh-copy-id $user@$node + ssh-keyscan "$node" 2>/dev/null >> "$HOME/.ssh/known_hosts" || \ + warn "ssh-keyscan failed for $node (will accept-new on first connect)" +done +[[ -f "$HOME/.ssh/known_hosts" ]] && sort -u "$HOME/.ssh/known_hosts" -o "$HOME/.ssh/known_hosts" +# Distribute the public key to each node. +for node in "${all[@]}"; do + info "Copying SSH key to $user@$node" + ssh-copy-id -i "$ssh_key.pub" -o StrictHostKeyChecking=accept-new "$user@$node" >/dev/null 2>&1 || \ + warn "ssh-copy-id to $node failed (key may already be present)" +done +ok "SSH ready for ${#all[@]} nodes" + +# ── Step 3/9 · Prepare nodes (time sync + prerequisites) ─────────────── +step "Step 3/9 · Prepare nodes (time sync + prerequisites)" +for node in "${all[@]}"; do + info "Preparing $node" + rc=0 + ssh "${ssh_opts[@]}" "$user@$node" 'sudo bash -s' <<'REMOTE' || rc=$? +set -e +# Resync time — VM snapshots drift, which breaks k3s/k3sup installs (issue #68). +timedatectl set-ntp off || true +timedatectl set-ntp on || true +if command -v apt-get >/dev/null 2>&1; then + export DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a + ok=0 + for i in $(seq 1 10); do + if apt-get update && apt-get install -y iptables sudo policycoreutils; then ok=1; break; fi + echo "apt busy, retry $i/10..."; sleep 3 + done + [ "$ok" = 1 ] || exit 91 +else + exit 90 +fi +REMOTE + case "$rc" in + 0) ok "$node prepared" ;; + 90) die "Node $node is not apt-based. This script targets Ubuntu/Debian; install iptables, sudo, and policycoreutils manually (see README)." ;; + 91) die "Node $node: apt failed after retries." ;; + *) die "Node $node: preparation failed (exit $rc)." ;; + esac done -# Install policycoreutils for each node -for newnode in "${all[@]}"; do - ssh $user@$newnode -i ~/.ssh/$certName sudo su </dev/null +info "Applying kube-vip RBAC" kubectl apply -f https://kube-vip.io/manifests/rbac.yaml +info "Rendering kube-vip manifest (interface=$interface, vip=$vip)" +curl -sfL "$manifest_base/kube-vip" -o "$HOME/kube-vip.src" +sed "s/REPLACE_INTERFACE/$interface/g; s/REPLACE_VIP/$vip/g" \ + "$HOME/kube-vip.src" > "$HOME/kube-vip.yaml" +for node in "${masters_all[@]}"; do + info "Placing kube-vip manifest on $node" + # scp to /tmp (absolute) — the mv below runs as root, where ~ is /root, + # not the ssh user's home where scp would otherwise land the file. + scp "${ssh_opts[@]}" "$HOME/kube-vip.yaml" "$user@$node:/tmp/kube-vip.yaml" >/dev/null + ssh "${ssh_opts[@]}" "$user@$node" 'sudo bash -s' <<'REMOTE' +set -e +mkdir -p /var/lib/rancher/k3s/server/manifests +mv /tmp/kube-vip.yaml /var/lib/rancher/k3s/server/manifests/kube-vip.yaml +chown root:root /var/lib/rancher/k3s/server/manifests/kube-vip.yaml +chmod 600 /var/lib/rancher/k3s/server/manifests/kube-vip.yaml +REMOTE +done +ok "kube-vip deployed to ${#masters_all[@]} control-plane nodes" +info "Pointing local kubeconfig at the VIP ($vip)" +# Rewrite the server URL from master1 to the VIP. The dots in the IP are +# unescaped regex metacharacters, but a false match would need the literal +# "https://:6443" string elsewhere in the kubeconfig, which does +# not occur (cert data is base64 — no scheme or port), so this is safe here. +sed -i "s#https://$master1:6443#https://$vip:6443#" "$HOME/.kube/config" -# Step 3: Download kube-vip -curl -sO https://raw.githubusercontent.com/JamesTurland/JimsGarage/main/Kubernetes/K3S-Deploy/kube-vip -cat kube-vip | sed 's/$interface/'$interface'/g; s/$vip/'$vip'/g' > $HOME/kube-vip.yaml +# Confirm the VIP is actually answering before any VIP-routed kubectl below. +# This is an early tripwire for a broken kube-vip (wrong interface, RBAC, etc.) +# instead of a confusing failure further down. +info "Waiting for the control-plane VIP ($vip) to answer" +vip_ready="" +for _ in $(seq 1 30); do + if kubectl --request-timeout=5s get --raw='/readyz' &>/dev/null; then + vip_ready=1; break + fi + sleep 2 +done +[[ -n "$vip_ready" ]] || die "Control-plane VIP $vip is not answering — check kube-vip on the masters: kubectl -n kube-system logs -l app.kubernetes.io/name=kube-vip-ds" +ok "Control-plane VIP is answering" -# Step 4: Copy kube-vip.yaml to master1 -scp -i ~/.ssh/$certName $HOME/kube-vip.yaml $user@$master1:~/kube-vip.yaml +# ── Step 6/9 · Fetch the cluster join token ──────────────────────────── +step "Step 6/9 · Fetch the cluster join token" +node_token="$(k3sup node-token --ip "$master1" --user "$user" --ssh-key "$ssh_key")" \ + || die "Failed to fetch node token from $master1" +[[ -n "$node_token" ]] || die "Empty node token from $master1" +ok "Join token fetched" - -# Step 5: Connect to Master1 and move kube-vip.yaml -ssh $user@$master1 -i ~/.ssh/$certName <<- EOF - sudo mkdir -p /var/lib/rancher/k3s/server/manifests - sudo mv kube-vip.yaml /var/lib/rancher/k3s/server/manifests/kube-vip.yaml -EOF - -# Step 6: Add new master nodes (servers) & workers -for newnode in "${masters[@]}"; do - k3sup join \ - --ip $newnode \ - --user $user \ +# ── Step 7/9 · Join control-plane and worker nodes ───────────────────── +step "Step 7/9 · Join control-plane and worker nodes" +# The prefetched --node-token is what keeps these joins from ever SSHing to +# the server: without it, k3sup would try --server-user (default: root) against +# --server-ip (the VIP, not an SSH target) and fail confusingly. Keep the token. +for node in "${masters[@]}"; do + info "Joining control-plane node $node" + k3sup_quiet join \ + --ip "$node" \ + --user "$user" \ --sudo \ - --k3s-version $k3sVersion \ --server \ - --server-ip $master1 \ - --ssh-key $HOME/.ssh/$certName \ - --k3s-extra-args "--disable traefik --disable servicelb --flannel-iface=$interface --node-ip=$newnode --node-taint node-role.kubernetes.io/master=true:NoSchedule" \ - --server-user $user - echo -e " \033[32;5mMaster node joined successfully!\033[0m" + --server-ip "$vip" \ + --node-token "$node_token" \ + "${k3s_selector[@]}" \ + --k3s-extra-args "$(server_extra_args "$node")" \ + --ssh-key "$ssh_key" + ok "Control-plane node $node joined" done - -# add workers -for newagent in "${workers[@]}"; do - k3sup join \ - --ip $newagent \ - --user $user \ +for node in "${workers[@]}"; do + info "Joining worker node $node" + k3sup_quiet join \ + --ip "$node" \ + --user "$user" \ --sudo \ - --k3s-version $k3sVersion \ - --server-ip $master1 \ - --ssh-key $HOME/.ssh/$certName \ - --k3s-extra-args "--node-label \"longhorn=true\" --node-label \"worker=true\"" - echo -e " \033[32;5mAgent node joined successfully!\033[0m" + --server-ip "$vip" \ + --node-token "$node_token" \ + "${k3s_selector[@]}" \ + --k3s-extra-args "--flannel-iface=$interface --node-ip=$node --node-label longhorn=true --node-label worker=true" \ + --ssh-key "$ssh_key" + ok "Worker node $node joined" done -# Step 7: Install kube-vip as network LoadBalancer - Install the kube-vip Cloud Provider -kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml +# Assert the full HA control plane is up before continuing. The VIP gate in +# Step 5 only proved master1 (a single-member etcd); a control-plane join that +# half-succeeds would otherwise sail through as success on a degraded cluster. +info "Waiting for all ${#masters_all[@]} control-plane nodes to be Ready" +kubectl wait --for=condition=Ready node \ + -l node-role.kubernetes.io/control-plane --timeout=180s || \ + die "Control-plane nodes did not all become Ready — check etcd/kube-vip on the masters: kubectl get nodes; kubectl -n kube-system logs -l app.kubernetes.io/name=kube-vip-ds" +cp_ready="$(kubectl get nodes -l node-role.kubernetes.io/control-plane --no-headers 2>/dev/null | awk '$2=="Ready"{c++} END{print c+0}')" +[[ "$cp_ready" -eq "${#masters_all[@]}" ]] || \ + die "Expected ${#masters_all[@]} control-plane nodes Ready, found $cp_ready — etcd quorum may be degraded" +ok "All ${#masters_all[@]} control-plane nodes Ready" -# Step 8: Install Metallb -kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml -kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml -# Download ipAddressPool and configure using lbrange above -curl -sO https://raw.githubusercontent.com/JamesTurland/JimsGarage/main/Kubernetes/K3S-Deploy/ipAddressPool -cat ipAddressPool | sed 's/$lbrange/'$lbrange'/g' > $HOME/ipAddressPool.yaml -kubectl apply -f $HOME/ipAddressPool.yaml +# ── Step 8/9 · Install MetalLB (service LoadBalancer) ────────────────── +step "Step 8/9 · Install MetalLB (service LoadBalancer)" +kubectl apply -f "https://raw.githubusercontent.com/metallb/metallb/$METALLB_VERSION/config/manifests/metallb-native.yaml" +info "Waiting for the MetalLB controller" +# rollout status waits on the Deployment (created synchronously by apply) and +# avoids the "no matching resources found" race that `wait --for=condition=ready +# pod` hits when the controller pod does not exist yet. +kubectl -n metallb-system rollout status deploy/controller --timeout=120s +info "Configuring address pool ($lbrange)" +curl -sfL "$manifest_base/ipAddressPool" -o "$HOME/ipAddressPool.src" +sed "s|REPLACE_LBRANGE|$lbrange|g" "$HOME/ipAddressPool.src" > "$HOME/ipAddressPool.yaml" +kubectl_apply_retry "$HOME/ipAddressPool.yaml" +curl -sfL "$manifest_base/l2Advertisement.yaml" -o "$HOME/l2Advertisement.yaml" +kubectl_apply_retry "$HOME/l2Advertisement.yaml" +ok "MetalLB configured" -# Step 9: Test with Nginx -kubectl apply -f https://raw.githubusercontent.com/inlets/inlets-operator/master/contrib/nginx-sample-deployment.yaml -n default -kubectl expose deployment nginx-1 --port=80 --type=LoadBalancer -n default - -echo -e " \033[32;5mWaiting for K3S to sync and LoadBalancer to come online\033[0m" - -while [[ $(kubectl get pods -l app=nginx -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}') != "True" ]]; do - sleep 1 +# ── Step 9/9 · Verify cluster and LoadBalancer ───────────────────────── +step "Step 9/9 · Verify cluster and LoadBalancer" +info "Waiting for the cluster to be ready" +k3sup_quiet ready --context "$context" --kubeconfig "$HOME/.kube/config" +info "Deploying nginx sample" +kubectl apply -n default -f https://raw.githubusercontent.com/inlets/inlets-operator/master/contrib/nginx-sample-deployment.yaml +# `|| true` — expose is imperative and errors with AlreadyExists on a re-run, +# which would abort the script under set -e. +kubectl expose deployment nginx-1 --port=80 --type=LoadBalancer -n default 2>/dev/null || true +info "Waiting for the nginx deployment" +kubectl -n default rollout status deploy/nginx-1 --timeout=120s +info "Waiting for the LoadBalancer IP" +lb_ip="" +for _ in $(seq 1 30); do + lb_ip="$(kubectl get svc nginx-1 -n default -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || true)" + [[ -n "$lb_ip" ]] && break + sleep 2 done -# Step 10: Deploy IP Pools and l2Advertisement -kubectl wait --namespace metallb-system \ - --for=condition=ready pod \ - --selector=component=controller \ - --timeout=120s -kubectl apply -f ipAddressPool.yaml -kubectl apply -f https://raw.githubusercontent.com/JamesTurland/JimsGarage/main/Kubernetes/K3S-Deploy/l2Advertisement.yaml - -kubectl get nodes -kubectl get svc -kubectl get pods --all-namespaces -o wide - -echo -e " \033[32;5mHappy Kubing! Access Nginx at EXTERNAL-IP above\033[0m" +kubectl get nodes -o wide || true +step "Done · cluster ready" +ready_nodes="$(kubectl get nodes --no-headers 2>/dev/null | awk '$2=="Ready"{c++} END{print c+0}')" || ready_nodes=0 +ok "$ready_nodes nodes Ready · context '$context'" +ok "API server: https://$vip:6443" +if [[ -n "$lb_ip" ]]; then + ok "nginx LoadBalancer IP: $lb_ip (curl http://$lb_ip)" +else + warn "nginx LoadBalancer IP not assigned yet — check: kubectl get svc -A" +fi +info "Next: kubectl get pods -A" diff --git a/Kubernetes/K3S-Deploy/kube-vip b/Kubernetes/K3S-Deploy/kube-vip index ae48e5d..ed9a0e3 100644 --- a/Kubernetes/K3S-Deploy/kube-vip +++ b/Kubernetes/K3S-Deploy/kube-vip @@ -1,10 +1,9 @@ apiVersion: apps/v1 kind: DaemonSet metadata: - creationTimestamp: null labels: app.kubernetes.io/name: kube-vip-ds - app.kubernetes.io/version: v0.8.2 + app.kubernetes.io/version: v1.2.1 name: kube-vip-ds namespace: kube-system spec: @@ -13,10 +12,9 @@ spec: app.kubernetes.io/name: kube-vip-ds template: metadata: - creationTimestamp: null labels: app.kubernetes.io/name: kube-vip-ds - app.kubernetes.io/version: v0.8.2 + app.kubernetes.io/version: v1.2.1 spec: affinity: nodeAffinity: @@ -36,36 +34,38 @@ spec: value: "true" - name: port value: "6443" + - name: vip_nodename + valueFrom: + fieldRef: + fieldPath: spec.nodeName - name: vip_interface - value: $interface - - name: vip_cidr + value: REPLACE_INTERFACE + - name: vip_subnet value: "32" + - name: dns_mode + value: first + - name: dhcp_mode + value: ipv4 - name: cp_enable value: "true" - name: cp_namespace value: kube-system - - name: vip_ddns - value: "false" - - name: svc_enable - value: "false" - - name: svc_leasename - value: plndr-svcs-lock - name: vip_leaderelection value: "true" - name: vip_leasename value: plndr-cp-lock - name: vip_leaseduration - value: "5" + value: "15" - name: vip_renewdeadline - value: "3" + value: "10" - name: vip_retryperiod - value: "1" + value: "2" - name: address - value: $vip + value: REPLACE_VIP - name: prometheus_server value: :2112 - image: ghcr.io/kube-vip/kube-vip:v0.8.2 - imagePullPolicy: Always + image: ghcr.io/kube-vip/kube-vip:v1.2.1 + imagePullPolicy: IfNotPresent name: kube-vip resources: {} securityContext: @@ -73,6 +73,8 @@ spec: add: - NET_ADMIN - NET_RAW + drop: + - ALL hostNetwork: true serviceAccountName: kube-vip tolerations: @@ -81,9 +83,3 @@ spec: - effect: NoExecute operator: Exists updateStrategy: {} -status: - currentNumberScheduled: 0 - desiredNumberScheduled: 0 - numberMisscheduled: 0 - numberReady: 0 - diff --git a/Kubernetes/K3S-Deploy/l2Advertisement.yaml b/Kubernetes/K3S-Deploy/l2Advertisement.yaml index b6f8c4d..ea303f0 100644 --- a/Kubernetes/K3S-Deploy/l2Advertisement.yaml +++ b/Kubernetes/K3S-Deploy/l2Advertisement.yaml @@ -1,8 +1,8 @@ apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: - name: example + name: l2-first-pool namespace: metallb-system spec: ipAddressPools: - - first-pool \ No newline at end of file + - first-pool diff --git a/Kubernetes/K3S-Deploy/readme.md b/Kubernetes/K3S-Deploy/readme.md deleted file mode 100644 index 386cd6f..0000000 --- a/Kubernetes/K3S-Deploy/readme.md +++ /dev/null @@ -1,5 +0,0 @@ -1. Snapshot your VMs! -1. Add certificates to your home directory -1. copy script to home directory and make executable - -1. Run the script, grab a coffee and enjoy :) (hopefully!)