microk8s-janitor
Skill4 starsChanged 5 months agoMITView on GitHub
Automates rolling upgrades and maintenance for HA MicroK8s clusters on Ubuntu via SSH. Supports node discovery, channel selection, pre-flight checks, and stateful recovery.
In this repository9 other files it gives its agents.
- agent-cprSkill
- anti-sycophancySkill
- cve-remediationSkill
- discord-coordinationSkill
- gui-test-leaseSkill
- infisical-proSkill
- nats-coordinationSkill
- rasp-pi-janitorSkill
- ubuntu-janitorSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
18 rules. About 0 are common, 7 are distinctive, and 11 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
11
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 24
Connectivity Check: Verify SSH and sudo access to every node in the cluster.
- Very specific: line 25
Channel Discovery: Run snap info microk8s on the seed node to list available tracking channels (e.g., 1.28/stable, latest/edge).
- Distinctive: line 29
Before any disruptive action, the janitor ensures the cluster is healthy:
- Very specific: line 31
All nodes must be in the Ready status.
- Very specific: line 32
Check for any "critical" pods (e.g., Longhorn, Calico, CoreDNS) that are currently in a non-running state.
- Very specific: line 33
Verify that dqlite (the HA backend) has a healthy quorum.
- Distinctive: line 40
Ask for confirmation before proceeding.
- Very specific: line 50
Health Check: Verify local pods are starting and node status is Ready.
- Distinctive: line 55
If a step fails:
- Distinctive: line 56
Abort: Stop immediately. Do not move to the next node.
- Very specific: line 63
When upgrading stateful applications (PostgreSQL, MongoDB, Redis) via Helm, changing certain fields (like storageClass or …
- Very specific: line 69
If data MUST be kept: Ensure the PVC has a Retain or Delete policy that allows it to persist (or orphan) after the StatefulSet is deleted.
- Distinctive: line 70
If a clean slate is intended: Prepare to delete the PVC as well.
- Very specific: line 71
Destructive Action: Delete the specific StatefulSet directly: \\\bash kubectl delete statefulset <name> -n <namespace> \\\
- Distinctive: line 79
Quorum First: Never upgrade more than one node at a time in a 3-node HA cluster to avoid losing quorum.
- Distinctive: line 80
Drain Timeout: If a drain hangs, report the specific pod causing the delay to the user.
- Very specific: line 81
Snap Rollback: If snap refresh fails, attempts snap revert microk8s if appropriate.
- Very specific: line 102
If docker push gets a 504 on a plain push, retry once — layers are already cached.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
MicroK8s Janitor
This skill manages the lifecycle of a MicroK8s cluster, specifically focusing on safe, rolling upgrades of nodes to ensure high availability is maintained throughout the process.
Prerequisites
- SSH access to at least one node in the cluster (the "seed" node).
- Passwordless
sudoor SSH key-based authentication for the user. - The
microk8ssnap must be installed on the target nodes.
Core Workflows
1. Cluster Discovery & Environment Setup
Starting from a single seed node provided by the user, the janitor discovers the full cluster state.
- Seed Connection: Connect to the seed node and run
microk8s kubectl get nodes -o json. - Node Mapping: Parse the output to identify all nodes, their roles, and current statuses.
- Connectivity Check: Verify SSH and
sudoaccess to every node in the cluster. - Channel Discovery: Run
snap info microk8son the seed node to list available tracking channels (e.g.,1.28/stable,latest/edge).
2. Pre-flight Checks (The "Dry Run")
Before any disruptive action, the janitor ensures the cluster is healthy:
- All nodes must be in the
Readystatus. - Check for any "critical" pods (e.g.,
Longhorn,Calico,CoreDNS) that are currently in a non-running state. - Verify that
dqlite(the HA backend) has a healthy quorum.
3. Interactive Planning
Present the plan to the user:
- Current version vs. Target channel.
- The order of nodes to be upgraded.
- Ask for confirmation before proceeding.
4. Rolling Upgrade Loop (Sequential Execution)
For each node in the sequence:
- Cordon:
microk8s kubectl cordon <node-name> - Drain:
microk8s kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force - Upgrade:
sudo snap refresh microk8s --channel=<channel> - Wait for Ready: Poll
microk8s status --wait-ready(max 5 minutes). - Health Check: Verify local pods are starting and node status is
Ready. - Uncordon:
microk8s kubectl uncordon <node-name>
5. Resume & Recovery
If a step fails:
- Abort: Stop immediately. Do not move to the next node.
- Log: Capture and display the error from the node.
- State Check: On re-invocation, the janitor detects if any nodes are still cordoned and offers to "Resume" the upgrade from the failed node.
Helm & StatefulSet Management
The "Immutable Wall"
When upgrading stateful applications (PostgreSQL, MongoDB, Redis) via Helm, changing certain fields (like storageClass or volumeClaimTemplates) will cause a Forbidden error because StatefulSet templates are immutable.
The Stateful Upgrade Workflow
To resolve immutable field errors:
- Identify: Confirm the error is due to immutable fields in a `StatefulSet`.
- Preservation Decision:
- If data MUST be kept: Ensure the PVC has a
RetainorDeletepolicy that allows it to persist (or orphan) after the StatefulSet is deleted. - If a clean slate is intended: Prepare to delete the PVC as well.
- If data MUST be kept: Ensure the PVC has a
- Destructive Action: Delete the specific StatefulSet directly:
```bash
kubectl delete statefulset
<name>-n<namespace>``` - Re-apply: Re-run the
helm upgradeorhelm installcommand. Helm will now successfully recreate the StatefulSet with the new spec.
Best Practices
- Quorum First: Never upgrade more than one node at a time in a 3-node HA cluster to avoid losing quorum.
- Drain Timeout: If a drain hangs, report the specific pod causing the delay to the user.
- Snap Rollback: If
snap refreshfails, attemptssnap revert microk8sif appropriate.
Cluster Access (bnaylor's cluster)
- Local kubectl:
kubectlworks directly from the Mac via~/.kube/config— no SSH needed for cluster operations. - API server:
https://10.3.2.4:16443(microk8s-cluster) - Docker Hub user:
bnaylor— already authenticated locally.
Deploying Services to This Cluster
General workflow for Python services (e.g. running-proxy)
Always cross-build for linux/amd64 — even pure Python services use pip packages with native binaries (uvicorn, pydantic-core, cryptography, greenlet, etc.) that are architecture-specific. Building natively on an ARM64 Mac produces an image that crashes with exec format error on the x86_64 cluster nodes.
docker buildx build --platform linux/amd64 -t bnaylor/<service>:latest --push .
kubectl rollout restart deployment/<service> -n <namespace>
kubectl rollout status deployment/<service> -n <namespace>--pushin the buildx command pushes directly, skipping a separatedocker pushstep.- If
docker pushgets a 504 on a plain push, retry once — layers are already cached.
running-proxy service
- Namespace:
running-proxy - Image:
bnaylor/running-proxy:latest - External IP:
10.3.2.132(LoadBalancer, port 80 → container 8000) - Deployment YAMLs:
~/k8s/running-proxy/yamls/ - Secrets:
running-proxy-secrets(spreadsheet-id),google-service-account-json(service-account.json) - Storage: NFS PVC
running-proxy-datamounted at/app/data(SQLite lives there); strategy isRecreateto avoid concurrent writers
LiteLLM (noc cluster)
- Namespace:
litellm - Image:
ghcr.io/berriai/litellm:main - External IP:
10.3.2.135(LoadBalancer, port 8000) - Config:
litellm-configConfigMap inlitellmnamespace.
Software Version Discovery (LiteLLM)
Determining the version of LiteLLM in the cluster is non-trivial because the CLI and package attributes are inconsistent.
Effective Methods:
- Check pyproject.toml (Most Reliable for source version):
kubectl exec -n litellm <pod-name> -- cat /app/pyproject.toml | grep version - Check pip list (Reliable for installed package):
kubectl exec -n litellm <pod-name> -- pip list | grep litellm - Check Git Commit (For development builds):
kubectl exec -n litellm <pod-name> -- cat /app/.git/refs/heads/main - Image Digest (Most definitive for release date):
kubectl get pod <pod-name> -n litellm -o jsonpath="{.status.containerStatuses[0].imageID}"
Known Failures (Don't use these):
- ❌
litellm --version(Throws an error or shows help) - ❌
python3 -c "import litellm; print(litellm.__version__)"(AttributeError: module 'litellm' has no attribute 'version') - ❌
/versionHTTP endpoint (Returns 404 in many configurations)
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

