K8sGPT with a Local 70B Model on Apple Silicon

Every vendor pitch deck right now has the same slide. “Autonomous cluster management.” An AI watches your Kubernetes cluster, spots problems, diagnoses them, and fixes them while you sleep. Platform engineers get to stop firefighting and the cluster heals itself.

I run a homelab specifically because I want to understand what’s actually happening, not trust a black box. So when I see a claim like that, my first instinct is to test it myself rather than believe the slide.

The catch with most of these tools is that “AI” means “your cluster metadata gets shipped to OpenAI.” For me that’s a non-starter. I self-host my infrastructure precisely so my data stays on my network. If the AIOps story only works when you hand your pod names, labels, and error messages to a third party, then it does not align with anything I care about.

So here’s the version I wanted to try: K8sGPT pointed at a Llama 3.3 70B model running locally on Apple Silicon. No cloud APIs. No data leaving the network. The question I actually want answered is whether a local model is good enough to be useful for cluster diagnosis, or whether sovereignty here costs more than it’s worth.

Disclaimer: This is a homelab experiment. I’m reporting what I tested and what I found. Read it as a “do not run this in production” writeup, because the security analysis below makes that case in detail.

Hardware and Software Stack

The Hardware

• Mac Studio M3 Ultra with 512GB unified memory
• The M3 Ultra has 80 GPU cores you can use for inference
• Unified memory means no copying between CPU and GPU RAM

At ~€10,000 this is an expensive box. I’m not pretending otherwise. But it’s about the only consumer hardware that can run a 70B model in Q8 quantization at speeds you’d tolerate. The unified memory is the whole trick: you don’t pay the copy tax between CPU and GPU RAM that kills you on a normal PC with a discrete card.

The Software Stack

Installation: vLLM with Metal Backend

vLLM has experimental Metal support for Apple Silicon. Installation:

# Create a dedicated conda environment
conda create -n vllm python=3.11
conda activate vllm

# Install vLLM with Metal support
pip install vllm

# Verify Metal backend
python -c "import vllm; print(vllm.__version__)"

Note: At time of writing, Metal support in vLLM is still experimental, and it shows. llama.cpp with the Metal backend is the more stable option, but K8sGPT wants an OpenAI-compatible API and vLLM gives me that out of the box. So I took the rougher path to avoid writing a shim.

Model Download

# Download the model (Q8 quantization, ~75GB)
huggingface-cli download meta-llama/Llama-3.3-70B-Instruct \
  --local-dir ./models/llama-3.3-70b-instruct

You need a Hugging Face account and must accept the Llama license.

Starting the vLLM Server

# Start the inference server
vllm serve ./models/llama-3.3-70b-instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype float16 \
  --max-model-len 8192 \
  --device mps

The --device mps flag forces Metal Performance Shaders. Without this flag, vLLM falls back to CPU.

Verify the server is running:

curl http://localhost:8000/v1/models

K8sGPT Operator Deployment

Install the K8sGPT operator in your cluster:

# Add the Helm repo
helm repo add k8sgpt https://charts.k8sgpt.ai/
helm repo update

# Install the operator
helm install k8sgpt-operator k8sgpt/k8sgpt-operator \
  -n k8sgpt-operator-system \
  --create-namespace

Configure a custom backend pointing to your local vLLM server:

# k8sgpt-config.yaml
apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
  name: k8sgpt-local
  namespace: k8sgpt-operator-system
spec:
  ai:
    enabled: true
    model: llama-3.3-70b-instruct
    backend: localai
    baseUrl: http://192.168.1.100:8000/v1  # IP of your Mac Studio
  noCache: false
  version: v0.3.40
  analyzers:
    - Pod
    - Deployment
    - Service
    - ReplicaSet
    - PersistentVolumeClaim
    - Ingress
    - StatefulSet
    - CronJob

kubectl apply -f k8sgpt-config.yaml

Test Scenarios

The setup runs. Now the part that matters: is it actually useful, or is it a very expensive way to get a wrong answer with confidence? I threw three deliberately broken things at it and graded the output the way I’d grade a junior engineer’s first pass.

Scenario A: CrashLoopBackOff Diagnosis

I introduce a deployment with a missing ConfigMap:

# broken-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broken-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: broken-app
  template:
    metadata:
      labels:
        app: broken-app
    spec:
      containers:
      - name: app
        image: nginx:1.25
        envFrom:
        - configMapRef:
            name: app-config  # This ConfigMap doesn't exist

kubectl apply -f broken-deployment.yaml

After a minute, the pod is in CrashLoopBackOff. K8sGPT analysis:

kubectl get results -n k8sgpt-operator-system -o yaml

Output (paraphrased):

Analysis: Pod broken-app-xxxx is in CreateContainerConfigError state.
The pod is referencing a ConfigMap 'app-config' that does not exist
in the namespace.

Suggested remediation:
1. Create the missing ConfigMap:
   kubectl create configmap app-config --from-literal=KEY=value
2. Or remove the configMapRef from the deployment spec
3. Verify the ConfigMap name spelling matches exactly

Score: Usable

The diagnosis is correct and the suggestions are practical. A junior engineer could work with this and not get led astray. The one thing it missed: it never suggests checking whether the ConfigMap lives in a different namespace, which is the mistake I make about once a month.

Scenario B: Resource Recommendations

A deployment without resource limits:

# no-limits.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: no-limits-app
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: no-limits
  template:
    metadata:
      labels:
        app: no-limits
    spec:
      containers:
      - name: app
        image: nginx:1.25
        # No resources defined

K8sGPT analysis:

Analysis: Deployment no-limits-app has containers without resource
requests or limits defined.

Issues identified:
- Container 'app' has no CPU requests/limits
- Container 'app' has no memory requests/limits

Suggested remediation:
Add resource specifications to ensure predictable scheduling and
prevent resource starvation:

resources:
  requests:
    memory: "64Mi"
    cpu: "100m"
  limits:
    memory: "128Mi"
    cpu: "200m"

Score: Partially usable

It spots the problem fine. The numbers it hands you are made up. They aren’t based on anything the workload actually does, because the model never saw a single metric. An experienced engineer would pull real data with kubectl top pods or Prometheus first, then rightsize from there. The LLM is guessing, and it doesn’t tell you it’s guessing.

What’s missing:

• No suggestion to use VPA (Vertical Pod Autoscaler)
• No warning that limits without requests can be problematic
• The requests:limits ratio (1:2) is arbitrary

Scenario C: Security Audit

A pod with poor security practices:

# insecure-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: insecure-pod
  namespace: default
spec:
  hostNetwork: true
  containers:
  - name: app
    image: nginx:1.25
    securityContext:
      privileged: true
      runAsRoot: true
    volumeMounts:
    - name: host-root
      mountPath: /host
  volumes:
  - name: host-root
    hostPath:
      path: /

K8sGPT analysis with Trivy integration:

Analysis: Pod insecure-pod has multiple security issues:

Critical:
- Container running in privileged mode (privileged: true)
- Container has access to host network (hostNetwork: true)
- Container mounts host root filesystem

High:
- No securityContext.readOnlyRootFilesystem
- No securityContext.allowPrivilegeEscalation: false

Suggested remediation:
1. Remove privileged: true unless absolutely necessary
2. Use a dedicated network namespace (remove hostNetwork: true)
3. Avoid mounting host paths, especially root filesystem
4. Add security hardening:

   securityContext:
     privileged: false
     readOnlyRootFilesystem: true
     allowPrivilegeEscalation: false
     runAsNonRoot: true
     capabilities:
       drop:
         - ALL

Score: Usable

This one surprised me. The issues are correctly identified and the remediation reads like what a security engineer would actually tell you. The Trivy integration earns its keep here by also flagging image vulnerabilities, so you get both the misconfiguration and the CVE story in one pass.

What’s missing:

• No suggestion for Pod Security Standards (restricted profile)
• No warning about Kyverno/OPA policies that should block this

Performance Metrics

Inference Speed

End-to-End Latency

From issue detection to report in K8sGPT:

Resource Usage

During inference:

Comparison with OpenAI API

The OpenAI API is faster and the output is marginally better. The price is that your cluster metadata leaves your network. For me that trade is the whole point of the exercise: I’ll happily wait 25 seconds instead of 5 to keep the data at home. Whether you make the same call depends on what’s in your pod labels.

Air-Gapped Deployment

Can this run with no internet at all? Yes, with some preparation up front. This is where the local-model approach pays off, because the air-gap case is impossible the moment you depend on a cloud API.

What You Need to Download Beforehand

# 1. Model weights (~75GB)
huggingface-cli download meta-llama/Llama-3.3-70B-Instruct \
  --local-dir ./airgap-bundle/models/

# 2. vLLM Python packages
pip download vllm -d ./airgap-bundle/packages/

# 3. K8sGPT container images
docker pull ghcr.io/k8sgpt-ai/k8sgpt-operator:latest
docker save ghcr.io/k8sgpt-ai/k8sgpt-operator:latest > ./airgap-bundle/images/k8sgpt-operator.tar

# 4. Helm charts
helm pull k8sgpt/k8sgpt-operator --destination ./airgap-bundle/charts/

Transport and Installation

# On the air-gapped machine:

# 1. Install Python packages offline
pip install --no-index --find-links=./airgap-bundle/packages/ vllm

# 2. Load container images
docker load < ./airgap-bundle/images/k8sgpt-operator.tar
# Or push to your local registry

# 3. Install Helm chart
helm install k8sgpt-operator ./airgap-bundle/charts/k8sgpt-operator-*.tgz \
  --set image.repository=your-local-registry/k8sgpt-operator

Air-Gap Friendly Components

Note: The Trivy vulnerability database needs to be updated and transported separately. Without a recent DB, K8sGPT will miss new CVEs.

Security Analysis and Threat Model

Here’s where the honesty matters. Running a local model gives you data sovereignty, but it does not magically make the setup secure. A Mac Studio sitting on your network as an inference server brings its own pile of problems, and the LLM brings a second pile that’s specific to it.

Platform Security Issues

A Mac Studio as inference server has fundamental limitations:

Conclusion: A Mac Studio is unsuitable for environments with strict compliance requirements (ISO27001 Annex A, NIS2, SOC2). For homelab and development it’s acceptable.

LLM-Specific Risks

Threat Model

Conclusions and Recommendations

Is a Local LLM Usable for Kubernetes Diagnosis?

Yes, under conditions.

It can:

• Correctly identify standard issues
• Provide usable remediation suggestions
• Detect security problems
• Do all this without data leaving your network

It cannot:

• Debug complex, multi-component issues
• Reliably do auto-remediation
• Understand the context of your specific setup
• Guarantee correctness

Recommendations per Use Case

Homelab / Learning

Recommendation: Go for it.

This is an excellent way to learn about:

• LLM inference infrastructure
• Kubernetes troubleshooting patterns
• The limits of AI-assisted operations

Risks are acceptable because the impact is limited.

Development / Staging

Recommendation: Usable with guardrails.

Implement:

• Output review before applying suggestions
• Logging of all LLM interactions
• No auto-remediation, diagnosis only

Production (not air-gapped)

Recommendation: Use cloud APIs.

Why:

• Better models (GPT-4, Claude)
• Lower latency
• No hardware investment
• Professional SLAs

The privacy trade-off is acceptable for most organizations if you don’t have PII in cluster metadata.

Production (air-gapped / sovereign)

Recommendation: Only as last resort.

If you truly cannot send data outside:

• Consider smaller, dedicated models
• Implement defense-in-depth for the inference server
• Treat all LLM output as untrusted
• Ensure extensive logging and audit trails
• Use this as assistance, never as authority

The State of Autonomous Cluster Management

Back to the slide that started this. “Autonomous cluster management” with LLMs is marketing today, not something you can run. What I actually tested is assisted diagnosis: a model that points at problems and offers suggestions. The human in the loop stays in the loop. That’s a requirement, not a setting you can turn off once you trust it enough.

And the technology is genuinely impressive. A 70B model running on hardware in my own office produced analyses I’d accept from a colleague. The gap between “good enough to read” and “good enough to act on a production cluster without me” is still wide, and it widened every time the model confidently invented a resource limit.

I got the answer I came for. A local model is good enough to be a useful diagnostic assistant, and running it locally means I never had to choose between AIOps and keeping my data at home. Treat the output the way you’d treat advice from a sharp colleague who has never seen your cluster before: worth hearing, never the final word.

Related posts:

• Sovereign Infrastructure - Why I self-host everything
• Why Privacy Matters - The context for local LLMs