TL;DR An e-commerce client moved a PHP monolith to EKS in October, two weeks before Black Friday, with resource limits guessed from the old server specs. Mid-morning the pods hit their memory limit, died in a loop for four hours, and the estimate was €240,000 in lost revenue. The second half is how to get those numbers from load tests and VPA recommendations instead of guessing them.
“We want to migrate to Kubernetes by November.”
It was September. The client was an e-commerce company, and their biggest sales event of the year was Black Friday, in late November. I said no. They asked if I knew someone who might take it on anyway.
I did. A fellow platform engineer, someone I respect and rate highly. I made the introduction and warned him about the timeline. He took the engagement, documented the same concerns I had, and got them signed off. The client went ahead anyway.
What happened next is his story, told here with his permission. It’s the clearest case I know of why you don’t migrate to Kubernetes without resource metrics, and it lands on something I keep coming back to on this blog. You can’t make good decisions about a system you don’t understand, and metrics are how you understand it. Here’s what he saw, and then how I’d do the sizing instead.
What happened on Black Friday
The client had a monolithic PHP application on four dedicated servers. Two app servers, one database server, one for Redis and background jobs. Boring, and it had handled Black Friday traffic for three years.
But the CTO had been to a conference. Kubernetes was the future. Microservices were the future. They wanted to containerize everything and deploy to EKS before Black Friday to “handle scale better.”
The CTO was new, brought in to drive innovation. The company had been coasting on the same tech stack for nearly a decade, profitable but stagnant. The board wanted modernization, and the CTO needed a win to show the investment was worth it.
The timeline wasn’t pure hubris, though. Their legacy hosting contract ended on December 1st. The provider had been acquired, the new owners were sunsetting the platform, and they wouldn’t extend. So either migrate before Black Friday, or go month-to-month on emergency hosting at 3x the cost and migrate during December, the one month when code freezes make everything harder and any outage is catastrophic.
The CTO did the math. A risky migration in October with two weeks to stabilize before Black Friday, against a certain cash drain and an even riskier migration in December. The CFO signed off. The CEO signed off. I’d call it a calculated bet rather than recklessness. They just did the calculation without the one input that mattered, because nobody had measured it.
What he warned them about
My colleague wrote four things down before he started.
They had no metrics. Basic Nagios monitoring, so CPU, memory and disk per server, and nothing at the application level. No request latency histograms, no idea how resource use tracked traffic.
They’d never load tested the application. Their only data point was “it survived last year’s Black Friday.”
There was no time for proper sizing. Kubernetes resource requests and limits are numbers, and without metrics those numbers would be guesses.
And there was no rollback plan. The old infrastructure was being decommissioned. If Kubernetes failed, there was nothing to fail back to.
He recommended extending the hosting contract, even at the higher price, to buy time for a proper run-up. They declined. The math had been done, the executives had signed, and the decision was final.
What they deployed
Without metrics, they made educated guesses from the current server specs:
# What they deployed (DON'T DO THIS)
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
Why set CPU limits at all? My colleague knows the arguments against them. The answer was policy. The client’s platform team required every deployment to specify both requests and limits for every resource, and the rule came from the CTO’s previous company, where a cryptominer had run on their servers for weeks before anyone noticed. Capping CPU on everything was one of the first policies they put in after joining, so a rogue process couldn’t eat unbounded resources. Reasonable, but it assumes you know what limits to set. Without historical data to push back with, and with the clock running, my colleague went along with it. That policy plus the guessed values is what made the day worse than it needed to be.
They set up a Horizontal Pod Autoscaler (HPA) on CPU utilization:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web-app
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web-app
minReplicas: 4
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
I’d have looked at that and nodded. Nothing in it is wrong on its own. It’s wrong for this application, and you can’t tell from the YAML.
Black Friday morning
Traffic started climbing at 6 AM. By 8 AM they were at 3x normal traffic. The HPA did its job and pods scaled from 4 to 12. Everything looked fine.
Then 10 AM hit.
Traffic jumped to 8x normal. The HPA tried to scale to 20 pods. What nobody knew was that the application was memory-bound. Each pod hit the 1Gi memory limit and got OOMKilled. The HPA saw low CPU utilization, because pods were dying before they could use any, and concluded that everything was fine.
New pods would start, load the PHP opcache, warm up, start serving traffic, run out of memory, die. Repeat. They had 20 pods cycling through CrashLoopBackOff.
Where the memory went
The application’s memory footprint depended heavily on the type of request:
- Browse category: ~200MB
- View product: ~300MB
- Add to cart: ~400MB
- Checkout: ~800MB+
On Black Friday, checkout traffic was 5x normal. Every pod was trying to handle checkout requests that needed 800MB, inside a 1Gi cap with no headroom.
It got worse. The CPU requests were too low as well, so the pods that did survive were being throttled. Response times went from 200ms to 8 seconds. Users started refreshing pages. More requests, more memory pressure. Death spiral.
The fallout
- 4 hours of degraded service during peak Black Friday hours
- Estimated lost revenue: €240,000
- Emergency scale-up of the node pool, which took 20 minutes to provision
- Manual intervention to raise memory limits, which meant a redeployment
- A post-incident review with very uncomfortable executives
The application was fine. Kubernetes was fine. The configuration was the problem, and the configuration was wrong because nobody had the data to make it right. I keep coming back to this bit. The HPA scaled. The scheduler placed pods. The kubelet killed pods that breached their memory limit. Every component did exactly what it was told. What was missing was someone who understood the workload well enough to tell them the right numbers.
How to size it with data
Here’s how I’d have done the sizing, given the time.
Instrument before you migrate
You can’t size Kubernetes resources without knowing how your application consumes them. Get at least 2-4 weeks of metrics before you plan anything, and at minimum:
For memory:
- Baseline memory usage when idle
- Working set memory under load
- Peak memory during traffic spikes
- Memory consumption by request type, if it varies
For CPU:
- CPU utilization under normal load
- CPU utilization during peaks
- Request latency at various CPU levels
- Whether the app is CPU-bound or IO-bound
For request patterns:
- Requests per second, RPS
- Request latency percentiles, p50, p95, p99
- Traffic patterns by time of day and day of week
- Historical peak traffic events
Load test one container first
Before you touch Kubernetes, load test the application in a single container:
# Run the app in a container with generous limits
docker run -d --name load-test \
--memory 4g \
--cpus 2 \
-p 8080:8080 \
your-app:latest
# Monitor resource usage during load test
docker stats load-test
Use a load testing tool to simulate realistic traffic:
# Example with k6
k6 run --vus 100 --duration 10m load-test.js
Write down:
- The memory high-water mark
- The CPU utilization percentage
- The point where performance degrades
Requests and limits from the numbers
With the load test data in hand, the arithmetic is short.
For memory:
requests.memory = p95 memory usage + 20% buffer
limits.memory = peak memory usage + 30% buffer
For CPU:
requests.cpu = average CPU under expected load
limits.cpu = peak CPU during burst (or omit to allow bursting)
For the e-commerce client, that would have come out as:
# What they SHOULD have deployed
resources:
requests:
memory: "1Gi" # p95 was ~800MB
cpu: "500m" # Average under load
limits:
memory: "2Gi" # Peak during checkout was ~1.5GB
# No CPU limit - allow bursting
A note on that missing CPU limit. I’m a proponent of not setting one. CPU is compressible, so if you run out you get throttled. Memory isn’t, so if you run out you get killed. The Google SRE book and many practitioners recommend against CPU limits in most cases.
Let VPA do the rightsizing
The Vertical Pod Autoscaler (VPA) watches actual resource usage and recommends better resource specifications, or applies them for you.
Install VPA:
# Clone the autoscaler repo
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
# Install VPA components
./hack/vpa-up.sh
Create a VPA resource in “Off” mode, so you get recommendations without them being applied:
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: web-app-vpa
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: web-app
updatePolicy:
updateMode: "Off" # Just recommend, don't apply
After a few days, check the recommendations:
kubectl describe vpa web-app-vpa
Output:
Recommendation:
Container Recommendations:
Container Name: web-app
Lower Bound:
Cpu: 200m
Memory: 800Mi
Target:
Cpu: 450m
Memory: 1200Mi
Upper Bound:
Cpu: 800m
Memory: 2Gi
The “Target” values are what VPA recommends as requests. Take those as your baseline, then add headroom for the limits.
HPA and VPA together, carefully
HPA and VPA can fight each other. HPA scales horizontally based on metrics, and VPA changes resource requests. If VPA raises the requests, HPA may decide the pods are underutilized and scale down.
The approach that doesn’t bite is to run VPA in “Off” mode for the recommendations, apply them by hand as your baseline, let HPA do the horizontal scaling on custom metrics rather than CPU alone, and revisit the VPA recommendations every so often.
For the e-commerce client, a better HPA would have looked like this:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web-app
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web-app
minReplicas: 8 # Higher minimum for Black Friday
maxReplicas: 50 # More headroom
metrics:
- type: Resource
resource:
name: memory # Scale on memory, not just CPU
target:
type: Utilization
averageUtilization: 70
- type: Pods
pods:
metric:
name: http_requests_per_second # Custom metric
target:
type: AverageValue
averageValue: "100"
The questions to answer before you migrate
Before any Kubernetes migration, answer these with data:
| Question | Data Source |
|---|---|
| What’s the application’s memory baseline? | Monitoring (Prometheus, Datadog, etc.) |
| What’s peak memory under load? | Load testing |
| Is the app CPU-bound or memory-bound? | Load testing + profiling |
| What’s p99 latency at expected load? | Load testing |
| What’s the traffic pattern (daily, weekly, seasonal)? | Historical metrics |
| What’s the highest traffic spike in the past year? | Historical metrics |
| How does the app behave at 2x, 5x, 10x load? | Load testing |
If you can’t answer them, you’re not ready to migrate. And sometimes the answer is that you shouldn’t, which is its own post, When Not to Use Kubernetes.
The second attempt
After Black Friday, the client agreed to the expensive emergency hosting they’d been trying to avoid. Three times the cost, but it bought them time. My colleague used December to do the migration properly.
He instrumented everything first. Prometheus, application metrics, custom metrics for the checkout flow. Then they load tested against simulated Black Friday traffic patterns before moving anything. They started with generous resources, overprovisioned on purpose, and used VPA to rightsize from there. They validated with canary deployments, running the new configuration alongside the old and comparing behaviour. And they kept a way back. The emergency hosting stayed available as a fallback for 30 days after the migration.
The migration itself, late December after the Christmas rush, was boring. Nothing broke.
Why I said no
Kubernetes doesn’t fix a scaling problem you haven’t measured. It runs the numbers you give it, guessed or measured, just as faithfully at Black Friday traffic.
Every number in a resource spec is a claim about how the application behaves. Requests, limits, the HPA target, all of them. Guessed from server specs, they’re claims nobody can back. Measured, they’re the reason the second attempt was boring.
I said no in September because I couldn’t see how anyone would get those numbers in time. My colleague took the job and got them the expensive way. It’s his story, and I’m grateful he let me tell it.