OpenShift in Production: What Nobody Tells You About Day-2 Operations

The day the cluster is delivered, everything works. Namespaces are clean, nodes are healthy, dashboards show green across the board. The problem starts three months later.

Someone needs to do an upgrade and nobody knows how. Logs pile up and fill the disk. A team deployed a pod that consumes all the memory on a node. And when something fails at 2 AM, the on-call engineer spends 40 minutes figuring out which namespace has the problem.

Why day-2 is harder than day-1

Installing OpenShift is a solved problem. Red Hat has exhaustive documentation, the installers are mature and there are hundreds of step-by-step guides. But operating OpenShift in production is an entirely different problem.

Upgrades: the silent terror

A production OpenShift cluster needs regular updates. Security patches, new versions, bug fixes. The problem is that every upgrade can break something:

• Operators that didn’t update: A custom operator that worked on 4.14 may fail on 4.15.
• Deprecated APIs: Kubernetes deprecates APIs regularly. If your manifests use an API that disappeared, your deployments break.
• Storage incompatibilities: CSI driver changes that affect existing volumes.
• Network policies that change behavior: OVN/Calico updates that modify how policies are evaluated.

The solution isn’t “don’t upgrade.” It’s having a tested upgrade process:

1. Staging environment that mirrors production — not a test cluster with 3 nodes and no traffic.
2. Automated smoke test suite that validates every critical workload after the upgrade.
3. Documented and tested rollback plan — not “we hope we won’t need it.”
4. Regular upgrade windows — the longer you wait, the more painful the jump.

Capacity management: between waste and fire

Without resource governance, OpenShift clusters degrade rapidly:

• Teams requesting 4 cores and 8GB per pod but using 0.2 cores and 500MB.
• ResourceQuotas nobody configured, allowing one namespace to consume the entire node.
• Pods without limits causing cascading OOM kills.
• PersistentVolumes that never get released after the application is deleted.

What works:

• ResourceQuotas and LimitRanges on every namespace — not optional, mandatory.
• Monitoring actual usage vs. requested with Kubecost or Prometheus.
• Monthly capacity reviews with data, not intuition.
• Correctly configured auto-scaling: HPA for pods, Cluster Autoscaler for nodes.

Observability: more than Prometheus and Grafana

Having metrics isn’t having observability. Real observability in OpenShift requires three correlated signals:

Metrics: CPU, memory, I/O, network latency, application throughput. Prometheus + Grafana is the standard, but alerts must be actionable — not 500 alerts everyone ignores.

Logs: Every pod generates logs that need to be collected, indexed and searchable. OpenShift’s default stack (Elasticsearch + Fluentd + Kibana) works, but at scale it needs aggressive tuning.

Traces: For distributed applications, traces are essential to understand the flow of a request across multiple services. Jaeger or OpenTelemetry integrated into the platform.

The key is correlation: when a latency alert fires, the engineer needs to see the pod metrics, relevant logs and the request trace — all in one place, not in three different tools.

Continuous security: it’s not a checkbox

Security in OpenShift doesn’t end with the initial configuration:

• Image scanning in the CI/CD pipeline before they reach the cluster.
• Security Context Constraints (SCCs) that prevent pods with unnecessary privileges.
• Network policies that implement zero-trust between namespaces.
• Granular RBAC — not everyone needs to be cluster-admin.
• Audit logging that records who did what and when.

GitOps: the answer to “who changed what”

The pattern with the most impact on day-2 operations is GitOps. With ArgoCD or Flux, all cluster configuration is in Git:

• Auditable changes: Every modification has a commit with author and reason.
• Trivial rollback: If something breaks, git revert and ArgoCD reconciles.
• Drift detection: If someone changes something manually in the cluster, it’s detected and reverted.
• Consistent environments: Dev, staging and production are managed with the same process.

We’ve seen teams go from “I’m afraid to touch production” to “we deploy five times a day” simply by implementing GitOps correctly.

Metrics of a well-operated cluster

How do you know if your day-2 operations are working? These are the metrics we monitor:

Metric	Good	Problematic
Upgrade frequency	Every 2-3 months	More than 6 months without updating
Incident MTTR	< 15 minutes	> 1 hour
% resources used vs. allocated	> 60%	< 30%
Actionable alerts vs. noise	> 80% actionable	More noise than signal
Deploys per day	5+	Weekly windows
New team onboarding time	< 1 day	> 1 week

What we’ve learned operating production clusters

After operating OpenShift platforms for telecom operators and enterprise companies, these are the lessons that stuck with us:

1. The most secure cluster is one that’s regularly updated. Every month without an upgrade is accumulated technical debt.
2. Without ResourceQuotas, the cluster is a tragedy of the commons. Every team consumes as much as possible because the resource is “free.”
3. Runbooks are useless if they’re in a wiki. Codify them in Git, test them periodically, execute them automatically.
4. Observability is an investment, not an expense. The cost of one hour of troubleshooting without visibility exceeds months of tool licenses.
5. GitOps is not optional for enterprise clusters. The alternative is manual kubectl apply at 2 AM, and that doesn’t scale.

---

Does your OpenShift cluster need better day-2 operations? Request an assessment of your platform and we’ll give you a concrete improvement plan.