Kubernetes Server-Side Apply: Stop GitOps and HPA From Fighting Over Your Replicas

Welcome to DevOps Inside, where we break things so you can scale them safely (and smile while doing it) 😄.

If you’ve run Kubernetes in production for any length of time, you’ve likely witnessed a tragic silent war: your Horizontal Pod Autoscaler (HPA) frantically scaling your workload up to handle a traffic surge, while your GitOps controller, like ArgoCD or Flux, repeatedly scales it right back down.

This endless tug-of-war usually ends in high latency, dropped requests, and a 3 AM incident call. The root cause is often not simply Client-Side Apply (CSA), though. The bigger problem is multiple controllers trying to manage the same field.

Enter Server-Side Apply (SSA): the Kubernetes feature that gives controllers a much better way to manage individual fields without blindly overwriting each other's changes.

The Legacy Problem: Client-Side Apply Chaos

Historically, running kubectl apply -f deployment.yaml relied on client-side logic. Your local kubectl binary, or your CI/CD runner, performed the merge calculation before sending the resulting request to the API server.

To track what was applied previously, Kubernetes stored a JSON representation of the applied configuration inside an annotation on your object called kubectl.kubernetes.io/last-applied-configuration.

Here is how a classic production problem can happen with Client-Side Apply:

1. The Initial Deploy: Your GitOps pipeline applies a Deployment manifest configured with replicas: 4.
2. The Traffic Spike: HPA detects high CPU usage and scales the live Deployment in the cluster from 4 to 8 replicas.
3. The GitOps Sync Loop: ArgoCD or Flux sees that the live state differs from the desired state stored in Git. If replicas: 4 is still present in Git, the GitOps controller can try to reconcile the Deployment back toward 4.
4. The Overwrite: GitOps changes the replica count back to 4. HPA sees the workload is still under pressure and scales it back toward 8. The cycle repeats.

The important point is that Server-Side Apply alone does not magically stop this conflict. If GitOps continues declaring ownership of spec.replicas, the controllers can still disagree about the desired state.

The real solution is to establish clear ownership of the field.

Enter Server-Side Apply (SSA): True Declarative Ownership

Server-Side Apply moves the apply and field-management logic into the kube-apiserver.

Instead of relying on the old last-applied-configuration annotation, Kubernetes maintains a fine-grained record of field ownership inside every object's metadata called managedFields.

With SSA, Kubernetes can track which field manager has ownership of specific fields.

For example, a live object can contain information similar to:

metadata:
  managedFields:
  - manager: argocd-controller
    operation: Apply
    fieldsV1:
      f:spec:
        f:template: {} # GitOps manages the workload template

  - manager: kube-controller-manager
    operation: Update
    fieldsV1:
      f:spec:
        f:replicas: {} # Controller manages replicas

The exact managers and field ownership you see in a real cluster can differ depending on the controllers and operations involved. The important part is that Kubernetes keeps track of field management in managedFields.

How SSA Keeps Everyone in Their Own Lane

With Server-Side Apply, Kubernetes introduces clear boundaries using field managers:

1. Field Managers: Different clients and controllers can identify themselves when modifying resources through a field manager.
2. Granular Ownership: Your GitOps manifest can manage the Deployment template, image, environment variables, and other configuration without continuously asserting control over spec.replicas.
3. Delegation: The HPA can then control the replica count based on observed workload metrics.
4. Conflict Detection: If an SSA operation attempts to change a field owned by another manager, Kubernetes can return a conflict instead of silently overwriting that manager's value. A forced operation can deliberately take ownership of the conflicting field.

This is the important difference: SSA gives Kubernetes a mechanism for identifying field ownership. You still need to design your manifests and controllers so that ownership is sensible.

Client-Side Apply vs. Server-Side Apply

Kubernetes documents Server-Side Apply as the mechanism for server-side field management and conflict handling, and it has been generally available since Kubernetes 1.22.

Practical Checklist: Implementing SSA in Production

To make full use of Server-Side Apply and avoid replica ping-ponging:

• Omit Replicas in Git: Once HPA is responsible for scaling a Deployment, don't continuously declare a fixed spec.replicas value in the GitOps manifest. Otherwise, GitOps still has a desired replica value to reconcile.
• Enable SSA in GitOps: Turn on Server-Side Apply in your deployment tools. For example, Argo CD supports the ServerSideApply=true sync option, while kubectl supports kubectl apply --server-side.
• Inspect Field Managers: Run kubectl get deployment <name> -o yaml whenever you suspect a field conflict and inspect metadata.managedFields to understand which managers are modifying the resource.
• Be Careful With Force Conflicts: --force-conflicts can deliberately take ownership of a field. It should not be treated as a generic fix for every SSA conflict because it can transfer ownership away from another manager.
• Check GitOps Diff Behavior: Even when SSA is enabled, your GitOps controller can still detect differences between Git and the live object. SSA is a field-management mechanism, not a replacement for GitOps reconciliation or diff configuration.

A Better GitOps + HPA Model

A cleaner setup looks something like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  # replicas intentionally omitted
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-app:v2

And the HPA manages the scaling decision:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 4
  maxReplicas: 10

Now the responsibilities are much clearer.

GitOps: "I manage how the application runs."

HPA: "I manage how many replicas are needed."

Kubernetes then has a much cleaner model to work with.

One Important Thing to Remember

Don't think of SSA as a magic switch that makes Kubernetes controllers automatically stop fighting.

The real principle is single responsibility per field.

If ArgoCD owns spec.replicas and HPA also needs to change spec.replicas, you still have a conflict waiting to happen. Switching from CSA to SSA doesn't change that architectural problem.

SSA gives you the tools to make ownership explicit and lets Kubernetes detect conflicts instead of silently hiding them.

That's where the real value comes from.

Wrapping Up: Keep Calm and Delegate Fields

Server-Side Apply turns Kubernetes field management from a chaotic free-for-all into a structured, audit-friendly system. By letting GitOps manage your templates and HPA manage your scale, you can eliminate unnecessary replica conflicts, reduce sync noise, and keep your production environments rock solid.

Got a favorite Kubernetes edge case or a GitOps sync horror story? Swing by DevOps Inside. We love breaking down complex cloud-native architectures so you can build them better!