Kubernetes Server-Side Apply: How to Stop GitOps and HPA From Fighting Over Replicas

Welcome to DevOps Inside, where we break things so you can scale them safely (and smile while doing it) 😄.

If you’ve run Kubernetes in production for any length of time, you’ve likely witnessed a tragic silent war: your Horizontal Pod Autoscaler (HPA) frantically scaling your workload up to handle a traffic surge, while your GitOps controller, like Argo CD or Flux, repeatedly tries to bring it back to the replica count defined in Git.

This can end in high latency, dropped requests, and a 3 AM incident call. But the root cause is often not simply Client-Side Apply (CSA). The bigger problem is multiple controllers or automation systems trying to manage the same field.

Enter Server-Side Apply (SSA): the Kubernetes feature that enables clients and controllers to track field ownership and detect conflicts when multiple managers try to change the same fields.

The important part, though, is this: SSA does not magically stop GitOps and HPA from fighting over replicas. You still need to decide who should manage spec.replicas and configure your GitOps workflow accordingly.

The Legacy Problem: Client-Side Apply Chaos

Historically, running kubectl apply -f deployment.yaml relied on client-side logic. Your local kubectl binary, or your CI/CD runner, performed the merge calculation before sending the resulting request to the API server.

To track what was applied previously, Kubernetes stored a JSON representation of the applied configuration inside an annotation on your object called kubectl.kubernetes.io/last-applied-configuration.

Here is how a classic production problem can happen with Client-Side Apply:

1. The Initial Deploy: Your GitOps pipeline applies a Deployment manifest configured with replicas: 4.
2. The Traffic Spike: HPA detects high CPU usage and scales the live Deployment in the cluster from 4 to 8 replicas.
3. The GitOps Sync Loop: Argo CD or Flux sees that the live state differs from the desired state stored in Git. If replicas: 4 is still present in Git, the GitOps controller can try to reconcile the Deployment back toward 4.
4. The Overwrite: GitOps changes the replica count back toward 4. HPA sees the workload is still under pressure and can scale it back toward 8. The cycle repeats.

Kubernetes itself describes HPA as managing the replica count of its target workload through the workload's scale subresource.

The important point is that Server-Side Apply alone does not solve this reconciliation problem. If GitOps continues trying to enforce a fixed value for spec.replicas, the underlying ownership and desired-state conflict still exists.

The real solution is to establish clear ownership of the field.

Enter Server-Side Apply (SSA): True Declarative Ownership

Server-Side Apply moves the apply and field-management logic into the kube-apiserver.

Instead of relying on the old last-applied-configuration annotation, Kubernetes maintains a fine-grained record of field management inside an object's metadata called managedFields. Server-Side Apply has been generally available since Kubernetes 1.22.

With SSA, Kubernetes can track which field manager has ownership of specific fields and can detect conflicts when an Apply operation attempts to change a field owned by another manager.

For example, a live object can contain information similar to:

metadata:
  managedFields:
  - manager: argocd-controller
    operation: Apply
    fieldsV1:
      f:spec:
        f:template: {} # GitOps manages the workload template

  - manager: another-controller
    operation: Update
    fieldsV1:
      f:spec:
        f:replicas: {} # Example of another manager changing replicas

The exact managers and field ownership you see in a real cluster can differ depending on the controllers and operations involved. The important part is that Kubernetes keeps track of field management in managedFields.

One important detail is that you should not assume managedFields will always show the HPA in exactly the way you expect. HPA operates through the workload's scale subresource, so the ownership information you see depends on the operation and resource involved.

How SSA Tracks Who Owns What

With Server-Side Apply, Kubernetes introduces clear boundaries using field managers:

1. Field Managers: Different clients and controllers can identify themselves when modifying resources through a field manager.
2. Granular Ownership: A GitOps manifest can manage the Deployment template, image, environment variables, and other configuration without continuously asserting control over a field that another controller is responsible for.
3. Delegation: Another controller can manage fields that belong to its own workflow.
4. Conflict Detection: If an SSA operation attempts to change a field owned by another manager, Kubernetes can return a conflict instead of silently overwriting that manager's value. A forced operation can deliberately take ownership of the conflicting field.

This is the important difference: SSA gives Kubernetes a mechanism for identifying field ownership and detecting conflicts. You still need to design your manifests and controllers so that ownership makes sense.

So What Actually Stops the HPA and GitOps Fight?

This is where the distinction between field ownership and GitOps reconciliation becomes important.

If HPA is responsible for deciding how many replicas your application needs, GitOps should not continuously try to enforce a fixed replica count.

For example, Argo CD can be configured to ignore changes to /spec/replicas. Argo CD also provides the RespectIgnoreDifferences=true sync option so that the ignored field is respected during synchronization rather than being reapplied from the desired state.

A simplified Argo CD configuration can look like this:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

This tells Argo CD to ignore differences in the Deployment's replica count when comparing and, with the sync option enabled, during synchronization as well.

So the responsibilities become much clearer:

GitOps: "I manage how the application is configured."

HPA: "I manage how many replicas are needed."

SSA: "I help Kubernetes track field management and detect conflicting Apply operations."

That separation is the important part. SSA and GitOps diff configuration solve related but different problems.

Client-Side Apply vs. Server-Side Apply

Server-Side Apply is not simply a replacement for GitOps diff configuration. Its main value here is server-side field management and conflict detection, especially when multiple actors are modifying different parts of the same Kubernetes resource.

Practical Checklist: Implementing SSA in Production

To make full use of Server-Side Apply and avoid replica ping-ponging:

• Decide Who Owns Replicas: If HPA is responsible for scaling a Deployment, don't continuously declare a fixed spec.replicas value in the GitOps workflow. Kubernetes itself recommends not manually setting replicas when an HPA manages the workload.
• Configure GitOps Diffing: With Argo CD, use ignoreDifferences for /spec/replicas when HPA should own scaling, and consider RespectIgnoreDifferences=true when you need that behavior during sync as well.
• Enable SSA Where It Makes Sense: Turn on Server-Side Apply in your deployment tools when you want Kubernetes to manage field ownership and detect Apply conflicts. For example, kubectl supports kubectl apply --server-side.
• Inspect Field Managers: Run kubectl get deployment <name> -o yaml whenever you suspect a field conflict and inspect metadata.managedFields to understand which managers are modifying the resource.
• Be Careful With Force Conflicts: --force-conflicts can deliberately take ownership of a field. It should not be treated as a generic fix for every SSA conflict because it can transfer ownership away from another manager.
• Check GitOps Diff Behavior: Even when SSA is enabled, your GitOps controller can still detect differences between Git and the live object. SSA is a field-management mechanism, not a replacement for GitOps reconciliation or diff configuration.

A Better GitOps + HPA Model

A cleaner setup looks something like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  # replicas intentionally omitted
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-app:v2

And the HPA manages the scaling decision:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 4
  maxReplicas: 10

Now the responsibilities are much clearer.

GitOps: "I manage how the application runs."

HPA: "I manage how many replicas are needed."

Kubernetes then has a much cleaner model to work with.

If you're using Argo CD, the ignoreDifferences configuration is another important part of this setup because simply removing replicas from the manifest does not mean GitOps will automatically stop caring about live replica changes in every workflow. The reconciliation behavior needs to be configured deliberately.

One Important Thing to Remember

Don't think of SSA as a magic switch that makes Kubernetes controllers automatically stop fighting.

The real principle is single responsibility per field.

If your GitOps system continuously wants spec.replicas to be 4 while HPA is trying to make it 8, there is still a reconciliation problem. Switching from CSA to SSA does not change that architectural decision.

SSA gives you the tools to make field ownership explicit and lets Kubernetes detect conflicts instead of silently hiding them.

GitOps configuration then determines how your deployment system responds to changes made by other controllers.

That's where the real value comes from.

Wrapping Up: Keep Calm and Delegate Fields

Server-Side Apply turns Kubernetes field management from a chaotic free-for-all into a more structured system. It gives Kubernetes a way to track which managers care about which fields and detect conflicts when those managers disagree.

But when it comes to GitOps and HPA, the solution is not simply "turn on SSA."

Decide who owns the replica count, configure your GitOps controller accordingly, and use SSA where you need proper server-side field management and conflict detection.

That way, GitOps can manage your application configuration while HPA handles scaling without both systems constantly stepping on each other's toes.

Got a favorite Kubernetes edge case or a GitOps sync horror story? Swing by DevOps Inside. We love breaking down complex cloud-native architectures so you can build them better!