Git & GitOps Study
Guide

Git tracks files across three distinct areas:

Working Directory — the files you actually see and edit on disk. Changes here are untracked by Git until you stage them. Run git status to see what's modified.

Staging Area (Index) — a snapshot of what will go into your next commit. You explicitly add files with git add. This lets you craft commits precisely — staging only the changes relevant to a single logical unit of work.

Repository (.git directory) — the permanent history of committed snapshots. Once committed, changes are in the local repo. git push syncs commits to the remote.

The flow is: edit → git add (working dir → staging) → git commit (staging → repo) → git push (local repo → remote repo).

git merge creates a new merge commit that joins two branch histories together. The history shows exactly when branches diverged and reconverged. This is non-destructive — existing commits are never changed.

git rebase replays your branch's commits on top of another branch, rewriting commit SHAs in the process. The result is a clean, linear history — as if you branched off the latest code and wrote your commits fresh.

When to use merge:

• Merging a completed feature back into main — preserves context of when work happened
• Any public/shared branch — never rewrite history others may have pulled
• When you want an explicit merge commit as an audit trail

When to use rebase:

• Updating a local feature branch with the latest main before opening a PR
• Cleaning up messy WIP commits before review (git rebase -i)
• Teams that prefer linear history for easier git log and git bisect

git cherry-pick <commit-sha> applies the changes from a specific commit onto your current branch without merging the entire source branch.

DevOps use cases:

• Hotfix backporting — a fix lands on main and you need it in the release/v2.3 branch too without merging all of main
• Selective feature delivery — picking a single ready commit from a larger branch when the rest isn't ready to ship
• Emergency patches — applying a security fix across multiple active release branches

git cherry-pick abc1234           # apply one commit
git cherry-pick abc1234 def5678   # apply multiple commits
git cherry-pick main~3..main      # apply a range

Cherry-pick creates a new commit with a different SHA, so the same logical change appears twice in history. Use it sparingly — over-reliance is a sign of branching strategy problems.

git bisect performs a binary search through commit history to find the exact commit that introduced a bug. You mark a known-good commit and a known-bad commit, and Git checks out the midpoint. You test and mark it good or bad, narrowing the range in half each round.

git bisect start
git bisect bad                     # current commit is broken
git bisect good v2.1.0             # last known good release tag
# Git checks out midpoint — you test, then:
git bisect good   # or: git bisect bad
# Repeat until Git identifies the culprit commit
git bisect reset  # return to original HEAD

Automation: provide a test script and Git runs it automatically:

git bisect run ./test.sh

Useful when a regression appeared "somewhere in the last 200 commits" — bisect finds it in ~8 steps (log2(200) ≈ 7.6) instead of checking commits one by one.

git reset — moves the current branch pointer backward, optionally changing the index and working directory. Three modes:

• --soft: moves HEAD, keeps staging area and working dir unchanged (commits disappear, changes stay staged)
• --mixed (default): moves HEAD, clears staging area, keeps working dir (changes appear unstaged)
• --hard: moves HEAD, clears staging area, discards working dir changes completely (destructive — changes are gone)

git revert <sha> — creates a new commit that is the inverse of a previous commit. The history is preserved; nothing is erased. Safe for shared/public branches.

git restore — discards changes in the working directory or unstages files. It does not touch commit history:

• git restore file.txt — discard working dir changes for a file
• git restore --staged file.txt — unstage a file (was: git reset HEAD file.txt)

Rule of thumb: revert for shared branches (safe), reset for local cleanup (destructive), restore for working dir/staging area only.

Step 1: Rotate the credential immediately. Assume the secret is compromised the moment it was pushed. Revoke it, issue a new one, update all consumers. This is non-negotiable regardless of how fast you clean up the repo.

Step 2: Remove from Git history. Two main approaches:

• git filter-repo (recommended over the deprecated BFG): git filter-repo --path secrets.txt --invert-paths rewrites all commits that touched the file
• BFG Repo Cleaner: faster for large repos, but requires a separate Java tool

git filter-repo --string 'AKIAIOSFODNN7EXAMPLE' --replace-with 'REDACTED'

Step 3: Force-push all branches. Coordinate with all contributors — their local clones still have the old history. Everyone must re-clone or hard-reset.

Step 4: Contact GitHub/GitLab support to clear cached views (pull request diffs, cached pages).

Prevention: Use git-secrets, trufflehog, or GitHub's secret scanning. Pre-commit hooks should block commits containing high-entropy strings or known key patterns.

Git Flow — structured model with long-lived branches: main, develop, feature, release, and hotfix branches. Suits teams with scheduled releases and strict versioning.

• ✓ Clear separation of in-progress vs. production-ready code
• ✗ Complex, merge-heavy; long-lived branches accumulate drift
• Use when: regulated industries, software with explicit version releases (SDKs, firmware)

GitHub Flow — simplified: main is always deployable. Feature branches are short-lived, opened as PRs, merged with squash or merge commit.

• ✓ Simple and lightweight; works well with CD
• ✗ Requires strong feature flags to hide incomplete work in main
• Use when: web apps, SaaS products, small-to-medium teams

Trunk-Based Development (TBD) — everyone commits directly to main (or merges very short-lived branches within a day or two). CI must be extremely fast and reliable.

• ✓ Eliminates merge hell; maximizes CI effectiveness
• ✗ Demands feature flags, comprehensive automated tests, and team discipline
• Use when: high-velocity teams, microservices, mature CI/CD platforms (Google, Meta)

Feature flags (also called feature toggles) are conditional blocks in code that activate or deactivate a feature at runtime without a new deployment.

if (featureFlags.isEnabled('new-checkout-flow', userId)) {
  return newCheckout();
} else {
  return legacyCheckout();
}

How they enable TBD: developers merge incomplete features into main behind a disabled flag. The code ships to production but is invisible to users. When the feature is complete and tested, the flag is enabled — no branch, no deployment needed.

Types of feature flags:

• Release toggles — hide work-in-progress (temporary, remove after launch)
• Experiment toggles — A/B testing, percentage rollouts
• Ops toggles — circuit breakers, kill switches for risky features
• Permission toggles — beta user access, internal-only features

Tools: LaunchDarkly, Unleash, AWS AppConfig, Flagsmith, or simple env vars for basic use cases.

This is the classic branching challenge in teams shipping both SaaS and versioned releases.

The pattern:

• Cut release branches from main at a stable tag: release/v2.4
• Only bug fixes and security patches land on release branches — no new features
• All fixes land on main first, then are cherry-picked into the relevant release branches
• Release branches are deployed to specific environments (staging-v2.4, prod) via CD

Key disciplines:

• Release branches should be short-lived — 1-2 sprint cycles before EOL
• Automate backporting: CI can open a cherry-pick PR automatically when a fix merges to main with a backport: v2.4 label
• Never merge release branches back into main — only cherry-pick specific commits

A monorepo stores multiple services, libraries, or apps in a single Git repository. A polyrepo gives each service its own dedicated repo.

Monorepo benefits:

• Atomic cross-service changes in a single commit/PR
• Unified CI/CD tooling and standards
• Shared code and libraries with no version management overhead
• Easier refactoring across service boundaries

Monorepo challenges:

• CI must be smart about only building/testing affected services (requires tools like Nx, Turborepo, Bazel, or Pants)
• Git performance degrades at massive scale (Google built a custom VFS)
• Access control is harder — everyone can see all code by default
• Large blast radius for a bad merge

Polyrepo benefits: strong isolation, independent CI, fine-grained access. Challenges: dependency versioning hell, duplicated config, harder cross-service refactors.

Trend: many mid-to-large teams are moving toward monorepos with affected-only CI (Nx, Turborepo) to get the best of both.

Branch protection rules prevent direct pushes to critical branches and enforce quality gates before merging.

GitHub branch protection settings for main:

• Require pull request before merging — 1-2 required approvals
• Dismiss stale reviews when new commits are pushed
• Require status checks (CI, linting, tests) to pass before merge
• Require branches to be up to date before merging
• Require signed commits (GPG/SSH) for regulated environments
• Restrict who can push to matching branches

At scale — CODEOWNERS: .github/CODEOWNERS maps file paths to teams, automatically requesting the right reviewer when a PR touches those files:

# .github/CODEOWNERS
/infra/          @org/platform-team
/services/auth/  @org/security-team
*.tf             @org/terraform-team

Enforcing at the organization level: GitHub Organization rulesets (new) or Terraform/Pulumi managing repository settings as code.

Conventional Commits is the de-facto standard for machine-readable commit messages:

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

Types: feat, fix, docs, style, refactor, test, chore, ci, perf

Why it matters for automation:

• Semantic versioning: tools like semantic-release parse commits to auto-bump versions — fix: → patch, feat: → minor, BREAKING CHANGE: → major
• Changelog generation: auto-generate CHANGELOG.md from commit history
• CI filtering: skip deployments for docs: or chore: commits
• Linking to issues: footers like Closes #123 auto-close GitHub issues on merge

Enforce with commitlint in a pre-commit hook or CI check:

npx commitlint --from=HEAD~1 --to=HEAD --verbose

Git hooks are scripts that Git executes automatically before or after events like commit, push, and merge. They live in .git/hooks/ as executable scripts.

Client-side hooks (run on developer machine):

• pre-commit — run linters, formatters, secret scanners before a commit completes
• commit-msg — validate commit message format (Conventional Commits enforcement)
• pre-push — run tests before pushing to prevent breaking shared branches

Server-side hooks (run on the remote):

• pre-receive — reject pushes that don't meet policy (force-push to main, large files)
• post-receive — trigger deployments after a push (legacy CD approach)

Team-wide distribution: .git/hooks isn't committed. Use Husky (Node.js projects) or git config core.hooksPath ./hooks to store hooks in the repo and configure on clone.

# package.json with Husky
"husky": {
  "hooks": {
    "pre-commit": "lint-staged",
    "commit-msg": "commitlint -E HUSKY_GIT_PARAMS"
  }
}

Both allow embedding one Git repository inside another, but they work very differently.

Git Submodules:

• The parent repo stores a pointer (a specific commit SHA) to an external repo
• The submodule directory is a separate git repo — not tracked in the parent's history
• Cloners must run git submodule update --init --recursive to fetch submodule content
• Updating a submodule = update the pointer commit, commit to parent repo
• Con: easy to forget to push submodule changes; teammates get detached HEAD confusion

Git Subtrees:

• The subtree's files are copied into the parent repo and committed as regular files
• No external dependency — clones just work; no extra setup step
• Harder to contribute changes back upstream (requires git subtree push)
• History can become messy with merge commits

DevOps verdict: most teams prefer subtrees or just vendor dependencies. Submodules are operationally fragile — CI pipelines forget --recurse-submodules, causing mysterious build failures.

Large repos slow down every clone, fetch, and CI run. Two approaches:

1. git filter-repo (rewrite history):

pip install git-filter-repo
git filter-repo --path-glob '*.zip' --invert-paths
git filter-repo --strip-blobs-bigger-than 10M

Rewrites entire history, removing blobs. Requires force-pushing and all contributors re-cloning. Permanent solution but disruptive.

2. Git LFS (Git Large File Storage) — preferred for ongoing work:

• Track large files in LFS: git lfs track "*.psd" — writes to .gitattributes
• LFS stores the file content in a separate object store (GitHub, Artifactory), commits only a tiny pointer
• On clone, LFS files download on-demand or via git lfs pull
• CI runners need the LFS client installed: git lfs install

Quick wins without history rewriting:

• Shallow clones in CI: git clone --depth=1 — no full history needed for builds
• Partial clone: git clone --filter=blob:none — skips blob downloads until needed

git reflog records every movement of HEAD — commits, checkouts, resets, rebases — for 90 days (default). It's your local safety net when history-rewriting operations go wrong.

git reflog
# Output:
# abc1234 HEAD@{0}: reset: moving to HEAD~3
# def5678 HEAD@{1}: commit: add authentication middleware
# ...

Recovery scenario — you ran git reset --hard HEAD~3 by mistake:

git reflog                        # find the commit SHA before the reset
git reset --hard def5678          # jump back to that state
# or create a new branch at that point:
git checkout -b recovery def5678

Recovery from bad rebase:

git reflog                        # find the SHA of HEAD before rebase started
git reset --hard HEAD@{5}         # restore pre-rebase state

Reflog is local-only — it is not pushed to remote and is not available after a fresh clone. It doesn't protect against git gc pruning orphaned commits beyond the retention window.

git stash takes your uncommitted changes (both staged and unstaged) and saves them on a stack, returning the working directory to a clean state matching HEAD.

git stash                         # save changes with auto-generated message
git stash push -m "auth work"     # save with a descriptive name
git stash list                    # view all stashes
git stash pop                     # apply and remove top stash
git stash apply stash@{2}         # apply specific stash (keep in list)
git stash drop stash@{2}          # remove specific stash
git stash branch feature/auth     # create branch from stash (cleaner recovery)

Stash vs. WIP commit:

• Use stash for quick context switches — you'll come back to this branch in minutes or hours. "I need to review a colleague's PR right now."
• Use a WIP commit (git commit -m "wip: do not merge") when you need to push to a remote, or the stash may sit for days and be forgotten. WIP commits are visible to teammates and survive laptop reboots/crashes.
• Stashes are local-only and easy to forget — git stash list regularly to avoid a pile-up

Git operations are often a significant chunk of CI runtime, especially for large repos.

Shallow clones — most impactful for CI:

git clone --depth=1 <repo-url>    # only latest commit, no full history
git fetch --depth=1 origin main   # shallow fetch on update

Partial clone (Git 2.22+) — skips large blobs until accessed:

git clone --filter=blob:none <url>          # no blobs on clone
git clone --filter=tree:0 <url>             # no trees (for monorepos)

Sparse checkout — only checkout the directories CI needs:

git sparse-checkout init
git sparse-checkout set services/api infra/

Clone caching between runs:

• GitHub Actions: actions/checkout@v4 with fetch-depth: 1 handles shallow clones automatically
• Cache the .git directory between CI runs to convert clones to fetches
• Use a local git mirror/proxy close to your CI runners for large teams

Avoid fetching all tags: git fetch --no-tags prevents downloading thousands of tags.

GitOps is an operational model where the desired state of your infrastructure and applications is declared entirely in Git, and an automated agent continuously reconciles the live system to match that declared state.

The four core principles (OpenGitOps):

• Declarative — the system is described as desired state (Kubernetes manifests, Helm charts, Terraform), not imperative steps
• Versioned and immutable — the desired state lives in Git, providing an immutable audit trail and rollback capability
• Pulled automatically — changes are pulled by an agent running inside the cluster, not pushed by external CI
• Continuously reconciled — an agent compares actual vs. desired state and corrects drift automatically

Traditional CI/CD (push-based): CI pipeline runs, builds an artifact, then pushes deployment to the cluster via kubectl apply or Helm. The cluster doesn't know about Git.

GitOps (pull-based): CI builds and pushes an image. An agent (ArgoCD, Flux) watches a Git repo for config changes and pulls them into the cluster. The cluster itself pulls its desired state.

Drift is any divergence between what is declared in Git (the desired state) and what is actually running in the cluster (the observed state).

How drift happens:

• Someone runs kubectl edit deployment or kubectl scale manually
• An operator or admission controller modifies a resource
• A pod restarts with stale config during a node failure
• A Terraform resource is modified via the cloud console instead of code

Reconciliation: the GitOps agent (ArgoCD, Flux) continuously compares the desired state in Git with the observed state in the cluster. When it detects drift, it either:

• Auto-heals — automatically applies the desired state (self-healing)
• Alerts — marks the application as Out of Sync and notifies operators (manual approval mode)

This eliminates "works on my machine" problems and snowflake servers — the cluster always converges to the Git-defined state.

The app-of-apps pattern (ArgoCD terminology) is a way to manage many ArgoCD Application resources declaratively using a single parent Application that points to a directory of Application manifests.

Without app-of-apps: you manually create each ArgoCD Application via the UI or CLI. 50 microservices = 50 manual registrations. Not GitOps-compliant because ArgoCD itself isn't managed via Git.

With app-of-apps:

# root-app/application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
spec:
  source:
    repoURL: https://github.com/org/gitops-repo
    path: apps/           # directory of Application manifests
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated: {}

ArgoCD deploys the root app, which creates all child Application resources from the apps/ directory. Adding a new microservice = add an Application YAML to apps/ and push.

Flux equivalent: Kustomization resources with dependencies fulfill the same role.

The industry-standard pattern is to separate application code from deployment configuration into two repos.

App repo (source of truth for code):

• Application source code
• Dockerfile
• CI pipeline (builds, tests, pushes image to registry)
• CI pipeline updates the image tag in the config repo after a successful build

Config/GitOps repo (source of truth for deployment state):

• Kubernetes manifests, Helm values, Kustomize overlays
• Environment directories: envs/dev/, envs/staging/, envs/prod/
• ArgoCD/Flux watches this repo and syncs the cluster
• Promotion = PR to update image tag in envs/prod/

Why separate?

• CI run frequency ≠ deploy frequency — decoupled lifecycles
• Clean audit trail for deployment changes separate from code changes
• Access control: devs may have write access to app repo but not config repo
• Config repo changes don't re-trigger CI builds

Promotion means taking a specific version of an application from one environment to the next (dev → staging → prod) in a controlled, auditable way.

Config repo structure:

gitops-config/
  base/               # shared Kubernetes manifests
  envs/
    dev/
      kustomization.yaml   # image: app:sha-abc123
    staging/
      kustomization.yaml   # image: app:sha-def456
    prod/
      kustomization.yaml   # image: app:sha-789xyz

Promotion workflow:

1. CI builds image, tags it with commit SHA, pushes to registry
2. CI opens a PR to update envs/dev/ image tag — auto-merges if tests pass
3. After dev verification, open PR to update envs/staging/ tag (manual or automated)
4. After staging sign-off, open PR to update envs/prod/ — requires approval
5. ArgoCD/Flux detects the merge and syncs the cluster

Tools that automate promotion: Flux Image Automation Controller, ArgoCD Image Updater, or custom CI scripts using the GitHub API to open PRs.

GitOps is powerful but comes with real constraints worth understanding:

Technical limitations:

• Secrets in Git — plain secrets can't be stored in Git. Requires a solution like Sealed Secrets, External Secrets Operator, or Vault + ESO to inject secrets at runtime
• Not well-suited for stateful workloads — databases with schema migrations, stateful sets with PVCs, require care and often imperative steps
• Bootstrap problem — something has to install ArgoCD/Flux before it can manage itself. Usually a one-time manual or Terraform-based bootstrap
• Drift from non-Kubernetes resources — GitOps with ArgoCD/Flux is Kubernetes-native. Reconciling cloud resources (RDS, S3) requires Crossplane or ACK, not just Argo/Flux

Organizational limitations:

• Requires team discipline — no more kubectl apply shortcuts; all changes go through Git
• Rollback is still a Git revert + PR, which has latency compared to an immediate helm rollback
• Config repo can become a sprawling "YAML hell" without good tooling (Helm, Kustomize)

Both are CNCF-graduated GitOps operators for Kubernetes, but with different philosophies:

ArgoCD:

• Ships with a full web UI out of the box — visual sync status, diff viewer, history, rollback
• Application-centric model — each app is a first-class ArgoCD Application CR
• Single controller with API server; multi-cluster via Application sets or separate ArgoCD instances
• Sync policies: manual or automated with optional self-healing and pruning
• Better for teams that want visibility and UI-driven operations

Flux v2 (GitOps Toolkit):

• Kubernetes-native, CLI-first — no UI included (Weave GitOps is the optional UI layer)
• Modular architecture: separate controllers for sources (GitRepository, HelmRepository), Kustomizations, HelmReleases, image automation
• Native multi-tenancy with namespaced resources
• Image Automation Controller can auto-update image tags in Git without CI involvement
• Better for teams embracing a pure Kubernetes-native, controller-per-concern model

Choosing: ArgoCD for teams wanting a visual control plane; Flux for teams preferring composable controllers and CLI workflows.

ArgoCD maintains a history of all sync operations — each sync is associated with a Git revision. Rollback means syncing to a previous revision.

Via UI: Navigate to the Application → History → select a previous revision → click Rollback

Via CLI:

argocd app history my-app              # list sync history with IDs
argocd app rollback my-app 3           # roll back to sync ID 3

What rollback does:

• Syncs the cluster back to the Kubernetes manifests at that Git revision
• The running state in the cluster reverts to the previous version
• The Git repo is NOT changed — the latest commit still has the newer version

Important caveat: if automated sync is enabled with self-healing, ArgoCD will immediately re-sync to the HEAD commit after rollback — undoing the rollback. To rollback durably with auto-sync enabled, you must either:

• Revert the commit in Git (proper GitOps approach)
• Temporarily disable auto-sync, roll back, then investigate and fix forward

An ApplicationSet is an ArgoCD CRD that generates multiple Application objects from a single template and a generator — automating multi-cluster, multi-environment, or multi-service deployments.

Common generators:

• List generator — iterate over a static list of clusters or environments
• Cluster generator — auto-generate an Application for every cluster registered in ArgoCD
• Git generator — scan a Git repo directory and generate an Application per subdirectory or per config file
• Matrix generator — combine two generators (e.g., all services × all environments)

Example — deploy to all clusters:

generators:
- clusters: {}      # all registered clusters
template:
  spec:
    source:
      path: '{{path}}'
    destination:
      server: '{{server}}'
      namespace: production

Use cases: deploying to 10 regional clusters, deploying all microservices from a monorepo, deploying per-PR preview environments.

ArgoCD itself manages sync state, but progressive delivery requires a traffic-splitting layer. The primary integration is Argo Rollouts.

Argo Rollouts replaces the Kubernetes Deployment object with a Rollout CRD that natively supports canary and blue-green strategies, integrated with service meshes and ingress controllers.

Canary with Argo Rollouts:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
      - setWeight: 10    # 10% traffic to new version
      - pause: {duration: 5m}
      - analysis:        # automated analysis before proceeding
          templates:
          - templateName: success-rate
      - setWeight: 50
      - pause: {}        # manual approval gate
      - setWeight: 100

Blue-green: Rollouts maintains two ReplicaSets (blue and green). Traffic is switched atomically after the green environment passes analysis checks. Rollback is instant — just switch traffic back.

ArgoCD integration: ArgoCD syncs the Rollout manifests from Git; Argo Rollouts handles the traffic shifting. They complement each other.

Flux Image Automation closes the loop between a new container image being pushed to a registry and the config repo being updated to reflect that new image — without any CI pipeline involvement in the config repo.

Components:

• ImageRepository — tells Flux to poll a container registry for new tags
• ImagePolicy — defines which tag to select (latest semver, latest by timestamp, regex)
• ImageUpdateAutomation — defines which Git repo and branch to update when a new tag matches the policy

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
spec:
  imageRepositoryRef:
    name: my-app
  policy:
    semver:
      range: '>=1.0.0'  # pick latest stable release

The flow:

1. CI builds and pushes my-app:1.2.3 to ECR
2. Flux ImageRepository detects the new tag via polling
3. ImagePolicy selects 1.2.3 as the new desired tag
4. ImageUpdateAutomation commits the tag update to the config repo
5. Flux Kustomization controller detects the new commit and syncs the cluster

Multi-cluster GitOps requires a strategy for both the GitOps operator placement and the config repo structure.

Hub-spoke model (most common): a single ArgoCD/Flux instance in a "management cluster" deploys to N target clusters. ArgoCD stores cluster credentials and targets each one. Pros: centralized visibility. Cons: single point of failure.

Decentralized model: each cluster runs its own ArgoCD/Flux instance, pointing at the same config repo but syncing its own directory. Pros: no cross-cluster credentials. Cons: more instances to manage.

Config repo structure for multi-cluster:

gitops-config/
  clusters/
    us-east-1/
      kustomization.yaml   # imports base + cluster-specific patches
    eu-west-1/
      kustomization.yaml
    ap-southeast-1/
      kustomization.yaml
  base/
    deployment.yaml        # shared base manifests

ArgoCD ApplicationSet cluster generator: automatically creates an Application for every registered cluster — no manual YAML per cluster.

Cluster identity: use cluster labels in ApplicationSet generators to target by region, tier (prod/non-prod), or capability.

Git is designed to be widely cloned and has immutable history — a terrible combination for secrets. A plaintext secret committed to Git:

• Persists in history even after deletion — accessible via git log or the reflog
• Is visible to anyone with repo access, including CI runners, contractors, and anyone the repo is ever shared with
• May be cached in GitHub's diff views, search indexes, or third-party Git analysis tools

The three main GitOps-compatible solutions:

1. Sealed Secrets (Bitnami) — encrypt Kubernetes Secrets with a cluster-scoped private key. The sealed (encrypted) Secret is safe to commit; only the in-cluster controller can decrypt it.

2. External Secrets Operator (ESO) — an ExternalSecret CR in Git points to a secret in AWS Secrets Manager, Vault, GCP Secret Manager, etc. ESO fetches and injects it as a Kubernetes Secret at runtime. Plaintext never touches Git.

3. Mozilla SOPS — encrypts secret values in YAML/JSON files using AWS KMS, GCP KMS, Age, or PGP. The encrypted file is committed; the CI/GitOps operator decrypts at deploy time.

External Secrets Operator (ESO) is a Kubernetes operator that reads secrets from external stores and materializes them as native Kubernetes Secret objects.

Setup:

# 1. SecretStore — how to authenticate to AWS
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:                      # IRSA — pod identity, no static keys
          serviceAccountRef:
            name: external-secrets-sa

# 2. ExternalSecret — what secret to fetch
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
  target:
    name: db-credentials          # creates this Kubernetes Secret
  data:
  - secretKey: password
    remoteRef:
      key: prod/myapp/db          # AWS Secrets Manager path
      property: password

ESO creates a Kubernetes Secret populated with the fetched value. Your app reads the standard Kubernetes Secret — no SDK or external store knowledge needed.

Key benefits: rotation in AWS Secrets Manager propagates to the cluster on the next refresh interval; no secrets in Git; works with any secret store via a unified API.

Sealed Secrets (Bitnami) is a Kubernetes operator that introduces a SealedSecret CRD. Secrets are encrypted locally using the cluster's public key; only the in-cluster controller can decrypt them using the private key.

Workflow:

# Fetch the cluster's public key
kubeseal --fetch-cert > public-key.pem

# Seal a regular Secret
kubectl create secret generic db-pass \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml | \
  kubeseal --cert public-key.pem > sealed-db-pass.yaml

# Safe to commit sealed-db-pass.yaml to Git
git add sealed-db-pass.yaml && git commit -m "add sealed db password"

The Sealed Secrets controller in the cluster decrypts the SealedSecret and creates a standard Kubernetes Secret.

Limitations:

• No central rotation — to rotate a secret you re-seal and commit. No automatic propagation from a vault.
• Key rotation — controller key rotation is manual and requires re-sealing all secrets
• Cluster-bound — a sealed secret is encrypted for one specific cluster. Multi-cluster deployments require sealing the same secret N times (or using a shared key)
• Disaster recovery — if the cluster is destroyed and the private key isn't backed up, sealed secrets are permanently unrecoverable

Secret rotation without downtime is a multi-step process that depends on the secret type and the secrets solution in use.

With External Secrets Operator (recommended):

• Rotate the secret in AWS Secrets Manager (via console, CLI, or Lambda)
• ESO picks up the new value on its next refresh cycle (configurable — 1m to 1h)
• Force immediate refresh: kubectl annotate externalsecret db-creds force-sync=$(date +%s) --overwrite
• Applications may need to reload the secret from the Kubernetes Secret (depends on how they read it — env vars require pod restart; volume mounts can reload live)

Zero-downtime rotation pattern (dual-active):

1. Create the new secret version — both old and new are active in the secrets store
2. Applications are updated to accept both versions (if protocol supports it)
3. Roll pods to pick up new secret
4. Verify all pods use new secret; deactivate old version

With Sealed Secrets: re-seal and commit (triggers a sync + rolling restart). No zero-downtime dual-active — requires coordinated deploy.

Defense in depth: multiple layers catching secrets at different points.

Layer 1 — Developer tooling (pre-commit hooks):

• detect-secrets (Yelp): scans for high-entropy strings and known secret patterns. Integrates with pre-commit framework.
• git-secrets (AWS): blocks commits matching configured patterns (AWS key formats, etc.)
• trufflehog: deep entropy-based scanning, also scans git history

Layer 2 — CI pipeline checks:

• Run trufflehog or gitleaks on every PR as a required status check
• Block merge if secrets are detected — fail the CI pipeline

Layer 3 — GitHub native features:

• GitHub Secret Scanning: automatically detects 200+ partner secret patterns in public repos (and private with GHAS)
• Push protection: blocks pushes containing detected secrets before they land in the repo

Layer 4 — Organizational policy:

• .gitignore templates blocking .env, *.pem, credentials.json
• Developer training and runbooks for "I committed a secret" scenarios

This is your behavioral closer. Use the STAR framework with a GitOps/workflow improvement angle:

Situation: describe the pain point. "We had 8 developers committing directly to main, deployments were manual, and rollbacks required SSH access to production servers."

Task: your role. "I was asked to establish a proper CI/CD pipeline and improve deployment reliability."

Action: what you specifically did.

• "Introduced branch protection on main — all changes via PR with 1 review required"
• "Added GitHub Actions workflow: lint → unit tests → integration tests → build image → push to ECR"
• "Deployed ArgoCD, migrated Helm charts to a config repo — cluster state became fully declarative"
• "Added pre-commit hooks with detect-secrets to prevent credential commits"

Result: concrete outcomes.

• "Deployment time dropped from 45 minutes (manual) to 8 minutes (automated)"
• "Zero production incidents from manual deploy mistakes in the following 6 months"
• "Rollbacks that used to require an on-call engineer became a 2-click ArgoCD operation"