Python
Study Guide

CPython is the reference implementation of Python written in C. It is what you get when you run python3 on most distributions — the interpreter executes your bytecode on a virtual machine.

Ops teams care because packaging, extension modules (many AWS SDK paths), and system packages (apt install python3) target CPython. Alternative implementations (PyPy, MicroPython) exist for speed or embedded use but are not interchangeable without testing extensions and wheels.

A virtual environment is an isolated directory with its own site-packages and usually a copy/symlink of the interpreter (python -m venv .venv). Imports resolve against that env first, not the system Python.

In production it prevents dependency clashes with OS-managed packages, makes upgrades rollback-friendly (replace the venv or image layer), and matches the “one app per env” model in containers and VMs.

env looks up python3 on PATH, so the script works across distros and local venv activation. A hard path like #!/usr/local/bin/python3 breaks on machines where Python lives elsewhere.

In systemd units or cron, you still often set PATH explicitly or call the venv’s interpreter directly (/opt/app/.venv/bin/python) for determinism.

PYTHONPATH prepends directories to sys.path so Python can import modules outside the default locations. Useful for monorepos or ad-hoc tooling.

Problems: shadowing installed packages, non-reproducible behavior between hosts, and security surprises if writable dirs appear earlier on the path. Prefer proper packages (pip install -e .), PYTHONPATH only when controlled and documented.

The Global Interpreter Lock (GIL) in CPython allows only one thread to execute Python bytecode at a time per process. CPU-bound parallel work in threads does not scale; I/O-bound work often does because threads release the GIL around blocking calls.

For DevOps scripts (boto3 calls, SSH, subprocess, HTTP), workloads are usually I/O bound — the GIL rarely matters. For heavy parallel CPU (parsing huge logs in threads), use multiprocessing, offload to native code, or pick another runtime.

pip installs into the active environment (venv or system) — libraries and apps tied to that project.

pipx installs CLI applications each into an isolated venv and links a single binary on PATH (pipx install black). Ideal for operator laptops, not for app dependency management inside a service repo.

pyproject.toml is the standard place to declare build system and project metadata (PEP 518 onward). PEP 621 defines the [project] table: name, version, dependencies, optional deps, scripts entry points.

Tools (Poetry, hatch, setuptools, uv) read or generate this file. It replaces scattered setup.py-only workflows for many teams.

requirements.txt — flat list (often pinned) of what pip installs. Simple but manual transitive updates.

requirements.in + pip-compile (pip-tools) — you edit top-level deps; compiler resolves and pins transitive versions into requirements.txt for reproducibility.

Poetry / PDM — poetry.lock or pdm.lock locks the full graph with metadata; installs are deterministic from the lockfile.

Editable installs add the project source to easy-install.pth (or equivalent) so code changes are immediately importable without reinstalling. Used when developing shared libraries or CLI tools in a monorepo.

Production images usually install a wheel or copy source intentionally — editable mounts in containers can hide “works on my machine” drift.

pip — universal, mature; slower cold installs unless cached.

Poetry — resolves deps, lockfile, good DX; resolver can be slower; still common in pipelines.

uv (Astral) — very fast Rust-based install and resolver; drop-in for many pip workflows; growing adoption for CI cache efficiency.

Reproducibility comes from locks or compiled requirements, not the installer brand alone.

argparse is in the stdlib — no extra deps, fine for small internal scripts and CI entrypoints.

Click / Typer add composable commands, better help text, and type-hint driven CLIs — worth it when the tool grows subcommands and plugins (kubectl-style UX).

With shell=True, the string passes through a shell, so metacharacters (; | $()) are interpreted. Untrusted input can become command injection.

Prefer shell=False with a list argv ["ansible-playbook", "site.yml"] and no user-controlled shell grammar.

Pass arguments as a list of strings to subprocess.run without shell=True so the OS passes argv directly to the program. Validate allowlists for hostnames, paths, and job names.

For environment variables, use env={**os.environ, "FOO": value} with controlled values. Never interpolate untrusted strings into shell one-liners.

pathlib.Path gives object-oriented paths, overloads / for joining, and clear methods (read_text, glob, mkdir(parents=True)). Fewer string bugs across Windows vs Linux when you must support both.

os.path still fine in legacy code; new internal tools should standardize on pathlib.

Plain text logs need fragile regex in Loki, CloudWatch, or ELK. JSON lines map cleanly to fields (level, trace_id, user) for filtering and alerting.

Libraries like structlog or logging formatters emit one JSON object per line; ship to the collector via stdout (12-factor).

pytest has concise asserts, powerful fixtures, parametrization, and a huge plugin ecosystem (pytest-cov, pytest-xdist). Less boilerplate than unittest classes for glue code and policy tests.

unittest remains valid (stdlib, some enterprises); pytest is the de facto modern default.

Common options: moto (decorators or server mode emulating many services), botocore.stub.Stubber for precise request/response fixtures, or dependency-inject a client interface and fake it in unit tests.

Integration tests may hit a real sandbox account with scoped credentials and cleanup fixtures.

coverage.py measures which lines ran during tests. Run pytest --cov=src --cov-report=xml and upload XML to Sonar, Codecov, or fail the job if coverage drops below threshold.

Use .coveragerc to omit generated or migration paths. Coverage is a guardrail, not proof of correctness.

pre-commit runs configured hooks (Ruff, Black, mypy, secret scanners) before a commit or in CI via pre-commit run --all-files. Same checks locally and in pipeline reduce “lint failed on main” churn.

Share .pre-commit-config.yaml in the repo; pin hook revisions for reproducibility.

Unit tests isolate functions (parsing, templating, IAM policy generation) with mocks — fast, deterministic.

Integration tests run against real-ish dependencies: Docker compose stack, k3d cluster, or account sandbox — slower, catch wiring and permission bugs.

Pyramid: many unit, fewer integration, rare full e2e pipeline runs.

boto3 is the AWS SDK for Python. A client is low-level: maps 1:1 to API operations (list_objects_v2), returns dicts.

A resource is a higher-level object-oriented facade (e.g. s3.Bucket) built on clients — ergonomic for simple CRUD, less control for every API edge case.

New code often uses clients + type hints; resources still appear in older scripts.

boto3 uses the default credential chain: env vars (AWS_ACCESS_KEY_ID), shared credentials file, SSO / IAM Identity Center, instance/container role, etc.

Lambda — execution role credentials injected automatically; never bake keys into the image.

CI — prefer OIDC federation to assume role (GitHub Actions → AWS) over long-lived access keys.

Use paginators: client.get_paginator('list_objects_v2') then paginate(Bucket=...) — handles NextToken / ContinuationToken correctly.

Manual loops with NextMarker are error-prone. Paginators are the standard pattern in automation and data pipelines.

boto3 (via botocore) can use configurable retries for transient errors and throttling. Not all APIs are safe to blindly retry — create operations may double-create without idempotency tokens.

Use client tokens where supported, check-before-create patterns, or Step Functions for orchestrated retries. Combine with exponential backoff and jitter at the application layer for custom loops.

Python IaC shines when you need loops, shared libraries, unit tests of infra logic, or tight coupling to application code in one language.

Terraform excels at broad provider ecosystem, declarative state, and team familiarity. Many orgs standardize on HCL and call Python only for glue (pre/post hooks, custom providers via CDKTF).

Tradeoff: abstraction power vs complexity and reviewability.

A builder stage installs compilers and build deps; a final runtime stage copies only wheels/venv and runs on a slim base (python:3.12-slim). Smaller images, fewer CVEs, faster pulls.

Avoid leaving gcc, git, or .git history in production layers.

Docker caches a layer if inputs unchanged. Copy requirements / lockfile first, run pip install, then COPY application source. Code changes won’t bust the dependency layer.

Anti-pattern: COPY . . before pip — any file edit reinstalls everything, slowing CI.

Automating CRUD on resources (Jobs, CronJobs, ConfigMaps), building internal portals, controllers/operators (often with kopf or framework), or glue between CI and the API server.

Uses in-cluster service account tokens or kubeconfig from CI secrets. Watch for RBAC least privilege on the ServiceAccount.

Build/push images in pipelines, prune images on hosts, introspect containers for diagnostics, integration-test compose stacks programmatically.

Prefer the CLI for human ops; SDK when tests or services must drive Docker without shelling out.

Cron / k8s CronJob — simple schedules on a box or cluster; good for batch scripts with clear idempotency.

Celery — distributed task queue with workers; when you need retries, routing, and backpressure inside an app ecosystem.

Step Functions (or similar) — visual/state-machine orchestration across Lambdas and services with built-in retries and observability.

Pick by complexity, team familiarity, and cloud vs self-hosted constraints.

Pin with lockfiles or compiled requirements; use pip-audit, Safety, GitHub Dependabot, or Snyk in CI. Fail builds on critical vulns or open tickets with SLA.

Combine with minimal base images and regular rebuilds — supply chain is layers + deps + base OS.

Env vars — 12-factor friendly, simple for non-rotating config; risk if leaked via logs or ps.

Secrets Manager / SSM Parameter Store — fetch at startup with IAM role, support rotation and auditing.

Vault — dynamic secrets, policy-heavy enterprises. Cache carefully and handle auth (K8s SA, AppRole).

Never commit secrets; inject via platform in production.

Run tests (unit + smoke) on the new interpreter in CI; build new base images; canary a subset of workloads; watch for deprecated stdlib removals and C-extension wheels (manylinux).

Document breaking changes (e.g. asyncio, encoding defaults). Roll forward with automated AMIs or GitOps image bumps; keep rollback tag.

Use pyupgrade or Ruff rules to modernize syntax after upgrade.

Wrong WorkingDirectory breaking relative paths; missing venv in ExecStart; buffered stdout hiding logs (use PYTHONUNBUFFERED=1 or -u); user permissions; Restart= policies masking crash loops.

Set EnvironmentFile or drop-ins for secrets; use Type=notify only if the app supports sd_notify.

Use a concise STAR outline:

Situation: Manual deploys, flaky config drift, or slow feedback.

Task: Automate validation, packaging, or cloud updates.

Action: Name concrete tools — pytest + moto, boto3 paginators, Docker multi-stage, GitHub Actions OIDC, pre-commit, structured logging.

Result: Quantify — time saved, fewer incidents, faster rollbacks.

Reflection: What you’d add next (integration tests, feature flags, better observability).