Software Engineering Handbook

Reliable correctness under retries, races and outages; protected data; predictable latency and cost; recoverability; clear ownership; and economics that fit the business.

• Evidence before claims

  Code, migrations, cloud config, executed tests and runtime metrics are the source of truth — not a README or an installed dependency.

• Separate the traffic numbers

  Daily users, concurrent connections, in-flight requests and requests per second are four different things.

• Correctness before speed

  No tenant leaks, duplicate payments, double bookings or unsafe downloads. A fast app that mispays is not production-ready.

• Scale progressively

  Baseline → optimise queries and cache → tune the pool and autoscale → split services or partition data only when measurements demand it.

Eight lenses for every design review

Every non-trivial decision gets an architecture decision record: context, at least two alternatives, constraints, the choice, expected signals, and a rollback plan.

Daily users, concurrent sessions, in-flight requests and RPS are not interchangeable — model the workload, then prove it.

rps = users × actions_per_min × api_calls_per_action ÷ 60

// illustration — not a benchmark:
100,000 users × 1 action / 30 s  ≈ 3,333 actions/s
× 4 uncached API calls            ≈ 13,333 API req/s
// …before retries, bots and background jobs

99.9%  uptime ≈ 43.2 min downtime / 30 days
99.95% uptime ≈ 21.6 min downtime / 30 days

The load-test sequence

1. 1Baseline
2. 2Ramp to peak
3. 3Spike
4. 4Soak
5. 5Stress
6. 6Failure injection

Never claim capacity that has not been demonstrated by a workload test and a recovery test at the target.

A representative scalable workload. Use managed services where possible — it is a map, not a shopping list.

• Edge

  DNS → CDN + DDoS protection + WAF + TLS termination

• Routing

  Reverse proxy / load balancer with health and readiness checks

• Application

  Stateless web / API / server-actions tier, plus an identity provider

• Data

  Managed relational database (pooled), optional read replica

• Async

  Transactional outbox → queue → workers → dead-letter queue

• Files

  Private object storage → signed URLs; CDN for public assets only

• Speed

  Redis for measured hot reads and rate-limit counters

• Insight

  Logs, metrics and traces — with secrets and PII redacted

What to adopt now, later, or almost never

Modular monolith first. Kubernetes, microservices and sharding are justified by measurements and team shape, not by appearances — see when they make sense.

A correctness-first model: invariants live in the database, and everything that can be retried must be safe to retry.

• Make invariants executable: UNIQUE, FOREIGN KEY, CHECK and NOT NULL constraints, clear transaction boundaries, audit fields.
• Pick isolation levels deliberately; test lost updates, write skew and concurrent approvals.
• Optimistic concurrency (version columns) for ordinary edits; locks or serialised commands where correctness demands it.
• Enforce idempotency and uniqueness on the server — a disabled button is not protection against double submit.
• Keep an authoritative ledger for money and payroll; recompute derived totals deterministically.
• Pool connections, bound query time, profile with EXPLAIN, and record every migration.
• Never hold a transaction open while calling an external API.

Scaling a database — in order

Zero-downtime schema changes

1. 1Expand — add compatible columns/indexes
2. 2Deploy code that reads both shapes; backfill in batches
3. 3Verify metrics and the rollback path
4. 4Contract — drop old fields after the rollback window

Cache, queues, files, search and APIs

• Caching

  A consistency decision: browser/CDN for public content, server cache for expensive reads, Redis only for shared ephemeral data or measured hot keys. Never cache one tenant's data where another can reach it; never serve stale permissions.

• Queues

  At-least-once delivery means duplicates. Use idempotent consumers, bounded retries with backoff and jitter, dead-letter queues, a transactional outbox and queue-age monitoring.

• File storage

  Private buckets and short-lived signed URLs for sensitive files; allowlisted types, size limits and malware scans for uploads; EXIF stripped from images.

• Search

  Start with indexed SQL and Postgres full-text. Add a dedicated engine only when relevance or scale needs it — with ACL filters so results never leak private data.

• APIs

  Versioned contracts, validation, typed errors, request IDs, bounded pagination, exact CORS allowlists, authenticated webhooks with replay windows and per-tenant quotas.

No system can promise to be unhackable. A defensible claim is that known risks are systematically reduced, tested, monitored and recoverable.

Threats to design against

• Broken object authorization (IDOR/BOLA) and tenant leakage
• SQL injection, XSS, SSRF, command injection, path traversal, malicious uploads
• Credential stuffing, session theft, phishing, OAuth misconfiguration
• Scraping, enumeration, spam sign-ups, cost exhaustion, denial of service
• Race conditions: double booking, duplicate payouts, replayed webhooks
• Secrets in code, CI logs or client bundles; public buckets
• Supply chain: vulnerable dependencies, compromised CI actions, unreviewed AI code
• Prompt injection and over-permissioned AI agents

Preventive, detective and recovery controls

Classify data as public, internal, confidential or highly sensitive; keep ID documents and payroll in private storage with signed access; review UAE and cross-border obligations with counsel. Verify against OWASP ASVS 5.0, the OWASP Top 10:2025 and the OWASP API Security Top 10.

Design around failure: every dependency will eventually be slow, down or wrong.

• Timeouts & budgets

  No hung call may use up every request slot.

• Bounded retries

  Exponential backoff + jitter + idempotency — never retry storms.

• Breakers & bulkheads

  Stop one failing dependency cascading through the system.

• Graceful degradation

  Cached catalogue stays up; payroll permissions are never served stale.

• Overload protection

  Queues, concurrency limits and pool caps shed non-critical work.

• Health checks

  Readiness, liveness and safe shutdown on every deploy.

The golden signals

Give each critical journey an SLI, an SLO, an error budget, an owner and a dashboard. State RPO and RTO per datastore, keep encrypted backups in an independent failure domain, and test the restore — a successful backup is not proof of a successful recovery.

A change reaches production only through a pipeline that proves it is safe — with humans approving what is destructive.

1. 1Pull request & code-owner review
2. 2Lint · typecheck · unit tests
3. 3Migrations from a clean baseline + RLS tests
4. 4Integration · contract · E2E
5. 5SAST · secrets · dependencies · SBOM
6. 6Staging deploy + smoke tests
7. 7Protected-branch gate & approval
8. 8Canary / feature flags & rollback decision

The test pyramid

Many engineers (and AI agents), one codebase

• One accountable technical lead owns scope, architecture and integration order.
• One branch or worktree per task, with assigned, non-overlapping files.
• Contracts and ADRs first → migrations → API → frontend → integration tests.
• One migration authority; only the integration owner merges or cherry-picks.
• Protected main, CODEOWNERS, required CI and environment approvals — no silent emergency merges.
• AI-written code is untrusted until reviewed; agents get limited credentials.

Sequence the work by risk and evidence, not by technology fashion.

1. P0

   Protect now

   Secret scanning, role/tenant authorization, financial invariants, backups, admin MFA, blocking CI.

   Exit: No known exploitable P0; critical data can be restored.
2. P1

   Stabilise

   Metrics and traces, query audit, connection pool, rate limits, bounded retries, private files.

   Exit: Critical SLOs measured; runbooks work.
3. P2

   Verify scale

   k6 workload models, CDN/cache, index fixes, controlled autoscaling, quota plan.

   Exit: Peak target sustained with margin and measured cost.
4. P3

   Specialise

   Workers/outbox, search index, replicas, partitioning if required.

   Exit: Bottleneck-specific benchmarks show value.
5. P4

   Expand platform

   Kubernetes, microservices, sharding, multi-region — only when justified.

   Exit: An ADR proves benefit exceeds risk and cost.

Anti-patterns to catch early

• Expensive technology with no measured bottleneck or trained owner
• Clients talking directly to privileged tables or open storage buckets
• Unbounded lists and full-table scans on every homepage load
• Long transactions holding locks while calling external APIs
• Retry loops without deadlines, idempotency or backoff
• Ignoring CI because the runner broke; migrations run out of order
• Several people or agents editing the same migration or production branch
• Calling it done because unit tests pass while authorization and recovery are untested

Audit first, redesign later. A dependency being installed proves nothing — verified configuration, tests and telemetry do.

• 1 · Pin & inventory

  Fix one app, commit and deployment; list routes, roles, jobs, migrations, integrations and unknowns — read-only.

• 2 · Verify independently

  Security, database and SRE checks run separately, against evidence.

• 3 · Capability matrix

  Need × status (implemented / partial / missing / unverified) × evidence × owner.

• 4 · Findings register

  P0–P3 severity, safe reproduction, fix, test plan and rollback for each.

• 5 · Fix & decide

  Small sequential PRs, staging tests, every CI gate green — then a release decision from results, not confidence.

Findings are ranked P0 (exploitable compromise or data corruption) to P3 (enhancement); maturity is scored 0–5 only when evidence backs it, and “unverified” is always an acceptable answer.

Domain checklists

• Rental marketplace

  Supplier/renter/admin isolation, availability concurrency, idempotent deposits and refunds, catalogue search, uploads and verification documents.

• Workforce & payroll

  Role scopes, local business-date and shift rules, overtime eligibility, approvals with audit, payroll lock, confidential HR documents.

• Corporate website

  CDN caching, Core Web Vitals, canonical URLs and sitemaps, anti-spam forms, attachment security, analytics consent.

• ERP / enquiry hub

  Deduplication across email, WhatsApp and forms, attachment scanning, audit trail, RBAC, reconciliation with accounting.

60 detailed guides cover each topic above — also available to ask the Devv assistant on this site.

• What does enterprise-grade software actually mean?

• How do I document architecture decisions (ADRs)?

• How do I plan for 100,000 users?

• How do I load-test an application?

• How do I design a scalable web architecture?

• Which infrastructure do I need now and which later?

• Do I need Kubernetes, microservices or sharding?

• What is the difference between a proxy, load balancer and API gateway?

• How do I keep data correct (ACID, constraints, concurrency)?

• How do I prevent duplicate payments and double bookings?

• Read replicas, partitioning or sharding?

• How do I deploy database changes with zero downtime?

• How should caching work in a production app?

• How do queues, retries and idempotency work?

• How do I store and serve files securely?

• When do I need a search engine like Elasticsearch?

• API design checklist for production?

• What are the most common application security threats?

• How do I secure authentication and authorization?

• How should I classify and protect sensitive data?

• How should an AI assistant be secured against prompt injection?

• How do I make an app reliable under failure?

• What are SLIs, SLOs and the golden signals?

• How do I plan backups and disaster recovery (RPO and RTO)?

• What should a CI/CD pipeline include?

• What tests does production software need?

• How do multiple developers or AI agents work on one codebase safely?

• How do I run a production-readiness audit?

• What is the right rollout roadmap for engineering maturity?

• What are common architecture anti-patterns?

• What should I check when building a rental or B2B marketplace?

• What should I check when building workforce, HR and payroll software?

• What should a production corporate website include?

• What is the software development lifecycle (SDLC)?

• How do I write good requirements and specifications?

• How should Git, branching and code review work in a team?

• How do I keep code clean and maintainable?

• What does a professional testing strategy look like?

• What is DevOps, and how do containers and infrastructure as code fit in?

• How do I choose and run workloads in the cloud?

• What networking fundamentals should every engineer know?

• What are data structures, algorithms and Big-O notation?

• How do frontend, backend and full-stack responsibilities differ?

• What is event-driven architecture and domain-driven design?

• How do data engineering, analytics and databases differ (SQL vs NoSQL, OLTP vs OLAP)?

• What does AI and machine-learning engineering involve?

• How do I manage technical debt, documentation and software maintenance?

• What are accessibility, internationalisation and performance standards for web products?

• What are the core principles of distributed systems (CAP, consistency, consensus)?

• What should every engineer know about concurrency and operating systems?

• How do I design a multi-tenant SaaS architecture?

• How do I build observability with OpenTelemetry, tracing and alerting?

• What is the process for incident management and on-call?

• How do SOC 2, ISO 27001, PCI-DSS and data-protection laws apply to software?

• How do I perform threat modelling with STRIDE?

• How do feature flags, canary releases and API versioning work?

• What is chaos engineering and resilience testing?

• What differs in mobile, embedded and game development?

• How should I choose a programming language and framework?

• What does technical leadership involve for a senior engineer?

Standards referenced

• AWS Well-Architected Framework
• OWASP ASVS 5.0
• OWASP Top 10:2025
• OWASP API Security Top 10 (2023)
• NIST SP 800-218 (SSDF)
• OWASP Secrets Management