By Yusuf Elborey

Executable Architecture: Replacing Static Reviews with Continuous Guardrails

architecture-governancegolden-pathspolicy-as-codeplatform-engineeringexecutable-architectureobservabilityguardrailssolution-architecture

Executable Architecture Guardrails

The architecture review board met for three hours last week.

They approved three services. They rejected one. They deferred two.

The rejected service had a missing failover strategy. The deferred services had basically the same problem, just dressed differently. One used a single Availability Zone. The other used a database without a backup plan. Both came back for re-review next month.

Meanwhile, five other teams shipped code to production that week without ever sitting in that room. Some skipped the review entirely because the board was booked out two weeks. Others got approval, then changed the design during implementation because the requirements shifted.

This is the normal state of architecture governance. Reviews are bottlenecks. Decisions become stale the moment they’re written. Non-functional requirements get documented in Confluence or Notion pages, then ignored.

The problem isn’t the board members. The problem is that architecture decisions live in documents instead of automation.

What executable architecture actually means

Executable architecture means your architectural decisions are encoded into things that run.

Templates. CI/CD checks. Infrastructure policies. Runtime SLOs. Observability dashboards. Service scorecards.

When a decision is encoded, it runs every time. It doesn’t depend on someone remembering to read a document. It doesn’t depend on a quarterly review. It executes.

This doesn’t mean architects stop designing. It means architects define the rules, and those rules are enforced automatically. The role shifts from “approver” to “policy author.”

The goal isn’t to remove human judgment. The goal is to handle the repetitive checks—health checks, authentication, resource limits, tag requirements—so the board can spend time on the hard problems: trade-offs, new patterns, strategic bets.

I’ve watched teams spend forty-five minutes in a review debating whether a service needs a /health endpoint. The answer was already decided three versions ago. The arguments were just painful because there was no automated enforcement. Make it mandatory in the template, and the conversation changes from “should we?” to “did we?”

Golden paths are architecture delivery vehicles

A golden path is a preconfigured, opinionated workflow for a common task. It’s the fastest way to do the right thing.

Create a new REST service. Create an event consumer. Provision a database. Expose an API. Add observability. Deploy to Kubernetes.

Each of these has a golden path. Each golden path embeds architectural decisions.

The REST service template includes OpenTelemetry initialization. The event consumer template includes idempotency keys and a dead-letter queue. The database provisioning template requires encryption at rest, automated backups, and point-in-time recovery.

Golden paths reduce cognitive load. Developers don’t need to remember twenty architectural rules. They use the template. The rules are already baked in.

But golden paths only work if they’re enforced. If teams can bypass them without friction, they will. Especially under delivery pressure. A senior engineer with a deadline will choose the shortcut every time. The question is whether the shortcut is safe.

This is where policy-as-code comes in. Golden paths guide. Policies enforce.

Mapping non-functional requirements to automated controls

Every non-functional requirement should have an automation point. This is where it gets checked, logged, and enforced.

Here’s the mapping we use:

NFRArchitecture ruleAutomation point
AvailabilityService must define health checksCI template validation
SecurityPublic APIs require auth policyAPI gateway policy check
ObservabilityEvery service emits traces, logs, metricsOpenTelemetry SDK check
CostWorkloads must define resource limitsKubernetes admission policy
ResilienceAsync consumers must be idempotentCode review checklist + tests
ComplianceData stores must declare classificationIaC policy scan

Each row has an owner and an enforcement mechanism. The CI template validation fails the build. The API gateway policy rejects the request at the edge. The Kubernetes admission controller rejects the manifest.

This turns architecture from a recommendation into a constraint. And constraints, when automated, scale.

The key is specificity. “Service must be secure” is a wish. “Service must have an auth policy attached at the gateway” is a check. The first generates debates. The second generates passes or failures.

Policy-as-code patterns

Policies are the rules that keep your architecture from drifting. They should run at every gate.

OPA/Rego for cross-cutting policies

Rego is the language of Open Policy Agent. It’s declarative, and it works everywhere: CI, Kubernetes admission, Terraform Cloud, API gateways.

The learning curve is real. Start with one policy. Get comfortable. Add more.

Here’s a policy that rejects any service exposing a public endpoint without authentication:

package architecture.api

deny[msg] {
    input.kind == "Service"
    input.spec.type == "LoadBalancer"
    not input.metadata.annotations["auth REQUIRED"] == "true"
    msg := sprintf("Service '%s' is public but missing auth requirement. Add annotation 'auth REQUIRED=true'.", [input.metadata.name])
}

This runs in CI and in the cluster. It doesn’t matter where the change comes from—a PR, a manual kubectl apply, or a controller. The policy executes.

The best policies are small and focused. Each policy should do one thing. If a policy has ten rules, split it. Smaller policies are easier to test, easier to explain, and easier to fix when they break.

Terraform and Bicep policy checks

Infrastructure-as-code tools need their own enforcement layer. Conftest evaluates Rego policies against Terraform plans. Azure Policy enforces rules on Bicep and ARM templates.

Conftest is the easiest place to start if you’re already using Terraform. It runs in CI, it’s fast, and the error messages are readable.

Here’s a Bicep policy that requires every resource to have an owner tag, conform to backup requirements, and use encryption:

// architecture-policies.bicepparam
param enforceTags = true
param enforceBackup = true
param enforceEncryption = true

resource validators 'Microsoft.Authorization/policyDefinitions@2021-06-01' = {
  name: 'require-owner-and-backup-tags'
  properties: {
    policyType: 'Custom'
    mode: 'Indexed'
    parameters: {
      tagKeys: {
        type: 'Array'
        defaultValue: ['Owner', 'Env', 'DataClassification']
      }
    }
    policyRule: {
      if: {
        allOf: [
          { field: 'tags[concat(parameters(''tagKeys''), 0)]', exists: 'false' },
          { field: 'type', notEquals: 'Microsoft.Resources/subscriptions/resourcegroups' }
        ]
      }
      then: {
        effect: 'deny'
        details: {
          message: 'Resource must have Owner, Env, and DataClassification tags.'
        }
      }
    }
  }
}

This blocks resources at deploy time. It doesn’t post a report to Slack. It denies the request. The developer sees the error immediately and fixes it before the resource is created.

Kubernetes admission controllers

Admission controllers are your runtime safety net. They catch things CI might miss: manual deployments, controller-generated resources, out-of-band changes.

Gatekeeper and Kyverno are the common choices. We prefer Kyverno for teams that want YAML instead of Rego. Use Gatekeeper if your policies are already in Rego or if you need OPA’s full power.

Here’s a Kyverno policy that requires every Pod to define resource requests and limits:

apiVersion: kyverno.io/v1
kind: Policy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-cpu-memory
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "CPU and memory limits are required."
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    memory: "?*"
                    cpu: "?*"
                  limits:
                    memory: "?*"
                    cpu: "?*"

Admission policies are the last line of defense. CI catches most problems early. Admission catches what CI misses. Together, they cover both planned and unplanned changes.

GitHub Actions pipeline gates

Your CI pipeline is where most architecture enforcement happens. It runs on every PR. It fails fast. It gives developers clear feedback.

Here’s a pipeline that validates service metadata, runs policy checks, validates OpenAPI security, and checks for an SLO file:

name: Architecture Guardrails

on:
  pull_request:
    paths:
      - 'services/**'
      - 'infra/**'
      - '.github/workflows/architecture-guardrails.yml'

jobs:
  validate-service-metadata:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Check service metadata exists
        run: |
          SERVICE_FILE="services/${{ github.event.repository.name }}/service.yaml"
          if [ ! -f "$SERVICE_FILE" ]; then
            echo "::error file=$SERVICE_FILE::Service metadata file is missing"
            exit 1
          fi

  policy-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install conftest
        run: |
          wget https://github.com/open-policy-agent/conftest/releases/download/v0.45.0/conftest_0.45.0_Linux_x86_64.tar.gz
          tar xzf conftest_0.45.0_Linux_x86_64.tar.gz
          sudo mv conftest /usr/local/bin
      - name: Run policy checks
        run: |
          conftest test infra/**/*.tf -p policies/terraform/
          conftest test k8s/**/*.yaml -p policies/kubernetes/

  validate-openapi-security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate security schemes exist
        run: |
          OPENAPI_FILE="services/${{ github.event.repository.name }}/openapi.yaml"
          if ! grep -q "securitySchemes" "$OPENAPI_FILE"; then
            echo "::error file=$OPENAPI_FILE::OpenAPI spec missing securitySchemes"
            exit 1
          fi

  check-slo-file:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Verify SLO file exists
        run: |
          SLO_FILE="services/${{ github.event.repository.name }}/slo.yaml"
          if [ ! -f "$SLO_FILE" ]; then
            echo "::warning file=$SLO_FILE::SLO definition missing. Required for critical services."
            exit 1
          fi

And the metadata file it validates? It should look like this:

service:
  name: payment-api
  owner: payments-team
  tier: critical
  data_classification: confidential
  slo:
    availability: 99.9
    latency_p95_ms: 300

Simple fields. Machine-readable. The CI pipeline reads them, the admission controller uses them, the scorecard displays them. One source of truth.

The key is that every check fails with a clear message. Developers know exactly what’s missing and where to fix it. Not “Policy check failed.” But “Service ‘payment-api’ is public but missing auth requirement.”

Architecture fitness functions

A fitness function is an architectural test. It asserts that your system has the characteristics you care about.

No service can deploy without an owner. No public endpoint without authentication. No production service without an SLO definition. No database without a backup policy. No event consumer without retry and DLQ configuration.

These aren’t suggestions. They’re assertions. They pass or fail.

You can implement fitness functions as CI checks, as admission policies, or as runtime probes. The mechanism matters less than the fact that they exist and run automatically.

For example, here’s a fitness function that enforces architectural ownership:

#!/bin/bash
# check-owner.sh - CI fitness function

SERVICE_FILE="services/$1/service.yaml"

if [ ! -f "$SERVICE_FILE" ]; then
    echo "FAIL: $1 is missing service.yaml"
    exit 1
fi

OWNER=$(yq '.owner' "$SERVICE_FILE")
if [ -z "$OWNER" ] || [ "$OWNER" == "null" ]; then
    echo "FAIL: $1 has no owner assigned"
    exit 1
fi

echo "PASS: $1 owned by $OWNER"

This script runs in CI. It stops deployment if a service has no owner. It’s a tiny check with high impact. Every production service must have someone responsible for it.

When we first added this check, three services failed. They had been running in production for months. Nobody owned them. Alerts fired into the void. The fitness function made the invisible visible.

Here’s another one that enforces OpenAPI security:

# check_openapi_security.py
import yaml
import sys

with open(sys.argv[1], 'r') as f:
    spec = yaml.safe_load(f)

if 'components' not in spec or 'securitySchemes' not in spec.get('components', {}):
    print("FAIL: OpenAPI spec missing securitySchemes")
    sys.exit(1)

print(f"PASS: OpenAPI spec has {len(spec['components']['securitySchemes'])} security schemes")
sys.exit(0)

Run it in CI against every service’s OpenAPI definition. If the securitySchemes section is missing, the build fails. No exceptions.

Runtime governance using telemetry

Architecture validation can’t stop at build time. Build-time checks confirm your intent. Runtime checks confirm your reality.

You might have a perfect health check in your code, but if the endpoint returns 500, the architecture is broken. You might have configured retries, but if the retry storm is consuming all your connections, the architecture is broken. You might have set resource limits, but if the service is still OOMKilled because the limit was too low, the architecture is broken.

Build-time checks tell you what you intended. Runtime telemetry tells you what’s happening.

You need runtime signals for:

  • Error rate
  • Latency
  • Saturation
  • Dependency failure rate
  • Queue lag
  • Retry storms
  • Cost anomalies

OpenTelemetry defines the standard signals: traces, metrics, logs, baggage, and profiles.

Here’s a minimal OpenTelemetry setup for a Python service using the standard SDK:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
import os

def setup_telemetry(app):
    trace_provider = TracerProvider()
    trace_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    )
    trace_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
    trace.set_tracer_provider(trace_provider)

    metric_reader = PeriodicExportingMetricReader(
        OTLPMetricExporter(endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"))
    )
    meter_provider = MeterProvider(metric_readers=[metric_reader])

    FlaskInstrumentor().instrument_app(app)
    RequestsInstrumentor().instrument()

    meter = meter_provider.get_meter(__name__)
    request_counter = meter.create_counter(
        "http.requests",
        description="Count of HTTP requests"
    )

Golden-path services should ship with this instrumentation included. No exceptions. If a service doesn’t emit traces and metrics, it shouldn’t reach production.

The instrumentation shouldn’t be an afterthought. It should be in the template, tested in CI, and verified by the deployment pipeline.

I once spent two weeks debugging a retry storm that only happened in production. The service had retry logic, but no metrics. We couldn’t see the retry rate until we added instrumentation. By then, we had already taken down a downstream dependency.

Runtime telemetry isn’t optional. It’s part of the architecture.

Service scorecards

Scorecards make architecture quality visible. They’re not about shaming teams. They’re about guiding improvement.

Here’s a practical scorecard for a payment API:

DimensionScoreNotes
Ownership100%owner: payments-team set in metadata
Security baseline90%mTLS enabled, auth policy attached; missing certificate rotation alert
Observability85%Traces and metrics shipping; logs missing structured JSON format
Resilience80%Retries and DLQ configured; missing circuit breaker on downstream DB
Cost hygiene70%Resource limits defined; no auto-scaling policy for off-peak
Operational readiness90%Runbook exists, SLOs defined, on-call rotation active

Scorecards should be automated. Pull the data from infrastructure metadata, CI results, and observability platforms. Don’t ask humans to fill out a spreadsheet.

The 70% on cost hygiene isn’t a failure. It’s a signal. It tells the architecture review where to focus. Ask the team: what’s blocking auto-scaling policy adoption? Often the answer is “we didn’t know we needed one.” The scorecard surfaces the gap.

Use scorecards in architecture reviews as a starting point. Review the low scores first. Review the trends. If observability dropped from 95% to 80% in one quarter, something changed. Find out what.

The goal is progress, not perfection. A scorecard that moves from 65% to 75% in a quarter is more valuable than one that stays at 90% because nobody is improving anything.

Avoiding guardrail failure modes

Automated guardrails fail in predictable ways. Watch for these.

Too many policies too early

Start with the top three risks. Enforce them well. Add more only after the team trusts the existing ones. A dozen policies that nobody reads might as well not exist.

We rolled out five policies in the first month. Health checks, ownership, authentication, resource limits, and data classification. We fixed the easy, high-impact problems first. By month three, teams were asking for more policies instead of resisting them.

Blocking without explanation

When a CI check fails, tell the developer what’s wrong and how to fix it. Not just “Policy check failed.” Give them the exact rule, the exact file, and a link to the golden path.

- name: Check service metadata
  run: |
    if [ ! -f "service.yaml" ]; then
      echo "::error::Missing service.yaml. Copy the template from https://internal.platform.dev/templates/service.yaml"
      exit 1
    fi

The error message is the UX of your guardrail. Make it good.

Outdated templates

Templates rot. Review them quarterly. If the golden path for creating a service still uses HTTP instead of gRPC for internal calls because “that’s how we’ve always done it,” the team will bypass it. Keep the paths current.

Platform engineers move fast. Services evolve. The template that was opinionated in January is archaic by June. Assign an owner to each template. Make updating them part of the platform team’s roadmap.

Platform teams as gatekeepers

The platform team’s job is to make the right path the easy path. If the platform becomes a bottleneck, the architecture enforcement backfires. Teams find workarounds. Compliance drops.

I’ve seen teams copy-paste YAML from production to get around an approval workflow. The workflow was slow. The YAML was wrong. The guardrail that was supposed to protect them became the reason they bypassed safety.

Measuring compliance instead of quality

If your scorecard only measures whether policies passed, you’ll optimize for passing policies, not for building good systems. A service can pass every check and still have a terrible latency SLO. Measure what matters: outcomes, not checkbox counts.

Compliance is a lagging indicator. Error rate is a leading indicator. Watch both, but weight the outcomes higher.

A 90-day implementation roadmap

Here’s how to roll this out without disrupting your organization.

Days 1–30: Identify the risks

Interview teams. Look at production incidents. Find the top five recurring architectural failures.

Go through your incident history for the last six months. List every architecture-related root cause. Was it a missing health check? A single-AZ deployment? A database without backups?

Common candidates:

  • Services without health checks.
  • Public endpoints without authentication.
  • Databases without backups.
  • Missing resource limits causing noisy neighbors.
  • No defined ownership.

Write them down. Get agreement from engineering leadership. If leadership doesn’t agree that these are problems, you’re not ready to automate them yet.

Days 31–60: Encode the first three

Pick the three highest-impact risks. Encode them into templates and CI checks.

Create the golden path for new services. Embed health checks, observability SDK, and resource limit defaults.

Add CI gates that fail when:

  • Health check endpoints are missing.
  • Public services lack authentication annotations.
  • Databases lack backup policies.

Document the rules. Announce them. Give teams two weeks to migrate existing services.

The migration window matters. Existing services need time to adapt. Don’t block their deploy on day one. Give them a grace period. Then enforce.

Days 61–90: Add runtime signals and scorecards

Deploy the service scorecard. Pull data from observability. Show it in your architecture review dashboard.

Add runtime validation: alert when a service’s error rate exceeds its SLO, or when a queue starts backing up and no one is paged.

Retrospect. What policies caused friction without preventing actual problems? Remove or adjust them. What incidents happened that your guardrails should have caught? Add new checks.

This isn’t a one-time project. It’s a feedback loop. The architecture evolves. The guardrails evolve with it.

The real goal

Architecture reviews served a purpose when delivery was slow and teams were small. Today, they’re a bottleneck that produces documents nobody reads.

Executable architecture doesn’t mean architects stop designing. It means architects stop writing documents that gather dust and start defining rules that run every deploy.

The board should debate strategy, trade-offs, and new patterns. They shouldn’t spend three hours approving services that could have passed automated checks in minutes.

Build the guardrails. Keep them current. Trust the automation. Reserve human judgment for the decisions that actually require it.

Discussion

Join the conversation and share your thoughts

Discussion

0 / 5000