# SG/Sentinel Architecture And Data Flows: How It All Fits Together

**version** v0.27.58
**date** 18 May 2026
**from** Human (project lead)
**to** Architect, Developer (lead), Security, @Dev
**type** Arch brief

---

## What This Is

The architecture document for SG/Sentinel: how the components fit together, how data flows between them, where the rule engines sit, how the development environment connects, and specifically how the two first use cases (logging and blocking obviously-bad traffic) work end to end. This is the companion to the TUI mockups document; together they form the acceptance criteria the dev team implements against.

The voice memo scoped this clearly: **the first two use cases to implement are logging and blocking unwanted traffic that is obviously bad.** And it set an important boundary for logging: **Sentinel's job ends when the data lands in an S3 bucket. We already have code that picks up from there, so Sentinel does not worry about what happens next; it worries about getting the data into S3.**

This brief maps the architecture, the data flows, the rule engines across the layers, the development environment, and the two first use cases in concrete detail.

## The Component Map

The major components of SG/Sentinel and how they relate:

```
┌──────────────────────────────────────────────────────────────────┐
│                          THE INTERNET                              │
└────────────────────────────────┬───────────────────────────────────┘
                                  │
                                  ▼
┌──────────────────────────────────────────────────────────────────┐
│                       CLOUDFRONT (CDN)                             │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │  LAYER 1: CloudFront Function (sub-ms, no I/O)              │  │
│  │  - Deterministic rules (banned IPs/regions, malformed)      │  │
│  │  - Logging (every hit)                                       │  │
│  │  - Fast allow/block decision                                │  │
│  └──────────────────────────┬─────────────────────────────────┘  │
│                             │ (passes / blocks)                    │
│  ┌──────────────────────────▼─────────────────────────────────┐  │
│  │  LAYER 2: Lambda@Edge (network + files; on request/miss)    │  │
│  │  - Request validation against known-good profile            │  │
│  │  - App-coupled rules                                         │  │
│  │  - Cached threat-intel lookups                              │  │
│  └──────────────────────────┬─────────────────────────────────┘  │
└─────────────────────────────┼──────────────────────────────────────┘
                              │ (valid requests only)
                              ▼
┌──────────────────────────────────────────────────────────────────┐
│                    THE PROTECTED APPLICATION                       │
│              (vault servers, static sites, APIs)                   │
└──────────────────────────────────────────────────────────────────┘

         ASYNC / OUT-OF-BAND (not in the request path)
┌──────────────────────────────────────────────────────────────────┐
│  LAYER 3: Async functions / our compute                           │
│  - Analysis (over a window, not inline)                            │
│  - LLM interpretation (Nova; never inline)                         │
│  - Rule generation; evidence-graph building                        │
│  - Threat-intel enrichment                                         │
└──────────────────────────────────────────────────────────────────┘

         STORAGE / EVIDENCE
┌──────────────────────────────────────────────────────────────────┐
│  S3 (logs land here; existing code picks up from here)            │
│  Evidence graph (per-user/IP vaults)                               │
│  Rule-set (rules-as-vault, versioned)                              │
└──────────────────────────────────────────────────────────────────┘

         CONTROL / DEVELOPMENT
┌──────────────────────────────────────────────────────────────────┐
│  TUI + TUI API (operate, observe, develop)                         │
│  CLI (deploy, create, destroy, teardown)                           │
│  Local environment (same code as production; lambda-as-container)  │
└──────────────────────────────────────────────────────────────────┘
```

The components in words:

| Component | Role |
|-----------|------|
| **Layer 1 (CloudFront Function)** | Sub-millisecond first responder; deterministic rules; logs every hit; fast allow/block |
| **Layer 2 (Lambda@Edge)** | Capable layer; request validation; app-coupled rules; runs on request or cache miss |
| **Layer 3 (async / our compute)** | Off-hot-path; analysis, LLM interpretation, rule generation, evidence building |
| **S3** | Where logs land; the handoff point to existing downstream code |
| **Evidence graph** | Per-user/IP vaults of accumulated evidence |
| **Rule-set** | The rules, as a versioned vault |
| **TUI + TUI API** | Operate, observe, develop; chatbot-friendly |
| **CLI** | Deploy, create, destroy, teardown |
| **Local environment** | Same code as production, for development and testing |

## The Logging Data Flow (First Use Case)

The first use case is logging. The flow, end to end:

```
Request arrives at CloudFront
    │
    ▼
Layer 1 (CloudFront Function) sees the request
    │
    ├─► Assigns/records the request ID (ours + AWS)
    ├─► Captures the log data (method, path, headers, timing, etc.)
    ├─► Applies privacy rules (e.g. IP logging on/off per config)
    │
    ▼
Log data is formatted (clean, structured)
    │
    ▼
Log data is delivered to S3
    │
    ▼  ◄─── SENTINEL'S JOB ENDS HERE
Existing downstream code picks up from S3
(Sentinel does not worry about what happens next)
```

The key architectural decisions for logging:

| Decision | Detail |
|----------|--------|
| **Capture point** | Layer 1 (sees every hit) |
| **Format** | Clean, structured, well-formatted (not raw mess) |
| **Privacy** | Configurable (IP logging on/off; threat-triggered capture) |
| **Destination** | S3 bucket |
| **Handoff** | Sentinel's responsibility ends at S3; existing code takes over |
| **Replaces** | The CloudWatch + Firehose pipeline (cheaper, real-time, controlled) |

The voice memo was explicit about the boundary: **Sentinel worries about getting the data into S3, cleanly formatted; it does not worry about what happens downstream, because we already have code that picks up from S3.** This is a clean separation of concerns: Sentinel is the capture-and-deliver-to-S3 component; the existing pipeline is the consume-from-S3 component.

This boundary keeps the logging use case tightly scoped and achievable. Sentinel does not need to build log analytics, dashboards, or retention; it needs to capture cleanly and land in S3. Everything downstream already exists.

## The Blocking Data Flow (Second Use Case)

The second use case is blocking obviously-bad traffic. The flow:

```
Request arrives at CloudFront
    │
    ▼
Layer 1 (CloudFront Function)
    │
    ├─► Check against deterministic block rules:
    │     - Is the source IP/region banned? (embedded list)
    │     - Is the request obviously malformed?
    │     - Is it an obvious-bad pattern? (/etc/passwd, PHP scan
    │       on a static site, known exploit probe, scan signature)
    │
    ├─► If obviously bad:
    │     ├─► Log the block (with reason, to S3)
    │     └─► Block / drop / deflect (per the rule's action)
    │
    └─► If not obviously bad:
          └─► Pass to Layer 2 (or to the application)
    │
    ▼
Layer 2 (Lambda@Edge) for requests needing more analysis
    │
    ├─► Validate against the known-good profile (app-coupled)
    ├─► Apply opinion rules (anomaly scoring)
    │
    ├─► If invalid / over threshold: block + log
    └─► If valid: pass to the application
    │
    ▼  (async, out-of-band)
Layer 3 analyses patterns over a window
    └─► Updates Layer 1 embedded lists / Layer 2 rules
        (detect-before-damage; feeds the fast layers)
```

The key architectural decisions for blocking:

| Decision | Detail |
|----------|--------|
| **Easy wins at Layer 1** | Obvious-bad patterns blocked sub-millisecond (banned IPs, `/etc/passwd`, PHP scans on static sites) |
| **App-coupled validation at Layer 2** | Known-good-profile validation; the no-invalid-request principle |
| **Anomaly scoring** | Opinion rules accumulate; block over threshold (the CRS-proven model) |
| **Block reason logged** | Every block has a clear, logged reason (to S3) |
| **Async feedback** | Layer 3 detects attackers over a window; updates the fast layers |
| **Actions** | Block, drop, deflect (wild goose chase); per the rule |

The blocking use case starts with the easy wins (the deterministic-certain rules from the rule-architecture brief) because they are high-value and low-risk. A `/etc/passwd` request or a PHP-page scan on a static site is unambiguously bad and trivially blockable. These get built first; the app-coupled validation and anomaly scoring follow.

## The Rule Engines Across The Layers

Each layer runs rules, with the same rule architecture (the rule-architecture brief) but different capabilities:

| Layer | Rule Types | Rule Capabilities |
|-------|-----------|-------------------|
| **Layer 1** | Deterministic, fast | Embedded-data lookups; no I/O; sub-ms; the easy wins |
| **Layer 2** | Logic, file-dependent, app-coupled | Network/file access; known-good validation; anomaly scoring |
| **Layer 3** | LLM, analytical, generative | Async; interpretation; rule generation; evidence building |

The rule graph (fractal, from the rule-architecture brief) spans the layers: entry rules at Layer 1 may direct deeper analysis to Layer 2 or Layer 3. The rules carry their layer/altitude as metadata, and the engine places each rule at the right altitude.

The rule-set itself is a versioned vault. The production bundle (phase-aware, from the interactivity brief) contains exactly the rules production runs, at pinned versions. Building a deployment means assembling the phase-appropriate rule bundle for each layer.

## The Development Environment

The development environment runs the whole of Sentinel locally, identically to production (the interactivity and delegation briefs):

```
LOCAL DEVELOPMENT
┌──────────────────────────────────────────────────────────────────┐
│  Same code as production (lambda-as-container)                     │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐                   │
│  │  Layer 1   │  │  Layer 2   │  │  Layer 3   │  (all local)      │
│  └────────────┘  └────────────┘  └────────────┘                   │
│                                                                    │
│  Dangerous dev-only rules available (enable/disable, full         │
│  inspection, what-do-you-know-about-me)                            │
│                                                                    │
│  Develop rules → test locally → push to local Sentinel →          │
│  push to live Sentinel → no surprises (deploy parity)             │
└──────────────────────────────────────────────────────────────────┘
         │
         ▼  (promote through phases)
   dev ──► main (= QA) ──► prod (main = production)
```

The development environment enables the workflows from the interactivity brief: develop rules, test them locally and differentially (enable/disable), promote through dev/main/prod, with deploy parity guaranteeing production behaves like QA. The dangerous dev-only rules (full inspection, what-do-you-know-about-me) are available locally and excluded from production by the bundle build.

## How It All Fits Together: The Unified Picture

Pulling the components, flows, rule engines, and development environment together:

1. **Requests flow** through Layer 1 (fast, deterministic, logs everything), Layer 2 (capable, app-coupled validation), to the application (valid requests only).
2. **Logs land in S3**, cleanly formatted; existing code picks up from there. Sentinel's logging job ends at S3.
3. **Blocks happen** at Layer 1 (easy wins) and Layer 2 (app-coupled), with reasons logged.
4. **Async analysis** (Layer 3) runs out-of-band, building the evidence graph, generating rules, detecting attackers over a window, feeding the fast layers.
5. **Rules** are a fractal graph, layer-tagged, carrying rich metadata, version-controlled as a vault.
6. **The development environment** runs everything locally, identically to production, enabling rule development with deploy parity.
7. **The TUI and TUI API** (the mockups document) operate, observe, and develop the whole system, chatbot-friendly throughout.
8. **The CLI** handles deployment lifecycle (create, destroy, teardown).

The two first use cases (logging to S3, blocking obvious-bad) exercise the core of this architecture: Layer 1 capture and blocking, the S3 handoff, the rule engine, the deployment, and the TUI visibility. They are the minimal end-to-end slice that proves the architecture works.

## The Acceptance-Criteria Role

This document, with the TUI mockups document, is the acceptance criteria for the dev team. The voice memo: **this is the acceptance criteria that we will ask the dev team to implement.** The architecture here defines what to build; the mockups define what the operator-facing surfaces look like. Together they specify the MVP concretely enough to implement.

## What This Does Not Try To Be

Deliberate scope limits:

- **Not the downstream log pipeline.** Sentinel ends at S3; existing code handles the rest.
- **Not all layers fully built in the first use cases.** Logging and blocking exercise Layer 1 heavily, Layer 2 partially, Layer 3 minimally.
- **Not multi-CDN.** CloudFront first.
- **Not the full rule library.** The easy-win rules first.
- **Not SSL termination.** Later, per the execution-model brief.

## Honest Risks

Three risks:

**Risk 1: The S3 handoff could become a bottleneck or cost issue.** High log volume to S3 has cost and rate implications. Mitigation: batch and compress; the logging is leaner than Firehose by design; measure.

**Risk 2: The layer boundaries could blur in implementation.** Without discipline, logic could leak between layers. Mitigation: the rule metadata (layer/altitude) is explicit; the horses-for-courses placement discipline; review.

**Risk 3: The two use cases could expand in scope.** Logging and blocking can each grow unbounded. Mitigation: the tight scoping (logging ends at S3; blocking starts with easy wins); resist scope creep in the MVP.

## Open Questions

| Question | Notes |
|----------|-------|
| S3 bucket structure for logs? | Connect to the existing downstream code's expectations |
| Layer 1 language: JavaScript (CloudFront Functions constraint)? | Yes, Layer 1 is JS; Layer 2 Python preferred |
| How are the embedded block-lists updated? | Redeploy the Layer 1 function; measure propagation |
| Anomaly-scoring threshold for blocking? | Start CRS-like; tune from traffic |
| How much Layer 3 in the first use cases? | Minimal; the hook exists; full async later |
| Where does the evidence graph live? | Per-user/IP vaults; the vault substrate |

## Relationship To Previous Briefs

| Date | Document | Relationship |
|---|---|---|
| 18 May | `v0.27.58__arch-brief__sg-sentinel-rule-architecture-strategy.md` | The rule graph this architecture runs |
| 18 May | `v0.27.58__dev-brief__edge-layer-mvp-visibility-blocking-deployment.md` | The MVP this architecture details |
| 18 May | `v0.27.58__arch-brief__edge-layer-execution-model-layered-responders.md` | The layers this maps |
| 18 May | `v0.27.58__arch-brief__sg-sentinel-interactivity-and-deployment-phases.md` | The development environment and phases |
| 18 May | `v0.27.58__arch-brief__sg-sentinel-time-as-first-class-dimension.md` | The async detect-before-damage feedback |
| 18 May | `v0.27.58__arch-brief__sg-sentinel-developer-friendliness-and-evidence-graph.md` | The evidence graph in the storage layer |
| 17 May | `v0.27.55__arch-brief__tui-api-structured-surface-for-text-uis.md` | The TUI + TUI API control surface |
| 16 May | `v0.27.45__dev-brief__on-demand-vault-provisioning-workflows.md` | The CloudFront plumbing |

---

## Acceptance Criteria

| # | Criterion | Verification |
|---|-----------|-------------|
| 1 | The component map is implemented (Layers 1/2/3, S3, evidence, rule-set, TUI, CLI) | All components exist |
| 2 | The logging flow works (capture at Layer 1, format, deliver to S3) | Logs land in S3 cleanly |
| 3 | Sentinel's logging job ends at S3; existing code picks up | Clean handoff |
| 4 | The blocking flow works (easy wins at Layer 1, app-coupled at Layer 2) | Obvious-bad blocked |
| 5 | Every block has a logged reason | Reasons in S3 |
| 6 | The rule engines run across the layers with the same architecture | Uniform rule model |
| 7 | The development environment runs everything locally (deploy parity) | Local = production |
| 8 | The async Layer 3 feedback hook exists | Architecture leaves room |
| 9 | The CLI handles the deployment lifecycle | Create/destroy/teardown |
| 10 | This + the mockups serve as implementable acceptance criteria | Dev team can build from it |

---

This document is released under the Creative Commons Attribution 4.0 International licence (CC BY 4.0).