Multi-Tier Permission Hierarchy

Have you ever looked at an authorization latency dashboard and asked yourself what portion of that time is actually spent evaluating security rules, and what portion is just pure architectural tax? I know this sounds like an obvious question about database indexing - and on paper, checking whether User A can perform Action B on Resource C looks like a single dictionary lookup. But once a system splits from a monolith into microservices, and roles stop being flat tags and turn into deeply nested organizational trees, what was once a trivial in-memory check quietly turns into the single heaviest bottleneck across your entire request pipeline.

When our architecture was split into microservices, authorization was centralized behind a dedicated auth service. On paper, this was clean domain boundary design. In production under load, it produced the classic "Chatty Microservices" crisis: every single incoming HTTP request across twenty downstream services had to make a synchronous gRPC call back to the auth service just to verify basic permissions. At peak traffic, the auth service wasn't just another dependency - it was the single point of failure where database connection pools saturated, CPU spiked, and p99 latency ballooned past 32ms.

Dropping that p99 latency to 2ms wasn't a matter of tweaking database indexes or throwing hardware at the problem. It required rethinking the entire evaluation pipeline: moving hot checks to a shared read cache, optimizing deep tree traversals in the persistence layer, and building guards against the most dangerous failure mode in caching - the cold-start stampede.

One thing to keep straight while reading, because it changes what each number means. The 32ms → 2ms figures are the production outcome, and production telemetry can't be published here. Every number after this point is from the runnable lab linked at the end - real MongoDB, Redis and Neo4j in containers on one machine - and is there to show the mechanism, not to restate production. Where the two would disagree, the lab is the one you can check.

The system context and constraints

The domain model required hierarchical Role-Based and Relationship-Based Access Control (RBAC/ReBAC). Permissions were rarely assigned directly to users; instead, users inherited permissions through complex organizational trees (departments, teams, project groups, and functional roles) spanning up to 6 levels of depth.

Evaluating whether a leaf user has a specific permission required a bottom-up traversal: starting at Level 6, walking all parent nodes up to the root, aggregating permission sets at each level, and resolving overrides.

Going into the redesign, the architectural constraints were strict:

• p99 Latency SLA: $\le 2\text{ms}$ on hot paths, $\le 32\text{ms}$ on worst-case cold cache misses under concurrent load.
• Scale: Over 5,400 role nodes and 4,000+ active users per tenant, with thousands of permission checks per second.
• Cache Invalidation Correctness: When an admin revokes a role or reassigns a permission, the "stale window" where a revoked permission remains usable must not exceed 100ms.
• Zero Cold-Start Stampedes: If a high-traffic cache key expires or is invalidated, thousands of concurrent requests must not cascade into the database and crash the upstream service.

The three-tier architecture

To satisfy these constraints, we structured the evaluation pipeline into three distinct tiers, each optimized for a completely different access pattern:

Tier 1: Redis PEP read-cache (the hot path)

The API Gateway acts as the Policy Enforcement Point. Instead of an RPC to the auth service on every request, it checks one flattened Redis Set, user:{userId}:permissions, with a single SISMEMBER.

• Measured: 0.128 - 0.208 ms per check in the lab, one round trip at a time.
• Role: absorbs 98%+ of evaluation volume, and - the part that matters more than the latency - skips the entire application pipeline behind it.

Tier 2: Neo4j graph persistence (mutation-path validator)

Neo4j is not on the read path. Its one job is answering a read-only question before a tree mutation commits: would this reparent create a cycle?

• Measured: a cycle rejection comes back in 2 - 3 ms in the lab.
• Role: pre-commit validator, nothing else. It never persists the mutation, and as shown below, it is not even the only cycle guard - it cannot be, because its copy of the tree can lag.

Tier 3: MongoDB Document Store (Sole Source of Truth & Ancestors Array)

MongoDB is where every mutation actually becomes durable. The naive way to resolve inheritance here - a while loop fetching one parent at a time - costs one network round trip per level, so a 6-level tree pays 7 sequential trips before it can answer anything. The depth is small; the round trips are what hurt, and they cannot be parallelised because you don't know the next parent until the current one comes back.

Why three stores instead of one

The honest version of this question is: why not just Postgres, with ltree or a recursive CTE, and skip Redis and Neo4j entirely? We looked at that first, because one fewer moving part is always the better default.

It falls apart on two things at once. A recursive CTE re-walks the tree on every read - fine when traffic is low, but it does work proportional to the depth on each request, where the ancestors array does one indexed lookup against a list it already has. Neither is free, and the honest framing is that no persistence engine reaches the hot-path SLA on its own. That's the real reason Tier 1 has to exist, no matter which document store sits behind it. Tree mutations are a separate problem - moving a department, checking whether a move would create a cycle. That's exactly the kind of query relational recursion is bad at, and graph traversal is built for. Neo4j earns its place on the mutation path for that reason alone. It's not there to make reads faster.

That leaves a question the earlier draft of this article dodged: if Neo4j checks tree mutations and MongoDB is the "source of truth," which one actually wins when a write happens? MongoDB does. Deliberately, and only MongoDB. MoveRole runs as two steps, not a two-database transaction: first Neo4j gets asked a read-only question - would this move create a cycle - and it answers without writing anything. Only once that comes back clean does MongoDB do the real write: update the role, insert an OutboxEvent, both in the same transaction. The Outbox Relay then replays that event to catch Neo4j's copy up. Neo4j never gets a write that MongoDB hasn't already committed first.

This isn't just tidiness. Two separate databases can't share a transaction. If Neo4j committed first, a crash between that write and the Mongo write would leave the two disagreeing forever, with nothing to say which one is right. Doing Mongo first removes that failure mode completely - Neo4j's graph isn't a second ledger, it's just a copy you can always rebuild. Lose it, corrupt it, wipe it to zero, and a single pass over the roles collection brings it back exactly. That's the real test for source of truth, not which store gets asked first - which one everything else can be rebuilt from.

Reads stay simple through all of this. If a MoveRole is mid-flight, or the outbox hasn't reached Neo4j yet, a permission check just sees the old tree - a little stale, never broken or half-written.

Two claims above matter too much to take on faith: that Neo4j is really disposable, and that doing Mongo first is actually safe. The runnable reproduction linked at the end of this article tests both, against real MongoDB and real Neo4j. First: it wipes Neo4j to zero and rebuilds the graph using nothing but MongoDB's role documents, then checks the node and edge counts match what was there before the wipe. Second: it runs two mutations back to back, holding back the first one's Neo4j resync on purpose, then checks whether the second mutation's cycle check - answered against that now-stale Neo4j - waves through something the current, already-committed MongoDB state would call a cycle.

Both come back the way the design claims. The wipe-and-rebuild ends at the exact node and edge counts it started with, every run - Neo4j really is disposable.

The race is worth drawing, because it only takes two mutations and one lagging copy:

Mutation A has committed in MongoDB. Neo4j's resync is still in flight, so the two stores briefly disagree about where Staff lives:

Mutation B now arrives asking: may I move NewDept under Staff? The same question, put to the two stores, gets opposite answers:

Neo4j isn't wrong here, and that's what makes it dangerous: it answers the question correctly, about a tree that stopped being current a few milliseconds ago. With only Neo4j guarding, mutation B is approved and a genuinely cyclic ancestors array lands in MongoDB. Not a hypothetical - an actual broken document sitting in the lab right now. The fix is free: newParentDoc.ancestors is already sitting in memory from the same read that fetched the role, and if the role being moved is already in that list, it's a cycle, no matter what Neo4j's copy says. Check that first and the gap closes before Neo4j is ever asked. For plain ancestry cycles it's just a better guard than the graph query is - which is also the honest answer to what Neo4j is actually for here: the complex ReBAC relationships an ancestors array can't express, not the simple tree cycles it can.

The persistence bottleneck: moving from recursive lookup to ancestors array

Before our refactor, fetching a role's full inheritance chain in MongoDB used a sequential loop fetching parent IDs one by one, or an unindexed $graphLookup aggregation. Under a 6-level tree with 5,460 roles, this was disastrous:

// BEFORE: Sequential parent traversal (Naive while-loop)
async getAncestorsNaive(roleId: string): Promise<string[]> {
  const ancestors: string[] = [];
  let currentId: string | null = roleId;
  while (currentId) {
    const role = await this.roleModel.findById(currentId).lean(); // N network round-trips!
    if (!role || !role.parentId) break;
    ancestors.push(role.parentId);
    currentId = role.parentId;
  }
  return ancestors;
}

Each iteration of that loop is a full network round trip, and the next one can't start until the current one returns. In the lab, one evaluation costs 10.4-12.0 ms this way, and under 50 concurrent callers the whole batch takes 208-276 ms - roughly 181-240 evaluations per second, for a check that is supposed to happen on every request.

We replaced this with the Materialized Path (Ancestors Array) Pattern: every role document stores an indexed array of all its ancestor IDs up to the root.

// AFTER: Materialized Path with Ancestors Array
export interface RoleDocument {
  id: string;
  name: string;
  parentId: string | null;
  ancestors: string[]; // e.g. ["root_01", "org_02", "dept_03", "team_04"]
  permissions: string[];
}

// Single-query resolution: fetch all ancestor documents in 1 indexed round-trip
async getEffectivePermissions(roleId: string): Promise<string[]> {
  const targetRole = await this.roleModel.findOne({ id: roleId }).lean();
  if (!targetRole) return [];

  const allRoleIds = [targetRole.id, ...targetRole.ancestors];
  const ancestorDocs = await this.roleModel
    .find({ id: { $in: allRoleIds } })
    .select('permissions')
    .lean();

  const permissionSet = new Set<string>();
  for (const doc of ancestorDocs) {
    for (const perm of doc.permissions) {
      permissionSet.add(perm);
    }
  }
  return Array.from(permissionSet);
}

What the lab actually measures

Everything below comes from the runnable lab linked at the end, against real MongoDB 7, real Redis 7 and real Neo4j 5 in containers, on a 6-level tree of 5,461 role documents. Ranges are the spread across three consecutive runs. Nothing here is a production number - see the limitations section for why.

One query instead of seven round trips. Resolving one leaf user's effective permissions:

That is 3.4-3.8× faster, and the reason is round trips, not query cleverness: seven sequential network hops collapse into one.

The gap widens under concurrency. Fifty evaluations fired at once:

The L1 cache is a different order of magnitude. A Redis SISMEMBER check measures 0.128-0.208 ms per operation, roughly 4,800-7,800 checks/sec. Worth reading carefully: that is one sequential connection doing one round trip at a time, not a pipelined throughput ceiling. The number that matters is the ratio - an L1 hit costs about 1/20th of a cold MongoDB resolution, and skips the entire application pipeline described in the next section.

Coalescing holds under a 50× surge. Fifty concurrent requests for the same uncached user produce exactly 1 MongoDB query; the other 49 wait on the leader's result. Total elapsed: 16-19 ms. Not "approximately one" - the lab counts the queries and the count is 1, every run.

One number here is deliberately absent. There is no Neo4j read benchmark, because Neo4j is not on the read path in this design - it validates mutations. Its measured numbers appear below, where it actually does work.

The application overhead reality: why frameworks are slower than raw databases

There's an honest distinction worth naming between "raw database query time" and "actual end-to-end HTTP response time." In our raw database benchmarks, querying the ancestors array in MongoDB took only $1.14\text{ms}$. But inside the live NestJS authentication service, end-to-end request latency measured between $8\text{ms}$ and $15\text{ms}$.

Why the gap? A junior engineer might assume the database is misconfigured or that the network is slow. But profiling the application under load reveals that over 85% of total request time is consumed by framework management, context propagation, domain mapping, and policy evaluation, not raw database I/O.

A production authorization service is a domain engine, not a database proxy. Each band in that chart is a layer doing real work. The useful question about each one is not "how slow is it" - you can read that off the chart - but "what breaks if I delete it?", because that is what separates a tax you could optimise away from one that is structural.

Six layers, in the order the request meets them.

1. The framework pipeline (~2ms). Before your code runs, NestJS has to work out which code to run. It reads the annotations on the route - @UseGuards(), @Permissions() - to discover what this endpoint requires, wraps the raw HTTP or gRPC request in one uniform object so the same handler works for both protocols, runs every global interceptor (logging, tracing, response shaping), then validates the request body field by field against its declared type. Doing that by reading annotations at runtime is what makes the framework convenient, and it is also why it is not free. Delete it? No. This is the routing and the validation. Deleting it means writing it again by hand.

2. Context propagation (~2ms, overlapping). In a multi-tenant system, almost every function needs to know which tenant is asking - but you don't want to pass tenantId through forty function signatures to get it there. Node's AsyncLocalStorage solves that by attaching the context to the async call chain itself, so any function can ask for it. The cost is that the runtime must now track that chain across every single await. Delete it? No. Tenant isolation is a security boundary; losing it is not a performance win.

3. Turning database bytes into domain objects (~3ms). MongoDB returns BSON - a binary format. That has to become JavaScript objects, and then those raw objects have to become domain objects: the role entity, its permission set, with the business rules checked as they're built. Two conversions, one after the other, and the second one allocates a lot of small objects. Delete it? Partly. Skipping the ORM's document-wrapping layer (.lean()) removes real overhead. Skipping the domain layer removes the invariant checks, which is a different thing than making it faster.

4. Compiling the policy (~2ms). A static role check only answers "does this role have docs:write?". Real authorization asks "can this user edit this document, given that it belongs to that department and isn't locked?" - a rule with conditions in it. Those conditions are stored as data, so on each check they have to be turned back into something executable and then run against this particular user and this particular document. Delete it? No. This is the authorization decision. Everything else in the pipeline exists to make this step possible.

5. Crossing the network (~2ms, split across the request). The gateway and the auth service are separate processes, so every call is serialised to Protobuf, framed over HTTP/2, decoded on the other side, and the same again for the response - including translating error states across the boundary. Delete it? Not while they are separate services. Merging them removes this cost and reintroduces the coupling the split was for.

6. Garbage collection (spread across everything). All five layers above allocate short-lived objects, per request. Under load, V8 spends real time reclaiming them, and that time lands unpredictably - which is why it shows up in p99 rather than in the average. Delete it? No. It is the consequence of the other five, not a layer you can remove on its own.

Read the "delete it?" answers together and the conclusion writes itself: almost none of this is removable. A secure auth service must validate, must isolate tenants, must evaluate real policies. Trying to make that whole pipeline finish in a fraction of a millisecond isn't an optimisation problem, it's a category error.

Which is exactly why the answer is a cache and not a faster service. An L1 hit at the gateway doesn't make this pipeline quicker - it never enters it. No gRPC hop, no BSON conversion, no policy compilation, because the answer was computed once already and stored as a flat set. The ~12ms is still there, waiting, for whoever misses the cache. The architecture's whole job is making sure that's 2 requests in 100 rather than 100.

Cache invalidation: the transactional outbox pattern

A multi-tier cache is only as good as its invalidation strategy. If an administrator revokes a user's role in MongoDB, how do we guarantee that L1 Redis immediately purges the cached permissions without creating dual-write inconsistencies?

We implemented the Transactional Outbox Pattern:

1. When a role mutation occurs, the updated role and an OutboxEvent are written to MongoDB within the same atomic transaction.
2. An Outbox Relay tails the change stream and publishes a RolePermissionUpdatedEvent to Kafka.
3. Two independent consumers subscribe to that same event: the Gateway Cache Invalidator (below), and a Neo4j Sync worker that replays the edge change into the graph - the same resync path described in the three-tier discussion above. Neither consumer's failure blocks the other; a Neo4j sync lag doesn't stall cache invalidation, and vice versa.
4. The Gateway Cache Invalidator consumes the event and deletes the affected users' cache entries.

That last step has a trap in it worth naming, because the obvious implementation is wrong. The tempting version is a Lua script that pattern-matches the keys to delete:

-- DO NOT DO THIS
local keys = redis.call('KEYS', 'user:*:permissions')
for i = 1, #keys do redis.call('DEL', keys[i]) end

KEYS walks the entire keyspace, and Redis runs commands on a single thread - so on a cache holding millions of entries this blocks every other client for the duration, turning a cache invalidation into a site-wide stall. It is also rejected outright on Redis Cluster, where a script may only touch keys declared up front.

The fix is to stop searching for the affected users and start already knowing them. Alongside user:{userId}:permissions, keep the mapping in the other direction too - role:{roleId}:members, a Redis Set listing every user who inherits that role. It is written by the same code path that already maintains the tree, so it costs nothing extra to keep current. Now invalidation doesn't have to look for anything:

-- Delete exactly the users this role affects, nothing scanned
local members = redis.call('SMEMBERS', KEYS[1])   -- role:{roleId}:members
for i = 1, #members do
  redis.call('DEL', 'user:' .. members[i] .. ':permissions')
end

Now the cost is proportional to the number of users actually affected, not to how big the cache happens to be - which is the difference between an invalidation that gets cheaper as it becomes more targeted and one that gets more expensive as the system succeeds. That keeps the stale window inside the 100ms budget without any periodic polling.

Defending against cold-start stampedes

The final piece of the puzzle is handling Cache Stampedes (Thundering Herd).

Consider what happens when a top-level role with 5,000 active users is invalidated: at peak traffic, hundreds of concurrent requests for those users will simultaneously encounter an L1 cache miss. If all 500 requests race directly into MongoDB, connection pools saturate, query latencies spike to seconds, and the database collapses.

To prevent this, we integrated Strict Request Coalescing (the singleflight pattern we explored in our previous note) at the cache-fill boundary:

When 500 concurrent requests miss the cache for the same user or role, exactly one request (the Leader) queries MongoDB. The other 499 requests wait for the leader's result in Redis and resolve within milliseconds. The database load remains perfectly flat regardless of concurrency spikes.

Experimental limitations and open questions

It is worth being explicit about what these numbers are - and what they are not.

Because strict non-disclosure obligations prevent publishing internal production telemetry, live tenant topologies, or proprietary cluster metrics, the benchmarks reported above were measured in a sanitized, single-machine reproduction environment running containerized instances on local loopback. The point of the benchmark is not to boast about raw hardware speeds or claim that local numbers translate linearly into a globally distributed production environment with cross-datacenter jitter. The point is to provide an apples-to-apples, reproducible comparison of one specific thing: what happens when you replace seven sequential round trips with one indexed lookup, under identical concurrency, on identical hardware.

Three places where this design stops being obviously right, and what would have to change:

The first of those is the one I actually find interesting, because it is the same shape as a problem I went and measured separately: choosing a parameter that has no single correct value under a workload that keeps changing.

Residual trade-offs and lessons learned

Every architecture is a balance of trade-offs, and being honest about what you gave up is just as important as celebrating what you gained:

1. Write Amplification on Tree Re-parenting: The Ancestors Array pattern optimizes reads at the cost of writes. If an administrator moves a department containing 1,000 sub-roles to a new parent, the system must rewrite the ancestors array of all 1,000 documents. Nothing about keeping Neo4j around helps with this - Neo4j never persists the mutation, so the write amplification is MongoDB's either way. What it actually buys is time: bulk re-parenting runs asynchronously in the background, and reads keep seeing a consistent older tree until it lands.
2. Network Topology Matters More Than Raw DB Speed: A database query that takes 1ms in isolation will still feel slow if your microservices are making 10 sequential network calls across distinct subnets. Colocating Redis on the same high-speed internal VPC network as the API Gateway was critical to hitting the sub-2ms p99 SLA.
3. Never Cache Permissions Without Coalescing Misses: Building a fast cache without singleflight protection is building a trap for yourself. The faster your cache makes your system during normal operations, the more catastrophic the failure will be when that cache goes cold.

Runnable Reproduction & Source Code

Every measured number in this article comes from here. A complete, runnable laboratory covering the ancestors array pattern, the naive sequential-traversal baseline, request-coalescing cache-fill, and the stale-Neo4j mutation race - against real MongoDB 7, Redis 7 and Neo4j 5 in containers:

• engineering-notebook-labs/permission-cache