API Gateway & Data-Sync Platform

• An empty repository and a short window. The sync had to be live first and the gateway after it.
• Eight organisations whose schemas differ and keep changing, so nothing could be hand-maintained per organisation.
• Callers need the outcome of a write in the same request, which rules out a queue in front of the gateway.
• Management wanted full traceability, because many more integrations were expected to follow this one.
• A cloud security policy that applied to my own setup, including how credentials and keys could be created.
• A small, known group of internal developers as the callers, not the public.

I designed this, wrote the specifications and decision records, set the rules the code had to follow, ran it in production and reviewed what shipped. Much of the code was produced from written plans I specified and approved, rather than typed line by line by me. Design, contracts, data model and operations are the parts I can defend in depth, so those are the parts I describe here.

Contract by reflection

I rejected the idea of a hand-maintained schema. The gateway reads each organisation’s live field and layout definitions from the CRM itself, stores the raw response, then a normalised form, with a time-to-live. A write is validated against that before any call to the CRM is made. When an organisation adds a field, the contract updates itself.

Permissions you can read

An API key has a client and a set of scopes. A scope has four dot-separated segments: tool, brand, module and operation. Wildcards work from the brand segment down, never on the tool, and matching is additive, so a key holds exactly what its scopes add up to. A denied request gets a precise 403.

crm.brandA.deals.create     one action in one brand
crm.brandA.deals.*          any operation on one module
crm.brandA.*                any module in one brand

Wildcards are convenient and dangerous, because they widen silently. A new module appears and an existing key now covers it. So the model has controls built in: wildcard keys are flagged as privileged, the audit records the concrete action rather than the wildcard that allowed it, and an expansion report surfaces grants that have widened without anyone choosing it. Keys are self-service, but only for developers on a small allow-list.

One schema per pipeline

The first real use exposed three vendor data-model gaps. The one I led the redesign for: the schema endpoint returned one merged set of fields per module, but each organisation runs several pipelines with different fields and different stage values. I switched reflection to the layouts definition, re-keyed the cache by layout with a zero-downtime migration, and added an optional layout parameter to the schema endpoint. I ruled that writes keep validating against the union of all layouts, so existing behaviour was not silently tightened. That is permissive, and I recorded it as open.

The second gap was a catalog that over-promised: a module nested under its parent record was listed as if it supported ordinary create, read, update and delete. I contained it immediately with the permission model, by issuing keys scoped only to the modules that behave regularly, so irregular ones were hidden from that key’s catalog. The permanent fix is written up as an open issue.

Idempotency keys that tell the truth

A batch that was sent twice reported the same record as “created” twice. I questioned the duplicate and traced it to idempotency keys derived from content: an identical batch replayed the original response, and the counts were inflated. I changed upsert objects to random per-batch keys, which is safe because the target upserts anyway, and wrote a rule that objects that are not idempotent must keep content-derived keys.

An audit trail that is a debugging tool

For every record the pipeline stores the source payload, the transformed payload and the target’s response, in one transaction, with 30-day retention. I built it to a management directive for traceability, and then it paid for itself the first time something failed. The story is in the hard problems below.

Decision log excerpts

The project has 18 numbered decisions and 15 rules. Three that I would defend in an interview:

• No dbt. The analytics service already merges every finance organisation and can run SQL, so the transforms run there. A separate warehouse would be one more thing to operate. The cost: transforms are limited to what that SQL can express, and I would revisit the choice if multi-step modelling appears.
• Synchronous only. Callers need the outcome in the same request, so the gateway answers directly instead of queueing. That is a documented exception to my own queue-only rule. The cost: nothing buffers if the vendor is slow, so a circuit breaker and throttle stand in.
• Native field names. My first idea was canonical commands translated per connector. Once I saw that callers already spoke the CRM’s own field names, I narrowed it to native mode. Callers are coupled to the vendor’s names, and insulating them is explicitly out of scope.

Two of the rules made extension cheap. Configuration lives only in config files or the registry, and a new capability may only land through four defined extension points. The proof is that eight organisations were onboarded with one migration and no code.

• An empty repository to a production gateway in under two months.
• Eight organisations onboarded with one migration and no code changes.
• The validation gate stopped 843 of 1,345 source records before they reached the partner system, each with a recorded reason. 502 were valid.
• The first live push created 100 records.
• Four production issues found in the first real use.
• 262 automated Java tests and 18 Python tests.

I have not yet pulled request volume, rejection counts or latency out of the audit tables, so those numbers are left out until I have run the queries.

• Make writes validate against the specific layout, not the union of all of them, so a wrong stage value for one pipeline is rejected.
• Fix the catalog properly, so it never lists operations a module cannot perform, instead of relying on scoped keys to hide them.
• Run the measurements I listed: invalid writes stopped before the vendor, duplicates absorbed, latency per endpoint and running cost.