Payment System Design — 1 Hour Interview Guide

Design a reliable payment system like Stripe. Covers exactly-once processing, idempotency keys, payment state machine, Saga pattern, double-entry ledger, webhook delivery, circuit breakers, multi-processor failover, reconciliation, and PCI-DSS compliance — all in a 1-hour interview format.

Payment System Design

1-Hour Interview Roadmap

Time	Topic
0 – 5 min	Requirements clarification
5 – 10 min	Capacity estimation
10 – 18 min	High-level architecture + core components
18 – 28 min	Payment processing flow — synchronous + asynchronous
28 – 36 min	Exactly-once processing — idempotency keys + state machine
36 – 44 min	Failure handling — retries, circuit breakers, failover
44 – 50 min	Database design + double-entry ledger
50 – 56 min	Webhooks + reconciliation
56 – 60 min	Trade-offs + failure scenarios

What Are We Building?

A payment processing platform (similar to Stripe) that:

Accepts payment requests from merchants and processes them through external payment processors
Guarantees exactly-once processing — no duplicate charges, no lost transactions
Handles idempotent retries safely when network failures cause merchants to retry
Notifies merchants of payment outcomes via webhooks
Maintains a complete, immutable double-entry ledger for financial compliance

Scale reference: Stripe processes ~1 billion transactions per year, handles 250 million API calls per day, and routes payments through Visa, Mastercard, and 100+ global payment processors. Design for 10 million transactions/day as our target scale.

Key unique challenges:

Exactly-once guarantee — unlike most distributed systems that accept at-least-once, payment systems cannot afford duplicate charges
Ambiguous processor responses — the processor may have charged the card but we never got confirmation; how do we know?
Zero tolerance for data loss — every transaction must be recorded before any response is returned
Financial compliance — double-entry ledger, PCI-DSS, audit trails

Step 1 — Requirements

Functional Requirements

#	Requirement
1	Merchants can create a payment with amount, currency, payment method, and an idempotency key
2	System processes the payment through an external payment processor (Stripe, Adyen)
3	Each payment is processed exactly once — retries with the same idempotency key return the original result
4	Merchants can query the status of any payment at any time
5	Merchants can initiate full or partial refunds for completed payments
6	System delivers real-time webhook notifications on payment events (success, failure, refund)
7	System maintains a double-entry ledger for every financial movement
8	System reconciles its records against processor settlement reports daily

Non-Functional Requirements

#	Requirement
1	Zero data loss — no transaction can be lost even if a service crashes
2	Exactly-once processing — no duplicate charges, no missed payments
3	High availability — 99.99% uptime (payment failures directly cost merchants money)
4	API response time < 2 seconds (p99)
5	Strong consistency — merchants see accurate status immediately after any change
6	PCI-DSS compliant — card data encrypted and tokenized; never stored raw

Out of Scope

Card tokenization and vault (assume tokenization done by a PCI-DSS vault service)
Subscription and recurring billing engine
Fraud detection ML pipeline
Currency conversion and FX rates
Marketplace payout splits

Step 2 — Capacity Estimation

Traffic Estimates

Daily transactions:     10 million
Avg TPS:                10M / 86,400 sec  = ~115 TPS
Peak TPS (10x):         ~1,150 TPS

Read/write ratio:       ~5:1
  Writes: 10M payment creations + 1M refunds/day  = ~127 TPS
  Reads:  Status checks + dashboard + webhooks     = ~580 QPS

Peak read load:         ~5,800 QPS

Storage Estimates

Per transaction:
  Transaction record:              ~500 bytes
  Ledger entries (2-4 per txn):    ~800 bytes
  Event log entries:               ~300 bytes
  Total per transaction:           ~1,600 bytes

Annual storage:
  10M txn/day × 1,600 bytes × 365 days = ~5.8 TB/year

Idempotency keys (Redis, 48h TTL):
  10M keys × ~200 bytes = ~2 GB active at any time

Key Insights

Write throughput is modest — 115 TPS average (1,150 peak) is well within PostgreSQL's capacity
Reads dominate — 5:1 read/write ratio; read replicas absorb reporting and status queries
Storage is predictable — ~6 TB/year enables precise capacity planning and archival scheduling
Peak handling matters — 10x peak factor requires queuing to protect against burst traffic

Step 3 — High-Level Architecture

flowchart TD
    MERCHANT[Merchant Server] --> AG[API Gateway\nAuth + Rate Limiting + TLS]

    AG --> PS[Payment Service]
    AG --> RS[Refund Service]

    PS --> IDEM[(Redis\nIdempotency Store)]
    PS --> PDB[(PostgreSQL\nPayment Database)]
    PS --> KF[Kafka\nPayment Queue]

    KF --> PW[Payment Worker]
    PW --> PSP[PSP Gateway\nProcessor Abstraction]
    PSP --> STRIPE[Stripe]
    PSP --> ADYEN[Adyen]
    PW --> PDB
    PW --> EQ[Kafka\nEvent Queue]

    EQ --> WS[Webhook Service]
    WS --> ME[Merchant Endpoint]

    PDB --> LDB[(PostgreSQL\nLedger Database)]

    style IDEM fill:#fff4e0,stroke:#f59e0b
    style KF fill:#ede9fe,stroke:#7c3aed
    style EQ fill:#ede9fe,stroke:#7c3aed
    style PDB fill:#f0fdf4,stroke:#16a34a
    style LDB fill:#f0fdf4,stroke:#16a34a

Why Synchronous + Asynchronous Split?

The architecture separates two paths:

Path	What It Does	Latency	Why
Synchronous	Validate → store → queue → respond	< 500ms	Merchant gets immediate acknowledgment
Asynchronous	Worker calls processor → update status → webhook	1–5 sec	External processors are slow and unreliable

External payment processors can take 1–5 seconds to respond. If the API blocked until the processor responded, merchants would see inconsistent latency and frequent timeouts. By returning immediately with status: processing and handling processor communication asynchronously, we provide predictable API latency while guaranteeing reliable processing.

Component Responsibilities

Component	Responsibility
API Gateway	Merchant authentication (API key), rate limiting, TLS termination
Payment Service	Validate request, check idempotency, create payment record, enqueue
Refund Service	Validate refund eligibility, create refund record, enqueue
Idempotency Store	Redis — fast duplicate detection; 48-hour TTL per key
Payment Database	PostgreSQL — payment records, events, refunds (ACID)
Ledger Database	PostgreSQL — double-entry ledger entries (append-only)
Kafka (Payment Queue)	Decouple payment acceptance from processor communication
Payment Worker	Consume queue, call PSP, update payment status
PSP Gateway	Abstraction layer over multiple external processors
Kafka (Event Queue)	Decouple payment outcomes from webhook delivery
Webhook Service	Deliver events to merchant endpoints with retry and exponential backoff

Step 4 — Payment Processing Flow

Full Payment Lifecycle

sequenceDiagram
    participant M as Merchant
    participant AG as API Gateway
    participant PS as Payment Service
    participant IDEM as Redis (Idempotency)
    participant PDB as PostgreSQL
    participant KF as Kafka
    participant PW as Payment Worker
    participant PSP as PSP Gateway

    M->>AG: POST /v1/payments {idempotency_key, amount, currency, payment_method}
    AG->>PS: Forward validated request

    PS->>IDEM: SET NX key:{merchant_id}:{idempotency_key} TTL=48h
    IDEM-->>PS: OK (new key — proceed) or EXISTS (duplicate — return cached)

    PS->>PDB: INSERT payment {status: PENDING}
    PDB-->>PS: payment_id

    PS->>KF: Publish payment.created {payment_id}
    PS-->>M: 201 {payment_id, status: "processing"}

    Note over PW,PSP: Asynchronous — happens after HTTP response
    KF->>PW: Consume message
    PW->>PDB: UPDATE payment status → PROCESSING
    PW->>PSP: Charge card {amount, card_token, idempotency_key}
    PSP-->>PW: {approved: true, processor_txn_id: "ch_xyz"}

    PW->>PDB: UPDATE payment status → SUCCEEDED, processor_txn_id
    PW->>PDB: INSERT ledger entries (debit customer, credit merchant, credit fees)
    PW->>KF: Publish payment.succeeded event

    KF->>WS: Deliver webhook to merchant

API Design

POST /v1/payments

{
  "idempotency_key": "550e8400-e29b-41d4-a716-446655440000",
  "amount":          5000,
  "currency":        "USD",
  "payment_method":  "pm_card_visa_xxxx4242",
  "merchant_id":     "merch_abc123",
  "description":     "Order #12345"
}

201 Created response:

{
  "payment_id": "pay_9f8e7d6c5b4a",
  "status":     "processing",
  "amount":     5000,
  "currency":   "USD",
  "created_at": "2024-01-15T10:30:00Z"
}

Why amounts in smallest currency unit? 5000 = $50.00 in USD. Using integers (cents) avoids floating-point precision errors and handles currencies with 0, 2, or 3 decimal places uniformly.

GET /v1/payments/{payment_id}

{
  "payment_id":            "pay_9f8e7d6c5b4a",
  "status":                "succeeded",
  "amount":                5000,
  "currency":              "USD",
  "processor_transaction_id": "ch_xyz789",
  "created_at":            "2024-01-15T10:30:00Z",
  "updated_at":            "2024-01-15T10:30:03Z"
}

POST /v1/refunds

{
  "idempotency_key": "refund-order-12345-001",
  "payment_id":      "pay_9f8e7d6c5b4a",
  "amount":          2500,
  "reason":          "Customer returned item"
}

Step 5 — Payment State Machine

Every payment transitions through a strict finite set of states. Enforcing valid transitions at the database level prevents corrupted state.

stateDiagram-v2
    [*] --> PENDING : Payment record created
    PENDING --> PROCESSING : Worker sends to processor
    PROCESSING --> SUCCEEDED : Processor approves
    PROCESSING --> FAILED : Processor declines or times out
    SUCCEEDED --> PARTIALLY_REFUNDED : Partial refund issued
    SUCCEEDED --> REFUNDED : Full refund issued
    PARTIALLY_REFUNDED --> REFUNDED : Remaining amount refunded
    FAILED --> [*]
    REFUNDED --> [*]

State Transition Rules

Current State	Allowed Next States	Trigger
PENDING	PROCESSING	Worker picks up and sends to processor
PROCESSING	SUCCEEDED, FAILED	Processor returns a definitive response
SUCCEEDED	PARTIALLY_REFUNDED, REFUNDED	Merchant initiates refund
PARTIALLY_REFUNDED	REFUNDED	Merchant refunds remaining amount
FAILED	(terminal)	No transitions allowed
REFUNDED	(terminal)	No transitions allowed

Why is FAILED a terminal state? Once a payment fails, the merchant must initiate a new payment request with a new idempotency key. Automatic retries after failure create confusion about which attempt succeeded and remove merchant control.

Safe State Transition (Optimistic Locking)

-- Only update if still in expected state — prevents race conditions
UPDATE payments
SET    status     = 'SUCCEEDED',
       processor_transaction_id = 'ch_xyz789',
       updated_at = NOW()
WHERE  payment_id = 'pay_9f8e7d6c5b4a'
  AND  status     = 'PROCESSING';    -- ← optimistic lock

-- If 0 rows affected → another worker already updated this payment → log and investigate

If two workers try to update the same payment simultaneously (which should never happen in a well-designed system), the second update affects zero rows. This database-level check is the final safety net.

Step 6 — Exactly-Once Processing and Idempotency

Idempotency is the single most important property of a payment system.

Why Idempotency Is Critical

Every one of these real-world scenarios causes duplicate payment requests:

Scenario	How It Happens	Without Idempotency
Network timeout	Merchant's HTTP client times out at 30s	Customer charged twice
Load balancer retry	Backend seemed slow; LB retried	Customer charged twice
Mobile app double-tap	User clicked "Pay" twice quickly	Customer charged twice
Server crash mid-request	Merchant's server crashed; client retried	Customer charged twice
Cloud provider network hiccup	Intermittent connectivity drops	Customer charged twice

How Idempotency Works

flowchart TD
    REQ[Request arrives\nidempotency_key = X] --> CHECK{Redis: does\nkey X exist?}
    CHECK -->|No| SET[SET NX key X\nstatus=processing\nTTL=48h]
    SET --> PROCESS[Process payment\nnormally]
    PROCESS -->|success| STORE_OK[Update Redis key:\nstatus=succeeded\nresult=...]
    PROCESS -->|failure| STORE_FAIL[Update Redis key:\nstatus=failed\nerror=...]

    CHECK -->|Yes, status=processing| R409[Return 409\nPayment in progress]
    CHECK -->|Yes, status=succeeded| RCACHE_OK[Return cached\nsuccess response]
    CHECK -->|Yes, status=failed| RCACHE_FAIL[Return cached\nfailure response]

    style R409 fill:#fefce8,stroke:#ca8a04
    style RCACHE_OK fill:#f0fdf4,stroke:#16a34a
    style RCACHE_FAIL fill:#fff1f2,stroke:#e11d48

Redis SET NX — The Atomic Gate

# Atomic check-and-set — "SET only if Not eXists"
SET idempotency:{merchant_id}:{key} '{"status":"processing"}' NX EX 172800

# NX = only set if key does not already exist
# EX 172800 = expire after 48 hours (172,800 seconds)
# Returns OK if set (we own this request)
# Returns nil if key existed (duplicate — return cached result)

The NX flag is crucial. It makes the check-and-set atomic — without it, two concurrent requests could both check "key not found" and both proceed to process the payment.

Idempotency Key Design

Property	Rule
Format	Opaque to the system; UUID recommended by best practice
Scope	Per merchant — two merchants can use the same key without collision
Lifetime	48 hours — long enough for any retry scenario
Immutability	Same key + different parameters → 409 Conflict

PostgreSQL Backup for Idempotency

Redis gives fast sub-ms checks but is not durable by default. PostgreSQL provides a durable fallback:

-- Unique constraint on (merchant_id, idempotency_key)
-- Attempting to insert a duplicate → unique constraint violation
-- We catch this and return the original payment's result

CREATE UNIQUE INDEX idx_idempotency
    ON payments (merchant_id, idempotency_key);

If Redis is unavailable: fall back to PostgreSQL unique constraint check. Slower (5–10ms vs < 1ms), but safe.

Edge Cases

Same key, different parameters:

Request 1: { "idempotency_key": "key-abc", "amount": 5000 }
Request 2: { "idempotency_key": "key-abc", "amount": 9900 }  ← different amount!
→ Return 409 Conflict — the key uniquely identifies a logical payment intent

Stuck in processing (worker crashed):

After 30 seconds in processing status, a retry with the same idempotency key:

Queries the processor directly using the stored processor_transaction_id
Updates our records to match the processor's authoritative answer
Returns the actual result

Step 7 — Failure Handling

Types of Processor Failures

Failure Type	What Happens	Strategy
Timeout	Processor doesn't respond within 30s	Query processor for status, then retry
5xx Error	Processor returns server error	Retry with exponential backoff
Network failure	Connection drops completely	Retry, then queue for later
Ambiguous response	Connection drops mid-response	Query processor for transaction status
4xx Decline	Processor says "no"	Return failure to merchant — not retryable

The hardest case is the ambiguous response. We sent the request, the processor received it and may have charged the card, but our connection dropped before we received the response. We must query the processor using the idempotency key we passed to them to find the authoritative answer.

Retry with Exponential Backoff + Jitter

def retry_with_backoff(attempt: int) -> float:
    base_delay    = 1.0   # seconds
    max_delay     = 64.0  # seconds
    jitter        = random.uniform(0, 1)
    delay         = min(base_delay * (2 ** attempt), max_delay)
    return delay + jitter

# Attempt 0: ~1s
# Attempt 1: ~2s
# Attempt 2: ~4s
# Attempt 3: ~8s
# Attempt 4: ~16s
# Attempt 5: ~32s
# Attempt 6: ~64s → fail permanently

Random jitter prevents the "thundering herd" — all workers retrying at exactly the same moment after a processor recovers, which would immediately overwhelm it again.

Circuit Breaker

When a processor is consistently failing, stop sending requests to it:

stateDiagram-v2
    [*] --> CLOSED
    CLOSED --> OPEN : failure rate > 50%\nin 10-second window
    OPEN --> HALF_OPEN : after 30 seconds
    HALF_OPEN --> CLOSED : test request succeeds
    HALF_OPEN --> OPEN : test request fails

    CLOSED : All requests pass through\nTrack failure rate
    OPEN : Fail immediately\nDo not call processor\nGives processor time to recover
    HALF_OPEN : Allow one test request\nDecide next state

State	Behaviour
CLOSED	Normal — all requests pass through; monitor failure rate
OPEN	Fail-fast — return error immediately without calling processor
HALF-OPEN	Recovery probe — allow one test request; reset if it succeeds

Multi-Processor Failover

flowchart TD
    REQ[Payment Request] --> CB{Circuit breaker\nfor Stripe CLOSED?}
    CB -->|Yes| STRIPE[Stripe\nPrimary processor]
    STRIPE -->|Success| OK[Return success]
    STRIPE -->|5xx / Timeout| RETRY{Retry ≤ 3x?}
    RETRY -->|Yes| STRIPE
    RETRY -->|No| ADYEN[Adyen\nFallback processor]
    CB -->|No - circuit OPEN| ADYEN
    ADYEN -->|Success| OK
    ADYEN -->|Failure| ERR[Return error\nto merchant]

    style OK fill:#f0fdf4,stroke:#16a34a
    style ERR fill:#fff1f2,stroke:#e11d48

Routing considerations:

Primary processor — default for most transactions (best rates, highest reliability)
Fallback processor — activated when primary's circuit opens
Geographic routing — EU processor for EU cards (lower latency, local compliance)
Cost optimization — some processors have better rates for specific card types (Amex, Discover)

Step 8 — Database Design

Database Selection

Data	Database	Reason
Payment records + events	PostgreSQL	ACID transactions; strong consistency; complex queries
Ledger entries	PostgreSQL	Append-only; ACID; financial audit requirements
Refund records	PostgreSQL	Joins to payments; referential integrity
Idempotency keys	Redis	Sub-ms lookups; ephemeral (48h TTL); no durability needed

Why PostgreSQL and not NoSQL?

Payment systems need ACID transactions. When a payment succeeds, we must atomically: update the payment status, create ledger entries, and store the processor response. A partial write (status updated but ledger entries missing) is catastrophic. NoSQL's eventual consistency model is fundamentally incompatible with financial data requirements.

Payments Table

Table: payments

  payment_id               UUID           PRIMARY KEY
  idempotency_key          VARCHAR(255)   NOT NULL
  merchant_id              UUID           NOT NULL   FK → merchants
  amount                   BIGINT         NOT NULL   (cents, always positive)
  currency                 CHAR(3)        NOT NULL   (ISO 4217)
  status                   VARCHAR(20)    NOT NULL   PENDING | PROCESSING | SUCCEEDED | FAILED | PARTIALLY_REFUNDED | REFUNDED
  payment_method_id        UUID           NOT NULL   (tokenized — never raw card data)
  processor_transaction_id VARCHAR(255)              (set when processor responds)
  failure_code             VARCHAR(50)               (e.g., card_declined, insufficient_funds)
  failure_message          TEXT
  metadata                 JSONB                     (merchant-provided, opaque to us)
  created_at               TIMESTAMP WITH TIME ZONE
  updated_at               TIMESTAMP WITH TIME ZONE

UNIQUE INDEX: (merchant_id, idempotency_key)
INDEX:        (merchant_id, created_at DESC)     ← merchant dashboard queries
INDEX:        (status) WHERE status IN ('PENDING','PROCESSING')  ← partial index for active payments
INDEX:        (processor_transaction_id)          ← reconciliation lookups

Payment Events Table (Audit Trail)

Table: payment_events

  event_id            UUID           PRIMARY KEY
  payment_id          UUID           FK → payments
  event_type          VARCHAR(30)    CREATED | PROCESSING | SUCCEEDED | FAILED | REFUNDED
  previous_status     VARCHAR(20)
  new_status          VARCHAR(20)
  processor_response  JSONB          ← raw response from processor (for debugging/disputes)
  created_at          TIMESTAMP WITH TIME ZONE

INDEX: (payment_id, created_at)

Storing the raw processor response as JSONB is intentional — when a customer disputes a charge, we need the exact processor decision. Anticipating every field is impossible, so JSONB captures everything.

Ledger Entries Table (Double-Entry)

Table: ledger_entries

  entry_id       UUID           PRIMARY KEY
  payment_id     UUID           FK → payments
  account_id     UUID           FK → accounts
  entry_type     CHAR(6)        DEBIT | CREDIT
  amount         BIGINT         NOT NULL  (always positive)
  currency       CHAR(3)
  balance_after  BIGINT                   (account balance after this entry)
  created_at     TIMESTAMP WITH TIME ZONE

-- CRITICAL: No UPDATE or DELETE on this table. Ever.
-- Corrections are new entries with opposite direction.

Step 9 — Double-Entry Ledger

Every serious payment system uses double-entry bookkeeping. This is not optional — it is the only way to ensure money is never created or destroyed.

The Core Principle

Every transaction has exactly two sides. The total of all debits must always equal the total of all credits. Any imbalance signals an error.

Ledger Entries for a $100 Payment (2.9% + $0.30 fee)

Entry	Account	Debit	Credit
1	Customer Receivable	$100.00
2	Merchant Payable		$96.80
3	Fee Revenue		$3.20

Total debits ($100.00) = Total credits ($96.80 + $3.20 = $100.00) ✅

The merchant receives $96.80 (after 2.9% fee = $2.90 + fixed fee = $0.30).

Ledger Entries for a Full Refund

Entry	Account	Debit	Credit
1	Merchant Payable	$96.80
2	Fee Revenue	$3.20
3	Customer Receivable		$100.00

This exactly reverses the original entries.

Account Types

Account Type	Normal Balance	Purpose
Customer Receivable	Debit	Money we expect to receive from customers
Merchant Payable	Credit	Money we owe to merchants
Fee Revenue	Credit	Processing fees we earn
Reserves	Credit	Funds held for potential chargebacks
Refund Liability	Credit	Money owed for pending refunds

Implementation: Atomic Ledger Insert

-- All ledger entries for one payment must be created together — or not at all
BEGIN;

INSERT INTO ledger_entries (entry_id, payment_id, account_id, entry_type, amount, currency)
VALUES
  (gen_random_uuid(), 'pay_xyz', 'acct_customer_receivable', 'DEBIT',  10000, 'USD'),
  (gen_random_uuid(), 'pay_xyz', 'acct_merchant_payable',    'CREDIT',  9680, 'USD'),
  (gen_random_uuid(), 'pay_xyz', 'acct_fee_revenue',         'CREDIT',   320, 'USD');

-- Verify the ledger balances before committing
DO $$
BEGIN
  IF (SELECT SUM(CASE WHEN entry_type = 'DEBIT' THEN amount ELSE -amount END)
      FROM ledger_entries WHERE payment_id = 'pay_xyz') != 0 THEN
    RAISE EXCEPTION 'Ledger imbalance detected for payment pay_xyz';
  END IF;
END $$;

COMMIT;

Three Immutable Principles

Append-only — ledger entries are never updated or deleted; corrections are new entries
Atomic insert — all entries for one payment are inserted in a single transaction
Balance validation — assert debits = credits before committing every payment

Step 10 — Webhook Delivery

Webhook Architecture

sequenceDiagram
    participant PW as Payment Worker
    participant KF as Kafka (Event Queue)
    participant WS as Webhook Service
    participant ME as Merchant Endpoint

    PW->>KF: Publish payment.succeeded {payment_id, amount, status}
    KF->>WS: Consume event
    WS->>WS: Build payload + sign with merchant's HMAC secret
    WS->>ME: POST https://merchant.com/webhooks {event_type, data, signature}

    alt 2xx Success
        ME-->>WS: 200 OK
        WS->>WS: Mark delivered
    else 5xx / Timeout
        ME-->>WS: 500 Error
        WS->>WS: Schedule retry (exponential backoff)
    else 4xx Client Error
        ME-->>WS: 404 Not Found
        WS->>WS: Mark failed — do not retry
    end

Webhook Events

Event	Trigger	Why Merchants Care
`payment.succeeded`	Processor approves payment	Update order status; send confirmation
`payment.failed`	Processor declines or error	Show error to customer; prompt retry
`refund.created`	Refund initiated	Update order status
`refund.succeeded`	Refund completed	Notify customer
`payment.dispute.created`	Chargeback filed	Gather evidence; contact customer

Retry Schedule (At-Least-Once Delivery)

Attempt 1:  immediate
Attempt 2:  + 1 minute
Attempt 3:  + 5 minutes
Attempt 4:  + 30 minutes
Attempt 5:  + 2 hours
Attempt 6:  + 6 hours
Attempt 7:  + 24 hours
→ After 7 failures: move to dead-letter queue; alert merchant

Why at-least-once and not exactly-once?

Exactly-once delivery across system boundaries is impossible. If a merchant processes the webhook but crashes before sending the HTTP acknowledgment, we have no way to know it succeeded. We will retry, and the merchant sees it twice. Therefore:

We guarantee at-least-once delivery
Merchants must handle duplicate events using the event_id field as a deduplication key

4xx vs 5xx handling:

4xx — the merchant's endpoint rejected the webhook (bad URL, auth issue). Retrying will not help. Mark as permanently failed and alert the merchant.
5xx — the merchant's server had a transient problem. Retry.

Webhook Security

Every webhook includes an HMAC signature the merchant can verify:

X-PayFlow-Signature: t=1705312200,v1=a1b2c3d4...

Signature computed as:
  HMAC-SHA256(
    key   = merchant_webhook_secret,
    data  = timestamp + "." + request_body
  )

Merchant verifies:
  1. Recompute signature using their secret
  2. Compare with header value
  3. Reject if timestamp > 5 minutes old (replay attack protection)

Step 11 — Reconciliation

Reconciliation ensures our internal records match the payment processor's settlement reports. This is critical for catching errors, missing transactions, and fee discrepancies.

Types of Reconciliation

Type	Frequency	What We Compare
Transaction-level	Per txn	Our record vs processor response
Daily settlement	End of day	Our totals vs processor settlement file
Bank reconciliation	Daily	Our records vs bank account statement

Reconciliation Architecture

flowchart LR
    OUR[(Our\nTransaction DB)] --> RE[Reconciliation Engine]
    PROC[Processor\nSettlement File] --> RE
    BANK[Bank\nStatement] --> RE

    RE --> MATCH[Matched Records\n✅ Mark reconciled]
    RE --> EXCEPT[Exceptions Queue\n⚠️ Human review]
    RE --> REPORT[Daily Report\nTotal matched, exceptions]

Common Discrepancies

Discrepancy	Likely Cause	Resolution
Missing in our records	Timeout during processing; worker crashed	Query processor; create record from their data
Missing in processor	We think succeeded; they do not	Investigate — may need to refund customer
Amount mismatch	Fee calculation error	Review fee structure; create adjustment entry
Status mismatch	Async update not received	Query processor for authoritative status
Duplicate in processor	Idempotency failure	Refund duplicate; root cause investigation

Reconciliation Best Practices

Never modify historical records — create adjustment ledger entries to correct errors; never edit or delete
Account for settlement timing — processors settle on T+1 or T+2; do not flag as exception before that window expires
Target 95%+ automated matching — humans should only review genuine exceptions, not routine matches
Alert on anomalies — if exception rate exceeds 0.1%, trigger immediate investigation

Step 12 — Scaling

Write Path Scaling

115 TPS average (1,150 peak) — a single PostgreSQL instance with good indexes handles this comfortably
Kafka buffers the burst between payment acceptance and processor calls
Payment Workers scale horizontally — add workers when queue depth grows
PostgreSQL partitioning — range-partition payments by created_at; current month = hot partition; older months archived to cold storage

Read Path Scaling

Read replicas absorb:
  - Merchant dashboard queries (by merchant_id + date range)
  - Payment status GET requests (by payment_id)
  - Reconciliation batch reads
  - Webhook retry lookups

Primary PostgreSQL handles only:
  - Payment INSERT (new payment created)
  - Payment status UPDATE (state transitions)
  - Ledger entry INSERT

Sharding (If Scale Demands It)

Table	Shard Key	Reason
Payments	`merchant_id`	All payments for one merchant on one shard
Ledger entries	`payment_id`	All entries for one payment co-located
Refunds	`payment_id`	Co-located with original payment

At our design target (10M txn/day), a single well-configured PostgreSQL instance handles the load comfortably. Sharding only becomes necessary at 100M+ txn/day.

Step 13 — Failure Scenarios

Failure	Impact	Mitigation
Redis (idempotency) down	Cannot verify duplicates	Return 503; fall back to PostgreSQL unique constraint check
Kafka down	New payments not queued	Payment Service buffers in PostgreSQL; replay when Kafka recovers
Payment Worker crash	Payment stuck in PROCESSING	Worker reads from Kafka offset — message reprocessed from queue
Processor timeout	Unknown if charge went through	Query processor with stored idempotency key; update records
PostgreSQL primary down	Payment writes fail	Failover to standby < 60s; Kafka retains messages during failover
Ambiguous processor response	Status unknown	Polling job queries processor every 30s until definitive answer
Webhook endpoint unreachable	Merchant doesn't know outcome	7-attempt retry schedule; merchant can also poll GET /v1/payments/{id}
Ledger imbalance detected	Financial inconsistency	INSERT with balance assertion fails; alert operations team immediately
Worker processes payment twice	Potential duplicate charge	Processor idempotency key prevents double-charge at processor level

Final Architecture

flowchart TD
    M[Merchant Servers] --> AG[API Gateway\nAuth + Rate Limiting + TLS]

    AG --> PS[Payment Service\nStateless]
    AG --> RS[Refund Service\nStateless]

    PS --> IDEM[(Redis Cluster\nIdempotency Keys\n48h TTL)]
    PS --> PDB[(PostgreSQL Primary\nPayments + Events)]
    PS --> KF1[Kafka\nPayment Queue]

    KF1 --> PW1[Payment Worker 1]
    KF1 --> PW2[Payment Worker 2]
    KF1 --> PW3[Payment Worker N]

    PW1 & PW2 & PW3 --> PSP[PSP Gateway]
    PSP --> STRIPE[Stripe]
    PSP --> ADYEN[Adyen]

    PW1 & PW2 & PW3 --> PDB
    PW1 & PW2 & PW3 --> LDB[(PostgreSQL\nLedger\nAppend-Only)]
    PW1 & PW2 & PW3 --> KF2[Kafka\nEvent Queue]

    KF2 --> WS1[Webhook Worker 1]
    KF2 --> WS2[Webhook Worker N]
    WS1 & WS2 --> ME[Merchant Endpoints]

    PDB --> RR[(PostgreSQL\nRead Replicas)]
    RR --> RECON[Reconciliation\nEngine]

    style IDEM fill:#fff4e0,stroke:#f59e0b
    style KF1 fill:#ede9fe,stroke:#7c3aed
    style KF2 fill:#ede9fe,stroke:#7c3aed
    style PDB fill:#f0fdf4,stroke:#16a34a
    style LDB fill:#f0fdf4,stroke:#16a34a

Technology Stack

Layer	Technology
API Gateway	NGINX / Envoy + API key auth + TLS termination
Payment Service	Java / Go (stateless, horizontally scalable)
Payment Workers	Java / Go (Kafka consumers)
Webhook Service	Node.js / Go (high I/O, many outbound connections)
Idempotency store	Redis Cluster (SET NX, 48h TTL)
Primary datastore	PostgreSQL (ACID, strong consistency)
Ledger	PostgreSQL (same cluster, separate schema, append-only)
Message queue	Apache Kafka (RF=3, durable, exactly-once semantics)
Processor abstraction	Internal PSP Gateway service
Reconciliation	Python batch jobs (daily, reads from read replicas)
Secrets + keys	HashiCorp Vault / AWS KMS
Monitoring	Prometheus + Grafana + PagerDuty
Deployment	Kubernetes multi-AZ (AWS)

Key Trade-Offs

Decision	Option A	Option B	Choice and Reason
Sync vs async processing	Block until processor responds	Accept request, process async	Async — processors take 1–5s; synchronous API would have unpredictable latency
Idempotency store	PostgreSQL only	Redis primary + PostgreSQL backup	Redis + PG — Redis gives sub-ms checks; PG provides durable fallback
Database type	NoSQL (DynamoDB)	PostgreSQL (ACID)	PostgreSQL — payment state changes must be atomic; ACID is non-negotiable
Ledger mutability	Allow updates	Append-only, corrections as new entries	Append-only — audit trails must be immutable; corrections are new entries
Webhook delivery guarantee	Exactly-once	At-least-once	At-least-once — exactly-once across network boundaries is impossible; merchants deduplicate by event_id
Processor failover	Manual switchover	Automatic circuit breaker	Circuit breaker — automatic failover reduces MTTR from minutes to seconds
Status final response timing	Block until processor answers	Return `processing`, push via webhook	Async + webhook — decouples merchant's UX from processor latency

Common Interview Mistakes

❌ Not requiring idempotency keys — without them, every network retry causes duplicate charges
❌ Calling the processor synchronously — 1–5s processor latency means unpredictable API response times
❌ No payment state machine — without enforced transitions, payments can end up in invalid states
❌ Using NoSQL for payments — eventual consistency is fundamentally incompatible with financial data
❌ Not explaining the ambiguous response problem — how do you know if the card was charged when the connection dropped?
❌ No double-entry ledger — without it, there is no way to detect or correct financial errors
❌ Mutable ledger — financial records must be immutable; corrections require new entries, not edits
❌ Exactly-once webhook delivery promise — impossible across network boundaries; always at-least-once
❌ No circuit breaker for processor failures — without it, a failing processor receives an ever-growing flood of retries
❌ No reconciliation design — without daily reconciliation, discrepancies between our records and the processor go undetected

Interview Questions

How do you guarantee exactly-once payment processing when network failures cause retries?
What happens if the payment processor times out — how do you know if the card was charged?
Walk me through what happens when a merchant's server crashes mid-payment and retries.
Why is the Redis SET NX flag critical for idempotency, and what happens without it?
Why return status: processing immediately instead of waiting for the processor's response?
What is the circuit breaker pattern and why is it essential for payment processor integration?
How does the payment state machine prevent corrupted data in concurrent updates?
Why is the ledger append-only? What do you do when you need to correct a ledger error?
Explain double-entry bookkeeping. What does a $100 payment look like in the ledger?
Why can't you guarantee exactly-once webhook delivery? How do merchants handle duplicates?
What is payment reconciliation and why does it matter?
How would you handle a processor returning a payment as successful that you believe failed?
What is the optimistic locking approach in state transitions, and what does a zero-row update mean?
How do you route payments across multiple processors for high availability?
How would you detect and handle a case where your ledger debits don't equal credits?

Summary

Concern	Solution
Exactly-once processing	Idempotency keys — Redis SET NX as atomic gate; PostgreSQL unique constraint backup
Duplicate charge prevention	Same idempotency key → return cached result; never reprocess
Async processing	Accept → store → queue → return `processing`; worker calls processor
Ambiguous responses	Query processor by idempotency key for authoritative status
Processor failures	Exponential backoff + jitter; circuit breaker; multi-processor failover
Payment state integrity	Optimistic locking — UPDATE WHERE status = expected; 0 rows = conflict
Financial accuracy	Double-entry ledger — atomic inserts; debits must equal credits
Ledger immutability	Append-only; corrections via new entries; no UPDATE/DELETE ever
Webhook reliability	At-least-once delivery; 7-attempt retry schedule; HMAC signatures
Reconciliation	Daily comparison vs processor settlement file; exceptions queue for human review
Scale	Kafka buffers burst; stateless workers scale horizontally; PostgreSQL read replicas

The core principle: A payment system has zero tolerance for two categories of error: duplicate charges (idempotency) and lost transactions (durability). Every design decision flows from these two requirements. Async processing with idempotency keys handles the first. Append-only PostgreSQL with Kafka durability handles the second. Everything else — circuit breakers, ledger, reconciliation — ensures the system stays correct when external dependencies fail.

CodeWithVenu