Enterprise File Processing System Design
Design a scalable Enterprise File Processing System — covering distributed ingestion, chunked uploads, virus scanning, format validation, document parsing, transformation pipelines, async job orchestration, S3 lifecycle management, and the operational concerns that keep large-scale file workflows reliable.
1-Hour Interview Roadmap
| Time | Topic |
|---|---|
| 0 – 5 min | Requirements clarification |
| 5 – 10 min | Capacity estimation |
| 10 – 18 min | High-level architecture + storage design |
| 18 – 28 min | Upload pipeline — chunked uploads, resumability, virus scanning |
| 28 – 36 min | Processing pipeline — parsing, validation, transformation |
| 36 – 43 min | Job orchestration — async jobs, retries, status tracking |
| 43 – 50 min | Database design + API design |
| 50 – 56 min | Compliance, security, retention, access control |
| 56 – 60 min | Trade-offs + common interview mistakes |
What Are We Building?
An enterprise File Processing System (FPS) that handles the complete lifecycle of documents and files in a business platform — from secure upload, through validation and virus scanning, through structured content extraction, format transformation, and downstream integration — at the scale of millions of files per day.
File processing is deceptively complex. The naive version is a file upload endpoint and an S3 bucket. The production version must handle large files (gigabytes) reliably over unreliable networks, enforce security controls (virus scanning, format validation, access control), extract structured data from unstructured documents (PDFs, images, spreadsheets), manage processing failures and retries without losing files, enforce regulatory retention requirements, and do all of this asynchronously at high throughput without blocking the user experience.
Real-world examples of this system:
| Use Case | Files processed/day | Typical file size |
|---|---|---|
| Insurance claims document intake | 500K–2M | 1–20 MB (PDFs, images) |
| Payroll file ingestion (bank) | 10K–100K | 1–500 MB (CSV, XML) |
| Medical record upload | 100K–500K | 1–50 MB (DICOM, PDF) |
| E-commerce product catalog | 1M–10M | 1–10 MB (images) |
| Tax document processing | 10M+ (peak season) | 100 KB–5 MB (PDFs) |
The five challenges that make this hard:
- Large file upload reliability — a 1 GB file uploaded over a mobile connection will fail partway. The system must support resumable chunked uploads with client-side verification that no data was lost.
- Security at the intake boundary — every file entering the system is an untrusted payload. Virus scanning, format validation, and content inspection must happen before the file is accessible to any internal service.
- Processing pipeline failures — document parsing, OCR, and format conversion are CPU-intensive and failure-prone. The pipeline must handle transient failures with retries, permanent failures with graceful degradation, and never lose files.
- Structured extraction from unstructured documents — extracting data from PDFs, images, and scanned documents requires OCR, layout analysis, and entity recognition. These are expensive operations that must be managed carefully for cost and latency.
- Compliance and retention — files in regulated industries (healthcare, finance, insurance) must be retained for specific periods (7 years, 10 years), must be immutable after acceptance, and must be auditable — every access logged.
Functional Requirements
Upload:
- Accept file uploads up to 10 GB in size
- Support chunked, resumable uploads (TUS protocol or equivalent)
- Validate file type, size, and content on upload
- Perform virus scanning before making files accessible
- Return upload status and a file ID to the caller
Processing pipeline:
- Extract metadata (file type, dimensions, page count, creation date)
- Extract structured content from PDFs, Word documents, spreadsheets, images (via OCR)
- Transform files to standard formats (PDF/A for archival, thumbnail generation for images)
- Support pluggable pipeline stages: validate → extract → transform → enrich → route
- Track processing status per file with progress events
Storage and lifecycle:
- Store files in object storage (S3-compatible) with content-addressed keys
- Apply lifecycle policies: hot tier → warm tier → cold tier (Glacier) → deletion
- Enforce immutability after acceptance (WORM — Write Once Read Many)
- Support versioning for mutable document types
Access control:
- Per-file access policies (owner, shared-with, org-level)
- Signed download URLs with expiry (pre-signed S3 URLs)
- Full audit log of every file access, download, and deletion
Downstream integration:
- Publish file-processed events to Kafka for downstream consumers
- Support webhook delivery on file processing completion
- Provide structured extraction results via API
Non-Functional Requirements
| Attribute | Target |
|---|---|
| Upload acceptance | < 2 seconds to acknowledge upload start (not processing complete) |
| Processing latency | p50 < 30 seconds, p99 < 5 minutes for standard documents |
| Throughput | 10,000 files/minute ingested |
| File size | Up to 10 GB per file |
| Availability | 99.9% for upload; 99.99% for stored files (S3 durability) |
| Durability | 11 nines (S3 standard); 3-AZ replication for hot files |
| Virus scan coverage | 100% of uploaded files before making accessible |
| Retention compliance | Configurable per document type; WORM enforcement |
| Audit log retention | 7 years minimum |
Capacity Estimation
Baseline assumptions:
- 5 million files uploaded/day
- Average file size: 2 MB
- Peak: 3× average = 15 million files in a 4-hour peak window
- 10% of files require OCR (scanned documents, images)
- 90% of files are PDFs or standard office documents (structured extraction)
Ingestion throughput:
5,000,000 / 86,400 = ~58 files/second average
Peak: 15,000,000 / (4 × 3600) = ~1,042 files/second
Design target: 5,000 files/second (5× headroom)
Storage:
5M files/day × 2 MB = 10 TB/day ingested
Monthly: 300 TB
1-year hot storage: 3.6 PB (before lifecycle tiering)
After lifecycle (90 days hot, rest cold): ~900 GB/day on Glacier × remaining 275 days = ~247 TB cold
S3 costs (rough): $0.023/GB × 900 GB/day × 90 days = ~$1,863/month hot tier
Processing capacity:
- Each standard document (PDF, 10 pages): ~2 seconds of CPU to extract text
- 5M files/day × 2s = 2.8 CPU-hours/second → 10 CPU cores sustained
- OCR files (10% = 500K/day): ~10 seconds/page × 5 pages avg = 50s/file
- 500K × 50s = 6.9 CPU-hours/second for OCR → 25 dedicated OCR cores
- Total: ~40 worker cores for processing (deploy 80 with headroom)
Chunk upload storage (temporary):
- 1 GB file in 5 MB chunks = 200 chunks
- Chunks live in S3 for max 24 hours before assembly or expiry
- With 5,000 concurrent large uploads: 5,000 × 1 GB = 5 TB temporary chunk storage
High-Level Architecture
flowchart TD
subgraph Clients
WEB[Web Client]
MOBILE[Mobile Client]
API_CLIENT[API Client / Service]
end
subgraph Upload
GW[API Gateway]
UPLOAD[Upload Service]
CHUNK[(S3: Chunks Bucket)]
ASSEMBLE[File Assembly Service]
end
subgraph Security
SCAN[Virus Scanner]
VALIDATE[Format Validator]
end
subgraph Storage
S3_HOT[(S3: Hot Tier)]
S3_WARM[(S3: Warm - IA)]
S3_COLD[(S3: Cold - Glacier)]
end
subgraph Processing
KAFKA[Kafka: file.uploaded]
WORKER[Processing Workers]
OCR[OCR Service]
EXTRACT[Extraction Service]
TRANSFORM[Transform Service]
end
subgraph Persistence
PG[(PostgreSQL)]
SEARCH[(Elasticsearch)]
META[(Metadata Cache - Redis)]
end
WEB --> GW
MOBILE --> GW
API_CLIENT --> GW
GW --> UPLOAD
UPLOAD --> CHUNK
UPLOAD --> ASSEMBLE
ASSEMBLE --> SCAN
SCAN -->|clean| VALIDATE
VALIDATE -->|valid| S3_HOT
VALIDATE -->|invalid| PG
S3_HOT -->|lifecycle| S3_WARM
S3_WARM -->|lifecycle| S3_COLD
VALIDATE -->|on acceptance| KAFKA
KAFKA --> WORKER
WORKER --> OCR
WORKER --> EXTRACT
WORKER --> TRANSFORM
WORKER --> PG
WORKER --> SEARCH
Core Service Responsibilities
| Service | Responsibility |
|---|---|
| API Gateway | Auth, rate limiting, request routing |
| Upload Service | Manage chunked upload sessions, receive chunks, trigger assembly |
| File Assembly Service | Assemble chunks into final file, compute checksum, initiate security scan |
| Virus Scanner | ClamAV / commercial AV engine; quarantine or reject infected files |
| Format Validator | Validate MIME type, file signature (magic bytes), structural validity |
| Processing Workers | Async pipeline execution — pull jobs from queue, dispatch to sub-services |
| OCR Service | Tesseract / cloud OCR; convert image/scanned PDFs to searchable text |
| Extraction Service | PDFBox / Apache POI; extract structured text, tables, form fields |
| Transform Service | LibreOffice / ImageMagick; convert formats, generate thumbnails, create PDF/A |
| PostgreSQL | File metadata, processing status, audit log, access control records |
| Elasticsearch | Full-text search over extracted document content |
| Redis | File metadata hot cache, upload session state, signed URL cache |
File Lifecycle State Machine
stateDiagram-v2
[*] --> UPLOAD_INITIATED: client starts chunked upload
UPLOAD_INITIATED --> UPLOADING: chunks being received
UPLOADING --> UPLOAD_COMPLETE: all chunks received
UPLOADING --> UPLOAD_ABANDONED: timeout / client cancel
UPLOAD_COMPLETE --> ASSEMBLING: file assembly started
ASSEMBLING --> SCANNING: assembly complete + checksum verified
ASSEMBLING --> ASSEMBLY_FAILED: checksum mismatch / corrupt chunk
SCANNING --> VALIDATING: virus scan passed (clean)
SCANNING --> QUARANTINED: virus detected
VALIDATING --> ACCEPTED: format valid + content safe
VALIDATING --> REJECTED: invalid format / policy violation
ACCEPTED --> PROCESSING: enqueued for content extraction
PROCESSING --> PROCESSED: extraction complete
PROCESSING --> PROCESSING_FAILED: extraction failed after retries
PROCESSED --> ARCHIVED: lifecycle policy moves to cold storage
ARCHIVED --> DELETED: retention period expired
QUARANTINED --> [*]
REJECTED --> [*]
DELETED --> [*]
Step 1: Chunked Upload with Resumability
A naive multipart upload fails on large files over unreliable connections. The production solution is a resumable chunked upload following the TUS protocol:
sequenceDiagram
participant CLIENT as Client
participant UPLOAD as Upload Service
participant S3 as S3 Chunks Bucket
participant ASSEMBLE as Assembly Service
CLIENT->>UPLOAD: POST /v1/uploads (metadata: filename, size, checksum)
UPLOAD->>UPLOAD: create upload session
UPLOAD-->>CLIENT: { upload_id, chunk_size: 5MB, expires_at }
loop For each chunk
CLIENT->>UPLOAD: PATCH /v1/uploads/{upload_id} (offset, chunk_data)
UPLOAD->>S3: store chunk (upload_id/chunk_N)
UPLOAD-->>CLIENT: 204 No Content + Upload-Offset header
end
CLIENT->>UPLOAD: POST /v1/uploads/{upload_id}/complete
UPLOAD->>ASSEMBLE: assemble(upload_id, expected_checksum)
ASSEMBLE->>S3: read all chunks in order
ASSEMBLE->>ASSEMBLE: concatenate + compute SHA-256
ASSEMBLE->>ASSEMBLE: verify checksum matches client-provided value
ASSEMBLE->>S3: write final file to staging bucket
ASSEMBLE-->>UPLOAD: assembly complete
UPLOAD-->>CLIENT: { file_id, status: "SCANNING" }
Resumability: If the client loses connection mid-upload, it calls HEAD /v1/uploads/{upload_id} to retrieve the current Upload-Offset. It resumes by sending the next chunk starting at that offset. The server uses the offset to identify which chunks have been received.
Chunk size: 5–10 MB is optimal — large enough to minimize per-request overhead, small enough to be retried cheaply on failure.
Upload session expiry: Upload sessions expire after 24 hours. Incomplete uploads are cleaned up by a background job that deletes orphaned chunks from S3 and marks the session abandoned.
Checksum verification: The client computes a SHA-256 of the complete file before upload and sends it in the initial request. After assembly, the server verifies the assembled file matches. This catches corruption at any stage — network, storage, or assembly bugs.
Step 2: Security — Virus Scanning and Format Validation
Every file must pass through two security gates before it is accepted into the system.
Gate 1: Virus Scanning
sequenceDiagram
participant ASSEMBLE as Assembly Service
participant SCAN as Virus Scanner
participant S3_STAGE as S3 Staging
participant S3_HOT as S3 Hot Tier
ASSEMBLE->>SCAN: scan_file(s3://staging/{file_id})
SCAN->>S3_STAGE: stream file in chunks
SCAN->>SCAN: run AV engine (ClamAV / commercial)
alt Clean
SCAN-->>ASSEMBLE: CLEAN
ASSEMBLE->>S3_HOT: copy file to hot tier
ASSEMBLE->>S3_STAGE: delete staging copy
else Infected
SCAN-->>ASSEMBLE: INFECTED (threat_name)
ASSEMBLE->>S3_STAGE: move to quarantine prefix
ASSEMBLE->>PG: update file status = QUARANTINED
ASSEMBLE->>KAFKA: file.quarantined event
end
The virus scanner runs as a separate service with its AV signatures updated hourly. The scanner streams the file from S3 rather than downloading it to a local disk — this prevents the infected file from ever touching a worker's filesystem.
Gate 2: Format Validation
Format validation checks:
- MIME type vs extension: does the file extension match the actual content type?
- Magic bytes verification: does the file start with the correct byte signatures for the declared type?
- Structural integrity: can the file be opened by the appropriate library without error?
- Policy compliance: is this file type allowed for this upload context? (e.g., only PDF/JPEG/PNG accepted for claim attachments)
# Example: format validation checks
ALLOWED_TYPES = {
"claim_attachment": ["application/pdf", "image/jpeg", "image/png"],
"payroll_file": ["text/csv", "application/vnd.ms-excel", "application/xml"],
}
def validate_format(file_path, upload_context):
detected_mime = magic.from_file(file_path, mime=True) # libmagic
if detected_mime not in ALLOWED_TYPES[upload_context]:
return ValidationResult(valid=False, reason=f"MIME type {detected_mime} not allowed for {upload_context}")
# Structural validation — attempt to open with appropriate library
if detected_mime == "application/pdf":
try:
PdfReader(file_path) # pypdf2
except PdfReadError as e:
return ValidationResult(valid=False, reason=f"Corrupt PDF: {e}")
return ValidationResult(valid=True)
Files that fail format validation are rejected immediately. The rejection record is persisted, the staging file is deleted, and the caller receives a structured error response indicating the reason.
Step 3: Content Extraction Pipeline
After a file passes security gates, it enters the async processing pipeline. A file.accepted event is published to Kafka. Processing workers pull from the queue and execute a configurable pipeline of stages:
Pipeline Configuration per Document Type
| Document Type | Pipeline Stages |
|---|---|
| PDF (digital) | metadata → text-extract → table-extract → thumbnail |
| PDF (scanned) | metadata → ocr → table-extract → thumbnail |
| Image | metadata → exif-extract → ocr (optional) → thumbnail |
| Spreadsheet | metadata → sheet-extract → data-validate → thumbnail |
| Word document | metadata → text-extract → table-extract → thumbnail |
| Video | metadata → thumbnail-generate → transcode-preview |
PDF Text Extraction
def extract_pdf(file_path):
reader = PdfReader(file_path)
result = {
"page_count": len(reader.pages),
"pages": [],
"is_searchable": False
}
for i, page in enumerate(reader.pages):
text = page.extract_text()
if text and len(text.strip()) > 50:
result["is_searchable"] = True
result["pages"].append({
"page_number": i + 1,
"text": text,
"word_count": len(text.split())
})
return result
If the extracted text is empty or very sparse (< 50 characters per page), the PDF is treated as a scanned document and routed to the OCR stage.
OCR Pipeline
OCR is the most resource-intensive stage. Architecture:
- Cloud OCR (AWS Textract / Google Vision) for high accuracy on business documents — costs ~$0.0015/page
- Local Tesseract for cost-sensitive workloads with lower accuracy requirements
- Decision: use cloud OCR for documents that are KYC/legal/financial; use local Tesseract for bulk image processing
sequenceDiagram
participant WORKER as Processing Worker
participant OCR as OCR Service
participant TEXTRACT as AWS Textract
participant PG as PostgreSQL
participant SEARCH as Elasticsearch
WORKER->>OCR: ocr_request(file_id, s3_path, quality=HIGH)
OCR->>TEXTRACT: StartDocumentTextDetection(s3_object)
TEXTRACT-->>OCR: job_id (async)
OCR->>OCR: poll for completion (max 5 min)
TEXTRACT-->>OCR: text blocks + confidence scores
OCR-->>WORKER: extracted_text, blocks, tables
WORKER->>PG: store extraction result
WORKER->>SEARCH: index full text (file_id, content, metadata)
WORKER->>KAFKA: file.processed event
Table Extraction
Tabular data in documents (financial statements, payroll summaries, claim forms) requires specialized extraction:
- PDFs: use pdfplumber for tabular structure detection
- Images: use Textract Tables API or local layout analysis
- Spreadsheets: use OpenPyXL for cell-by-cell extraction with formulas resolved
Extracted tables are stored as structured JSON:
{
"tables": [
{
"page": 2,
"rows": [
["Employee", "Hours", "Rate", "Gross"],
["John Smith", "80", "$35.00", "$2,800.00"],
["Jane Doe", "80", "$42.00", "$3,360.00"]
],
"confidence": 0.94
}
]
}
Step 4: Job Orchestration
Processing jobs must survive worker failures. The job queue is the foundation:
Job Queue Architecture
flowchart LR
KAFKA[Kafka: file.accepted] --> JOB_MGMT[Job Management Service]
JOB_MGMT --> PG[(Job State - PostgreSQL)]
JOB_MGMT --> QUEUE[SQS / Redis Queue]
QUEUE --> WORKER_1[Worker 1]
QUEUE --> WORKER_2[Worker 2]
QUEUE --> WORKER_N[Worker N]
WORKER_1 -->|complete| JOB_MGMT
WORKER_1 -->|fail| RETRY[Retry with backoff]
RETRY --> QUEUE
WORKER_1 -->|max retries| DLQ[Dead-Letter Queue]
Job Schema
CREATE TABLE processing_jobs (
job_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
file_id UUID NOT NULL REFERENCES files(file_id),
job_type VARCHAR(50) NOT NULL, -- EXTRACT, OCR, TRANSFORM, THUMBNAIL
status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
priority INT NOT NULL DEFAULT 5, -- 1 (high) to 10 (low)
attempt_count INT NOT NULL DEFAULT 0,
max_attempts INT NOT NULL DEFAULT 3,
worker_id VARCHAR(100),
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ,
failed_at TIMESTAMPTZ,
failure_reason TEXT,
result JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_jobs_pending ON processing_jobs (priority DESC, next_attempt_at ASC)
WHERE status IN ('PENDING', 'RETRY');
CREATE INDEX idx_jobs_file_id ON processing_jobs (file_id, job_type);
CREATE INDEX idx_jobs_worker ON processing_jobs (worker_id, status);
Workers use SELECT FOR UPDATE SKIP LOCKED to claim jobs without contention:
UPDATE processing_jobs
SET status = 'RUNNING', worker_id = $1, started_at = NOW(), attempt_count = attempt_count + 1
WHERE job_id = (
SELECT job_id FROM processing_jobs
WHERE status IN ('PENDING', 'RETRY')
AND next_attempt_at <= NOW()
ORDER BY priority DESC, next_attempt_at ASC
LIMIT 1
FOR UPDATE SKIP LOCKED
)
RETURNING *;
Retry with Exponential Backoff
def next_retry_delay(attempt_count):
# Exponential backoff: 30s, 2min, 8min, 30min, 2hr
return min(30 * (4 ** (attempt_count - 1)), 7200)
def handle_job_failure(job_id, error):
job = get_job(job_id)
if job.attempt_count >= job.max_attempts:
update_job(job_id, status="FAILED", failure_reason=str(error))
publish_dlq_event(job)
else:
delay = next_retry_delay(job.attempt_count)
update_job(job_id, status="RETRY", next_attempt_at=now() + delay)
Database Schema
-- Core file record
CREATE TABLE files (
file_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
upload_id UUID UNIQUE, -- upload session ID
owner_id UUID NOT NULL, -- user or service that uploaded
org_id UUID NOT NULL,
filename VARCHAR(500) NOT NULL,
original_filename VARCHAR(500) NOT NULL,
mime_type VARCHAR(100) NOT NULL,
file_size_bytes BIGINT NOT NULL,
checksum_sha256 CHAR(64) NOT NULL,
storage_key VARCHAR(500), -- S3 object key
storage_bucket VARCHAR(100),
storage_tier VARCHAR(20) DEFAULT 'HOT', -- HOT, WARM, COLD
status VARCHAR(30) NOT NULL, -- see lifecycle state machine
document_type VARCHAR(50), -- CLAIM_ATTACHMENT, PAYROLL, etc.
scan_result VARCHAR(20), -- CLEAN, INFECTED, SKIPPED
validation_result VARCHAR(20), -- VALID, INVALID
page_count INT,
is_searchable BOOLEAN DEFAULT FALSE,
is_immutable BOOLEAN DEFAULT FALSE,
retention_until DATE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
accepted_at TIMESTAMPTZ,
processed_at TIMESTAMPTZ,
deleted_at TIMESTAMPTZ
);
CREATE INDEX idx_files_owner ON files (owner_id, created_at DESC);
CREATE INDEX idx_files_org_status ON files (org_id, status, created_at DESC);
CREATE INDEX idx_files_checksum ON files (checksum_sha256);
CREATE INDEX idx_files_retention ON files (retention_until) WHERE deleted_at IS NULL;
-- Extraction results
CREATE TABLE file_extractions (
extraction_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
file_id UUID NOT NULL REFERENCES files(file_id),
extraction_type VARCHAR(30) NOT NULL, -- TEXT, TABLE, METADATA, FORM_FIELDS
extractor VARCHAR(50) NOT NULL, -- pdfbox, textract, tesseract
result JSONB NOT NULL,
confidence NUMERIC(4,3),
page_count INT,
word_count INT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_extractions_file_id ON file_extractions (file_id, extraction_type);
-- Upload sessions
CREATE TABLE upload_sessions (
upload_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
owner_id UUID NOT NULL,
filename VARCHAR(500) NOT NULL,
expected_size BIGINT NOT NULL,
expected_checksum CHAR(64) NOT NULL,
chunk_size INT NOT NULL DEFAULT 5242880, -- 5 MB
total_chunks INT NOT NULL,
received_chunks INT NOT NULL DEFAULT 0,
upload_context VARCHAR(50), -- claim_attachment, payroll_file
status VARCHAR(20) NOT NULL DEFAULT 'IN_PROGRESS',
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
expires_at TIMESTAMPTZ NOT NULL,
completed_at TIMESTAMPTZ
);
CREATE INDEX idx_uploads_owner ON upload_sessions (owner_id, created_at DESC);
CREATE INDEX idx_uploads_expiry ON upload_sessions (expires_at) WHERE status = 'IN_PROGRESS';
-- File access audit log
CREATE TABLE file_access_log (
log_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
file_id UUID NOT NULL REFERENCES files(file_id),
accessor_id UUID NOT NULL,
access_type VARCHAR(20) NOT NULL, -- DOWNLOAD, VIEW, SHARE, DELETE
ip_address INET,
user_agent TEXT,
accessed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
) PARTITION BY RANGE (accessed_at);
-- Partition monthly for efficient archiving
CREATE TABLE file_access_log_2026_07 PARTITION OF file_access_log
FOR VALUES FROM ('2026-07-01') TO ('2026-08-01');
API Design
POST /v1/uploads — Initiate Upload Session
{
"filename": "claim_documentation_2026.pdf",
"file_size": 4718592,
"checksum_sha256": "a3f1b2c4d5e6...",
"mime_type": "application/pdf",
"upload_context": "claim_attachment",
"metadata": {
"claim_id": "clm_xyz789",
"document_type": "SUPPORTING_EVIDENCE"
}
}
Response:
{
"upload_id": "upl_a1b2c3d4",
"chunk_size": 5242880,
"total_chunks": 1,
"upload_url_base": "https://api.example.com/v1/uploads/upl_a1b2c3d4",
"expires_at": "2026-07-16T20:00:00.000Z"
}
PATCH /v1/uploads/{upload_id} — Upload Chunk
Headers:
Content-Type: application/offset+octet-stream
Upload-Offset: 0
Content-Length: 5242880
Response: 204 No Content with Upload-Offset: 5242880
POST /v1/uploads/{upload_id}/complete — Finalize Upload
Response:
{
"file_id": "fil_b2c3d4e5",
"filename": "claim_documentation_2026.pdf",
"file_size": 4718592,
"status": "SCANNING",
"estimated_processing_time_seconds": 45
}
GET /v1/files/{file_id} — Get File Metadata and Status
{
"file_id": "fil_b2c3d4e5",
"filename": "claim_documentation_2026.pdf",
"mime_type": "application/pdf",
"file_size": 4718592,
"status": "PROCESSED",
"page_count": 12,
"is_searchable": true,
"extraction": {
"word_count": 4821,
"tables_found": 3,
"confidence": 0.97
},
"storage_tier": "HOT",
"accepted_at": "2026-07-16T10:30:05.000Z",
"processed_at": "2026-07-16T10:30:52.000Z"
}
GET /v1/files/{file_id}/download — Generate Pre-Signed Download URL
{
"download_url": "https://s3.amazonaws.com/...?X-Amz-Signature=...",
"expires_at": "2026-07-16T11:30:00.000Z",
"filename": "claim_documentation_2026.pdf",
"content_type": "application/pdf"
}
The download endpoint logs the access (accessor_id, IP, timestamp) before generating the signed URL. This ensures every download is audited even though the actual byte transfer goes directly from S3 to the client.
GET /v1/files/{file_id}/extractions — Get Extracted Content
{
"file_id": "fil_b2c3d4e5",
"extractions": [
{
"type": "TEXT",
"pages": [
{ "page": 1, "text": "INSURANCE CLAIM FORM...", "word_count": 312 },
{ "page": 2, "text": "Section 1: Claimant Information...", "word_count": 289 }
]
},
{
"type": "TABLE",
"tables": [
{
"page": 4,
"rows": [
["Date of Service", "Provider", "Amount"],
["2026-06-15", "City Medical Center", "$1,250.00"]
]
}
]
}
]
}
Kafka Event Architecture
| Topic | Producer | Consumers | Retention |
|---|---|---|---|
files.upload.initiated |
Upload Service | Analytics, Rate Limiter | 7 days |
files.upload.completed |
Upload Service | Assembly Service | 7 days |
files.scan.completed |
Virus Scanner | Validation Service, Analytics | 30 days |
files.accepted |
Validation Service | Processing Workers, Notifier, Consumers | 90 days |
files.rejected |
Validation Service | Notifier, Audit | 90 days |
files.quarantined |
Virus Scanner | Security Team, Notifier | 1 year |
files.processed |
Processing Workers | Downstream consumers, Notifier | 90 days |
files.processing.failed |
Processing Workers | Alerting, Retry Orchestrator | 30 days |
files.lifecycle.transitioned |
Lifecycle Job | Analytics, Billing | 1 year |
S3 Lifecycle Policy Design
Bucket: enterprise-files-hot
├── staging/ ← assembled files awaiting virus scan (24hr auto-delete)
├── quarantine/ ← infected files (90-day retention, then delete)
├── accepted/ ← clean, validated files
│ ├── {org_id}/
│ │ ├── {year}/{month}/
│ │ │ └── {file_id}
Lifecycle rules on accepted/:
- Day 0–90: S3 Standard (HOT) — immediate access, $0.023/GB
- Day 90–180: S3 Intelligent-Tiering or IA — $0.0125/GB
- Day 180+: S3 Glacier Instant Retrieval — $0.004/GB
- Deletion: based on retention_until field; enforced by daily job
WORM (Write Once Read Many):
For regulated document types (insurance claims, financial records), enable S3 Object Lock in Compliance mode. Once a file is accepted, it cannot be deleted or overwritten until the retention period expires — even by the storage administrator. Deletion requests before expiry are rejected with a 403 Access Denied.
Observability
Key Metrics
| Metric | Alert Threshold |
|---|---|
upload.success_rate |
< 99% (5-min window) |
upload.p99_chunk_receive_ms |
> 5,000 ms |
scan.infected_files_per_hour |
> 5 (alert security team) |
scan.scan_latency_p99_seconds |
> 60 s |
processing.queue_depth |
> 50,000 jobs |
processing.job_failure_rate |
> 1% |
processing.ocr_latency_p99_seconds |
> 300 s |
storage.staging_bucket_size_gb |
> 10,000 GB (cleanup issue) |
storage.hot_tier_cost_daily_usd |
> $2,000 (budget alert) |
retention.files_overdue_for_deletion |
> 0 |
Design Trade-offs
1. Synchronous vs Asynchronous Processing
Option A (Sync): Block the upload API response until the file is scanned, validated, and extracted. Caller gets complete results in one call.
Option B (Async): Accept the upload, return immediately with a file_id and status: SCANNING, process in the background. Caller polls or receives a webhook.
Decision: Async for all processing beyond virus scanning. Virus scanning takes 2–30 seconds depending on file size — acceptable for a synchronous gate. But content extraction (OCR, table parsing) can take minutes for large multi-page documents. Blocking an HTTP connection for minutes is unreliable and wasteful. Return a file ID after scan, deliver processing results asynchronously via webhook or polling.
2. Chunked Upload Service vs Pre-Signed S3 URLs
Option A (Chunked service): Clients upload chunks to the application server, which relays to S3. Full control over chunk validation, progress tracking, and rate limiting.
Option B (Pre-signed S3 URLs): Generate a pre-signed multi-part upload URL and let clients upload directly to S3. No data flows through the application server.
Decision: Hybrid — pre-signed S3 multi-part for files > 100 MB; application-mediated for smaller files. For files over 100 MB, routing through the application server adds unnecessary latency and costs network egress. Pre-signed multi-part uploads go directly to S3 at full S3 throughput. For smaller files, application-mediated uploads simplify the client implementation and allow inline validation.
3. Local OCR (Tesseract) vs Cloud OCR (Textract/Vision)
Option A (Local Tesseract): Runs on-premise, $0 per page, ~75–85% accuracy on clean documents.
Option B (Cloud OCR): AWS Textract, ~$0.0015/page for text, ~$0.015/page for form/table extraction. ~95–98% accuracy.
Decision: Cloud OCR for regulated documents; local Tesseract for bulk low-stakes processing. For insurance claim forms, KYC documents, and legal contracts, OCR errors have real business consequences (misread claim amounts, missed required fields). The $0.0015/page cost is justified. For bulk e-commerce product image alt-text generation or internal document indexing, local Tesseract at high throughput is cost-effective.
4. Content-Addressed Storage vs Path-Based Storage
Option A (Content-addressed, SHA-256 key): S3 key is the file's SHA-256 hash. Identical files deduplicate automatically.
Option B (UUID-based key): S3 key is {org}/{year}/{month}/{file_id}. Each upload is a unique object.
Decision: UUID-based for user-uploaded files; content-addressed for generated assets (thumbnails, transformed formats). User-uploaded files with identical content are often distinct business documents (two different claimants uploading the same blank form). Deduplication based on content hash could incorrectly merge them. But generated thumbnails and PDF/A conversions of identical source files are truly identical — content-addressing saves storage cost for generated assets.
Common Interview Mistakes
-
Treating file upload as a single POST request. A POST request with a 1 GB body will time out on any normal HTTP infrastructure. Chunked resumable uploads are not a nice-to-have for large files — they are the only viable design.
-
Skipping virus scanning. Every interviewer expects virus scanning to be mentioned. Accepting untrusted file uploads without scanning is a security vulnerability. The scanning architecture (streaming, isolation from the production network, quarantine flow) should be discussed.
-
Proposing synchronous processing for content extraction. OCR of a 100-page scanned document takes 5–10 minutes. Holding an HTTP connection open for 10 minutes is not a production architecture. Async processing with status polling or webhooks is required.
-
Storing files in a relational database. Proposing to store file bytes as BLOBs in PostgreSQL shows a misunderstanding of object storage. Files always go to S3-compatible object storage. PostgreSQL stores metadata.
-
Missing checksum verification. Files transmitted over networks get corrupted. A checksum (SHA-256) computed before upload and verified after assembly is the only way to confirm the file arrived intact. Missing this leads to undetected corruption in production.
-
Ignoring retention and WORM requirements. For regulated industries, retention compliance is a legal obligation. An architecture that allows deletion before retention expiry could expose the business to regulatory penalties. S3 Object Lock and lifecycle policies should be mentioned.
-
Designing a single processing stage. "The file is processed after upload" is not an answer. The processing pipeline has multiple stages (scan, validate, extract, transform, index) with different failure modes, retry policies, and latencies for each stage.
-
Forgetting the access audit log. Every file download in a regulated system must be logged. Generating a pre-signed URL without logging the access means the system has no record of who downloaded what and when — a compliance failure.
Summary
flowchart LR
A[Client Upload] --> B[Chunked Upload Service]
B --> C[S3 Staging]
C --> D[Assembly + Checksum]
D --> E[Virus Scan]
E -->|clean| F[Format Validation]
E -->|infected| Z1[Quarantine]
F -->|valid| G[S3 Hot Tier]
F -->|invalid| Z2[Rejected]
G --> H[Kafka: file.accepted]
H --> I[Processing Workers]
I --> J[OCR / Extract / Transform]
J --> K[PostgreSQL + Elasticsearch]
J --> L[Kafka: file.processed]
L --> M[Downstream Consumers / Webhooks]
G -->|lifecycle| N[Warm Tier → Cold Tier → Delete]
Design Principles:
- Intake is the security boundary — virus scanning and format validation must happen before any internal service sees a file; never relax these gates
- Async processing is the only viable architecture — content extraction is too slow and too failure-prone for synchronous handling; always acknowledge the upload, process in the background
- Checksums are not optional — compute a SHA-256 of every file before upload and verify after assembly; network and storage corruption happens in production
- Every download must be audited — generate signed URLs only after logging the access attempt; the byte transfer itself can bypass the application server, but the authorization event must not
- Lifecycle policies must be designed, not defaulted — every file type needs an explicit retention period; S3 Object Lock enforces it; a background job monitors for overdue deletions
- Extraction accuracy drives downstream value — the quality of content extraction determines whether the file system is just a storage bucket or a structured data source that enables downstream automation