Backend interview prep

Start

Serverless Best Practices Interview Questions and Answers

Learn production-ready serverless best practices covering function design, security, scalability, cold starts, idempotency, retries, observability, event-driven architecture, cost optimization, deployment, testing, and disaster recovery.

Module Navigation

Previous: Step Functions QA | Parent: Serverless Learning Path | Next: Monitoring

Introduction

Serverless computing allows teams to build and run applications without directly provisioning or maintaining servers.

Cloud providers manage much of the infrastructure, including:

  • Runtime provisioning
  • Operating-system maintenance
  • Automatic scaling
  • Availability
  • Request routing
  • Infrastructure patching
  • Capacity allocation

However, serverless does not remove architecture responsibility.

Developers and architects are still responsible for:

  • Function design
  • Application security
  • Identity and permissions
  • Reliability
  • Data consistency
  • Error handling
  • Observability
  • Cost control
  • Performance
  • Deployment safety
  • Disaster recovery

A poorly designed serverless application can suffer from:

  • Excessive cold starts
  • Duplicate processing
  • Uncontrolled scaling
  • Database connection exhaustion
  • High cloud cost
  • Hidden failures
  • Event loops
  • Security vulnerabilities
  • Vendor lock-in
  • Difficult troubleshooting

This lesson explains the most important serverless best practices for building secure, scalable, reliable, and production-ready systems.

Serverless Best-Practice Architecture

flowchart TB
    Client[Web or Mobile Client] --> CDN[CDN]
    CDN --> WAF[Web Application Firewall]
    WAF --> Gateway[API Gateway]

    Gateway --> Auth[Identity Provider]
    Gateway --> FunctionA[Serverless Function]

    FunctionA --> Queue[Message Queue]
    FunctionA --> Database[(Managed Database)]
    FunctionA --> Cache[(Distributed Cache)]

    Queue --> FunctionB[Background Function]
    FunctionB --> EventBus[Event Bus]
    EventBus --> Consumers[Independent Consumers]

    FunctionA --> Secrets[Secret Manager]
    FunctionB --> Secrets

    Gateway --> Logs[Centralized Logging]
    FunctionA --> Logs
    FunctionB --> Logs

    Logs --> Monitoring[Metrics and Alerts]
    Monitoring --> Operations[Operations Team]

Shared Responsibility in Serverless

The cloud provider manages the underlying platform, but the customer still manages the application.

Cloud Provider Responsibility Customer Responsibility
Physical infrastructure Application code
Host operating system Function configuration
Runtime infrastructure Identity and permissions
Platform availability Input validation
Automatic scaling platform Scaling boundaries
Infrastructure patching Dependency updates
Managed-service durability Data design
Service-level monitoring Business monitoring
Runtime isolation Secret management
Regional infrastructure Disaster-recovery design

Serverless reduces infrastructure work but does not eliminate operational responsibility.

Keep Functions Small and Focused

A function should perform one clearly defined responsibility.

Good examples include:

  • Validate an order
  • Process a payment
  • Resize an image
  • Send a notification
  • Update customer status
  • Publish an audit event

Avoid building one large function that:

  • Validates input
  • Processes payment
  • Updates inventory
  • Sends notifications
  • Creates reports
  • Handles every error path
flowchart LR
    Request[Order Request] --> Validate[Validation Function]
    Validate --> Payment[Payment Function]
    Payment --> Inventory[Inventory Function]
    Inventory --> Notification[Notification Function]

Benefits of focused functions include:

  • Easier testing
  • Independent deployment
  • Clear ownership
  • Smaller packages
  • Reduced startup time
  • Simpler permissions
  • Easier scaling
  • Better failure isolation

Avoid Over-Fragmentation

Functions should be small, but splitting every line of logic into a separate function creates unnecessary complexity.

Excessive fragmentation can cause:

  • Increased network calls
  • Higher latency
  • More state transitions
  • More monitoring complexity
  • Larger cloud bills
  • Distributed debugging challenges
  • More deployment resources

A function boundary should represent a meaningful:

  • Business capability
  • Scaling boundary
  • Security boundary
  • Deployment boundary
  • Failure boundary

Stateless Function Design

Serverless function instances are temporary.

A function must not assume that:

  • The same instance will process the next request
  • Local files will remain available
  • In-memory state will persist
  • A warm instance will always exist
  • Only one request uses the instance

Persistent state should be stored in managed services such as:

  • DynamoDB
  • Firestore
  • Cosmos DB
  • Cloud SQL
  • Amazon RDS
  • Cloud Storage
  • Amazon S3
  • Redis
  • Managed message queues
flowchart LR
    Function[Stateless Function] --> Database[(Persistent Database)]
    Function --> Storage[(Object Storage)]
    Function --> Cache[(Distributed Cache)]

Local memory may be used for temporary processing or reusable clients, but not as the system of record.

Reuse Expensive Resources

Resources that are safe to reuse should be initialized outside the handler.

Examples include:

  • Database connection pools
  • HTTP clients
  • Cloud SDK clients
  • Configuration objects
  • JSON serializers
  • Encryption clients
public class OrderFunction {

    private static final OrderRepository ORDER_REPOSITORY =
            new OrderRepository();

    public OrderResponse handle(OrderRequest request) {
        return ORDER_REPOSITORY.process(request);
    }
}

This can reduce:

  • Repeated initialization
  • Connection creation
  • CPU usage
  • Invocation latency

Reusable objects must be thread-safe when the platform allows concurrent requests per instance.

Cold-Start Optimization

A cold start occurs when the platform creates and initializes a new function instance.

Cold starts are influenced by:

  • Runtime choice
  • Package size
  • Dependency count
  • Framework startup
  • Memory and CPU
  • Network initialization
  • VPC configuration
  • Container initialization
  • Static startup logic

Reducing Cold Starts

Recommended techniques include:

  • Minimize dependencies.
  • Remove unused libraries.
  • Use lightweight frameworks.
  • Reduce package size.
  • Avoid unnecessary startup tasks.
  • Lazy-load rarely used components.
  • Reuse initialized clients.
  • Allocate appropriate memory and CPU.
  • Keep runtime versions current.
  • Use provisioned or minimum instances for critical APIs.
  • Avoid long initialization network calls.
  • Measure real cold-start latency.
flowchart LR
    Request[Incoming Request] --> Instance{Warm Instance?}
    Instance -->|Yes| Execute[Execute Immediately]
    Instance -->|No| Initialize[Initialize Runtime]
    Initialize --> Execute

Provisioned capacity reduces cold starts but introduces baseline cost.

Choose the Right Runtime

Runtime selection affects:

  • Startup time
  • Memory usage
  • Library availability
  • Development speed
  • Security patching
  • Team expertise
  • Long-term support

Choose a runtime based on the workload rather than popularity alone.

For example:

  • Java may be suitable for enterprise libraries and complex business logic.
  • Node.js may be suitable for lightweight I/O-driven APIs.
  • Python may be suitable for automation and data processing.
  • Go may be suitable for fast startup and low memory usage.
  • .NET may be suitable for Microsoft-focused enterprise environments.

The team must monitor runtime deprecation dates and upgrade before support ends.

Allocate Memory and CPU Correctly

Serverless platforms commonly allocate CPU in relation to memory.

Increasing memory may:

  • Reduce execution time
  • Improve startup time
  • Provide more CPU
  • Reduce timeout risk
  • Sometimes reduce total cost
flowchart LR
    LowMemory[Low Memory] --> Slow[Longer Execution]
    HigherMemory[Higher Memory] --> Fast[Shorter Execution]

    Slow --> CostA[Duration-Based Cost]
    Fast --> CostB[Potentially Lower Total Cost]

Functions should be load-tested at multiple memory configurations rather than using the default blindly.

Configure Timeouts

Every function should have a realistic timeout.

Timeouts that are too high can:

  • Hide downstream problems
  • Increase cost
  • Hold resources
  • Delay retries
  • Exhaust connections

Timeouts that are too low can:

  • Interrupt valid processing
  • Produce duplicate retries
  • Create partial transactions

Timeout layers should be aligned:

flowchart LR
    Client[Client Timeout] --> Gateway[Gateway Timeout]
    Gateway --> Function[Function Timeout]
    Function --> Database[Database Timeout]
    Function --> External[External API Timeout]

A downstream timeout should generally be shorter than the function timeout so the function has time to handle the failure.

Use Asynchronous Processing

Long-running or non-immediate work should not remain inside synchronous API requests.

Examples include:

  • Report generation
  • Email delivery
  • File conversion
  • Video processing
  • Large database updates
  • Partner integration
  • Batch processing
sequenceDiagram
    participant Client
    participant API as API Gateway
    participant Function as Request Function
    participant Queue
    participant Worker
    Client->>API: Submit request
    API->>Function: Validate request
    Function->>Queue: Publish work item
    Function-->>API: Return request ID
    API-->>Client: HTTP 202 Accepted
    Queue->>Worker: Process asynchronously

Asynchronous processing improves:

  • API response time
  • Reliability
  • Backpressure handling
  • Scalability
  • Retry control

Use Queues for Buffering

A queue separates event production from processing capacity.

flowchart LR
    Producers[Multiple Producers] --> Queue[Managed Queue]
    Queue --> ConsumerA[Function Instance 1]
    Queue --> ConsumerB[Function Instance 2]
    Queue --> ConsumerC[Function Instance 3]

Queues provide:

  • Buffering
  • Backpressure
  • Retry support
  • Consumer decoupling
  • Traffic smoothing
  • Dead-letter handling

Use a queue when downstream systems cannot scale as quickly as incoming traffic.

Event Bus vs Queue

Event Bus Queue
Routes events Stores work items
Supports many consumers Commonly distributes work to consumers
Best for business-event fan-out Best for buffering and controlled processing
Consumers react independently Messages remain until consumed or expired
Content-based routing Delivery and backlog management

A production system may route an event through an event bus and then place it into a queue for processing.

flowchart LR
    Producer --> EventBus[Event Bus]
    EventBus --> Rule[Matching Rule]
    Rule --> Queue[Processing Queue]
    Queue --> Function[Worker Function]

Design Idempotent Functions

Serverless event systems commonly provide at-least-once delivery.

A function may receive the same request or event more than once because of:

  • Delivery retries
  • Network failures
  • Client retries
  • Timeout ambiguity
  • Queue redelivery
  • Event replay
  • Workflow retries

An idempotent function produces the same intended business result when a request is repeated.

flowchart LR
    Request[Request with Idempotency Key] --> Function[Function]
    Function --> Check{Already Processed?}

    Check -->|Yes| Existing[Return Stored Result]
    Check -->|No| Process[Process Request]

    Process --> Store[Store Result and Key]
    Store --> Response[Return Response]

Idempotency Techniques

Common approaches include:

  • Idempotency keys
  • Event IDs
  • Database unique constraints
  • Conditional writes
  • Transaction records
  • Processed-event tables
  • Version checks
  • Business-state validation

Example:

Idempotency-Key: order-payment-10025

For financial operations, the idempotency record should be stored atomically with the business result where possible.

Retry Only Transient Failures

Retries are appropriate for temporary failures such as:

  • Throttling
  • Short network interruptions
  • Temporary service unavailability
  • Database connection problems
  • Rate-limit responses

Retries are not appropriate for permanent failures such as:

  • Invalid request data
  • Missing mandatory fields
  • Unsupported operation
  • Business-rule rejection
  • Invalid authentication
  • Resource permanently deleted
flowchart TD
    Failure[Function Failure] --> Type{Failure Type}

    Type -->|Transient| Retry[Retry with Backoff]
    Type -->|Permanent| Reject[Reject or Dead-Letter]

Exponential Backoff and Jitter

Retries should use exponential backoff.

Example delays:

2 seconds
4 seconds
8 seconds
16 seconds

Jitter adds randomness so that many clients do not retry at exactly the same time.

flowchart LR
    Failure --> Wait1[Wait 2 Seconds Plus Jitter]
    Wait1 --> Retry1[Retry]
    Retry1 --> Wait2[Wait 4 Seconds Plus Jitter]
    Wait2 --> Retry2[Retry]

This protects downstream services from retry storms.

Dead-Letter Queues

A dead-letter queue stores events or messages that cannot be processed successfully after repeated attempts.

flowchart LR
    Queue[Primary Queue] --> Function[Worker Function]
    Function -->|Success| Complete[Complete]
    Function -->|Repeated Failure| DLQ[Dead-Letter Queue]
    DLQ --> Review[Investigate and Replay]

Every production DLQ should have:

  • Monitoring
  • Alerts
  • Ownership
  • Investigation procedure
  • Replay mechanism
  • Retention policy

A DLQ without monitoring only hides failures.

Prevent Poison-Message Loops

A poison message always fails because its data is invalid or incompatible.

Without bounded retries, it may be processed repeatedly.

Recommended controls include:

  • Maximum retry count
  • Dead-letter queue
  • Payload validation
  • Schema validation
  • Error classification
  • Manual or automated remediation
  • Controlled replay

Validate Inputs Early

All external input must be treated as untrusted.

Validate:

  • Required fields
  • Data types
  • String length
  • Numeric ranges
  • Allowed values
  • File type
  • File size
  • Content type
  • Identifiers
  • Authentication claims
flowchart TD
    Input[Incoming Input] --> Validate{Valid?}
    Validate -->|Yes| Process[Process Request]
    Validate -->|No| Reject[Return Validation Error]

Validation should occur before:

  • Database calls
  • External API calls
  • Event publication
  • Expensive processing

Use Schema Contracts

Events and API requests should follow documented schemas.

Schema contracts help prevent:

  • Consumer failures
  • Unexpected field changes
  • Data-type mismatches
  • Breaking deployments
  • Silent data corruption

A versioned event may include:

{
  "eventId": "evt-10025",
  "eventType": "order.created",
  "schemaVersion": "1.0",
  "eventTime": "2026-07-18T15:00:00Z",
  "data": {
    "orderId": "ORD-10025",
    "customerId": "CUS-7001"
  }
}

Prefer backward-compatible changes such as adding optional fields.

Avoid Recursive Event Loops

An event loop occurs when a function updates the same resource that triggers it.

flowchart LR
    Storage[Storage Update] --> Function[Triggered Function]
    Function --> Storage

Examples include:

  • Storage function writing back to the same input location
  • Database trigger updating the same watched record
  • Event consumer republishing an identical matching event
  • Notification function triggering itself

Prevention techniques include:

  • Separate input and output locations
  • Add processing-status fields
  • Filter event types
  • Use source identifiers
  • Validate whether processing already occurred
  • Use separate event buses or topics

Control Automatic Scaling

Serverless functions can scale faster than databases and external services.

flowchart LR
    Traffic[Traffic Spike] --> Functions[Thousands of Function Invocations]
    Functions --> Database[(Limited Database Capacity)]

Uncontrolled scaling can cause:

  • Database connection exhaustion
  • Third-party rate-limit violations
  • Higher cost
  • Service instability
  • Cascading failures

Protection techniques include:

  • Maximum concurrency
  • Maximum instance limits
  • Queue buffering
  • Reserved concurrency
  • Rate limiting
  • Batch processing
  • Circuit breakers
  • Connection-pool limits

Protect Database Connections

Opening a new database connection for every invocation can overwhelm the database.

Recommended practices include:

  • Reuse connection pools.
  • Limit pool size.
  • Configure maximum function concurrency.
  • Use database proxies where available.
  • Keep transactions short.
  • Close failed connections.
  • Monitor active connections.
  • Use serverless-native databases where appropriate.
flowchart LR
    Functions[Function Instances] --> Proxy[Database Proxy]
    Proxy --> Database[(Relational Database)]

The maximum possible connection count should be calculated across all function instances.

Use NoSQL Carefully

NoSQL databases often integrate naturally with serverless workloads, but poor key design can still create problems.

Consider:

  • Partition-key distribution
  • Hot partitions
  • Read and write capacity
  • Access patterns
  • Secondary indexes
  • Item size
  • Transaction requirements
  • Consistency requirements

Serverless compute does not automatically correct poor database design.

Apply Least-Privilege IAM

Every function should have its own runtime identity where practical.

The function should receive only the permissions required for its responsibility.

flowchart LR
    PaymentFunction[Payment Function] --> PaymentRole[Payment Role]
    PaymentRole --> PaymentTable[(Payment Table)]

    NotificationFunction[Notification Function] --> NotifyRole[Notification Role]
    NotifyRole --> Topic[Notification Topic]

Avoid assigning:

  • Administrator roles
  • Owner roles
  • Editor roles
  • Wildcard actions
  • Wildcard resources

Separate identities for:

  • Deployment
  • Build
  • Runtime
  • Trigger invocation
  • Administration

Secure Public Endpoints

Public serverless APIs should be protected by:

  • HTTPS
  • API Gateway
  • Authentication
  • Authorization
  • WAF
  • Rate limiting
  • Request validation
  • DDoS protection
  • CORS restrictions
  • Logging and monitoring
flowchart LR
    Client --> DDoS[DDoS Protection]
    DDoS --> WAF[Web Application Firewall]
    WAF --> Gateway[API Gateway]
    Gateway --> Function[Serverless Function]

Direct public function URLs should be used only when their security and management limitations are understood.

Authentication and Authorization

Authentication confirms identity.

Authorization confirms permission.

Common serverless identity options include:

  • OAuth 2.0
  • OpenID Connect
  • JWT
  • IAM authentication
  • Managed identity
  • Service accounts
  • Mutual TLS
  • Signed requests

The function should not trust user-supplied roles or identifiers without validating signed identity claims.

Store Secrets Securely

Do not hardcode secrets in:

  • Source code
  • Git repositories
  • Deployment templates
  • Container images
  • Environment files
  • Log statements

Use managed secret stores such as:

  • AWS Secrets Manager
  • Azure Key Vault
  • Google Secret Manager
  • Secure parameter stores
flowchart LR
    Function[Serverless Function] --> Identity[Runtime Identity]
    Identity --> SecretManager[Secret Manager]
    SecretManager --> Secret[Authorized Secret]

Secret access should be logged and limited through IAM.

Environment Variables

Environment variables are appropriate for non-secret configuration such as:

  • Environment name
  • Topic name
  • Bucket name
  • Feature flag
  • Service URL
  • Log level

Sensitive values should be injected from a secret manager.

Configuration should be externalized so the same code can be promoted across environments.

Encrypt Data

Sensitive data should be protected:

  • In transit using TLS
  • At rest using managed encryption
  • Through customer-managed keys when required
  • Through key rotation
  • Through restricted key policies

Encryption keys should not be embedded in application code.

Minimize Sensitive Data

Do not include unnecessary sensitive information in:

  • Events
  • Queue messages
  • Workflow input
  • Logs
  • Traces
  • Error responses
  • Dead-letter queues

Serverless events may be:

  • Retried
  • Archived
  • Replayed
  • Routed to multiple consumers
  • Stored for debugging

Publish the minimum information required.

Use Structured Logging

Structured logs make it easier to search, filter, and correlate events.

Example:

{
  "severity": "INFO",
  "function": "process-order",
  "correlationId": "corr-10025",
  "orderId": "ORD-10025",
  "status": "PAYMENT_COMPLETED",
  "durationMs": 245
}

Important log fields include:

  • Timestamp
  • Severity
  • Function name
  • Environment
  • Correlation ID
  • Trace ID
  • Event ID
  • Business identifier
  • Error code
  • Processing duration

Avoid Sensitive Logging

Never log:

  • Passwords
  • Access tokens
  • Private keys
  • Full payment-card data
  • Secret values
  • Unmasked personal information
  • Authorization headers

Use masking and redaction controls.

Distributed Tracing

A single business request may cross:

  • API Gateway
  • Multiple functions
  • Queues
  • Event buses
  • Workflows
  • Databases
  • External APIs
flowchart LR
    Client --> Gateway[API Gateway]
    Gateway --> FunctionA[Order Function]
    FunctionA --> Queue[Message Queue]
    Queue --> FunctionB[Payment Function]
    FunctionB --> Database[(Database)]

    Gateway --> Trace[Distributed Trace]
    FunctionA --> Trace
    FunctionB --> Trace

Trace context should be propagated through:

  • HTTP headers
  • Event metadata
  • Message attributes
  • Workflow context
  • Log fields

Monitor Technical and Business Metrics

Technical metrics include:

  • Invocation count
  • Error rate
  • Duration
  • Timeout count
  • Throttling
  • Concurrent executions
  • Cold starts
  • Queue depth
  • Dead-letter messages
  • Database connections

Business metrics include:

  • Orders completed
  • Payments failed
  • Claims processed
  • Notifications delivered
  • Files rejected
  • Accounts created

A technically successful function may still produce incorrect business outcomes.

Recommended Alerts

Create alerts for:

  • High error rate
  • Timeout increase
  • Throttled invocations
  • DLQ messages
  • Queue backlog
  • Missing expected events
  • Sudden traffic spikes
  • Sudden traffic drops
  • Database connection exhaustion
  • High latency
  • Cost anomalies
  • Authentication failures

A drop to zero events may indicate a broken upstream producer.

Define Service-Level Objectives

A production serverless application should define objectives such as:

  • Availability
  • Success rate
  • P95 latency
  • P99 latency
  • Maximum processing delay
  • Event-delivery delay
  • Recovery time
  • Error-budget thresholds

Example:

99.9% of order requests must complete successfully.
95% of synchronous requests must finish within 500 milliseconds.
99% of queued orders must begin processing within 60 seconds.

Use Infrastructure as Code

Serverless infrastructure should be defined using tools such as:

  • AWS CloudFormation
  • AWS SAM
  • AWS CDK
  • Terraform
  • Azure Bicep
  • Google Cloud deployment tools

Infrastructure as code should define:

  • Functions
  • Roles
  • Triggers
  • Queues
  • Event buses
  • API routes
  • Environment variables
  • Alarms
  • Dead-letter queues
  • Network settings

Benefits include:

  • Repeatability
  • Reviewability
  • Version control
  • Environment consistency
  • Faster recovery
  • Reduced manual errors

CI/CD Pipeline

flowchart LR
    Developer[Developer] --> Repository[Source Repository]
    Repository --> Build[Build Pipeline]
    Build --> UnitTests[Unit Tests]
    UnitTests --> Scan[Security Scan]
    Scan --> Package[Package Function]
    Package --> Deploy[Deploy to Test]
    Deploy --> Integration[Integration Tests]
    Integration --> Approval[Production Approval]
    Approval --> Production[Production Deployment]

A production pipeline should include:

  • Compilation
  • Unit tests
  • Integration tests
  • Dependency scanning
  • Secret scanning
  • Static analysis
  • Infrastructure validation
  • Deployment approvals
  • Smoke tests
  • Rollback capability

Use Safe Deployment Strategies

Versioning

Deploy immutable function versions.

Aliases or Revisions

Use stable names that point to deployed versions.

Canary Deployment

Send a small portion of traffic to the new version.

flowchart LR
    Traffic[Incoming Traffic] --> Router[Traffic Router]
    Router -->|90 Percent| Stable[Stable Version]
    Router -->|10 Percent| Canary[New Version]

Blue-Green Deployment

Maintain old and new environments, then switch traffic.

New deployments should be monitored for:

  • Error rate
  • Latency
  • Timeouts
  • Business failures
  • Resource usage

Keep Deployments Small

Large packages can increase:

  • Build time
  • Upload time
  • Cold-start latency
  • Security surface
  • Dependency conflicts

Recommended techniques include:

  • Remove unused dependencies.
  • Use production-only dependencies.
  • Separate shared libraries carefully.
  • Avoid packaging development tools.
  • Use optimized container layers.
  • Keep one deployment focused on one responsibility.

Dependency Management

Dependencies must be:

  • Version controlled
  • Scanned for vulnerabilities
  • Updated regularly
  • Compatible with the runtime
  • Kept minimal
  • Licensed appropriately

Avoid unpinned dependency versions that may produce unpredictable builds.

Testing Strategy

Serverless testing should include multiple layers.

Test Type Purpose
Unit test Tests function logic
Contract test Validates API and event schemas
Integration test Tests managed-service integration
End-to-end test Tests the full business flow
Load test Measures scaling and latency
Failure test Validates retry and recovery
Security test Validates permissions and input handling
Resilience test Tests dependency failures

Unit Testing

Business logic should be separated from platform-specific handler code.

public class OrderHandler {

    private final OrderService orderService;

    public OrderHandler(OrderService orderService) {
        this.orderService = orderService;
    }

    public OrderResponse handle(OrderRequest request) {
        return orderService.process(request);
    }
}

This makes it easier to test logic without deploying the function.

Integration Testing

Integration tests should validate:

  • Function invocation
  • IAM permissions
  • Queue integration
  • Event routing
  • Database access
  • Secret access
  • Retry behavior
  • Dead-letter routing

Local emulation can help development, but managed-cloud integration tests are still required.

Test Failure Scenarios

Test what happens when:

  • The database is unavailable.
  • The external API times out.
  • The queue redelivers a message.
  • The same event arrives twice.
  • The function reaches maximum concurrency.
  • The DLQ receives a message.
  • A secret cannot be retrieved.
  • A deployment must be rolled back.
  • A region becomes unavailable.

Happy-path tests alone are not sufficient.

Handle Partial Failures

Batch-processing functions may receive multiple records.

If one record fails, retrying the entire batch can duplicate successfully processed records.

flowchart TD
    Batch[Batch of Records] --> Process[Process Each Record]
    Process --> R1[Record 1 Success]
    Process --> R2[Record 2 Failure]
    Process --> R3[Record 3 Success]

    R2 --> Retry[Retry Failed Record]

Use partial-batch failure reporting where supported.

Each record should remain idempotent.

Use Batch Processing Carefully

Batching can improve:

  • Throughput
  • Cost efficiency
  • Database efficiency
  • Network utilization

Large batches can increase:

  • Processing time
  • Memory usage
  • Retry impact
  • Duplicate work
  • Failure complexity

Batch size should be tuned using production-like load testing.

Circuit Breaker Pattern

A circuit breaker prevents continuous calls to an unhealthy dependency.

flowchart TD
    Request[Request] --> Circuit{Circuit State}

    Circuit -->|Closed| Dependency[Call Dependency]
    Dependency --> Result{Success?}

    Result -->|Yes| Complete[Complete]
    Result -->|No| Count[Increase Failure Count]

    Count --> Threshold{Threshold Reached?}
    Threshold -->|Yes| Open[Open Circuit]
    Threshold -->|No| Dependency

    Circuit -->|Open| FailFast[Fail Fast]

Circuit breaking can be implemented through:

  • Application libraries
  • API gateways
  • Service meshes
  • Workflow logic
  • Managed integration services

Bulkhead Pattern

The bulkhead pattern isolates resources so one workload cannot consume all capacity.

Examples include:

  • Separate queues for high- and low-priority requests
  • Reserved concurrency per function
  • Separate database pools
  • Separate event buses by domain
  • Separate deployment accounts
flowchart LR
    HighPriority[High-Priority Requests] --> HighQueue[High-Priority Queue]
    LowPriority[Low-Priority Requests] --> LowQueue[Low-Priority Queue]

    HighQueue --> HighFunction[Reserved Function Capacity]
    LowQueue --> LowFunction[Limited Function Capacity]

Multi-Region Design

Serverless services are regional unless a global service is explicitly used.

A resilient architecture may require:

  • Multiple regions
  • Global DNS
  • Replicated data
  • Cross-region event routing
  • Regional API endpoints
  • Failover procedures
flowchart TB
    Users[Global Users] --> DNS[Global DNS]

    DNS --> RegionA[Region A API]
    DNS --> RegionB[Region B API]

    RegionA --> FunctionA[Regional Functions]
    RegionB --> FunctionB[Regional Functions]

    FunctionA --> DatabaseA[(Regional Database)]
    FunctionB --> DatabaseB[(Regional Database)]

Data consistency and failover behavior must be explicitly designed.

Disaster Recovery

Serverless disaster recovery should define:

  • Recovery Time Objective
  • Recovery Point Objective
  • Backup policy
  • Data replication
  • Infrastructure recreation
  • DNS failover
  • Secret replication
  • Event replay
  • Queue recovery
  • Operational ownership

Infrastructure as code is essential for rebuilding the environment.

Cost Optimization

Serverless does not automatically mean inexpensive.

Cost may come from:

  • Function invocations
  • Execution duration
  • Memory allocation
  • API Gateway requests
  • Workflow transitions
  • Queue operations
  • Event routing
  • Logging
  • Data transfer
  • NAT gateways
  • Database usage
  • Provisioned concurrency
  • Secret retrieval

Common Cost Problems

Excessive Logging

Large debug logs can cost more than function execution.

Chatty Architecture

Too many functions and state transitions increase request cost and latency.

Long Execution Times

Slow database queries and external calls increase duration-based billing.

Uncontrolled Retries

Repeated failures create additional invocations.

NAT Gateway Dependency

Private functions accessing the internet through NAT may create significant cost.

Over-Provisioned Warm Capacity

Provisioned or minimum instances introduce baseline charges.

Cost Optimization Practices

  • Right-size memory and CPU.
  • Reduce execution duration.
  • Avoid unnecessary function calls.
  • Use direct service integrations.
  • Set log-retention policies.
  • Use sampling for verbose logs.
  • Control retries.
  • Configure maximum concurrency.
  • Use queues to smooth traffic.
  • Review API Gateway and workflow costs.
  • Avoid unnecessary NAT traffic.
  • Monitor cost by service and function.
  • Configure budget alerts.
  • Remove unused functions and versions.

Portability and Vendor Lock-In

Serverless applications often depend on provider-specific services.

Examples include:

  • Trigger formats
  • IAM models
  • Workflow definitions
  • Event buses
  • Monitoring tools
  • Database APIs

Complete provider independence may reduce the benefits of managed services.

A balanced approach includes:

  • Separate business logic from handlers.
  • Use standard protocols where practical.
  • Use portable event schemas.
  • Encapsulate provider SDK calls.
  • Document provider dependencies.
  • Choose managed services intentionally.

Avoid adding abstraction solely to claim portability when migration is unlikely.

Production Use Case: Order Processing Platform

flowchart TB
    Client[Customer Application] --> CDN[CDN]
    CDN --> WAF[Web Application Firewall]
    WAF --> Gateway[API Gateway]

    Gateway --> Auth[Identity Provider]
    Gateway --> OrderFunction[Order Function]

    OrderFunction --> Orders[(Orders Database)]
    OrderFunction --> EventBus[Order Event Bus]

    EventBus --> PaymentRule[Payment Rule]
    EventBus --> InventoryRule[Inventory Rule]
    EventBus --> NotificationRule[Notification Rule]

    PaymentRule --> PaymentQueue[Payment Queue]
    InventoryRule --> InventoryQueue[Inventory Queue]

    PaymentQueue --> PaymentFunction[Payment Function]
    InventoryQueue --> InventoryFunction[Inventory Function]

    PaymentFunction --> PaymentProvider[Payment Provider]
    InventoryFunction --> InventoryDB[(Inventory Database)]

    NotificationRule --> NotificationFunction[Notification Function]

    PaymentQueue --> PaymentDLQ[Payment DLQ]
    InventoryQueue --> InventoryDLQ[Inventory DLQ]

    OrderFunction --> SecretManager[Secret Manager]
    PaymentFunction --> SecretManager

    OrderFunction --> Observability[Logs Metrics and Traces]
    PaymentFunction --> Observability
    InventoryFunction --> Observability

Production Processing Flow

  1. The client authenticates and submits an order.
  2. The API Gateway validates and rate-limits the request.
  3. The order function validates the business input.
  4. The order is stored using an idempotency key.
  5. An order-created event is published.
  6. Event rules route the event to payment, inventory, and notification consumers.
  7. Queues buffer payment and inventory workloads.
  8. Worker functions process messages using controlled concurrency.
  9. Transient errors are retried with exponential backoff.
  10. Permanent failures are sent to dead-letter queues.
  11. Every function emits structured logs, metrics, and traces.
  12. Alerts notify the operations team when business or technical thresholds fail.

Interview Questions and Answers

1. What are the most important serverless best practices?

Answer

The most important practices include:

  • Keep functions focused.
  • Design functions to be stateless.
  • Make event handlers idempotent.
  • Reuse expensive clients and connections.
  • Configure realistic timeouts.
  • Retry only transient failures.
  • Use queues for buffering.
  • Configure dead-letter queues.
  • Apply least-privilege IAM.
  • Store secrets in managed secret stores.
  • Use structured logging and distributed tracing.
  • Control automatic scaling.
  • Deploy through CI/CD.
  • Monitor cost and business outcomes.

2. Why should serverless functions be stateless?

Answer

Function instances are temporary and can be created or removed at any time.

The same instance may not process the next request, and multiple instances may run simultaneously.

Persistent state should therefore be stored in external managed services such as databases, object storage, or distributed caches.

In-memory state may be used only for temporary processing or safe client reuse.

3. Why is idempotency important in serverless systems?

Answer

Serverless platforms, queues, event buses, and clients may retry requests.

The same event can therefore be processed multiple times.

Without idempotency, duplicate processing may cause:

  • Duplicate payments
  • Duplicate orders
  • Repeated emails
  • Incorrect inventory updates
  • Duplicate database records

Idempotency keys, event IDs, conditional writes, and uniqueness constraints help prevent duplicate business outcomes.

4. How do you reduce serverless cold starts?

Answer

Cold starts can be reduced by:

  • Reducing package size
  • Minimizing dependencies
  • Avoiding heavy startup logic
  • Reusing initialized clients
  • Selecting an efficient runtime
  • Allocating appropriate memory and CPU
  • Using provisioned or minimum instances for critical APIs
  • Avoiding unnecessary startup network calls

Cold-start performance should be measured under realistic idle and scale-out conditions.

5. How do you prevent a serverless function from overwhelming a database?

Answer

Recommended controls include:

  • Maximum function concurrency
  • Queue buffering
  • Controlled batch size
  • Database connection pooling
  • Database proxy services
  • Short transactions
  • Connection monitoring
  • Rate limiting
  • Backpressure

The maximum number of possible connections should be calculated across all concurrent function instances.

6. What is the correct retry strategy for serverless applications?

Answer

Retry only transient failures.

Use:

  • Exponential backoff
  • Jitter
  • Maximum retry count
  • Maximum event age
  • Dead-letter queues
  • Idempotent handlers

Do not repeatedly retry permanent errors such as invalid input, unauthorized requests, or business-rule rejections.

7. How do you secure a serverless application?

Answer

A secure serverless architecture should:

  • Require authentication.
  • Apply authorization.
  • Use least-privilege runtime identities.
  • Store secrets in managed secret stores.
  • Validate all inputs.
  • Encrypt data in transit and at rest.
  • Protect public APIs with WAF and rate limiting.
  • Restrict network access.
  • Scan dependencies.
  • Mask sensitive logs.
  • Enable audit logging.
  • Separate build, deployment, and runtime identities.

8. How do you monitor serverless applications?

Answer

Serverless monitoring should include:

  • Centralized structured logs
  • Invocation metrics
  • Error rates
  • Timeouts
  • Latency percentiles
  • Cold starts
  • Throttling
  • Queue backlog
  • DLQ messages
  • Distributed traces
  • Business-success metrics
  • Cost anomalies

Correlation IDs should be propagated across API gateways, functions, queues, events, and workflows.

9. How do you optimize serverless cost?

Answer

Cost optimization techniques include:

  • Right-size memory and CPU.
  • Reduce execution duration.
  • Avoid unnecessary invocations.
  • Use direct service integrations.
  • Control retries.
  • Configure log retention.
  • Avoid excessive debug logging.
  • Use asynchronous processing.
  • Avoid unnecessary NAT traffic.
  • Review provisioned concurrency.
  • Remove unused functions and versions.
  • Configure budgets and cost alerts.

Cost should be evaluated across the complete architecture, not only function execution.

10. When should serverless not be used?

Answer

Serverless may not be the best option for:

  • Continuously running workloads
  • Extremely predictable high-utilization compute
  • Applications requiring complete operating-system control
  • Specialized hardware requirements
  • Large monolithic applications
  • Workloads requiring persistent local state
  • Ultra-low-latency processing where cold starts are unacceptable
  • Workloads exceeding platform execution limits
  • Applications where provider-specific constraints create unacceptable risk

Containers, Kubernetes, virtual machines, or managed batch platforms may be more suitable.

Common Interview Follow-Up Questions

  • How do serverless cold starts occur?
  • What is reserved concurrency?
  • What is provisioned concurrency?
  • Why should functions be idempotent?
  • What is a poison message?
  • What is the purpose of a dead-letter queue?
  • What is exponential backoff?
  • How do you avoid database connection exhaustion?
  • How do you prevent recursive function invocations?
  • What should be logged in a serverless function?
  • How do you test event-driven systems?
  • How do you deploy serverless applications safely?
  • How do you design multi-region serverless systems?
  • How do you control serverless cost?
  • What are the main serverless security risks?

Common Mistakes

Building Large Monolithic Functions

Large functions are harder to test, deploy, secure, and scale.

Creating Too Many Tiny Functions

Over-fragmentation increases latency and operational complexity.

Assuming Exactly-Once Delivery

Retries and duplicate delivery must be expected.

Ignoring Idempotency

Duplicate events can produce duplicate business transactions.

Unlimited Scaling

Rapid scaling can overwhelm databases and external systems.

Opening Connections Per Invocation

This can exhaust database capacity.

Hardcoding Secrets

Secrets should be stored in managed secret services.

Logging Sensitive Data

Logs may remain available for long periods and be accessed by multiple teams.

Missing DLQ Alerts

A dead-letter queue must be actively monitored.

Retrying Permanent Failures

Invalid data will not become valid through repeated retries.

Ignoring Cold Starts

Latency-sensitive workloads require startup optimization.

Testing Only the Happy Path

Retries, timeouts, duplicate events, and dependency failures must be tested.

Troubleshooting Serverless Applications

Function Has High Latency

Check:

  • Cold starts
  • Memory and CPU
  • Dependency size
  • Initialization logic
  • Database queries
  • External API latency
  • VPC networking
  • Concurrency
  • Package size

Function Is Timing Out

Check:

  • Downstream timeouts
  • Slow database queries
  • External APIs
  • Infinite loops
  • Batch size
  • Network connectivity
  • Function timeout setting

Duplicate Records Are Created

Check:

  • Idempotency-key handling
  • Queue redelivery
  • Event retries
  • Client retries
  • Unique database constraints
  • Timeout ambiguity

Database Connections Are Exhausted

Check:

  • Connection-pool size
  • Maximum concurrency
  • Connection reuse
  • Long transactions
  • Database proxy usage
  • Failed connection cleanup

Events Are Sent to the DLQ

Check:

  • Payload format
  • Function exceptions
  • Permissions
  • Timeout
  • Retry count
  • Dependency availability
  • Schema compatibility

Cost Increased Unexpectedly

Check:

  • Invocation volume
  • Retry loops
  • Excessive logs
  • Long execution duration
  • NAT traffic
  • Provisioned capacity
  • Recursive events
  • Workflow transitions

Serverless Best-Practices Checklist

Design

  • Keep functions focused.
  • Avoid excessive fragmentation.
  • Keep functions stateless.
  • Separate business logic from handlers.
  • Use asynchronous processing where appropriate.
  • Use queues for backpressure.
  • Use event buses for fan-out.
  • Design event schemas.

Reliability

  • Make handlers idempotent.
  • Retry transient errors only.
  • Use exponential backoff and jitter.
  • Configure dead-letter queues.
  • Configure timeouts.
  • Handle partial-batch failures.
  • Prevent event loops.
  • Test compensation and replay.

Security

  • Use least-privilege IAM.
  • Use dedicated runtime identities.
  • Store secrets securely.
  • Validate all inputs.
  • Require authentication.
  • Apply authorization.
  • Restrict CORS.
  • Protect APIs with WAF.
  • Mask sensitive logs.

Performance

  • Reduce cold starts.
  • Minimize dependencies.
  • Reuse initialized clients.
  • Right-size memory and CPU.
  • Tune concurrency.
  • Protect database connections.
  • Tune batch sizes.

Operations

  • Use structured logging.
  • Propagate trace IDs.
  • Monitor technical metrics.
  • Monitor business metrics.
  • Alert on DLQs and queue backlogs.
  • Define SLOs.
  • Document runbooks.
  • Test disaster recovery.

Delivery

  • Use infrastructure as code.
  • Use automated CI/CD.
  • Scan dependencies and secrets.
  • Use immutable versions.
  • Use canary or blue-green deployments.
  • Validate rollback procedures.

Cost

  • Monitor invocation volume.
  • Reduce execution duration.
  • Control retries.
  • Set log retention.
  • Review warm-capacity settings.
  • Avoid unnecessary NAT traffic.
  • Configure budget alerts.

Quick Revision

Topic Key Point
Focused function Performs one clear responsibility
Stateless design Stores persistent data externally
Cold start New runtime-instance initialization
Idempotency Prevents duplicate business outcomes
Retry Used for temporary failures
Backoff Gradually increases retry delay
Jitter Randomizes retry timing
DLQ Stores repeatedly failed messages
Queue Provides buffering and backpressure
Event bus Routes events to multiple consumers
Concurrency limit Protects downstream capacity
Least privilege Grants only required permissions
Secret manager Stores credentials securely
Structured logging Produces searchable log fields
Trace ID Connects distributed operations
Infrastructure as code Recreates serverless resources consistently
Canary deployment Sends limited traffic to a new version
RTO Target recovery time
RPO Acceptable data-loss window
Cost optimization Controls total architecture spending

Key Takeaways

  • Serverless removes server management but not architecture and operational responsibility.
  • Functions should be focused, stateless, idempotent, secure, and observable.
  • Cold starts should be measured and optimized according to business latency requirements.
  • Queues provide buffering, backpressure, and controlled processing.
  • Event buses provide routing and fan-out.
  • Retries should use exponential backoff, jitter, and bounded attempts.
  • Dead-letter queues require monitoring, ownership, and replay procedures.
  • Automatic scaling must be limited to protect databases and downstream APIs.
  • Least-privilege IAM and managed secret storage are essential security controls.
  • Structured logs, distributed traces, technical metrics, and business metrics are all required.
  • Infrastructure as code, CI/CD, immutable versions, and safe rollout strategies reduce deployment risk.
  • Testing must include failures, duplicates, retries, timeouts, and disaster-recovery scenarios.
  • Serverless cost must be evaluated across functions, APIs, queues, workflows, logging, networking, and databases.
  • Serverless is a strong architecture choice when its scaling, operational, and event-driven characteristics match the workload.

Module Complete

You have completed the Serverless Module.

Completed Lessons

  1. Serverless Basics QA
  2. AWS Lambda QA
  3. Azure Functions QA
  4. Google Cloud Functions QA
  5. API Gateway QA
  6. EventBridge QA
  7. Step Functions QA
  8. Serverless Best Practices QA