Module Navigation

Previous: EventBridge QA | Parent: Serverless Learning Path | Next: Serverless Best Practices QA

---

Introduction

AWS Step Functions is a serverless workflow-orchestration service used to coordinate distributed applications, microservices, AWS services, and business processes.

Instead of writing custom code to manage the sequence of operations, developers define a workflow as a state machine.

The state machine describes:

• Which task runs first
• Which task runs next
• How data moves between tasks
• Which conditions determine the next step
• Which tasks can run in parallel
• How failures should be retried
• How errors should be handled
• How long the workflow should wait
• How compensation should occur
• When the workflow succeeds or fails

AWS Step Functions is commonly used for:

• Serverless application orchestration
• Order-processing workflows
• Payment processing
• Data-processing pipelines
• Human approval workflows
• Microservice coordination
• Machine-learning pipelines
• Batch-job orchestration
• Infrastructure automation
• Long-running business processes

---

What Is AWS Step Functions?

AWS Step Functions allows developers to define workflows that coordinate multiple services.

A workflow can call services such as:

• AWS Lambda
• Amazon ECS
• AWS Fargate
• AWS Batch
• Amazon DynamoDB
• Amazon SQS
• Amazon SNS
• Amazon EventBridge
• Amazon API Gateway
• AWS Glue
• Amazon SageMaker
• Other AWS APIs
• External HTTPS APIs

flowchart LR
    Client[Client Application] --> API[Amazon API Gateway]
    API --> Workflow[AWS Step Functions]

    Workflow --> Validate[AWS Lambda Validation]
    Workflow --> Payment[Payment Service]
    Workflow --> Inventory[Inventory Service]
    Workflow --> Notification[Amazon SNS]

Step Functions manages the workflow state, transitions, retries, error handling, and execution history.

---

Why Use Step Functions?

Without a workflow orchestrator, developers may create one Lambda function or service that:

• Calls several downstream services
• Maintains workflow state
• Implements retries
• Handles timeouts
• Manages conditional logic
• Coordinates parallel operations
• Tracks failures
• Performs compensation
• Stores intermediate results

This produces tightly coupled and difficult-to-maintain code.

Step Functions moves workflow-control logic into a declarative state-machine definition.

flowchart TB
    Workflow[Step Functions State Machine]

    Workflow --> Sequence[Task Sequencing]
    Workflow --> Conditions[Conditional Routing]
    Workflow --> Parallel[Parallel Execution]
    Workflow --> Retry[Retries]
    Workflow --> Errors[Error Handling]
    Workflow --> State[Workflow State]

---

Orchestration vs Choreography

Orchestration

A central workflow controls each processing step.

flowchart LR
    StepFunctions[Step Functions] --> Payment[Payment Service]
    StepFunctions --> Inventory[Inventory Service]
    StepFunctions --> Shipping[Shipping Service]

The orchestrator knows:

• Which services participate
• The execution order
• Failure behavior
• Compensation behavior

Choreography

Services react independently to events.

flowchart LR
    Order[Order Service] --> EventBridge[Amazon EventBridge]
    EventBridge --> Payment[Payment Service]
    EventBridge --> Inventory[Inventory Service]
    EventBridge --> Notification[Notification Service]

Orchestration	Choreography
Central workflow coordinator	Services respond independently
Easier to visualize	More loosely coupled
Centralized error handling	Distributed error handling
Suitable for ordered processes	Suitable for event fan-out
Step Functions is commonly used	EventBridge is commonly used

An enterprise application may use both patterns.

---

State Machine

A state machine is the workflow definition executed by Step Functions.

It contains:

• A starting state
• One or more states
• Transitions between states
• Input and output processing
• Terminal success or failure states

flowchart LR
    Start[Start] --> Validate[Validate Order]
    Validate --> Payment[Process Payment]
    Payment --> Inventory[Reserve Inventory]
    Inventory --> Notify[Send Notification]
    Notify --> Success[Success]

Each state represents one stage of the workflow.

---

Amazon States Language

Workflows are defined using Amazon States Language, commonly called ASL.

ASL is a JSON-based language used to define:

• States
• Transitions
• Tasks
• Conditions
• Retries
• Error handling
• Parallel processing
• Input and output transformations

---

Basic ASL Example

{
  "Comment": "Simple order workflow",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "validate-order",
        "Payload": "{% $states.input %}"
      },
      "Next": "ProcessPayment"
    },
    "ProcessPayment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Arguments": {
        "FunctionName": "process-payment",
        "Payload": "{% $states.input %}"
      },
      "End": true
    }
  }
}

Important fields include:

Field	Purpose
Comment	Describes the workflow
StartAt	Identifies the first state
States	Contains state definitions
Type	Defines the state type
Resource	Identifies the integrated service
Next	Defines the next state
End	Marks a terminal state

---

Step Functions State Types

The main state types are:

State	Purpose
Task	Performs work
Choice	Selects a processing path
Parallel	Executes multiple branches concurrently
Map	Processes a collection of items
Wait	Delays execution
Pass	Passes or transforms data
Succeed	Ends the workflow successfully
Fail	Ends the workflow with failure

---

Task State

A Task state performs one unit of work.

It can:

• Invoke Lambda
• Send an SQS message
• Publish an SNS message
• Update DynamoDB
• Start an ECS task
• Run an AWS Batch job
• Start another Step Functions workflow
• Call an AWS API
• Invoke an HTTPS endpoint

flowchart LR
    Input[Workflow Input] --> Task[Task State]
    Task --> Lambda[AWS Lambda]
    Lambda --> Output[Task Output]

Example:

{
  "ProcessPayment": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Arguments": {
      "FunctionName": "payment-function",
      "Payload": "{% $states.input %}"
    },
    "Next": "ReserveInventory"
  }
}

---

Choice State

A Choice state provides conditional routing.

flowchart TD
    Order[Order] --> Choice{Order Amount}

    Choice -->|Less Than 1000| Standard[Standard Processing]
    Choice -->|1000 or More| Review[Manual Review]

Example:

{
  "CheckOrderAmount": {
    "Type": "Choice",
    "Choices": [
      {
        "Condition": "{% $states.input.amount >= 1000 %}",
        "Next": "ManualReview"
      },
      {
        "Condition": "{% $states.input.amount < 1000 %}",
        "Next": "ProcessOrder"
      }
    ],
    "Default": "RejectOrder"
  }
}

Choice states are useful for:

• Business-rule routing
• Approval decisions
• Error-code handling
• Customer-type routing
• Payment-method selection

---

Parallel State

A Parallel state runs multiple branches at the same time.

flowchart LR
    Start[Order Confirmed] --> Parallel[Parallel State]

    Parallel --> Email[Send Email]
    Parallel --> SMS[Send SMS]
    Parallel --> Analytics[Update Analytics]

    Email --> Complete[Complete]
    SMS --> Complete
    Analytics --> Complete

The workflow waits for all branches to complete unless one branch fails and the error is not handled.

Use Parallel when operations:

• Are independent
• Can run concurrently
• Must all complete before continuing

---

Map State

A Map state processes each item in a collection.

flowchart LR
    Items[Order Items] --> Map[Map State]

    Map --> Item1[Process Item 1]
    Map --> Item2[Process Item 2]
    Map --> Item3[Process Item 3]

    Item1 --> Results[Combined Results]
    Item2 --> Results
    Item3 --> Results

Common use cases include:

• Processing order items
• Resizing images
• Validating records
• Sending notifications
• Processing files
• Transforming datasets

---

Distributed Map

Distributed Map is designed for large-scale parallel processing.

It runs iterations as child workflow executions.

flowchart TB
    Dataset[Large Dataset in Amazon S3] --> DistributedMap[Distributed Map]

    DistributedMap --> Child1[Child Workflow 1]
    DistributedMap --> Child2[Child Workflow 2]
    DistributedMap --> Child3[Child Workflow 3]
    DistributedMap --> ChildN[Child Workflow N]

    Child1 --> Results[Result Storage]
    Child2 --> Results
    Child3 --> Results
    ChildN --> Results

Distributed Map is useful when:

• The input dataset is large
• High concurrency is required
• Workflow-history limits may be exceeded
• Data is stored in Amazon S3
• Each item needs an independent child execution

Typical use cases include:

• Large file processing
• Data migration
• Batch transformation
• Security scanning
• Machine-learning preprocessing

---

Wait State

A Wait state pauses execution for a specified duration or until a timestamp.

flowchart LR
    Submit[Submit Request] --> Wait[Wait 24 Hours]
    Wait --> Check[Check Status]

Wait states are useful for:

• Cooling-off periods
• Delayed notifications
• Polling intervals
• Subscription expiration
• Payment settlement
• Approval deadlines

Example:

{
  "WaitBeforeRetry": {
    "Type": "Wait",
    "Seconds": 30,
    "Next": "CheckStatus"
  }
}

Step Functions manages the wait without keeping a server or Lambda function running.

---

Pass State

A Pass state performs no external work.

It can be used to:

• Transform data
• Add static values
• Test workflow paths
• Provide default values
• Improve workflow readability

{
  "AddProcessingStatus": {
    "Type": "Pass",
    "Result": {
      "status": "PROCESSING"
    },
    "Next": "ProcessOrder"
  }
}

---

Succeed State

A Succeed state ends the workflow successfully.

{
  "OrderCompleted": {
    "Type": "Succeed"
  }
}

---

Fail State

A Fail state ends the workflow unsuccessfully.

{
  "OrderRejected": {
    "Type": "Fail",
    "Error": "OrderValidationFailed",
    "Cause": "The order did not pass validation."
  }
}

---

Standard Workflows

Standard Workflows are designed for durable, auditable, and potentially long-running processes.

Common use cases include:

• Payment workflows
• Order processing
• Human approvals
• Account onboarding
• Insurance claims
• Long-running batch processes
• Saga transactions

Important characteristics include:

• Long-running execution support
• Durable workflow state
• Detailed execution history
• Visual execution tracking
• Support for callback tasks
• Support for job-run integration
• Pricing based primarily on state transitions

---

Express Workflows

Express Workflows are designed for high-volume, short-duration processing.

Common use cases include:

• IoT event processing
• High-volume data transformation
• Streaming-event enrichment
• Synchronous API workflows
• Short microservice orchestration
• Mobile application backends

Express Workflows are available in two execution models:

• Asynchronous Express
• Synchronous Express

---

Standard vs Express Workflows

Feature	Standard	Express
Best for	Durable business workflows	High-volume short workflows
Execution duration	Long-running	Short-duration
Execution history	Managed by Step Functions	Commonly viewed through logs
Execution model	Exactly-once workflow execution semantics unless retried by caller	At-least-once for asynchronous; at-most-once for synchronous
Callback pattern	Supported	Not supported in the same way
Job-run .sync pattern	Supported	Limited by integration model
Pricing focus	State transitions	Executions, duration, and memory
Typical use	Payments and approvals	Event processing and APIs

The workflow type cannot be changed after the state machine is created. Create a new state machine when changing workflow type.

---

Workflow Execution Models

Asynchronous Execution

The caller starts the workflow and immediately receives an execution identifier.

sequenceDiagram
    participant Client
    participant API as API Gateway
    participant SF as Step Functions
    participant Worker
    Client->>API: Submit request
    API->>SF: Start execution
    SF-->>API: Return execution ARN
    API-->>Client: HTTP 202 Accepted
    SF->>Worker: Process workflow

Synchronous Execution

The caller waits for the workflow result.

sequenceDiagram
    participant Client
    participant SF as Step Functions
    participant Service
    Client->>SF: Start synchronous execution
    SF->>Service: Execute tasks
    Service-->>SF: Return result
    SF-->>Client: Return workflow result

Use synchronous execution only when the entire workflow can complete within the client and integration timeout limits.

---

Step Functions Service Integrations

Step Functions can integrate with AWS services without requiring a Lambda function for every operation.

Examples include:

• Write an item to DynamoDB
• Send a message to SQS
• Publish to SNS
• Run an ECS task
• Start an AWS Batch job
• Start a Glue job
• Invoke SageMaker
• Publish an EventBridge event
• Start another state machine

flowchart LR
    Workflow[Step Functions] --> DynamoDB[(DynamoDB)]
    Workflow --> SQS[Amazon SQS]
    Workflow --> SNS[Amazon SNS]
    Workflow --> ECS[Amazon ECS]
    Workflow --> EventBridge[Amazon EventBridge]

Direct integrations reduce:

• Lambda code
• Deployment packages
• Maintenance
• Cold starts
• Additional failure points
• Unnecessary cost

---

Service Integration Patterns

Step Functions supports three major service-integration patterns.

Request Response

Step Functions calls a service and continues after receiving the initial response.

flowchart LR
    Workflow --> Service[Call AWS Service]
    Service --> Response[Immediate Response]
    Response --> Next[Next State]

Run a Job

Step Functions starts a job and waits for the job to finish.

This is commonly represented by the .sync integration pattern.

flowchart LR
    Workflow --> Job[Start Batch or ECS Job]
    Job --> Wait[Wait for Completion]
    Wait --> Next[Next State]

Callback with Task Token

Step Functions pauses until an external process returns a task token.

flowchart LR
    Workflow --> Task[Create Task Token]
    Task --> External[External System or Human]
    External --> Callback[Return Task Token]
    Callback --> Continue[Continue Workflow]

---

Callback Pattern

The callback pattern is useful when the workflow must wait for:

• Human approval
• External partner response
• Legacy-system completion
• Manual investigation
• Long-running custom processing

A task token uniquely identifies the paused task.

The external system completes the task by calling:

• SendTaskSuccess
• SendTaskFailure

---

Human Approval Workflow

flowchart TD
    Request[Submit Request] --> Notify[Send Approval Request]
    Notify --> Wait[Wait with Task Token]

    Wait --> Decision{Approver Decision}
    Decision -->|Approved| Process[Process Request]
    Decision -->|Rejected| Reject[Reject Request]
    Decision -->|Timeout| Expire[Expire Request]

Typical use cases include:

• High-value transactions
• Insurance claims
• Access requests
• Loan approvals
• Production deployments
• Expense approvals

A timeout should always be configured so the workflow does not wait indefinitely.

---

Error Handling

Distributed workflows can fail because of:

• Lambda exceptions
• Service throttling
• Network problems
• Invalid input
• Missing resources
• Timeouts
• Permission failures
• Business validation errors

By default, an unhandled state failure causes the workflow to fail.

Step Functions supports:

• Retry
• Catch
• Timeout
• Heartbeat
• Fail states
• Compensation tasks

---

Retry

A Retry block automatically retries a failed state.

{
  "ProcessPayment": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Retry": [
      {
        "ErrorEquals": [
          "Lambda.ServiceException",
          "Lambda.TooManyRequestsException"
        ],
        "IntervalSeconds": 2,
        "MaxAttempts": 3,
        "BackoffRate": 2
      }
    ],
    "Next": "ReserveInventory"
  }
}

Important retry settings include:

Setting	Purpose
ErrorEquals	Errors that should be retried
IntervalSeconds	Delay before the first retry
MaxAttempts	Maximum retry count
BackoffRate	Multiplier applied to retry delays
MaxDelaySeconds	Optional maximum retry interval
JitterStrategy	Helps distribute repeated retry traffic

Retry only transient errors.

Do not repeatedly retry permanent errors such as:

• Invalid input
• Unsupported operation
• Missing mandatory data
• Business-rule rejection

---

Catch

A Catch block routes an error to another state after retries are exhausted or when retry is not configured.

{
  "ProcessPayment": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Catch": [
      {
        "ErrorEquals": ["PaymentDeclined"],
        "Next": "HandlePaymentDecline"
      },
      {
        "ErrorEquals": ["States.ALL"],
        "Next": "HandleUnexpectedFailure"
      }
    ],
    "Next": "ReserveInventory"
  }
}

flowchart TD
    Payment[Process Payment] --> Result{Successful?}
    Result -->|Yes| Inventory[Reserve Inventory]
    Result -->|Declined| Decline[Handle Decline]
    Result -->|Technical Error| Failure[Handle Failure]

---

Retry and Catch Flow

flowchart TD
    Task[Execute Task] --> Success{Successful?}

    Success -->|Yes| Next[Next State]
    Success -->|No| Retry{Retry Available?}

    Retry -->|Yes| Wait[Apply Backoff]
    Wait --> Task

    Retry -->|No| Catch{Catch Handler?}
    Catch -->|Yes| Recovery[Recovery State]
    Catch -->|No| Failed[Workflow Failed]

---

Timeout and Heartbeat

Timeout

A timeout defines the maximum time a task may run.

Heartbeat

A heartbeat confirms that a long-running worker is still active.

flowchart LR
    Task[Long-Running Task] --> Heartbeat[Periodic Heartbeat]
    Heartbeat --> Workflow[Step Functions]

    Workflow -->|Heartbeat Received| Continue[Continue Waiting]
    Workflow -->|Heartbeat Missing| Timeout[Heartbeat Timeout]

Timeouts and heartbeats prevent workflows from waiting forever for failed workers.

---

Data Flow Between States

Each state receives JSON input and produces JSON output.

flowchart LR
    Input[State Input] --> Task[Task Processing]
    Task --> Result[Task Result]
    Result --> Output[State Output]
    Output --> Next[Next State]

Data can be:

• Selected
• Filtered
• Transformed
• Combined
• Assigned to variables
• Passed to the next state

Step Functions supports JSONPath and JSONata-based data processing, depending on the workflow definition.

---

Context Object

The Step Functions Context object provides workflow metadata.

It can contain information such as:

• Execution ID
• Execution name
• State-machine ID
• State name
• Retry count
• Execution start time
• Task token
• Map item information

This information is useful for:

• Correlation IDs
• Logging
• Idempotency
• Audit records
• Task-token callbacks

---

Nested Workflows

One state machine can start another state machine.

flowchart LR
    Parent[Parent Workflow] --> Payment[Payment Child Workflow]
    Parent --> Shipping[Shipping Child Workflow]
    Parent --> Notification[Notification Child Workflow]

Nested workflows help:

• Reuse common processes
• Reduce state-machine complexity
• Separate domain ownership
• Improve testing
• Isolate failures
• Apply independent permissions

Avoid creating excessive nesting that makes tracing difficult.

---

Saga Pattern

A Saga manages a distributed business transaction across multiple services.

Each local transaction has a corresponding compensation action.

Example order process:

1. Create order
2. Charge payment
3. Reserve inventory
4. Arrange shipment

If shipment creation fails:

1. Release inventory
2. Refund payment
3. Cancel order

---

Saga Architecture

flowchart TD
    Start[Start Order] --> Create[Create Order]
    Create --> Payment[Charge Payment]
    Payment --> Inventory[Reserve Inventory]
    Inventory --> Shipping[Create Shipment]

    Shipping --> Success{Shipment Created?}

    Success -->|Yes| Complete[Order Completed]
    Success -->|No| Release[Release Inventory]
    Release --> Refund[Refund Payment]
    Refund --> Cancel[Cancel Order]

Step Functions is well suited for orchestrated Sagas because it clearly defines:

• Forward transactions
• Compensation transactions
• Retry behavior
• Failure paths
• Workflow state

---

Compensation Transactions

Compensation is not always a database rollback.

Examples include:

Forward Action	Compensation Action
Charge payment	Refund payment
Reserve inventory	Release inventory
Create shipment	Cancel shipment
Allocate reward points	Remove reward points
Activate account	Deactivate account

Compensation tasks should also be:

• Idempotent
• Retry-safe
• Observable
• Auditable

---

Step Functions with EventBridge

EventBridge can start Step Functions when a business event occurs.

flowchart LR
    Application[Application] --> EventBridge[Amazon EventBridge]
    EventBridge --> Rule[Matching Rule]
    Rule --> Workflow[Step Functions Workflow]

Step Functions can also publish events to EventBridge during or after execution.

flowchart LR
    Workflow[Step Functions] --> EventBridge[Amazon EventBridge]
    EventBridge --> Notification[Notification Consumer]
    EventBridge --> Analytics[Analytics Consumer]

Use EventBridge for event routing and Step Functions for controlled workflow orchestration.

---

Step Functions with API Gateway

flowchart LR
    Client[Client] --> APIGateway[API Gateway]
    APIGateway --> StepFunctions[Step Functions]
    StepFunctions --> Services[Backend Services]

API Gateway can:

• Start an asynchronous workflow
• Start a synchronous Express workflow
• Return an execution identifier
• Expose a status endpoint

For long-running Standard Workflows, return HTTP 202 Accepted and an execution or business request ID.

---

Step Functions with SQS

flowchart LR
    Workflow[Step Functions] --> SQS[Amazon SQS]
    SQS --> Worker[Worker Service]
    Worker --> Callback[Task Token Callback]
    Callback --> Workflow

SQS provides:

• Buffering
• Backpressure
• Consumer decoupling
• Retry handling
• Dead-letter queues

---

Step Functions with ECS and Fargate

Step Functions can start container workloads for processing that is not suitable for Lambda.

flowchart LR
    Workflow[Step Functions] --> Fargate[ECS Fargate Task]
    Fargate --> Processing[Heavy Processing]
    Processing --> S3[(Amazon S3)]
    Fargate --> Complete[Task Completion]
    Complete --> Workflow

Use ECS or Fargate for:

• Large dependencies
• Long-running compute
• Custom container environments
• CPU-intensive processing
• Memory-intensive workloads

---

Step Functions with AWS Batch

flowchart LR
    Workflow[Step Functions] --> Batch[AWS Batch Job]
    Batch --> Compute[Batch Compute Environment]
    Compute --> Results[(Amazon S3)]
    Results --> Workflow

This integration is useful for:

• Large data processing
• Scientific calculations
• Financial simulations
• Report generation
• Media processing

---

Production Architecture

flowchart TB
    Client[Web or Mobile Client] --> WAF[AWS WAF]
    WAF --> API[Amazon API Gateway]
    API --> StartLambda[Request Lambda]

    StartLambda --> Workflow[AWS Step Functions]

    Workflow --> Validate[Validation Lambda]
    Workflow --> Fraud[Fraud Check Lambda]
    Workflow --> Payment[Payment Service]
    Workflow --> Inventory[Inventory Service]
    Workflow --> Shipping[ECS Shipping Task]

    Payment --> PaymentDB[(Payment Database)]
    Inventory --> InventoryDB[(Inventory Database)]
    Shipping --> S3[(Amazon S3)]

    Workflow --> EventBridge[Amazon EventBridge]
    EventBridge --> Notification[Notification Lambda]
    EventBridge --> Audit[SQS Audit Queue]

    Workflow --> CloudWatch[Amazon CloudWatch]
    Workflow --> XRay[AWS X-Ray]

---

Production Use Case: Order Processing

Consider an e-commerce order-processing workflow.

Processing Flow

1. The client submits an order.
2. API Gateway validates the initial API request.
3. A Lambda function starts the Step Functions workflow.
4. The workflow validates the order.
5. A fraud-check task evaluates the transaction.
6. A Choice state determines whether manual review is required.
7. The payment service charges the customer.
8. The inventory service reserves products.
9. A shipping task creates the shipment.
10. EventBridge publishes an order-completed event.
11. Notification and audit consumers process the event.
12. If a later step fails, compensation tasks reverse completed operations.

---

Order Workflow

flowchart TD
    Start[Order Submitted] --> Validate[Validate Order]
    Validate --> Fraud[Fraud Check]

    Fraud --> Decision{Fraud Risk}
    Decision -->|Low| Payment[Charge Payment]
    Decision -->|High| Review[Manual Review]

    Review --> Approval{Approved?}
    Approval -->|Yes| Payment
    Approval -->|No| Reject[Reject Order]

    Payment --> Inventory[Reserve Inventory]
    Inventory --> Shipping[Create Shipment]

    Shipping --> Result{Successful?}
    Result -->|Yes| Event[Publish Order Completed]
    Result -->|No| Release[Release Inventory]

    Release --> Refund[Refund Payment]
    Refund --> Cancel[Cancel Order]

    Event --> Complete[Workflow Succeeded]

---

Security

Step Functions security includes:

• IAM execution roles
• Least-privilege permissions
• Resource-level permissions
• Encryption
• CloudTrail auditing
• Logging controls
• Network security for connected services
• Secure handling of workflow data

---

Execution Role

Every state machine requires an IAM role allowing it to access workflow resources.

For example, the role may permit Step Functions to:

• Invoke selected Lambda functions
• Send messages to specific SQS queues
• Publish to approved SNS topics
• Start specific ECS tasks
• Read or write selected DynamoDB tables

Avoid using broad permissions such as:

{
  "Action": "*",
  "Resource": "*"
}

Grant only the actions and resources required by the workflow.

---

Sensitive Data

Workflow input, output, execution history, and logs may contain application data.

Avoid including:

• Passwords
• Access tokens
• Private keys
• Full card numbers
• Unnecessary personal information
• Secrets
• Security credentials

Store secrets in AWS Secrets Manager or Systems Manager Parameter Store and allow the task to retrieve only the required secret.

---

Monitoring

Step Functions integrates with:

• Amazon CloudWatch Metrics
• CloudWatch Logs
• AWS X-Ray
• AWS CloudTrail
• EventBridge execution-status events

Important metrics include:

Metric	Purpose
ExecutionsStarted	Number of workflow executions started
ExecutionsSucceeded	Number of successful executions
ExecutionsFailed	Number of failed executions
ExecutionsTimedOut	Number of timed-out workflows
ExecutionsAborted	Number of stopped executions
ExecutionThrottled	State-transition throttling
ExpressExecutionBilledDuration	Express billed duration
ExpressExecutionBilledMemory	Express billed memory

---

Observability Architecture

flowchart LR
    Workflow[Step Functions] --> Metrics[CloudWatch Metrics]
    Workflow --> Logs[CloudWatch Logs]
    Workflow --> Traces[AWS X-Ray]
    Workflow --> Audit[AWS CloudTrail]

    Metrics --> Alarm[CloudWatch Alarms]
    Alarm --> SNS[Amazon SNS]
    SNS --> Operations[Operations Team]

---

Logging

Logs should include:

• Execution ID
• State-machine name
• State name
• Correlation ID
• Business transaction ID
• Error name
• Retry count
• Processing duration

Do not log unmasked sensitive workflow input or output.

For Express Workflows, CloudWatch Logs are particularly important because they provide execution-level troubleshooting data.

---

AWS X-Ray

AWS X-Ray helps trace workflow requests across supported services.

It can help identify:

• Slow workflow states
• Lambda latency
• Downstream-service latency
• Failed service calls
• Processing bottlenecks

Trace and correlation identifiers should be propagated across asynchronous components.

---

Cost Considerations

Step Functions cost depends on the workflow type.

Standard Workflow Cost Drivers

• Number of state transitions
• Number of executions
• Retry transitions
• Polling-loop transitions
• Nested workflows

Express Workflow Cost Drivers

• Number of executions
• Execution duration
• Allocated or consumed memory

Cost optimization techniques include:

• Choose the correct workflow type.
• Avoid unnecessary Pass states.
• Avoid tight polling loops.
• Use optimized service integrations.
• Use Express for high-volume short workflows.
• Reduce excessive retries.
• Avoid passing unnecessarily large payloads.
• Store large data in S3 and pass references.

---

Interview Questions and Answers

1. What is AWS Step Functions?

Answer

AWS Step Functions is a serverless workflow-orchestration service.

It allows developers to define state machines that coordinate:

• AWS Lambda functions
• Microservices
• AWS services
• Containers
• External APIs
• Human approval processes

Step Functions manages workflow state, transitions, retries, errors, parallel execution, and execution history.

---

2. What is a state machine?

Answer

A state machine is the workflow definition executed by Step Functions.

It contains:

• A starting state
• Task states
• Conditional states
• Transitions
• Error handling
• Terminal success or failure states

State machines are defined using Amazon States Language.

---

3. What is the difference between Standard and Express Workflows?

Answer

Standard Workflows are suitable for durable, auditable, and long-running business processes.

Express Workflows are suitable for high-volume and short-duration processing.

Use Standard for:

• Payments
• Human approvals
• Order processing
• Saga transactions

Use Express for:

• Streaming-event processing
• High-volume APIs
• Short data transformations
• IoT workloads

---

4. What are the main Step Functions state types?

Answer

The main state types are:

• Task
• Choice
• Parallel
• Map
• Wait
• Pass
• Succeed
• Fail

Each state has a specific purpose within the workflow.

---

5. How does Step Functions handle errors?

Answer

Step Functions supports:

• Retry for transient failures
• Catch for routing failures to recovery states
• Timeouts
• Heartbeats
• Fail states
• Compensation transactions

A production workflow should distinguish between transient technical failures and permanent business failures.

---

6. What is the callback pattern?

Answer

The callback pattern allows a workflow to pause until an external system returns a task token.

It is useful for:

• Human approvals
• Partner-system processing
• Legacy applications
• Long-running external tasks

The external process calls SendTaskSuccess or SendTaskFailure using the task token.

---

7. What is the difference between Parallel and Map states?

Answer

A Parallel state runs a fixed set of different branches concurrently.

A Map state runs the same processing logic for every item in a collection.

Use Parallel for:

• Send email
• Send SMS
• Update analytics

Use Map for:

• Process every order item
• Process every uploaded file
• Validate every record

---

8. What is the Saga pattern?

Answer

The Saga pattern manages a distributed transaction as a sequence of local transactions.

If one step fails, compensation transactions reverse previously completed work.

For example:

• Charge payment → refund payment
• Reserve inventory → release inventory
• Create shipment → cancel shipment

Step Functions can explicitly coordinate both forward and compensation steps.

---

9. When should Step Functions be used instead of EventBridge?

Answer

Use Step Functions when:

• Steps must run in a defined order
• Workflow state must be maintained
• Conditional branching is required
• Centralized retries and compensation are needed
• The complete workflow must be visualized

Use EventBridge when:

• Producers publish events
• Multiple consumers react independently
• Many-to-many routing is required
• A centralized workflow coordinator is unnecessary

They are frequently used together.

---

10. When should Step Functions not be used?

Answer

Step Functions may not be necessary when:

• Only one simple service call is required
• Basic event routing is sufficient
• A queue and worker solve the problem
• The workflow is better represented as stream processing
• Extremely low-latency in-process logic is required
• Orchestration complexity exceeds the business value

The architecture should not introduce a state machine when normal application code is simpler and clearer.

---

Common Interview Follow-Up Questions

• What is Amazon States Language?
• What is a Task state?
• What is the difference between Retry and Catch?
• How does exponential backoff work?
• What is a task token?
• What is the .sync integration pattern?
• How do Standard and Express pricing models differ?
• What is Distributed Map?
• How do you implement a human approval process?
• How do you implement compensation transactions?
• How do you monitor workflow failures?
• How do nested workflows work?
• How does Step Functions integrate with ECS?
• How do you pass data between states?
• How do you secure a state machine?

---

Common Mistakes

Using Lambda for Every Integration

Use direct AWS service integrations where possible.

Choosing the Wrong Workflow Type

Standard and Express serve different execution patterns.

Retrying Permanent Errors

Invalid input and business rejection should not be retried repeatedly.

Missing Timeouts

External tasks and callback workflows should always have timeout controls.

Non-Idempotent Tasks

Retries can cause the same task to execute more than once.

Passing Large Payloads

Store large data in S3 and pass object references.

Exposing Sensitive Data

Workflow history and logs may contain input and output data.

Creating Tight Polling Loops

Frequent Wait and Task transitions can increase cost.

Missing Compensation Logic

Distributed transactions require recovery for partially completed work.

Giving the Execution Role Broad Permissions

Apply least-privilege permissions to every state machine.

---

Troubleshooting Step Functions

Workflow Fails Immediately

Check:

• ASL syntax
• Starting state
• State transitions
• Resource ARN
• IAM permissions
• Input format

Lambda Task Fails

Check:

• Lambda exception
• Lambda timeout
• Function permissions
• Payload structure
• Retry configuration
• CloudWatch Logs

Workflow Is Stuck

Check:

• Callback task token
• Missing heartbeat
• Wait state
• Long-running job
• External system availability
• Timeout configuration

Access Denied Error

Check:

• Step Functions execution role
• Target resource policy
• Lambda invocation permission
• KMS permissions
• Cross-account trust

Data Is Missing Between States

Check:

• Input transformation
• Output transformation
• JSONPath or JSONata expression
• Task integration result
• Payload nesting

Express Execution Cannot Be Found

Check:

• CloudWatch Logs configuration
• Log group permissions
• Execution logging level
• Correlation identifiers

---

Best Practices

• Choose Standard or Express based on workload requirements.
• Keep workflow definitions understandable.
• Use direct service integrations.
• Make every task idempotent.
• Retry only transient failures.
• Use exponential backoff and jitter.
• Configure timeouts.
• Configure heartbeat timeouts for workers.
• Use Catch blocks for controlled recovery.
• Design compensation transactions.
• Use Map for collection processing.
• Use Distributed Map for large datasets.
• Use callback tokens for external processing.
• Use nested workflows for reusable domains.
• Store large payloads in S3.
• Pass references instead of entire files.
• Apply least-privilege IAM.
• Avoid sensitive data in workflow history.
• Enable CloudWatch monitoring.
• Enable X-Ray where useful.
• Add alarms for failures and timeouts.
• Propagate correlation identifiers.
• Use infrastructure as code.
• Test failure and compensation paths.
• Monitor state-transition and execution costs.

---

Quick Revision

Topic	Key Point
Step Functions	Serverless workflow orchestration
State machine	Workflow definition
ASL	JSON-based workflow language
Task	Performs one unit of work
Choice	Conditional routing
Parallel	Runs fixed branches concurrently
Map	Processes collection items
Distributed Map	Large-scale parallel processing
Wait	Delays workflow execution
Pass	Passes or transforms data
Succeed	Ends successfully
Fail	Ends unsuccessfully
Standard	Durable and long-running
Express	High-volume and short-running
Retry	Repeats transiently failed tasks
Catch	Routes errors to recovery states
Task token	Supports external callback
Saga	Distributed transaction pattern
Compensation	Reverses completed operations
X-Ray	Distributed tracing

---

Key Takeaways

• AWS Step Functions orchestrates serverless functions, containers, microservices, AWS services, and external APIs.
• Workflows are defined as state machines using Amazon States Language.
• Standard Workflows are designed for durable and auditable business processes.
• Express Workflows are designed for high-volume, short-duration workloads.
• Task, Choice, Parallel, Map, Wait, Pass, Succeed, and Fail are the primary state types.
• Retry and Catch provide declarative error handling.
• Callback task tokens support human approvals and external processing.
• Direct service integrations reduce unnecessary Lambda functions.
• Distributed Map supports large-scale parallel workloads.
• Step Functions is suitable for orchestrated Saga transactions and compensation processing.
• EventBridge handles event routing, while Step Functions controls ordered workflow execution.
• IAM, monitoring, idempotency, timeouts, compensation, and cost controls are essential for production workflows.

---