Backend interview prep

Start

Kubernetes Advanced - Complete Interview Guide

Learn advanced Kubernetes concepts including scheduler, controllers, StatefulSets, DaemonSets, Jobs, CronJobs, RBAC, Network Policies, Autoscaling, Helm, Operators, CNI, CSI, GitOps, Observability, and enterprise production best practices.

Introduction

Running a few Pods is easy. Running thousands of containers across hundreds of nodes with high availability, security, monitoring, autoscaling, and zero downtime is where Kubernetes truly shines.

This chapter focuses on advanced Kubernetes concepts used in enterprise production environments.

These topics are commonly discussed in Senior Software Engineer, DevOps Engineer, Platform Engineer, SRE, Cloud Engineer, and Solution Architect interviews.

Learning Objectives

After completing this chapter, you should understand:

  • Kubernetes Internals
  • Scheduler
  • Controllers
  • StatefulSets
  • DaemonSets
  • Jobs
  • CronJobs
  • Horizontal Pod Autoscaler
  • Vertical Pod Autoscaler
  • Cluster Autoscaler
  • Resource Requests & Limits
  • Node Affinity
  • Pod Affinity
  • Taints & Tolerations
  • Network Policies
  • RBAC
  • Service Accounts
  • Pod Security
  • CSI
  • CNI
  • Helm
  • Operators
  • CRDs
  • Observability
  • GitOps
  • CI/CD
  • Enterprise Best Practices

Kubernetes Reconciliation Loop

Kubernetes constantly compares the Desired State with the Current State.

If they differ, controllers automatically make corrections.

Desired State
      │
      ▼
API Server
      │
      ▼
Controllers
      │
      ▼
Current Cluster State
      │
      ▼
Match Desired State

Kubernetes Scheduler

The Scheduler decides where Pods should run.

Scheduling considers:

  • Available CPU
  • Available Memory
  • Node Health
  • Labels
  • Taints
  • Affinity Rules
  • Resource Requests
  • Policies

Scheduling Flow

Pod Created
      │
      ▼
API Server
      │
      ▼
Scheduler
      │
      ▼
Best Worker Node
      │
      ▼
Pod Running

Controllers

Controllers continuously monitor Kubernetes resources.

Common Controllers:

  • Deployment Controller
  • ReplicaSet Controller
  • StatefulSet Controller
  • Job Controller
  • Node Controller
  • Endpoint Controller

StatefulSets

StatefulSets manage stateful applications.

Examples:

  • MySQL
  • PostgreSQL
  • MongoDB
  • Kafka
  • Elasticsearch
  • Redis Cluster

Features:

  • Stable Hostnames
  • Stable Storage
  • Ordered Deployment
  • Ordered Scaling

Deployment vs StatefulSet

Deployment StatefulSet
Stateless Stateful
Dynamic Pod Names Fixed Pod Names
Shared Storage Dedicated Storage
Independent Pods Ordered Pods

DaemonSets

DaemonSets ensure one Pod runs on every Worker Node.

Common Use Cases:

  • Fluent Bit
  • Prometheus Node Exporter
  • Log Collection
  • Monitoring Agents
  • Security Agents
Node A → Monitoring Pod

Node B → Monitoring Pod

Node C → Monitoring Pod

Jobs

Jobs execute one-time tasks.

Examples:

  • Database Migration
  • Data Import
  • Batch Processing
  • Report Generation

CronJobs

CronJobs execute Jobs on schedules.

Examples:

  • Nightly Backup
  • Cleanup Tasks
  • Scheduled Reports
  • Database Maintenance

Resource Requests

Requests define the minimum resources required.

Examples:

  • CPU
  • Memory

Scheduler uses Requests when placing Pods.

Resource Limits

Limits define the maximum resources a Pod can consume.

Benefits:

  • Prevent resource abuse
  • Better cluster stability
  • Predictable performance

Horizontal Pod Autoscaler (HPA)

HPA automatically changes the number of Pods.

Metrics:

  • CPU
  • Memory
  • Custom Metrics
High CPU

↓

Scale Out

↓

More Pods

Vertical Pod Autoscaler (VPA)

VPA adjusts CPU and Memory assigned to Pods.

Useful when workloads change over time.

Cluster Autoscaler

Cluster Autoscaler automatically adds or removes Worker Nodes.

Benefits:

  • Cost Optimization
  • Automatic Scaling
  • Better Resource Utilization

Node Affinity

Node Affinity schedules Pods on specific Nodes.

Examples:

  • GPU Nodes
  • SSD Nodes
  • Production Nodes

Pod Affinity

Pod Affinity places Pods close together.

Common for:

  • Microservices
  • Low latency communication

Pod Anti-Affinity

Prevents Pods from running on the same Node.

Improves:

  • High Availability
  • Fault Tolerance

Taints

Taints prevent Pods from running on specific Nodes.

Example:

GPU Node

↓

Only GPU Workloads

Tolerations

Tolerations allow Pods to run on tainted Nodes.

Service Accounts

Service Accounts provide identities for Pods.

Applications use them to access Kubernetes APIs securely.

RBAC

Role-Based Access Control controls user permissions.

Components:

  • Role
  • ClusterRole
  • RoleBinding
  • ClusterRoleBinding

Benefits:

  • Least Privilege
  • Better Security
  • Fine-grained Access Control

Network Policies

Network Policies control Pod-to-Pod communication.

Benefits:

  • Zero Trust Networking
  • Traffic Isolation
  • Security

CSI (Container Storage Interface)

CSI standardizes storage integration.

Examples:

  • AWS EBS
  • Azure Disk
  • GCE Persistent Disk
  • NFS
  • Ceph

CNI (Container Network Interface)

CNI standardizes container networking.

Popular Plugins:

  • Calico
  • Flannel
  • Cilium
  • Weave
  • Canal

Helm

Helm is Kubernetes' package manager.

Components:

  • Charts
  • Templates
  • Values
  • Releases
  • Repositories

Benefits:

  • Simplified Deployments
  • Version Control
  • Reusability

Helm Architecture

Helm Chart

      │

Templates

      │

Rendered YAML

      │

Kubernetes Cluster

Custom Resource Definitions (CRDs)

CRDs extend Kubernetes with custom resource types.

Example:

Database

Cache

MessageQueue

Operators

Operators automate application management.

Responsibilities:

  • Installation
  • Scaling
  • Upgrades
  • Backup
  • Recovery

Examples:

  • Prometheus Operator
  • Kafka Operator
  • MongoDB Operator

Observability

Observability helps understand cluster behavior.

Components:

  • Metrics
  • Logs
  • Traces
  • Events

Monitoring Stack

Common tools:

  • Prometheus
  • Grafana
  • Alertmanager
  • kube-state-metrics
  • Metrics Server

Logging Stack

Popular solutions:

  • Fluent Bit
  • Fluentd
  • Elasticsearch
  • Loki
  • Kibana

Distributed Tracing

Common tools:

  • Jaeger
  • Zipkin
  • OpenTelemetry

Useful for:

  • Microservices
  • API Performance
  • Root Cause Analysis

GitOps

Git becomes the single source of truth.

Popular tools:

  • Argo CD
  • Flux CD

Workflow:

Git Repository

↓

GitOps Controller

↓

Kubernetes Cluster

Kubernetes Security

Enterprise security includes:

  • RBAC
  • Network Policies
  • Pod Security Standards
  • Secrets Encryption
  • Admission Controllers
  • Image Scanning
  • Service Accounts

CI/CD Integration

Typical enterprise pipeline:

Developer

↓

Git Repository

↓

CI Pipeline

↓

Docker Image

↓

Image Registry

↓

Helm

↓

Kubernetes Cluster

↓

Production

Production Best Practices

  • Use Namespaces for isolation.
  • Define CPU and Memory Requests.
  • Always configure Resource Limits.
  • Enable Liveness and Readiness Probes.
  • Store secrets in Kubernetes Secrets.
  • Avoid running containers as root.
  • Use Network Policies.
  • Apply RBAC.
  • Use Helm for deployments.
  • Enable monitoring and logging.
  • Use GitOps for deployments.
  • Keep clusters updated regularly.

Common Production Issues

  • CrashLoopBackOff
  • ImagePullBackOff
  • Pending Pods
  • OOMKilled
  • Failed Scheduling
  • Node Not Ready
  • DNS Resolution Failure
  • PVC Pending
  • High CPU Usage
  • High Memory Usage
  • Failed Health Checks
  • Network Policy Blocking Traffic
  • etcd Failure
  • Deployment Rollout Failure

Production Troubleshooting

Typical troubleshooting workflow:

  1. Check Pod status.
  2. Review Pod events.
  3. Inspect logs.
  4. Verify Deployment.
  5. Check Resource Requests and Limits.
  6. Validate Service and Ingress.
  7. Inspect Network Policies.
  8. Verify Persistent Volumes.
  9. Review Node health.
  10. Check cluster events and metrics.

Enterprise Kubernetes Workflow

Developer
      │
Git Repository
      │
CI Pipeline
      │
Docker Image
      │
Container Registry
      │
Helm Chart
      │
Argo CD
      │
Kubernetes Cluster
      │
Monitoring
      │
Production

Interview Summary

After completing this chapter, you should understand:

  • Kubernetes Scheduler
  • Controllers
  • StatefulSets
  • DaemonSets
  • Jobs
  • CronJobs
  • Resource Requests
  • Resource Limits
  • HPA
  • VPA
  • Cluster Autoscaler
  • Node Affinity
  • Pod Affinity
  • Taints
  • Tolerations
  • RBAC
  • Service Accounts
  • Network Policies
  • CSI
  • CNI
  • Helm
  • CRDs
  • Operators
  • Observability
  • Monitoring
  • Logging
  • Tracing
  • GitOps
  • Kubernetes Security
  • CI/CD Integration
  • Production Troubleshooting

Next Chapter

➡️ Kubernetes Interview Questions

Topics include:

  • 100+ Kubernetes Interview Questions
  • Kubernetes Architecture
  • Scheduling Scenarios
  • Networking Questions
  • Storage Questions
  • Security Questions
  • Helm Questions
  • GitOps Questions
  • Production Troubleshooting
  • Enterprise Best Practices