Backend interview prep

Start

Linux Interview Questions and Answers — Scenario-Based & Senior Level

Advanced scenario-based Linux interview questions for DevOps Engineers, SREs, Platform Engineers, and Java Developers. Covers production troubleshooting, networking, security, performance, shell scripting, and cloud-native Linux.

Introduction

This guide focuses on scenario-based and senior-level Linux interview questions — the kind asked in Staff Engineer, SRE, DevOps Engineer, and Platform Engineer interviews. It complements:

Here you will find questions that test applied understanding, production judgment, and architectural thinking.

Section 1 — Production Troubleshooting Scenarios

Q1. A server's disk is 100% full. How do you resolve it without restarting the service?

Answer

Step 1 — Identify which filesystem is full

df -h

Step 2 — Find the largest directories

du -ah / --max-depth=3 | sort -rh | head -20

Step 3 — Identify large files

find / -type f -size +500M 2>/dev/null

Step 4 — Check for deleted-but-open files (a common gotcha — the space is not released until the file handle is closed)

lsof | grep deleted

If you find any, restart only that specific process or send it SIGHUP to rotate logs.

Step 5 — Common quick wins

# Rotate and clear application logs
> /var/log/app/application.log

# Clean package cache
apt clean           # Debian/Ubuntu
dnf clean all       # RHEL/Rocky

# Remove old kernel packages
sudo apt autoremove

# Clear systemd journal older than 7 days
journalctl --vacuum-time=7d

Key insight: Deleting a file while a process has it open does not free disk space until the process closes the file descriptor. Use lsof | grep deleted to find these "ghost" files.

Q2. A production server's memory is exhausted and the OOM killer is firing. What do you do?

Answer

Immediate triage

# Confirm OOM events
dmesg | grep -i oom
journalctl -k | grep -i "oom\|killed process"

# Check current memory
free -h
vmstat -s

# Identify top memory consumers
ps aux --sort=-%mem | head -15

Identify the offending process

# Find PID of high-memory process
top -o %MEM

# Inspect Java heap (if Java process)
jmap -heap <PID>
jstat -gcutil <PID> 1000 5

Short-term relief

# Drop page cache without affecting running processes
sync && echo 3 > /proc/sys/vm/drop_caches

# Reduce swappiness to keep more in RAM
sysctl vm.swappiness=10

Protect critical processes from OOM killer

echo -1000 > /proc/<critical-PID>/oom_score_adj

Root cause checklist

  • Memory leak in application code
  • Insufficient heap/JVM settings
  • Missing memory limits on containers
  • Too many processes on undersized instance

Q3. SSH connection to a server is extremely slow. How do you diagnose it?

Answer

Step 1 — Test with verbose output

ssh -vvv user@host

Look for where the handshake stalls — key exchange, authentication, or post-login.

Step 2 — Common causes and fixes

Cause Symptom in -vvv Fix
Reverse DNS lookup Stalls at debug1: resolving... Add UseDNS no to /etc/ssh/sshd_config
GSSAPI auth attempt Stalls at debug1: Authentications that can continue Add GSSAPIAuthentication no to ~/.ssh/config
Slow /etc/hosts lookup Stalls at connecting Check /etc/nsswitch.conf
Server load Slow after login Check top / uptime on server

Step 3 — Check sshd logs on the server

journalctl -u sshd -f
# or
tail -f /var/log/auth.log

Step 4 — Check TCP connection time

time ssh -o ConnectTimeout=5 user@host echo ok

Q4. A cron job is not executing. How do you debug it?

Answer

# Step 1 — Check if cron daemon is running
systemctl status cron     # Debian/Ubuntu
systemctl status crond    # RHEL/Rocky

# Step 2 — View cron logs
grep CRON /var/log/syslog
journalctl -u cron

# Step 3 — Verify the crontab entry
crontab -l
crontab -l -u <username>

# Step 4 — Test the script manually as the cron user
sudo -u <username> /path/to/script.sh

# Step 5 — Redirect output in crontab for debugging
* * * * * /path/to/script.sh >> /tmp/cron_debug.log 2>&1

Common cron gotchas

Issue Explanation
Missing PATH Cron has a minimal PATH. Use absolute paths in scripts.
No execute permission chmod +x script.sh
Environment variables missing Export them at the top of the script or crontab
Wrong user Script requires root but runs as another user
Syntax error in crontab Use crontab -e and validate timing with crontab.guru

Q5. A network port is not reachable. How do you systematically diagnose it?

Answer

# Step 1 — Check if the service is listening on the server
ss -tulnp | grep <port>
# or
netstat -tulnp | grep <port>

# Step 2 — Check firewall rules
iptables -L -n -v | grep <port>
firewall-cmd --list-all       # firewalld
ufw status                    # UFW

# Step 3 — Test local connectivity
curl -v http://localhost:<port>
telnet localhost <port>

# Step 4 — Test from a remote host
nc -zv <host> <port>
telnet <host> <port>

# Step 5 — Trace the network path
traceroute <host>
mtr <host>

# Step 6 — Capture traffic to confirm packets arrive
tcpdump -i eth0 port <port>

Section 2 — Networking

Q6. Explain the difference between ss, netstat, and lsof for network inspection.

Answer

Tool Purpose Notes
ss Socket statistics — fast, from kernel directly Preferred modern tool
netstat Legacy socket and routing stats Deprecated; reads /proc
lsof -i Shows which process has which file/socket open Detailed per-process view 
# Show all listening TCP sockets with PID
ss -tulnp

# Show all established connections
ss -tnp state established

# Show which process is using port 8080
ss -tulnp | grep :8080
lsof -i :8080

# Show all open network sockets by a Java process
lsof -p <PID> | grep IPv

Q7. What is the difference between /etc/hosts, /etc/resolv.conf, and /etc/nsswitch.conf?

Answer

File Purpose
/etc/hosts Static local name-to-IP mappings, checked before DNS
/etc/resolv.conf Defines which DNS servers to query
/etc/nsswitch.conf Controls resolution order (files → DNS → other sources) 
# View resolution order (hosts line controls name resolution)
cat /etc/nsswitch.conf | grep hosts
# hosts: files dns myhostname

# Add a local override
echo "192.168.1.10 myservice.local" >> /etc/hosts

# Check which DNS server is being used
cat /etc/resolv.conf

# Test DNS resolution
dig google.com
nslookup google.com

Q8. How does iptables work and how do you add a rule to allow a port?

Answer

iptables is the Linux kernel firewall. It uses chains of rules to decide what to do with packets:

Chain Applies to
INPUT Packets destined for the local machine
OUTPUT Packets generated by the local machine
FORWARD Packets routed through the machine 
# View all current rules with line numbers
iptables -L -n -v --line-numbers

# Allow inbound TCP on port 8080
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT

# Block a specific IP
iptables -A INPUT -s 192.168.1.100 -j DROP

# Save rules (so they survive reboot)
iptables-save > /etc/iptables/rules.v4

# Prefer firewalld on RHEL/Rocky
firewall-cmd --permanent --add-port=8080/tcp
firewall-cmd --reload

Section 3 — Shell Scripting

Q9. Write a shell script that monitors a Java process and restarts it if it dies.

Answer

#!/bin/bash
# monitor-java.sh — Restarts java app if not running

APP_JAR="/opt/app/app.jar"
LOG_FILE="/var/log/app/app.log"
PID_FILE="/var/run/app.pid"
CHECK_INTERVAL=30   # seconds

start_app() {
    echo "[$(date)] Starting application..."
    nohup java -jar "$APP_JAR" >> "$LOG_FILE" 2>&1 &
    echo $! > "$PID_FILE"
    echo "[$(date)] Started with PID $(cat $PID_FILE)"
}

while true; do
    if [ -f "$PID_FILE" ]; then
        PID=$(cat "$PID_FILE")
        if ! kill -0 "$PID" 2>/dev/null; then
            echo "[$(date)] Process $PID is dead. Restarting..."
            start_app
        fi
    else
        echo "[$(date)] PID file not found. Starting..."
        start_app
    fi
    sleep "$CHECK_INTERVAL"
done

Production note: Use systemd service units with Restart=always instead of a custom monitor script — it is more reliable and integrates with journald logging.

Q10. What does set -euo pipefail do in a shell script?

Answer

This is a best-practice safety header for production shell scripts.

Option Meaning
set -e Exit immediately if any command returns a non-zero exit code
set -u Treat unset variables as errors (prevents silent bugs)
set -o pipefail Catch failures in piped commands (without this, false | true returns 0) 
#!/bin/bash
set -euo pipefail

# Without -u, this silently uses an empty string:
echo "Hello $UNDEFINED_VAR"

# With -u, it exits with:
# bash: UNDEFINED_VAR: unbound variable

# Without -o pipefail:
cat nonexistent.log | grep ERROR   # returns 0 (hidden failure)

# With -o pipefail:
# Returns non-zero — script exits safely

Q11. How do you pass arguments to a shell script and validate them?

Answer

#!/bin/bash
set -euo pipefail

# $0 = script name, $1 = first arg, $# = number of args
if [ "$#" -lt 2 ]; then
    echo "Usage: $0 <environment> <version>"
    echo "  environment: dev | staging | prod"
    echo "  version:     e.g. 1.2.3"
    exit 1
fi

ENV="$1"
VERSION="$2"

# Validate environment
case "$ENV" in
    dev|staging|prod)
        echo "Deploying version $VERSION to $ENV"
        ;;
    *)
        echo "Error: invalid environment '$ENV'"
        exit 1
        ;;
esac

# Validate semver format
if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then
    echo "Error: version must be in semver format (e.g. 1.2.3)"
    exit 1
fi

Q12. How do you handle errors and cleanup in a shell script?

Answer

Use trap to register a cleanup function that runs on exit or error.

#!/bin/bash
set -euo pipefail

TMPDIR=$(mktemp -d)

cleanup() {
    echo "Cleaning up temporary directory: $TMPDIR"
    rm -rf "$TMPDIR"
}

# Register cleanup to run on EXIT (success or failure)
trap cleanup EXIT

# Register additional handler for SIGINT / SIGTERM
trap 'echo "Interrupted!"; exit 1' INT TERM

echo "Working in $TMPDIR"
cp /etc/hosts "$TMPDIR/"
# ... rest of script
echo "Done."
# cleanup() runs automatically here

Section 4 — Performance & Observability

Q13. How do you identify and resolve high I/O wait on a Linux server?

Answer

Identify I/O wait

# I/O wait is shown in the 'wa' column
top
# or
vmstat 1 10
# wa > 10–15% is typically significant

Identify which process is causing I/O

# Install iotop
sudo iotop -o

# Or use pidstat
pidstat -d 1 5

Identify which disk/filesystem

iostat -xz 1 5
# Look at %util, await, and r/s + w/s columns

Common root causes

Cause Fix
Log files written too aggressively Reduce log verbosity; use async logging
Database doing full table scans Add indexes; tune query
/tmp full or on slow disk Mount /tmp as tmpfs
Application writing synchronously Use buffered/async I/O
Disk hardware degrading Check dmesg for I/O errors 
# Check for disk errors
dmesg | grep -iE "error|fail|reset|ata"

Q14. How do you use strace and lsof in production debugging?

Answer

strace — traces system calls made by a process.

# Trace all syscalls of a running process
strace -p <PID>

# Count syscall frequency and time
strace -cp <PID>

# Trace only file-related syscalls
strace -e trace=file -p <PID>

# Trace a command from start
strace -o /tmp/trace.txt ls /tmp

Caution: strace adds significant overhead. Use only briefly in production.

lsof — lists open files (files, sockets, pipes, devices).

# Files opened by a specific process
lsof -p <PID>

# Which process is using a port
lsof -i :8080

# Files opened by a user
lsof -u appuser

# Find deleted files still held open (disk space issue)
lsof | grep deleted

# List all network connections
lsof -i

Q15. What is the difference between vmstat, iostat, and sar?

Answer

Tool What it measures Package
vmstat CPU, memory, swap, I/O, processes — overview procps
iostat Per-disk I/O statistics sysstat
sar Historical system performance — CPU, memory, I/O, network sysstat 
# vmstat — 1-second intervals, 10 samples
vmstat 1 10

# iostat — extended disk stats
iostat -xz 1 5

# sar — CPU usage for today
sar -u 1 10

# sar — memory usage
sar -r 1 10

# sar — read historical data (yesterday's CPU)
sar -u -f /var/log/sa/sa$(date -d yesterday +%d)

vmstat column reference

Column Meaning
r Processes waiting for CPU (run queue)
b Processes blocked on I/O
si / so Swap in / swap out per second
wa CPU time spent waiting for I/O
cs Context switches per second

Section 5 — Security

Q16. How do you harden SSH access on a Linux server?

Answer

Edit /etc/ssh/sshd_config:

# Disable root login
PermitRootLogin no

# Disable password authentication (use SSH keys only)
PasswordAuthentication no

# Only allow specific users
AllowUsers deploy admin

# Change default port (security through obscurity — optional)
Port 2222

# Limit authentication attempts
MaxAuthTries 3

# Disable empty passwords
PermitEmptyPasswords no

# Disable X11 forwarding unless needed
X11Forwarding no

# Set idle timeout (seconds)
ClientAliveInterval 300
ClientAliveCountMax 2

# Apply changes
systemctl restart sshd

# Test configuration before restarting
sshd -t

Q17. How do you find recently modified files on a Linux system?

Answer

# Files modified in the last 24 hours
find / -type f -mtime -1 2>/dev/null

# Files modified in the last 10 minutes
find / -type f -mmin -10 2>/dev/null

# Files modified in /etc in the last 7 days
find /etc -type f -mtime -7

# Sort by modification time (newest first)
find /var/log -type f -printf '%T@ %p\n' | sort -rn | head -20

# Files with SUID set (security audit)
find / -perm /4000 -type f 2>/dev/null

# World-writable files (security risk)
find / -type f -perm -o+w 2>/dev/null | grep -v /proc

Q18. What is sudo and how do you grant granular permissions?

Answer

sudo allows a permitted user to run commands as another user (typically root) without sharing the root password.

# Edit sudoers safely (validates syntax before saving)
visudo

# Grant user 'deploy' permission to restart nginx only
deploy ALL=(ALL) NOPASSWD: /bin/systemctl restart nginx

# Grant a group full sudo without password
%admins ALL=(ALL) NOPASSWD: ALL

# Allow 'monitor' user to run specific read-only commands
monitor ALL=(ALL) NOPASSWD: /bin/ps, /usr/bin/top, /bin/df

# Check what sudo rules apply to current user
sudo -l

Best practice: Grant the least privilege needed. Avoid ALL=(ALL) NOPASSWD: ALL in production except for automation service accounts with other compensating controls.

Section 6 — Linux in Containerised & Cloud Environments

Q19. How does Linux support containers — what is the role of namespaces and cgroups?

Answer

Containers are not a separate OS feature — they are Linux processes with isolated views and resource limits provided by:

Mechanism What it provides
Namespaces Isolation — what a container can see
cgroups Limits — what a container can use
Union filesystem (OverlayFS) Layered image filesystem 
Host Linux Kernel
├── Container A
│   ├── PID namespace   (PID 1 = app process, not systemd)
│   ├── Net namespace   (own eth0, own IP)
│   ├── Mount namespace (own root filesystem view)
│   └── cgroup          (CPU: 0.5 cores, Memory: 512 MB)
│
└── Container B
    ├── PID namespace
    ├── Net namespace
    └── cgroup

# Verify container namespaces from the host
ls -la /proc/<container-PID>/ns

# Check cgroup limits for a Docker container
docker inspect <container> | grep -A5 "Memory\|Cpu"

# From inside a container — confirm PID namespace isolation
ps aux   # PID 1 = your app

Q20. A Docker container keeps restarting. How do you troubleshoot it?

Answer

# Step 1 — Check container status and restart count
docker ps -a

# Step 2 — View exit code (non-zero = crash)
docker inspect <container> | grep '"ExitCode"'

# Step 3 — Read the logs (last 100 lines)
docker logs --tail 100 <container>

# Step 4 — Follow logs in real time
docker logs -f <container>

# Step 5 — Run an interactive shell with the same image
docker run -it --entrypoint /bin/bash <image>

# Step 6 — Check resource limits (OOM kill = ExitCode 137)
docker stats <container>
docker inspect <container> | grep -i memory

Exit code quick reference

Exit Code Meaning
0 Clean exit
1 Application error
137 OOM kill (128 + SIGKILL 9)
143 SIGTERM (128 + 15)
126 Permission error
127 Command not found

Q21. How do you check and tune kernel parameters with sysctl?

Answer

sysctl reads and writes kernel parameters at runtime from /proc/sys/.

# View all current kernel parameters
sysctl -a

# View a specific parameter
sysctl vm.swappiness
sysctl net.ipv4.ip_forward

# Change a parameter at runtime (does not survive reboot)
sysctl -w vm.swappiness=10
sysctl -w net.ipv4.ip_forward=1

# Make permanent (survives reboot)
echo "vm.swappiness=10" >> /etc/sysctl.d/99-custom.conf
sysctl -p /etc/sysctl.d/99-custom.conf

Commonly tuned parameters

Parameter Default Use Case
vm.swappiness 60 Lower (10) on app servers to prefer RAM
vm.overcommit_memory 0 Set to 1 for Redis, 2 for strict limits
net.ipv4.ip_forward 0 Set to 1 on Docker hosts / routers
fs.file-max varies Raise for high-connection servers
net.core.somaxconn 128 Raise for high-throughput web servers
net.ipv4.tcp_tw_reuse 0 Set to 1 to reuse TIME_WAIT sockets

Section 7 — Senior-Level & Architectural Questions

Q22. What is the difference between a process, a thread, and a coroutine in the Linux context?

Answer

Concept OS Visibility Memory Context Switch Cost Example
Process Full kernel process Separate address space High JVM, Nginx worker
Thread (kernel thread) Scheduled by kernel Shared within process Medium Java thread, pthread
Green thread / coroutine User-space only Shared Very low Java Virtual Threads, Go goroutines
  • Linux kernel schedules kernel threads (not user-space coroutines).
  • Java's Virtual Threads (JDK 21+) are user-space threads multiplexed onto a small pool of carrier (OS) threads — drastically reducing context switch overhead for I/O-bound workloads.
  • Coroutines/green threads are invisible to the kernel — it only sees the carrier OS thread.
# See all kernel threads for a Java process
ps -eLf | grep java | wc -l

# With Virtual Threads, OS thread count stays low regardless of concurrency

Q23. How would you design a Linux-based system to handle 100,000 concurrent connections?

Answer

This is the C10k (or C100k) problem. Key Linux tuning steps:

1 — Increase open file descriptor limits

# Per-process limit
ulimit -n 1000000

# System-wide limit
echo "fs.file-max = 2000000" >> /etc/sysctl.d/99-tuning.conf
sysctl -p

2 — Tune TCP stack

# /etc/sysctl.d/99-tcp.conf
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535

3 — Use epoll-based I/O

  • Nginx uses epoll (event-driven) — handles 100k connections with a small thread count.
  • Traditional blocking thread-per-connection model (old Tomcat BIO) does not scale beyond a few thousand.

4 — Application layer

  • Use non-blocking / reactive frameworks (Netty, Reactor, Vert.x) or JDK 21 Virtual Threads.
  • Keep connection handlers I/O-bound and non-blocking.

Q24. What is epoll and why is it better than select or poll?

Answer

All three are Linux mechanisms for monitoring multiple file descriptors for I/O readiness.

Mechanism How it works Scalability
select Copies FD set to kernel on every call; scans all FDs O(n) — poor above ~1024 FDs
poll No FD limit, but still O(n) linear scan O(n) — better but still linear
epoll Kernel maintains watch list; only returns ready FDs O(1) per event — scales to millions 
epoll workflow:
  epoll_create() → create epoll instance
  epoll_ctl()    → register FDs to watch
  epoll_wait()   → block until FDs are ready (returns only ready ones)

Nginx, Redis, Node.js, and Java NIO all use epoll on Linux. This is why they handle massive concurrency with very few threads.

Q25. How does Linux manage virtual memory and what is the page cache?

Answer

Virtual memory gives each process its own private address space. The kernel maps virtual addresses to physical RAM pages on demand.

Concept Explanation
Virtual address space Each process sees 0 to max address; kernel manages mapping
Page fault Access to unmapped virtual page → kernel loads it from disk/file
Page cache RAM used to cache filesystem data for faster reads
Dirty pages Modified pages not yet written to disk
kswapd Kernel thread that reclaims memory by writing dirty pages to disk 
# View page cache usage
free -h
# The "buff/cache" column IS the page cache — it is not wasted memory

# View detailed memory stats
cat /proc/meminfo | grep -E "MemTotal|MemFree|Cached|Dirty|Writeback"

# Drop page cache (for benchmarking — never in production under load)
sync && echo 3 > /proc/sys/vm/drop_caches

Interview key point: On Linux, "available" memory includes page cache that can be evicted. A server showing 95% memory used may still be perfectly healthy if most is page cache.

Interview Summary

Category Topics Covered
Production Troubleshooting Disk full, OOM, slow SSH, broken cron, port unreachable
Networking ss vs netstat, DNS resolution chain, iptables / firewalld
Shell Scripting Process monitors, set -euo pipefail, argument validation, trap
Performance & Observability I/O wait, strace, lsof, vmstat, iostat, sar
Security SSH hardening, sudo least privilege, file audit
Containers & Cloud Namespaces + cgroups, Docker restarts, sysctl kernel tuning
Senior / Architecture Process vs thread vs coroutine, C100k tuning, epoll, virtual memory

Mastering these areas positions you to answer any Linux question at Staff Engineer, SRE, or Principal DevOps Engineer level — connecting low-level internals to production impact.