Week 8: Kubernetes replication, health probes and failure experiments #6

Merged

23365978 merged 4 commits from week8-kubernetes into main 2026-10-27 14:41:59 +00:00

Conversation 1 Commits 4 Files changed 27 +876 -12

23365978 commented 2026-10-27 14:08:02 +00:00

Owner

Copy link

Change

Implement the original Week 8 Kubernetes replication lab in the cumulative Labs repository. Add a two-replica Deployment, Service, ConfigMap and local Secret template; expose Kafka on a pod-reachable advertised listener. Replace destructive create-drop with lab update so restarting a replica preserves shared rows. Keep the deliberate aggregate liveness/readiness probes needed for the MySQL outage experiment.

Add Q1-Q11 design answers, an acceptance matrix, a deployment guide and redacted execution evidence. The original brief has no student-designed feature requirement. Preserve Week 7 JWT/bcrypt/RBAC and its documented registration-role shortcut. Include the existing PR #5 Kafka test-shutdown fix as a baseline dependency.

Verification

• Baseline: 46 tests passed. Updated full Gradle tests and build: 49 passed, no failures/errors/skips.
• Source Dockerfile and Dockerfile.ci builds passed.
• Actual two-pod JWT/RBAC and Kafka -> Elasticsearch/Redis/DynamoDB checks passed before and after recovery.
• Pod deletion produced a healthy replacement without losing shared data.
• Broken readiness left two old pods serving; EndpointSlices excluded the new pod; 10/10 ClusterIP health requests succeeded; rollback and source repair passed.
• Four replicas became unready 36 s after MySQL stopped; all four containers restarted by 97 s; all recovered 52 s after MySQL restarted. Account and link persistence verified.
• Temporary native-Linux standalone kind cluster and datastores cleaned up; existing volumes retained.

Review limits

Local cluster verification used the locally built image and a host alias. Docker Desktop kubeadm execution and LA discussion/sign-off remain manual. Q11 recovery targets are proposals for the current LinkLab, not achieved production guarantees. Production deployment remains outside the lab. The user explicitly requested the PR merge after validation.

Final remote verification

• Final head: 650d27df3543ee4033b150da7c4f29a79f60acfa.
• Forgejo CI run #9: success for tests, JAR build, and Docker build/push. Downloaded test-results artifact confirms 49 tests, 0 failures, 0 skipped.
• Published week8 and sha-650d27d tags; main's existing latest remains unchanged.
• Published image: cs4297.skynet.ie/23365978/cs4297-linklab@sha256:71ae6c32e8d3eccaa58a7ad59ffcbc136e2cbfca1a214002e264f6cdc1739865. OCI revision label matches the final PR head.
• Verified registry authentication and actual Kubernetes pulls using the forgejo-registry Secret and imagePullPolicy: Always, without loading a local app image into this second cluster.
• Both registry-backed pods reached 1/1 Ready with zero restarts. The complete two-pod smoke passed again: same-token create=201 and identity=200 on each pod; invalid credentials/tokens rejected; RBAC enforced; Kafka indexing/leaderboard delivery and DynamoDB click persistence passed; admin reindex passed.
• The first smoke attempt hit a kubectl port-forward stream timeout (Error creating error stream / lost connection to pod). Replacing the forwards restored transport; the full unchanged smoke passed. Pods remained healthy with zero restarts.
• Native Linux verification uses the checked-in registry deployment plus a temporary hostAliases mapping for host.docker.internal. Docker Desktop kubeadm and LA sign-off remain manual.

An agent source review has been completed; this is not a human approval. The user subsequently requested merging this PR after review and green PR CI. The baseline includes existing unmerged PR #5 and its earlier Week 7 answer annotations; those existing edits were preserved.

Registry verification cleanup completed: namespace (including both Secrets) and temporary cluster deleted; the five lab datastores stopped; existing volumes retained. The working tree is clean and the final local/remote branch SHA matches.

Merge

Merged at the user's explicit request using merge commit e09d70059a6c2286b9e8d740a884d89e8f66bb24. Local main was fast-forwarded to match origin/main, with a clean working tree. Post-merge CI run #10 was triggered; its current status is available at that link. PR-head CI run #9 had already passed all three stages and 49 tests before merging.

## Change Implement the original Week 8 Kubernetes replication lab in the cumulative Labs repository. Add a two-replica Deployment, Service, ConfigMap and local Secret template; expose Kafka on a pod-reachable advertised listener. Replace destructive `create-drop` with lab `update` so restarting a replica preserves shared rows. Keep the deliberate aggregate liveness/readiness probes needed for the MySQL outage experiment. Add Q1-Q11 design answers, an acceptance matrix, a deployment guide and redacted execution evidence. The original brief has no student-designed feature requirement. Preserve Week 7 JWT/bcrypt/RBAC and its documented registration-role shortcut. Include the existing PR #5 Kafka test-shutdown fix as a baseline dependency. ## Verification - Baseline: 46 tests passed. Updated full Gradle tests and build: 49 passed, no failures/errors/skips. - Source Dockerfile and Dockerfile.ci builds passed. - Actual two-pod JWT/RBAC and Kafka -> Elasticsearch/Redis/DynamoDB checks passed before and after recovery. - Pod deletion produced a healthy replacement without losing shared data. - Broken readiness left two old pods serving; EndpointSlices excluded the new pod; 10/10 ClusterIP health requests succeeded; rollback and source repair passed. - Four replicas became unready 36 s after MySQL stopped; all four containers restarted by 97 s; all recovered 52 s after MySQL restarted. Account and link persistence verified. - Temporary native-Linux standalone kind cluster and datastores cleaned up; existing volumes retained. ## Review limits Local cluster verification used the locally built image and a host alias. Docker Desktop kubeadm execution and LA discussion/sign-off remain manual. Q11 recovery targets are proposals for the current LinkLab, not achieved production guarantees. Production deployment remains outside the lab. The user explicitly requested the PR merge after validation. ## Final remote verification - Final head: `650d27df3543ee4033b150da7c4f29a79f60acfa`. - [Forgejo CI run #9](https://cs4297.skynet.ie/23365978/cs4297-linklab/actions/runs/9): **success** for tests, JAR build, and Docker build/push. Downloaded test-results artifact confirms **49 tests, 0 failures, 0 skipped**. - Published `week8` and `sha-650d27d` tags; main's existing `latest` remains unchanged. - Published image: `cs4297.skynet.ie/23365978/cs4297-linklab@sha256:71ae6c32e8d3eccaa58a7ad59ffcbc136e2cbfca1a214002e264f6cdc1739865`. OCI revision label matches the final PR head. - Verified registry authentication and actual Kubernetes pulls using the `forgejo-registry` Secret and `imagePullPolicy: Always`, without loading a local app image into this second cluster. - Both registry-backed pods reached 1/1 Ready with zero restarts. The complete two-pod smoke passed again: same-token create=201 and identity=200 on each pod; invalid credentials/tokens rejected; RBAC enforced; Kafka indexing/leaderboard delivery and DynamoDB click persistence passed; admin reindex passed. - The first smoke attempt hit a kubectl port-forward stream timeout (`Error creating error stream` / `lost connection to pod`). Replacing the forwards restored transport; the full unchanged smoke passed. Pods remained healthy with zero restarts. - Native Linux verification uses the checked-in registry deployment plus a temporary `hostAliases` mapping for `host.docker.internal`. Docker Desktop kubeadm and LA sign-off remain manual. An agent source review has been completed; this is not a human approval. The user subsequently requested merging this PR after review and green PR CI. The baseline includes existing unmerged PR #5 and its earlier Week 7 answer annotations; those existing edits were preserved. Registry verification cleanup completed: namespace (including both Secrets) and temporary cluster deleted; the five lab datastores stopped; existing volumes retained. The working tree is clean and the final local/remote branch SHA matches. ## Merge Merged at the user's explicit request using merge commit `e09d70059a6c2286b9e8d740a884d89e8f66bb24`. Local `main` was fast-forwarded to match `origin/main`, with a clean working tree. [Post-merge CI run #10](https://cs4297.skynet.ie/23365978/cs4297-linklab/actions/runs/10) was triggered; its current status is available at that link. PR-head CI run #9 had already passed all three stages and 49 tests before merging.

23365978 added 3 commits 2026-10-27 14:08:02 +00:00

Small edits to answers cc140a10e4

test: prevent Kafka broker retries in CI 7ace34bd40

Complete Week 8 Kubernetes replication lab and verification 249649465b

23365978 added 1 commit 2026-10-27 14:09:12 +00:00

Record Week 8 pull request publication 650d27df35

23365978 reviewed 2026-10-27 14:32:20 +00:00

23365978 left a comment

Copy link

Agent source review completed for the Week 8 changes: deployment selectors/ports and Secret/ConfigMap wiring; host/Compose/Kubernetes Kafka listeners; shared JWT and RBAC regression boundaries; shared-schema persistence; rolling-update/readiness behavior; and branch-specific registry tagging. Validation: CI run #9 is green with 49 passing tests; actual registry-backed two-pod smoke passed. Known teaching limits are documented: public registration role selection, lab schema update, dependency-sensitive liveness and single-instance datastores. Human approval and LA discussion remain outstanding. No merge performed.

Agent source review completed for the Week 8 changes: deployment selectors/ports and Secret/ConfigMap wiring; host/Compose/Kubernetes Kafka listeners; shared JWT and RBAC regression boundaries; shared-schema persistence; rolling-update/readiness behavior; and branch-specific registry tagging. Validation: CI run #9 is green with 49 passing tests; actual registry-backed two-pod smoke passed. Known teaching limits are documented: public registration role selection, lab schema update, dependency-sensitive liveness and single-instance datastores. Human approval and LA discussion remain outstanding. No merge performed.

23365978 merged commit e09d70059a into main 2026-10-27 14:41:59 +00:00

23365978 referenced this pull request from a commit 2026-10-27 14:42:01 +00:00

Merge pull request 'Week 8: Kubernetes replication, health probes and failure experiments' (#6) from week8-kubernetes into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

23365978/cs4297-linklab!6

No description provided.