LinkLab — Week 6 Blueprint

CS4297 Applied System Design · Async Messaging with Apache Kafka

Complete Week 5 solution included. Clone this and start here.

---

What you are building this week

Part	What happens	Time
A	Measure synchronous latency baseline	10 min
A.5	Simulate burst traffic — feel backpressure with sync approach	15 min
B	Implement two Kafka producers, replace sync fan-out	60 min
C	Partitions, consumer parallelism, when NOT to use Kafka	15 min
Optional 1	Consumer lag monitoring in Grafana	30 min
Optional 2	Dead letter queue for failed events	30 min
Optional 3	Outbox pattern for guaranteed delivery	30 min

---

Prerequisites

Windows: use Git Bash, not PowerShell. Every command here is written for a Unix shell. In PowerShell curl is an alias for something else and rejects -s and -X, and \ line continuations do nothing, so commands run truncated. If you must use PowerShell: write curl.exe, keep each command on one line, and use .\gradlew.bat for ./gradlew.

• JDK 21 — check with java -version
• Docker Desktop with at least 6 GB RAM (Settings → Resources → Memory)
• kotlinc on PATH — needed for the seed-links script in Part A.5
  • Mac: brew install kotlin
  • Windows: sdkman.io or JetBrains Toolbox
  • Linux: snap install kotlin
  • Check: kotlinc -version

---

Getting started

1. Start the infrastructure

docker compose up -d mysql redis dynamodb-local dynamodb-admin elasticsearch kafka

Kafka runs in KRaft mode (no Zookeeper needed) and starts in ~15 seconds.

docker compose ps   # kafka AND elasticsearch should show "healthy"

Wait for both before starting the application. On first start the app creates its ElasticSearch index and allows five seconds for that; if ElasticSearch is not ready, or cannot allocate shards because your disk is over 90% full, the app aborts with 5,000 milliseconds timeout on connection http-outgoing-0.

2. Run the application

From IntelliJ: open LinkLabApplication.kt and click ▶

From terminal (Mac/Linux): ./gradlew bootRun From terminal (Windows): .\gradlew.bat bootRun

You should see in the logs:

Started LinkLabApplication
KafkaMessageListenerContainer ... started
Creating DynamoDB table 'linklab-click-events'

If you see Connection to node -1 could not be established, Kafka is not yet healthy. Wait a few seconds and restart the app.

3. Run the tests

./gradlew test

• All Week 5 tests pass (solution complete)
• KafkaProducerTest — 4 tests fail until you implement the producers. That is expected.

---

What to implement

src/main/kotlin/ie/ul/cs4297/linklab/kafka/
├── KafkaTopics.kt           # Complete — topic name constants
├── KafkaConfig.kt           # Complete — producer/consumer config
├── LinkEvents.kt            # Complete — event data classes
├── LinkCreatedProducer.kt   # WEEK 6 TASK: implement publish()
├── LinkClickedProducer.kt   # WEEK 6 TASK: implement publish()
├── LinkCreatedConsumer.kt   # Complete — do not modify
├── LinkClickedConsumer.kt   # Complete — do not modify
├── ConsumerLagService.kt    # WEEK 6 OPTIONAL TASK 1
├── KafkaDlqConfig.kt        # WEEK 6 OPTIONAL TASK 2
└── OutboxPoller.kt          # WEEK 6 OPTIONAL TASK 3

Implement in this exact order (the same order as the lab sheet, Part B):

1. LinkCreatedProducer.publish() — one kafkaTemplate.send() in a try/catch
2. LinkClickedProducer.publish() — same pattern. All 35 tests pass once both are done; do not move on until they do.
3. In LinkService.shorten():
   • Delete the two "WEEK 5 approach" try/catch blocks (ES index + leaderboard)
   • Uncomment the "WEEK 6 approach" Kafka block
   • Verify POST /api/links still returns 201 before continuing
4. In LinkController.redirect():
   • Delete the "WEEK 5 approach" clickEventService.recordClick() call
   • Uncomment the "WEEK 6 approach" Kafka publish block
   • Verify GET /{code} still returns 302 before continuing

---

Architecture after Week 6

POST /api/links
  └── MySQL save (source of truth)
  └── Kafka publish (link-created)  ← you implement this
            └── [async] LinkCreatedConsumer
                  ├── ES index
                  └── Redis leaderboard

GET /{code}
  └── Redis cache lookup (returns if hit)
  └── MySQL lookup (only on cache miss)
  └── Kafka publish (link-clicked)  ← you implement this (happens on EVERY redirect)
            └── [async] LinkClickedConsumer
                  └── DynamoDB write

---

Useful Kafka CLI commands

# List topics
docker exec linklab-kafka kafka-topics \
  --bootstrap-server localhost:9092 --list

# Watch events in real time
docker exec linklab-kafka kafka-console-consumer \
  --bootstrap-server localhost:9092 --topic link-created --from-beginning

# Check consumer lag
docker exec linklab-kafka kafka-consumer-groups \
  --bootstrap-server localhost:9092 \
  --describe --group linklab-created-consumer

# Increase partitions. Do this on a SCRATCH topic, as Part C says: partition
# counts can only go up, never down, and changing the count of a live topic
# changes which partition each key lands in, so events for one link would
# stop arriving together. Part C.1 creates partition-demo with 3; this takes
# it to 6, and shows you that the command only ever increases.

docker exec linklab-kafka kafka-topics \
  --bootstrap-server localhost:9092 \
  --alter --topic partition-demo --partitions 6

---

Forgejo pipeline

Push your Week 6 work to your Forgejo repo — the pipeline from Week 5 should still go green. The pipeline runs ./gradlew test and builds the Docker image. Make sure all tests pass locally before pushing.

git checkout -b week6-kafka
git add . && git commit -m "Week 6: Kafka producers"
git push origin week6-kafka

Then open a Pull Request on Forgejo from week6-kafka → main and merge it. The pipeline runs on pushes to main, so merging the PR is what fires it.

---

Week-by-week progression

Week	Content	Blueprint
2	REST API + MySQL	Stubs for LINK-001
3	Docker · Redis · NGINX · Grafana	Complete Week 2
4	SQL vs NoSQL · DynamoDB	Complete Week 3
5	CI/CD · ElasticSearch · Fan-out	Complete Week 4
6	Kafka async messaging + scale	← you are here
7	JWT + OAuth2	Complete Week 6
8	Kubernetes	Complete Week 7
9	Capstone: full integration	Complete Week 8

---

Common issues

Kafka stays unhealthy / kafka-topics --list times out KRaft needs a valid CLUSTER_ID. It's set in docker-compose.yml already. If Kafka won't start, replace just that container: docker compose rm -sf kafka && docker compose up -d kafka. It is healthy again in about 20 seconds and every other service keeps running. Your topics and their contents go with it — that is the reset. Don't use docker compose down -v: it stops every service and deletes the MySQL and ElasticSearch volumes too.

Producer implemented but events never appear in Kafka You implemented publish() but forgot to uncomment the Kafka block in LinkService.shorten() or LinkController.redirect(). Check both files — the "WEEK 6 approach" block must be uncommented AND the "WEEK 5 approach" blocks must be deleted.

Consumer never processes events / search not updating Check the app logs for INFO Received link-created. If you don't see it, the consumer is not receiving events. Verify the topic name in LinkCreatedProducer matches KafkaTopics.LINK_CREATED exactly.

KafkaProducerTest fails with UnsatisfiedDependencyException KafkaAutoConfiguration is not excluded in the test profile. Check application-test.properties includes it in spring.autoconfigure.exclude.

ElasticSearch not indexing after Kafka migration After deleting the sync linkSearchService.index() call, indexing now happens via LinkCreatedConsumer. Wait 1–2 seconds after creating a link before searching — the consumer processes events asynchronously.

./gradlew bootRun fails with IllegalArgumentException Gradle is on the wrong JDK — it must be Java 21, not 17 and not 25. Run ./gradlew --stop (.\gradlew.bat --stop on Windows), then set Gradle JVM to Java 21 in IntelliJ Settings → Build Tools → Gradle → Gradle JVM.