Timeline
10-12 hours
PRO · part 01 free previewStreaming trackP24
Build the
streaming feature
spine StreamGuard runs on.
FinServe needs millisecond features for its fraud model. Build the full real-time feature spine: Feast 0.37 with 22 features across customer / account / merchant entities, Spark batch + Parquet for offline, Redis 7 + FastAPI for online (p99 < 10ms validated by Locust), Kafka 7.5 + Confluent Schema Registry + Avro contracts + Spark Structured Streaming with watermarks and 1m / 5m / 1h sliding windows, Evidently AI drift detection, and a Helm / K8s deployment with GitHub Actions feature-validation CI.
The “walk me through how you’d ship features to a fraud model in milliseconds” question — asked at any fintech with real-time risk decisions on the critical path (Stripe, Plaid, Chime, Robinhood).
By the end you will have shipped
- A Feast 0.37 project with 22 features across 3 entities (customer / account / merchant)
- Spark batch + Parquet offline store and Redis 7 + FastAPI online (p99 < 10ms validated by Locust)
- Kafka 7.5 + Confluent Schema Registry + Avro contracts + a 6-partition transaction topic
- Spark Structured Streaming with 1m / 5m / 1h sliding windows and 2-minute watermarks for late events
- Evidently AI drift detection (PSI + KS) + Prometheus exporter + Grafana dashboards
- Helm chart + Kubernetes manifests + a GitHub Actions feature-validation CI workflow
PREREQComfortable with Python (pandas, PySpark basics), Docker, and Kafka producer/consumer concepts. We recommend the Feature Stores curriculum first if Feast is new — it covers the primitives this project composes into a production streaming feature platform.
finserve.streaming.* · 22 features wired
watermark armedOffline
Streaming
Online · <10ms
Prod ops
customer_txns.parquet
merchant_risk.parquet
account_velocity.parquet
feast registry22 features · 3 entities
PIT-correct training set
spark batch · parquet
schema registryavro · BACKWARD compat
spark streaming1m / 5m / 1h windows
watermark · 2 min
stream-batch parity
the differentiator
materialize_incremental
redis_sink · writeStream
fastapi · pool=50
locust · p99 < 10ms
batch + stream converge
prometheus · gauges
grafana · dashboards
helm + k8s · probes
GH Actions · feast plan
ops spine
# Streaming windowing core
withWatermark("event_timestamp", "2 min")
window(col, "1 min", "15 sec") sliding
avro contracts · BACKWARD compat in CI
→ late events handled, then sunk into Redis
● Production ops spine
evidently DataDriftPreset · PSI > 0.25
alert: drift_share > 0.3 fires
helm chart · readiness probes · smoke tests
→ GH Actions runs feast plan on every PR
22
Features · 3 entities
p99 < 10ms
Locust-validated
1m / 5m / 1h
Sliding windows
Why streaming feature engineering, why now
Real-time features are now their own engineering specialty.
Fraud, trust-and-safety, and personalization teams all need features in milliseconds. The bar isn't writing Spark jobs — it's owning the streaming spine: Schema Registry contracts that don't break, watermarks that handle late events, sliding windows that match the fraud model's expectations, and drift detection that catches feature decay before the model does. That's the engineer companies pay senior+ for in 2026.
Real-time feature engineering is its own discipline
It's not the data engineer who writes the dbt model and not the ML engineer who trains the classifier. It's the platform engineer who owns the streaming spine in between — and that's the role hiring managers actually call out.
Schema governance stops streaming feature outages
Confluent Schema Registry + Avro + BACKWARD compatibility is what reviewers look for. Hand-typed Kafka payloads aren't — they're the production incident that takes the fraud model offline at 3 AM.
Watermarks + windowing > batch every 5 minutes
If a fraud event arrives 90 seconds late, batch jobs miss it. Spark Structured Streaming with watermarks emits the right window even when the data shows up out-of-order. That's the difference between catching fraud in real-time and catching it in tomorrow's report.
Drift detection is now table stakes
Evidently's PSI + KS-test pattern caught feature decay weeks before models started misbehaving. Without it, you're flying blind. With it, you have a Prometheus gauge + a Grafana panel + an alert rule the security team will sign off on.
Curriculum · 4 parts · 10-12 hours
Part 01 is free. The rest unlocks with PRO.
Try the first ~3 hours — set up Feast, define entities + feature views, build Spark batch features over Parquet sources, and ship a point-in-time correct training set. If it clicks, upgrade to unlock the Redis serving layer, the Kafka + Schema Registry + Spark Streaming spine, and the production ops layer (Evidently drift + Helm/K8s + GH Actions CI).
P24 · 10-12 hours · 4 parts
Free preview PRO required
Part 01 is free — no card required. See the Feast offline store + PIT-correct training set before paying.
M01
✓Foundation — Feast offline store + point-in-time joins
Set up Feast 0.37 with FileSource Parquet sources. Define 3 entities (customer / account / merchant) and 3 feature views (22 features total). Build Spark batch transforms. Generate point-in-time correct training sets with `store.get_historical_features()`. Validate with pytest.
M02
⊘Online store — Redis materialization + FastAPI <10ms serving
Stand up Redis 7 with persistence and LRU eviction. Build initial + incremental materialization scripts (with watermark tracking). FastAPI serving endpoint with connection pooling and TTL caching. Locust load test validating p99 < 10ms under concurrent load.
M03
⊘Streaming spine — Kafka + Schema Registry + Spark Streaming windowing
Kafka 7.5 + Zookeeper + Confluent Schema Registry. Avro schema registration with BACKWARD compatibility. Transaction producer simulating 10 TPS + fraud bursts. Spark Structured Streaming with `withWatermark('event_timestamp', '2 minutes')` and 1m / 5m / 1h sliding windows. Stream-batch consistency tests. Feature versioning + deprecation.
M04
⊘Production — Evidently drift + Helm/K8s + GH Actions CI
Evidently AI `DataDriftPreset` (PSI + KS test) emitting Prometheus gauges. Grafana dashboards (drift score, freshness, materialization timing, Redis memory). GitHub Actions feature-validation workflow (`feast plan` on PR). Helm chart deploying Feast + Redis + Kafka. Smoke tests + readiness probes.
3 parts locked · Unlock all PRO content for $29/mo
Upgrade to PRO →Backed by curriculum
Feature Stores for ML
The Feature Stores curriculum covers the primitives — this project shows you how to compose them into a production streaming feature platform on a real fintech dataset.
The build, in 3 phases
Three sprints. Three checkpoints. One millisecond feature spine.
Each phase ends with a tagged commit, a passing acceptance gate, and an artifact a senior platform reviewer would actually accept.
01~6h
Stand up offline + online
Feast registry with 22 features. PIT-correct training set generation. Redis 7 online store with materialization. FastAPI serving validated by Locust at p99 < 10ms.
- ✓Feast project: 3 entities + 3 feature views + 22 features over Parquet sources
- ✓PIT-correct training sets via store.get_historical_features()
- ✓Redis 7 + materialize_incremental + FastAPI + Locust validating p99 < 10ms
02~3h
Wire the streaming spine
Kafka 7.5 + Confluent Schema Registry + Avro contracts. Transaction producer with fraud-burst injection. Spark Structured Streaming with watermarks + 1m / 5m / 1h sliding windows. Stream-batch consistency tests passing.
- ✓Kafka 6-partition transaction topic + Confluent Schema Registry + Avro contracts
- ✓Spark Structured Streaming with withWatermark + sliding windows + Redis sink
- ✓Feature versioning + schema evolution + deprecation handlers
03~3h
Ship to production
Evidently AI drift detection (PSI + KS) emitting Prometheus gauges. Grafana dashboards. GitHub Actions feature-validation CI. Helm chart deploying the full stack to K8s with readiness probes + smoke tests.
- ✓Evidently drift detection + Prometheus exporter + Grafana dashboards + alert rules
- ✓GitHub Actions feature-validation workflow (feast plan on PR)
- ✓Helm chart + K8s manifests + smoke tests + readiness probes
Project setup · 10 minutes
One starter kit. Feast + Kafka + Spark + Redis + K8s, wired.
The starter kit ships a complete project skeleton — Feast registry with 22 feature definitions, sample Parquet sources (5K customers / 800 merchants / 6K accounts / 90 days), Kafka producer + Confluent Schema Registry config, Helm chart, and the GitHub Actions feature-validation workflow.
What lives in the repo
Everything you need to walk all 4 parts on your laptop — including the FinServe sample dataset that simulates a multi-merchant retailer at the row counts this project actually uses (not the inflated marketing numbers).
- feast-feature-store/ — Feast project with 3 entities + 3 feature views (22 features) + sources.yml
- data/raw/ — sample Parquet sources (5K customers / 800 merchants / 6K accounts / 90 days)
- src/streaming/ — Kafka producer, Avro schemas, Spark Structured Streaming windowing job
- src/serving/ — FastAPI app + connection pool + TTL cache + Locust load test
- monitoring/ — Evidently drift detector + Prometheus exporter + Grafana dashboard JSON
- helm/ + k8s/ — Helm chart + manifests + readiness probes + smoke tests
Download · Starter Kit
StreamGuard Feature Spine Starter Kit
Pre-built Feast project with 22 features, sample fintech data (5K customers / 800 merchants / 6K accounts), Kafka + Schema Registry + Avro config, Helm chart, and GitHub Actions workflows. docker-compose up + dbt-style apply in under 10 minutes.
~/projects/streamguard-anomaly-detection — zsh
1. Unzip and start the local stack (Redis + Kafka + Schema Registry)
$ unzip streamguard-anomaly-detection-starter.zip
$ cd streamguard-feature-spine
$ docker-compose up -d
2. Set up Python env + register the Feast feature views
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install -r requirements.txt
$ cd feast-feature-store && feast apply
3. Materialize the first 30 days into Redis
$ python scripts/materialize_initial.py --start 2024-01-01 --end 2024-01-31
4. Start the FastAPI feature-serving endpoint
$ uvicorn serving.app:app --host 0.0.0.0 --port 8000
5. Run the Locust load test against the serving endpoint
$ locust -f tests/load_test.py --host http://localhost:8000
5K
customers
800
merchants
6K
accounts
22
features
What changes vs a tutorial Feast project
The same feature spine — but built for the real fraud model.
Tutorials run a single materialization, against a single Redis, with hand-typed Kafka payloads and no late-event handling. Production requires watermarks, schema contracts, drift detection, and a Helm chart that survives a rolling upgrade. Here’s the diff, with the real APIs you reach for.
Tutorial versionWhat you ship in 10-12 hrs
×
Materialization
store.materialize() once, full window
×
Kafka payloads
Hand-typed JSON
×
Late events
Dropped or counted twice
×
Redis serving
Single client per request
×
Latency targets
Hope it's fast enough
×
Drift detection
Manual SQL spot-checks
×
Deployment
kubectl apply against a YAML pile
Production versionParts 02-04 + ops
✓
Materialization
store.materialize_incremental() with watermark tracking via Airflow DAG✓
Kafka payloads
Confluent Schema Registry + Avro with
BACKWARD compatibility enforced in CI✓
Late events
withWatermark("event_timestamp", "2 minutes") + sliding windows emit correct results out-of-order✓
Redis serving
Connection pool
max_connections=50 + cachetools.TTLCache in-process cache✓
Latency targets
Locust 50-100 concurrent users validating
p99 < 10msagainst the serving endpoint✓
Drift detection
Evidently
DataDriftPreset + PSI computation + Prometheus gauges + alert rule on drift_share > 0.3✓
Deployment
Helm chart with per-env
values.yaml + readiness probes + smoke tests in the CI pipelinePRO benefit · code review
Real review from ML platform engineers who’ve owned this spine.
Submit your repo, get line-by-line feedback within 48 hours from someone who has actually owned the streaming feature layer for a fraud model in production. The kind of review that's quietly worth thousands of dollars in time-to-staff.
CR
4 reviews / month
Submit a PR, a refactor proposal, or a full repo. Reviewer is matched to your domain — Feast / Kafka / Spark Streaming for this project. Async, comments inline, average turnaround 31 hours.
31h
avg turnaround
9.2/10
helpfulness
94%
return next month
OH
2 office hours / month
Live 30-min sessions with a senior ML platform engineer. Architecture questions, whiteboard a streaming windowing decision, mock a system-design interview on real-time feature serving. Group sessions also available.
30 min
per session
2 / mo
included
+ group
unlimited
What PRO unlocks
One subscription. 15+ projects, all curriculum, code review.
PRO is built for engineers who want production-grade builds and feedback loops — not more tutorials.
What you getFREEPROEXPERT
Projects
Production-grade builds
2
15+
8
Curriculum modules
All 7 tracks
Phase 1 only
All
All + bonus
Code review credits
Senior engineer review
0
4 / month
Unlimited
Career path access
5 paths × full plans
1 path
All 5
All 5 + 1:1
Certificate
Verifiable on LinkedIn
—
Yes
Yes + portfolio review
Community
Discord + office hours
Read-only
Full + 2/mo
Full + 4/mo
$29/mo
billed monthly · cancel anytime
or annual
$249/yr save 28%
Who this is for
Pick this if you want to own the millisecond feature layer.
MP
ML platform engineers
You own the layer between data engineering and ML. This project is the streaming feature spine you'll defend in code review and on a system-design whiteboard.
DE
Senior data engineers crossing into ML
You ship dbt + Airflow but the next interview loop wants you to talk Kafka + Spark Streaming + Schema Registry. After this you can defend windowing trade-offs and watermark tuning from first principles.
SD
DEs prepping for streaming + schema interviews
Take-homes increasingly ship a Kafka topic and ask for a real-time feature pipeline. After this you can produce one in an afternoon and walk through schema evolution + late-event handling without hand-waving.
OP
MLOps engineers focusing on the feature layer
You've shipped models. You haven't owned the feature spine that feeds them. This is the project that lets you talk about drift, freshness, and feature CI as concretely as you talk about model rollbacks.
Related curriculum
Going deeper? Three tracks back this project.
The Feature Stores curriculum is the spine. These three let you go deeper on the layers it touches — Kafka primitives behind the streaming spine, MLOps platform patterns behind the production layer, and streaming theory behind the watermarking + windowing decisions.
FAQ
Quick answers.
No — and that's the most important thing to know upfront. This project ships the streaming feature spine that an anomaly model would consume (Feast offline + online + Kafka + Spark Streaming + Schema Registry). The model itself — Isolation Forest, autoencoder, XGBoost classifier — is intentionally out of scope. For the full MLOps lifecycle (model training + BentoML serving + drift + AI integration), see /projects/predictflow-feature-store (P07). The two projects are designed to pair.
P07 is the full MLOps lifecycle (MLflow experiment tracking + Feast + scikit-learn/XGBoost training + BentoML serving + canary rollout + Claude API/RAG integration in part 5). This project (P24) goes deeper on the streaming feature side specifically — Confluent Schema Registry + Avro contracts + Spark Structured Streaming with watermarks + 1m/5m/1h sliding windows. P07 is the systems integrator's project; P24 is the streaming feature engineer's project. Most learners do both.
The catalog entry was wrong — the tutorials use Spark Structured Streaming, not Flink. We've corrected it. For a Flink-specific deep-dive, see /projects/flink-fraud-detection (P01) — keyed state, exactly-once via 2PC, Flink K8s Operator with ZK HA. Different stack, different positioning.
No. The starter kit ships docker-compose with Kafka + Zookeeper + Confluent Schema Registry + Redis, and the data generator simulates 10 TPS + fraud bursts locally. Helm + K8s deployment in Part 04 runs against a local kind/minikube cluster. The patterns transfer to MSK or Confluent Cloud with config changes only.
Parts 02-04 (Redis + FastAPI + Locust + Kafka + Schema Registry + Spark Streaming + Evidently + Helm/K8s + GH Actions CI), the starter kit ZIP, plus 4 code-review credits + 2 office-hours sessions per month and access to all 15+ PRO projects + curriculum across all 7 tracks. Cancel anytime.
Especially the system-design rounds. After this you can whiteboard a streaming feature pipeline end-to-end, defend a Kafka partition + Spark window choice, explain why watermarks matter for fraud, and reason about Evidently PSI thresholds with concrete numbers. Those are the four conversations that decide most senior+ ML platform offers.
Ready to build the feature spine?
Start with Part 01 — free, no card. About 3 hours. By the end you'll have Feast running with 22 features across 3 entities, point-in-time correct training sets generated, and pytest validating the registry.
P24 · StreamGuard Feature Spine · PRO · part 01 freeUpgrade to PRO →