ClaudeSuperPower
SubmitLog in

databricks-spark-structured-streaming

Skill

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoint

0Data Engineering

Install

git clone https://github.com/databricks/databricks-agent-skills.git ~/.claude/skills/databricks-spark-structured-streaming

What is databricks-spark-structured-streaming?

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

What this can do

Capabilities declared in this component's own frontmatter — not inferred.

Inherit all session tools

Declares no tool restrictions — inherits every session tool

~105 tokens of context used while enabled, before you invoke anything

Documentation

README · ~1 min read

Spark Structured Streaming

Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.

Quick Start

from pyspark.sql.functions import col, from_json

# Basic Kafka to Delta streaming
df = (spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")

Reviews

Log in to leave a review.

No reviews yet — be the first.

Explore related

Other things in this space — across every part of the ecosystem, not just skills.