aidp-spark-optimization
SkillUse when a Spark job/notebook is slow, missing an SLA, spilling, OOMing, generating too many small files, or shuffling/skewing heavily; when reviewing Spark/PySpark code or a Spark UI for performance; or before running a large Spark workload. Covers open-source Apache Spark 3.5.0 (+ Delta Lake) tuni
Install
git clone https://github.com/oracle-samples/oracle-aidp-samples.git ~/.claude/skills/aidp-spark-optimizationWhat is aidp-spark-optimization?
Use when a Spark job/notebook is slow, missing an SLA, spilling, OOMing, generating too many small files, or shuffling/skewing heavily; when reviewing Spark/PySpark code or a Spark UI for performance; or before running a large Spark workload. Covers open-source Apache Spark 3.5.0 (+ Delta Lake) tuning -- partitions, shuffle, joins, skew, file layout, memory, codegen, caching, AQE, compression, and configuration.
What this can do
Capabilities declared in this component's own frontmatter — not inferred.
Inherit all session tools
Declares no tool restrictions — inherits every session tool
~104 tokens of context used while enabled, before you invoke anything
Documentation
README · ~6 min readSpark Optimization
Field-tested techniques for optimizing open-source Apache Spark 3.5.0 (and open-source Delta Lake) workloads. Every technique lists what it does, why it matters, patterns (when it helps), anti-patterns (when it does not help or hurts), the exact config/code, and impact evidence from real reproductions.
Applies to plain OSS Spark 3.5.0. Vendor-only features (Databricks Photon / predictive optimization / Delta disk cache, AWS Glue/EMR runtime tricks) are excluded or explicitly flagged. (Note: Delta Lake liquid clustering, OPTIMIZE/ZORDER, deletion vectors, CDF are open-source — GA in Delta 3.2.0 — see references/08-delta-lake.md.) Some examples run on the Oracle AI Data Platform (AIDP); AIDP-specific execution rules are in references/aidp-notes.md.
In this plugin (
oracle-ai-data-platform-workbench-engineer-agent): measure/apply on AIDP via the bundledscripts/aidp_sql.py(one SparkSession per cluster -- set/revertspark.confexplicitly; seereferences/aidp-notes.md); pull Spark-UI stage/task metrics via theaidp-spark-debuggingskill; right-size compute + set cluster-create-only configs (e.g.spark.memory.fraction) viaaidp-cluster-ops; run DeltaOPTIMIZE/VACUUM/ZORDERDDL viaaidp-sql-ddl/aidp-table-management. Adapted from theai-data-engineer-agentspark-optimizationskill (Oracle).
Core principle
Reviews
Log in to leave a review.
No reviews yet — be the first.
Explore related
Other things in this space — across every part of the ecosystem, not just skills.
Skillssimilar to this one
All skills →ECC
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
235.3K stars
ui-ux-pro-max-skill
An AI SKILL that provide design intelligence for building professional UI/UX multiple platforms
111.3K stars
claude-mem
Persistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
88.9K stars
Subagents
All subagents →ECC
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
235.3K stars
code-architect
Designs feature architectures by analyzing existing codebase patterns and conventions, then providing comprehensive implementation blueprints with specific files to create/modify, component designs, data flows, and build sequences
32.8K stars
plugin-validator
Use this agent when the user asks to "validate my plugin", "check plugin structure", "verify plugin is correct", "validate plugin.json", "check plugin files", or mentions plugin validation. Also trigger proactively after user creates or modifies plugin components. Examples: <example> Context: User f
32.8K stars
Commands
All commands →create-plugin
Guided end-to-end plugin creation workflow with component design, implementation, and validation
32.8K stars
release-testing
Run automated UAT tests (UI or API) using a test profile on the AWS DevOps Agent
2.2K stars
quick-build
Creates a WordPress block theme from a description with an index.html landing page template and deploys it to a local Studio site
Plugins
All plugins →claude-md-management
Tools to maintain and improve CLAUDE.md files - audit quality, capture session learnings, and keep project memory current.
32.9K stars
frontend-design
Create distinctive, production-grade frontend interfaces with high design quality. Generates creative, polished code that avoids generic AI aesthetics.
32.9K stars
playground
Creates interactive HTML playgrounds — self-contained single-file explorers with visual controls, live preview, and prompt output with copy button. Includes templates for design playgrounds, data explorers, concept maps, and document critique.
32.9K stars
MCP Servers
All mcp servers →codebase-memory-mcp
High-performance code intelligence MCP server. Indexes codebases into a persistent knowledge graph — average repo in milliseconds. 158 languages, sub-ms queries, 99% fewer tokens. Single static binary, zero dependencies.
36.7K stars
Figma-Context-MCP
MCP server to provide Figma layout information to AI coding agents like Cursor
15.5K stars
infranodus-mcp-server-infranodus
Map text into knowledge graphs to create a structured representation of conceptual relations and t…
95 stars