Instructions for developing a connector in Rill

Introduction

Connectors are resources that contain credentials and settings for connecting to external systems. They are typically found at the root of the project's DAG, providing access to data sources and services that power downstream resources like models and metrics views.

Connectors are usually lightweight resources. When reconciled, they validate the connection to the external system but do not move or process data. The main exception is managed OLAP connectors (with managed: true), which trigger database provisioning.

Driver capabilities

Each connector uses a driver that implements one or more capabilities:

• OLAP database: Can power metrics views and dashboards (e.g., duckdb, clickhouse, druid, pinot, starrocks, snowflake, bigquery, databricks as live connectors)
• SQL database: Can run SQL queries as model inputs (e.g., postgres, bigquery, snowflake)
• Information schema: Can list tables and their schemas (e.g., duckdb, bigquery)
• Object store: Can list, read, and write flat files (e.g., s3, gcs)