Databricks MCP Server: Give AI Agents Access to Notebooks, Jobs, and Unity Catalog

Why Give AI Access to Databricks?

Databricks runs data lakehouse workloads for thousands of enterprises. When jobs fail, notebooks need debugging, or you need to understand table lineage, you switch to the Databricks workspace to check job runs, browse Unity Catalog, and read notebook outputs. Each switch breaks flow.

An MCP server for Databricks lets your AI agent check job status, query tables, inspect notebooks, and browse catalog metadata directly from Claude, Cursor, or Windsurf through DataFaucet.

What Becomes Possible

Once your AI agent has Databricks access:

• "Which jobs failed in the production workspace last night?"
• "Show me the schema for the gold.customers table in Unity Catalog"
• "What's the run history for the daily_etl_pipeline job?"
• "Are there any long-running clusters I should terminate?"
• "Show me the output of the last cell in the data_quality notebook"
• "What tables in the silver layer haven't been updated in 48 hours?"

How It Works

1. Go to DataFaucet and enter your Databricks workspace URL
2. Authenticate with your Databricks credentials
3. Browse jobs, notebooks, catalog as you normally would
4. Each API call becomes an MCP tool your AI can invoke
5. Deploy and connect to Claude, Cursor, or Windsurf

The AI agent accesses whatever workspace resources your Databricks permissions allow.

Security Best Practices

• Read-only service principal — create a Databricks service principal with viewer permissions
• No cluster creation — prevent AI from spinning up compute resources
• No job triggers — block AI from starting job runs
• Workspace-scoped — limit to specific workspaces, not entire account
• No secret access — exclude Databricks Secrets scope
• Unity Catalog RBAC — use fine-grained table permissions to limit data access

Use Cases by Role

Data engineers: Job debugging. "Why did the ETL pipeline fail? Show me the error from the last run and the notebook cell that failed" while writing the fix.

ML engineers: Experiment tracking. "What was the accuracy of the last model training run? Show me the MLflow metrics for the latest experiment."

Analytics engineers: Catalog discovery. "What tables exist in the gold schema? When were they last updated? What columns does customer_360 have?"

Platform engineers: Cost management. "Which clusters have been running for more than 4 hours? What's the total DBU usage this week by workspace?"

Databricks UI vs AI Agent

Data Stack Integration

Databricks connects to dbt, Airflow, and downstream BI tools. With DataFaucet MCP servers for each:

• Correlate Databricks job failures with Airflow DAG status
• Check dbt model freshness against Databricks table update times
• Verify Unity Catalog lineage matches dbt documentation

Getting Started

Set up read access for your AI agent:

1. Create a Databricks service principal with minimal permissions:

- Workspace: Can view jobs, notebooks, and clusters

- Unity Catalog: Can browse schemas and read table metadata

- SQL Warehouses: Can execute read-only queries

- No create, delete, or run permissions

1. Point DataFaucet at your Databricks workspace
2. Authenticate with the service principal token
3. Browse jobs, catalog, and notebooks as you normally would

Related: Connect Databricks to AI for the quick-start guide, dbt MCP Server for data modeling, Airflow MCP Server for pipeline orchestration.