Skip to main content
Version: devel

Workspace MCP Server - current status

dltHub

This page is for dltHub Feature, which requires a license. Join our early access program for a trial license.

The server can do the following:

  • list pipelines in workspace
  • inspect table schema and data for dataset in particular pipeline
  • do sql queries

It is the same server that is called the open-source dlt in the documentation below.

Since all mcp clients work with sse transport, it is the default when running the server. Before we were struggling with launching mcp as a part of client process. There was no way to pass right Python virtual environment and dlt run context. There were also issues with stdio pollution from print statement (overall that was IMO a dead end, mcp is a server by nature.)

To launch the server in workspace context:

dlt workspace mcp

INFO:     Started server process [24925]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:43654 (Press CTRL+C to quit)

Workspace mcp server has 43654 as default port and is configured without any path (ie /sse) so user can just copy the link above in the right client.

To launch the server in pipeline context:

dlt pipeline fruitshop mcp

Starting dlt MCP server
INFO:     Started server process [28972]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:43656 (Press CTRL+C to quit)

Pipeline mcp server has 43656 as default port. Pipeline is already attached when mcp server starts. Both pipeline and workspace mcps can work side by side.

Example client configurations

Cursor, Cline

{
  "mcpServers": {
    "dlt-workspace": {
      "url": "http://127.0.0.1:43654/"
    },
    "dlt-pipeline-mcp": {
      "url": "http://127.0.0.1:43656/"
    }
  }
}

Continue: for some reason it does see mcp configuration created in dev container. Maybe someone will make it work...

name: dlt mcps
version: 0.0.1
schema: v1
mcpServers:
  - name: dlt-workspace
    type: sse
    url: "http://localhost:43654"

Configuration​

Server can still be started with stdio transport and different port using the command line. The plan is to allow to configure mcp deeply via dlt configuration system.

[workspace.mcp]
path="/sse"
port=888
[pipelines.fruitshop.mcp]
transport="stdio"

Interactions with Runtime​

This is a heads-up on how we host mcps on runtime. To be deleted.

  • deployed workspace dashboard has two routes /app to see the notebook and /mcp to connect to mcp server
  • workspace dashboard in single pipeline mode /app/fruitshop and /mcp/fruitshop
  • I'm also pondering exposing some kind of mcp attached to each marimo notebook

Project MCP server

dltHub

This page is for dltHub Feature, which requires a license. Join our early access program for a trial license.

This is our "project" mcp (integrates with dltHub features below) and can be launched with:

dlt project mcp

It gets 43655 port and project context is obtained before launching the server.

MCP Server

dltHub

This page is for dltHub Feature, which requires a license. Join our early access program for a trial license.

Currently, dltHub is building two MCP servers that you can run locally and integrate with your preferred IDE. One server is for the open-source dlt library and the other integrates with dltHub features (Learn more).

This page gives an overview of what we're building and includes detailed instructions to install the MCP in your favorite IDE.

warning

🚧 This feature is under development, and the interface may change in future releases. Interested in becoming an early tester? Join dltHub early access.

Model Context Protocol​

The Model Context Protocol (MCP) is a standard initiated by Anthropic to connect large language models (LLMs) to external data and systems.

In the context of the MCP, the client is built-in the user-facing application. The most common clients are LLM-enabled IDEs or extensions such as Continue, Cursor, Claude Desktop, Cline, etc. The server is a process that handles requests to interact with external data and systems.

Core constructs​

  • Resources are data objects that can be retrieved by the client and added to the context (i.e., prompt) of the LLM request. Resources will be manually selected by the user, or certain clients will automatically retrieved them.

  • Tools provided a way to execute code and provide information to the LLM. Tools are called by the LLM; they can't be selected by the user or the client.

  • Prompts are strings, or templated strings, that can be injected in the conversation. Prompts are selected by the user. They provide shortcuts for frequent commands, or allow to ask the LLMs to use specific tools.

note

The MCP is progressively being adopted and not all clients support all the features used by the dlt MCP server. See this page for an overview of client capabilities

Features​

The dlt and dltHub MCP servers aim to be a toolbox to help developers build, maintain, and operate dlt pipelines. There are two primary avenues:

  • Code generation: LLMs are good at writing Python code, but they don't know everything about dlt. The MCP provides resources and tools to provide up-to-date information to the LLm about the dlt library and the specifics of your project.

  • Assistance: The MCP resources, tools, and prompts can also be used to learn more about dlt. The built-in knowledgebase combined with your project's context brings Q&A support inside your IDE.

The next sections are a non-exhaustive documentation of existing and upcoming features.

Tools​

  • Pipeline metadata: read your pipeline directory (default: ~/.dlt/pipelines) to know available pipelines, available tables, table schemas.

  • Operational metadata: read your pipeline directory to identify last load date, schema changes, load errors, and inspect load packages.

  • dlt configuration: use an instantiated pipeline to inspect the dlt configuration (sources, resources, destinations).

  • Datasets: connect to the destination and execute SQL queries to retrieve data tables via light text-to-SQL.

Resources​

  • LLM-optimized dlt documentation pages. These can be selected and added to your prompt to help the LLM generate valid dlt code.

Prompts​

  • Tutorial-like instructions that puts the LLM in "assistant mode". For example, the LLM can ask you questions about your data and "think" with you to select the right loading strategy.

  • Command-like instructions that gives the LLM a task to complete. For example, the LLM can initialize a new pipeline. This is akin to a conversational command line tool.

Installation​

The dlt and dltHub MCP servers are intended to run locally on your machine and communicate over standard I/O. Typically, the MCP server process is launched by the MCP client, i.e., the IDE. We will use the uv package manager to launch the MCP server.

The next sections include client-specific instructions, references, and snippets to configure the MCP server. They are mainly derived from this uv command:

uv tool run --with "dlthub[mcp]==0.9.0" dlt mcp run

To explain each part:

  • uv tool run executes the command in an isolated virtual environment ---with PACKAGE_NAME specifies the Python dependencies for the command that follows
  • dlthub[mcp] ensures to get all the extra dependencies for the MCP server
  • dlthub==0.9.0 pins a specific dlthub version (where the MCP code lives). We suggest at least pinning the dlthub version to provide a consistent experience
  • dlt mcp run is a CLI command found in dltHub that starts the dlt MCP server. Use dlt mcp run_plus to

Then, to enable the MCP server and tool usage, several IDEs require you to enable "tool/agent/mcp mode".

dltHub MCP server​

To run the dltHub MCP server, you will need to set your dltHub License globally in ~/.dlt/secrets.toml or in an environment variable (must be set before lauching the IDE) and use dlt mcp run_plus in your configuration. If the dltHub license is missing, the dlt MCP server will be launched instead. You can tell the two apart by the tools, resources, and prompts available­.

Continue​

With Continue, you can use Continue Hub for a 1-click install of the MCP, or a local config file. Select Agent Mode to enable the MCP server.

Continue Hub​

See the dltHub page and select the dlt or dltHub Assistants. This bundles the MCP with additional Continue-specific features. You can also select the dlt or dltHub MCP blocks to install the server exclusively.

Local​

You can define an assistant locally with the same YAML syntax as the Continue Hub by adding files to $PROJECT_ROOT/.continue/assistants. This snippet creates an assistant with the MCP only.

# local_dlt.yaml
name: dlt MCP  # can change
version: 0.0.1  # can change
schema: v1
mcpServers:
  - name: dlt  # can change
    command: uv
    args:
      - tool
      - run
      - --with
      - dlthub[mcp]==0.9.0
      - dlt
      - mcp
      - run

There's also a global configuration specs in JSON

{
  "experimental": {
    "modelContextProtocolServers": [
      {
        "transport": {
          "type": "stdio",
          "command": "uv",
          "args": [
            "tool",
            "run",
            "--with",
            "dlthub[mcp]==0.9.0",
            "dlt",
            "mcp",
            "run"
          ]
        }
      }
    ]
  }
}

Claude Desktop​

You need to add a JSON configuration file on your system. See our full Claude Desktop tutorial

{
  "mcpServers": {
    "dlt": {
      "command": "uv",
      "args": [
        "tool",
        "run",
        "--with",
        "dlthub[mcp]==0.9.0",
        "dlt",
        "mcp",
        "run"
      ]
    }
  }
}

Cursor​

Select Agent Mode to enable the MCP server. The configuration follows the same JSON specs as Claude Desktop

{
  "mcpServers": {
    "dlt": {
      "command": "uv",
      "args": [
        "tool",
        "run",
        "--with",
        "dlthub[mcp]==0.9.0",
        "dlt",
        "mcp",
        "run"
      ]
    }
  }
}

Cline​

Follow this tutorial to use the IDE's menu to add MCP servers.

Manual server launch (advanced)​

The following methods allow the user to manually launch the MCP server from their preferred directory and Python environment before connecting to the IDE. The basic installation methods let the IDE launch the server over STDIO in isolation from the current project directory and Python environment.

We won't use uv tool run because we want to use the local Python environment instead of an isolated one. We assume that dlthub[mcp] is installed in the environment, giving access to the dlt mcp command.

warning

🚧 The MCP, IDE features, and the dlt MCP server are all rapidly evolving and some details are likely to change.

SSE transport​

To launch the server using Server-Sent Events (SSE) transport, modify the CLI command to:

dlt mcp run_plus --sse --port 43655

Then, configure your IDE to connect to the local connection at http://127.0.0.1:43655/sse (the /sse path is important). Many IDEs don't currently support SSE connection, but Cline does.

Proxy mode​

Since many IDEs don't currently support SSE, we use a workaround with two MCP servers using the mcp-proxy library:

  • the proxy server is launched by the IDE and communicates with the IDE over STDIO
  • the dlt server is launched by the user in the desired directory and environment and communicates with the proxy server over SSE

The benefit of this approach is that you will never need to update the IDE configuration; it always launches the same proxy server. Then, you can launch the dlt server from any context.

To launch the proxy server, follow the basic installation method (Cursor, Continue, Claude, etc.), but change the command to

uv tool run mcp-proxy "http://127.0.0.1:43655/sse"

For example, Cursor would use this config.

{
  "mcpServers": {
    "dlt": {
      "command": "uv",
      "args": [
        "tool",
        "run",
        "mcp-proxy",
        "http://127.0.0.1:43655/sse"
      ]
    }
  }
}

To launch the dlt server, use this command to start communication over SSE. The --port value should match the IDE config.

dlt mcp run_plus --sse --port 43655
warning

The proxy server typically fails at startup if the dlt server is not already running. After launching the dlt server, use the "reconnect" button in the IDE to have the proxy connect to it.

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.