Today, I attempted to install Mage.ai via Docker as part of my familiarization with Mage.ai. This is currently (as of 2024-01-26) the only scenario for running Dbt together with Mage.ai within pipelines natively. Of course, there is a possibility to run dbt model using custom python code anyway (in case you use pip/conda installed mage)
This guide also covers a solution to an issue related to the incompatible dbt-sqlserver adapter – “dbt was unable to connect to the specified database. The database returned the following error: >SQLServerConnectionManager.execute() got an unexpected keyword argument ‘limit’
Installation of Mage.ai – Docker, SQL Server Adapter
1) Install Docker.
2) Create a folder where you want to set up your Mage project, wherever it makes sense to you. Open your terminal or command line and navigate to this folder.
3) In the Mage documentation, you will find a command for starting the Docker container and running the Mage project. Our project will be named mage-ai:
- For Mac/Linux – Run the command: docker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start mage-ai
- For Windows – Run the command: docker run -it -p 6789:6789 -v “%cd%:/home/src” mageai/mageai /app/run_app.sh mage start mage-ai
4) Mage is now running in the container and is accessible at http://localhost:6789/.
Initialization of dbt via Docker, dbt Debug, and Troubleshooting Errors
1) Dbt is already installed in the Docker container, as mentioned in the previous section. So, all you need to do is open your terminal, navigate to the dbt folder using dbt init dbt_project, and then select the adapter that corresponds to your database platform. In my case, it’s SQL Server.
2) You need to configure the profiles.yml file and dbt_project.yml file. The containerized Mage-ai with dbt expects the profiles.yml file inside the dbt project (not in .dbt or elsewhere). It may be possible to configure it differently, but I simply moved the file.
3)After configuring the connection, it’s time to test it using dbt debug. Here, I encountered an error – SQLServerConnectionManager.execute() got an unexpected keyword argument ‘limit’.
4) The problem lies in the potentially incompatible version of the dbt-sqlserver dbt adapter compared to dbt-core, which is present in the container after installation via the mageai/mageai image. You can confirm this by running the command: “dbt –version”.
5) To fix the dbt-sql adapter, update it with “pip install –upgrade dbt-sqlserver”, and upgrade any other incompatibilities.
