In this guide, we will take a look at how to configure the io_config.yaml file in Mage.ai. We will also explore how to hide and encrypt access passwords so that they are not readily available in this configuration file.
Mage.ai io_config.yaml Configuration and Location
The io_config.yaml file is the main configuration file for setting up profiles that are then used in ETL pipelines. You can find this file in the project’s root directory. In my case: /home/src/mage-ai
When you open this file for the first time, you will see a profile named ‘default,’ which contains templates for configuring various sources, such as:
- AWS
- MSSQL
- MongoDb
- Mysql
- PostgresSQL
- Azure
- Chroma, Clickhouse, Druid, DuckDb, etc.
For my tutorials, I use a connection to AdventureWorks on Azure as the source (thanks to sqlservercentral.com) and a local SQL Server database as the destination. So, my configuration will look like this:
I have created two new profiles in Mage 2
- AdventureWorks – freely available database
- AW_FULL_LOAD – My database where I load data
You may notice that my database does not contain any configuration details but includes a reference to a secret.
Managing Project Secrets (Passwords, Addresses, Accounts) in Mage.ai
As you can see in the previous screenshot, it is not desirable to have login credentials exposed for security reasons. Mage.ai allows the creation of secrets that can be dynamically referenced in code or configuration files.
Secrets in Mage.ai have a project scope. However, according to the documentation, there is planned functionality to allow different scopes – environment, pipeline, and more. You can view secrets in the right sidebar:
Here, you can create a secret and then reference it using the parameter – “{{ mage_secret_var(”) }}”
The secret is stored in the local environment’s database, where it is encrypted. The key is stored in the ~/.mage_data file. So, if you are working in a team with each member having Mage.ai in their local environment and developing for a remote repository, each team member must set up their secret separately (the secret is in the local environment, and the encryption key is in the project directory). More about secrets in Mage doc
