In this guide, we will take a look at how to configure the io_config.yaml file in Mage.ai. We will also explore how to hide and encrypt access passwords so that they are not readily available in this configuration file.

Mage.ai io_config.yaml Configuration and Location

The io_config.yaml file is the main configuration file for setting up profiles that are then used in ETL pipelines. You can find this file in the project’s root directory. In my case: /home/src/mage-ai

mage.ai io_config

When you open this file for the first time, you will see a profile named ‘default,’ which contains templates for configuring various sources, such as:

  • AWS
  • Google
  • MSSQL
  • MongoDb
  • Mysql
  • PostgresSQL
  • Azure
  • Chroma, Clickhouse, Druid, DuckDb, etc.

For my tutorials, I use a connection to AdventureWorks on Azure as the source (thanks to sqlservercentral.com) and a local SQL Server database as the destination. So, my configuration will look like this:

io_config.yaml configuration and database settings

I have created two new profiles in Mage 2

  • AdventureWorks – freely available database
  • AW_FULL_LOAD – My database where I load data

You may notice that my database does not contain any configuration details but includes a reference to a secret.

Managing Project Secrets (Passwords, Addresses, Accounts) in Mage.ai

As you can see in the previous screenshot, it is not desirable to have login credentials exposed for security reasons. Mage.ai allows the creation of secrets that can be dynamically referenced in code or configuration files.

Secrets in Mage.ai have a project scope. However, according to the documentation, there is planned functionality to allow different scopes – environment, pipeline, and more. You can view secrets in the right sidebar:

mage.ai secrets settings

Here, you can create a secret and then reference it using the parameter – “{{ mage_secret_var(”) }}”

The secret is stored in the local environment’s database, where it is encrypted. The key is stored in the ~/.mage_data file. So, if you are working in a team with each member having Mage.ai in their local environment and developing for a remote repository, each team member must set up their secret separately (the secret is in the local environment, and the encryption key is in the project directory). More about secrets in Mage doc

5/5 - (1 vote)

Ing. Jan Zedníček - Data Engineer & Controlling

My name is Jan Zedníček and I have been working as a freelancer for many companies for more than 10 years. I used to work as a financial controller, analyst and manager at many different companies in field of banking and manufacturing. When I am not at work, I like playing volleyball, chess, doing a workout in the gym.

🔥 If you found this article helpful, please share it or mention me on your website

Leave a Reply

Your email address will not be published. Required fields are marked *