Skip to main content
Version: 0.16.16

How to configure credentials

This guide will explain how to populate credentials either through an environment variable, or configure your Great Expectations project to load credentials from either a YAML file or a secret manager.

If your Great Expectations deployment is in an environment without a file system, refer to How to instantiate an Ephemeral Data Context.

Prerequisites​

Using Environment Variables​

The quickest way to get started is by setting up your credentials as environment variables.

First set values by entering export ENV_VAR_NAME=env_var_value in the terminal or adding the commands to your ~/.bashrc file:

export MY_DB_PW=password
export POSTGRES_CONNECTION_STRING=postgresql://postgres:${MY_DB_PW}@localhost:5432/postgres

These can then be loaded into the connection_string parameter when we are adding a datasource to the Data Context.

# The password can be added as an environment variable
pg_datasource = context.sources.add_or_update_sql(
    name="version-0.16.16 my_postgres_db",
    connection_string="postgresql://postgres:${MY_DB_PW}@localhost:5432/postgres",
)

# Alternately, the full connection string can be added as an environment Variable
pg_datasource = context.sources.add_or_update_sql(
    name="version-0.16.16 my_postgres_db", connection_string="${POSTGRES_CONNECTION_STRING}"
)

Using YAML or Secret Manager​

Using the config_variables.yml file​

A more advanced option is to use the config variables YAML file. YAML files make variables more visible, easily editable, and allow for modularization (e.g. one file for dev, another for prod).

If using a YAML file, save desired credentials or config values to great_expectations/uncommitted/config_variables.yml:

my_postgres_db_yaml_creds: postgresql://localhost:${MY_DB_PW}@$localhost:5432/postgres
note
  • If you wish to store values that include the dollar sign character $, please escape them using a backslash \ so substitution is not attempted. For example in the above example for Postgres credentials you could set password: pa\$sword if your password is pa$sword. Say that 5 times fast, and also please choose a more secure password!
  • You can also have multiple substitutions for the same item, e.g. database_string: ${USER}:${PASSWORD}@${HOST}:${PORT}/${DATABASE}

Then the config variable can be loaded into the connection_string parameter when we are adding a datasource to the Data Context.

# Variables in config_variables.yml can be referenced in the connection string
pg_datasource = context.sources.add_or_update_sql(
    name="version-0.16.16 my_postgres_db", connection_string="${my_postgres_db_yaml_creds}"
)

Additional Notes​

  • The default config_variables.yml file located at great_expectations/uncommitted/config_variables.yml applies to deployments using FileSystemDataContexts.
  • To view the full script used in this page, see it on GitHub: how_to_configure_credentials.py