Os environ python error

Environment variables provide a great way to configure your Python application, eliminating the need to edit your source code when the configuration changes. Common configuration items that are often passed to application through environment variables are third-party API keys, network ports, database servers, and any custom options that your application may need to work properly.

In this article I’m going to share some of the techniques and tools available in Python to work with environment variables.

How to access environment variables from Python

Using the os.environ dictionary

In Python, the os.environ dictionary contains all the environment variables. The simplest way to retrieve a variable from inside your application is to use the standard dictionary syntax. For example, this is how you can access an environment variable named USER:

>>> import os
>>> user = os.environ['USER']
>>> user
'miguel'

>>> database_url = os.environ['DATABASE_URL']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/miguel/.pyenv/versions/3.8.6/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'DATABASE_URL'

Using os.environ.get()

Getting the KeyError is a good idea for environment variables that your program requires, but there are situations where you may want to allow some variables to be optional. To avoid the error you can use the dictionary’s get() method, which returns None when the requested key does not exist in the dictionary:

>>> database_url = os.environ.get('DATABASE_URL')

Adding a default value if the variable is not defined

If you’d like to provide a default value for the missing variable that is not None, you can add it as a second argument:

>>> database_url = os.environ.get('DATABASE_URL', 'sqlite:///')
>>> database_url
'sqlite:///'

Using the os.getenv() function

Python also provides the os.getenv() function to access environment variables. This function works in a very similar way to the os.environ.get() method. Here is how to access a variable with it:

>>> user = os.getenv('USER')
>>> user
'miguel'

>>> database_url = os.getenv('DATABASE_URL', 'sqlite://')
>>> database_url
'sqlite://'

Is os.getenv() better than os.environ? That is really up to you. I personally prefer to use the os.environ dictionary, since it gives me the option of halting the program with a KeyError if a required variable is missing.

How to set environment variables

In this section I’m going to give you a quick summary of how to set environment variables in a terminal or command prompt window. If you want to know all the possible ways to set environment variables, my colleague Dominik Kundel has written a very detailed blog post on this subject titled How to Set Environment Variables.

Unix and MacOS

There are two basic ways to set an environment variable from a bash or zsh terminal session. One is using the export keyword:

A variable that is set in this way will be passed on to any programs or scripts that you start from this terminal session. Keep in mind that environment variables are not saved anywhere outside of the context of the shell session, so they will go away when you close the terminal session.

An alternative way to define an environment variable is to set it in the same line where the target application is executed:

DEBUG=true python my_cool_application.py

This second form has the benefit that the variable is only set in the environment space of the intended application.

Microsoft Windows

If you are using Windows you have a few options. If you are interested in setting environment variables via the control panel, see the article linked above.

If you are in a command prompt window, you can set an environment variable using the set command:

Like in the Unix case, the variable is not stored or remembered beyond the current session.

If you are using the newer PowerShell console, the syntax for setting environment variables is completely different:

Finally, if you are using a Unix compatibility layer such as WSL or Cygwin, then you must go to the Unix section above and use any of the methods listed there for bash or zsh.

Using .env files

Are you confused by all the different ways to set environment variables? I personally find it inconvenient that each platform or shell requires a different procedure.

In my opinion, a better way to manage your environment variables is to store them in a .env (pronounced dot env) file. A .env file is a text file in which the variables are defined, one per line. The format of a .env file is exactly the same under all operating systems, so .env files make working with environment variables uniform across all platforms. And as if this isn’t enough, having your environment variables written in a file that is automatically imported by Python means that you don’t have to manually set them every time you start a new shell.

Here is a short .env file example with two variables:

DEBUG=true
DATABASE_URL=sqlite:///mydb.sqlite

You can create a .env file in the root directory of each of your projects, and that way you can keep all the variables that are needed by each project neatly organized!

The python-dotenv package allows a Python application to import variables defined in a .env file into the environment. You can install python-dotenv in your virtual environment using pip:

pip install python-dotenv

>>> from dotenv import load_dotenv
>>> load_dotenv()

The load_dotenv() function will look for a file named .env in the current directory and will add all the variable definitions in it to the os.environ dictionary. If a .env file is not found in the current directory, then the parent directory is searched for it. The search keeps going up the directory hierarchy until a .env file is found or the top-level directory is reached.

If you want to prevent python-dotenv from searching for a .env file through your directories, you can pass an explicit path to your file as an argument to load_dotenv():

>>> from dotenv import load_dotenv
>>> load_dotenv('/home/miguel/my_project/.env')

There are some additional arguments that you can use when you call the load_dotenv() function. If you want to learn about them consult the documentation.

Once the .env file is imported, you can access environment variables using any of the methods shown above.

A note about .env file security

In many cases you will be adding environment variables that contain sensitive information to your .env files, like passwords or API keys. For that reason, in general you do not want to add these files to your project’s source control repository.

The standard practice is to add an exception for files with this name, so that they are not committed to source control by mistake. For git, you can add a line with the name of the file to the .gitignore file in the root of your repository.

But if you cannot commit the .env file, how do you tell users of the project which variables need to be set? For this you can add an example .env file to your repository with a name such as .env.example, that contains the list of variables that the project requires, but without including their values. This serves as guidance for users of your project, without giving away sensitive information.

Conclusion

I hope this article was useful to help you understand how to use environment variables to configure your Python projects.

Do you have any other ways to work with environment variables? I’d love to know!

Miguel Grinberg is a Python Developer for Technical Content at Twilio. Reach out to him at mgrinberg [at] twilio [dot] com if you have a cool Python project you’d like to share on this blog!