Docker Deployment

Overview
The Ed-Fi ODS Docker deployment scripts install and configure components of the Ed-Fi ODS / API on Docker containers. The example Docker Compose files automate the numerous configuration steps required to get the ODS / API up and running on Linux containers backed by PostgreSQL or an external SQL Server. These scripts can be used for scenarios where a default ODS / API is needed without extensions or customizations.
Please have a look at Ed-Fi Docker Compose Architecture for more information on the components and configuration options provided by the Ed-Fi Docker solution.

The Ed-Fi Alliance does not have expertise in production deployments. Before you deploy into a production environment, please review and adjust the orchestration appropriately for your needs.
If running in a cloud provider, it might be wise to use native services to host the containers. In such situations, using the provided NGiNX container as a reverse proxy might not be necessary or advisable. Similarly, PgBouncer might not be required in some cases. The roles played by NGiNX and PgBouncer are explained in the following section.
Running in Docker Desktop
Docker Containers have the added benefit of running anywhere (e.g., VMs, on-premises, in the cloud), which is a massive advantage for both development and deployment. Leading cloud providers, including Google Cloud, Amazon Web Services (AWS), and Microsoft Azure, have adopted it. For simplicity, the steps below describe how to use Docker Compose to deploy the Ed-Fi ODS/API and related tools on Docker Desktop.
Setup
Step 1. Download the Source Code or Clone the Repo
The Ed-Fi ODS Docker deployment source code is in the Ed-Fi repository hosted by GitHub. A link to the repository is provided in the download panel on the right. You can clone the repository or download the source code as a ZIP file.
Step 2. Setup Runtime Environment
2a. Docker Desktop
To work with the material in this repository, set up a Docker environment by referring to the article Set Up Your Docker Environment.
2b. Microsoft SQL Server

Experimental - this is not a widely tested deployment path.
The default configurations use PostgreSQL. The Ed-Fi Alliance does not distribute pre-configured Microsoft SQL Server (MSSQL) containers. To run with MSSQL instead of PostgreSQL, whether MSSQL is running in Docker, as a managed cloud service, or on a "bare metal" / on-prem virtual machine, you must install the databases yourself. The following notes will help guide a decision and approach,

MSSQL Deployment Options...
Run installations from the official binary distributions and documentation:
PRO: can get the minimal or populated template
CON: can only run on localhost - requires a migration process

Follow steps 1 and 2 in Sandbox Installation Steps: Step 2, only download the EdFi.Suite3.RestApi.Databases package (ignore the package links for WebApi, SwaggerUI, and SandboxAdmin).
Export data-tier applications (bacpac) for each of the created databases.
Import the data-tier applications onto your destination server.
Use experimental EdFi-MSSQL-Install-Scripts for a lighter-weight deployment process. This set of scripts uses duplicate installation files but orchestrates them differently to avoid installing the Admin API and to allow remote deployments.

PRO: should be able to install into any destination MSSQL database you can reach
CON: no sample data (minimal or populated template)
Step 3. Setup Your Environment Variables
Configure your deployments using an environment file. The repository includes a .env.example listing the supported environment variables. Copy .env.example file and name it .env. Update the values as desired.
Note: Compose supports declaring default environment variables in an environment file named .env, placed in the folder where the docker compose command is executed (current working directory). If you desire a different name or location for the .env file, use the --env-file argument as shown below:

docker compose --env-file .env.dev -f (docker-compose-filename) up

Sample .env files for a PostgreSQL and SQL Server are provided below. Please change these values to more secure settings before using them in a deployed environment.

The LOGS_FOLDER is a local directory for permanent storage of log files, and thus, the file path is on the Docker Desktop host, not inside a container. In Windows, use forward slashes rather than backslashes.
Values containing tokens such as ${xyz} are variable insertions you do not need to change.

PostgreSQL

ADMIN_USER=admin@example.com ADMIN_PASSWORD=Admin1 LOGS_FOLDER=c:/tmp/logs MINIMAL_KEY=minimal MINIMAL_SECRET=minimalSecret POPULATED_KEY=populated POPULATED_SECRET=populatedSecret POSTGRES_USER=postgres POSTGRES_PASSWORD=P@ssw0rd PGBOUNCER_LISTEN_PORT=6432 ODS_VIRTUAL_NAME=api SANDBOX_ADMIN_VIRTUAL_NAME=admin TPDM_ENABLED=true ODS_CONNECTION_STRING_ENCRYPTION_KEY=<base64-encoded 256-bit key> TAG=<ODS API image tag. Use 7 or 7-5.0.0 for DS 5.0 and use 7-4.0.0 for DS 4.0> GATEWAY_TAG=v3 ADMINAPI_TAG=<Admin API image tag> ADMINAPI_DB_TAG=<Admin API db image tag> # Specify a health check URL for ODS API, Admin App, Sandbox, and Swagger. # RECOMMENDED: To use the default internal health check endpoints, set: API_HEALTHCHECK_TEST="curl -f http://localhost/health" SANDBOX_HEALTHCHECK_TEST="curl -f http://localhost/health" SWAGGER_HEALTHCHECK_TEST="curl -f http://localhost/health" # The following are only needed for Admin API ADMIN_API_MODE=< Nonempty string value other than 'sandbox'> ADMIN_API_VIRTUAL_NAME=adminapi ADMIN_DB_IMAGE=<admin db image name> AUTHORITY=http://localhost/${ADMIN_API_VIRTUAL_NAME} ISSUER_URL=https://localhost/${ADMIN_API_VIRTUAL_NAME} SIGNING_KEY=<base64-encoded 256-bit key> PAGING_OFFSET=0 PAGING_LIMIT=25 ADMIN_API_HEALTHCHECK_TEST="url -f http://${ADMIN_API_VIRTUAL_NAME}/health" API_INTERNAL_URL=http://${ODS_VIRTUAL_NAME} # Admin API - MultiTenancy SWAGGER_DEFAULT_TENANT=<Tenant-Id>

SQL Server

ADMIN_USER=admin@example.com ADMIN_PASSWORD=Admin#1 LOGS_FOLDER=c:/tmp/logs MINIMAL_KEY=minimal MINIMAL_SECRET=minimalSecret POPULATED_KEY=populated POPULATED_SECRET=populatedSecret SQLSERVER_ODS_DATASOURCE=<DNS or IP Address of the SQL Server Instance, i.e. sql.somedns.org or 10.1.5.9,1433> SQLSERVER_ADMIN_DATASOURCE=<DNS or IP Address of the SQL Server Instance that contains the Admin/Security/Master databases, i.e. sql.somedns.org or 10.1.5.9,1433> SQLSERVER_USER=<SQL Username with access to SQL Server Ed-Fi databases, edfiadmin> SQLSERVER_PASSWORD=<SQL Password for the SQLSERVER_USER with access to SQL Server Ed-Fi databases, password123!> ODS_VIRTUAL_NAME=api SANDBOX_ADMIN_VIRTUAL_NAME=admin TPDM_ENABLED=true ODS_CONNECTION_STRING_ENCRYPTION_KEY=<base64-encoded 256-bit key> TAG=<ODS API image tag. Use 7-mssql or 7-5.0.0-mssql for DS 5.0 and use 7-4.0.0-mssql for DS 4.0> GATEWAY_TAG=v3 ADMINAPI_TAG=<admin api image tag> ADMINAPI_DB_TAG=<admin api db image tag> # Specify a health check URL for ODS API, Admin App, Sandbox, and Swagger. # RECOMMENDED: To use the default internal health check endpoints, set: API_HEALTHCHECK_TEST="curl -f http://localhost/health" SANDBOX_HEALTHCHECK_TEST="curl -f http://localhost/health" SWAGGER_HEALTHCHECK_TEST="curl -f http://localhost/health" # The following are only needed for Admin API ADMIN_API_MODE=< Nonempty string value other than 'sandbox'> ADMIN_API_VIRTUAL_NAME=adminapi ADMIN_DB_IMAGE=<admin db image name> AUTHORITY=http://localhost/${ADMIN_API_VIRTUAL_NAME} ISSUER_URL=https://localhost/${ADMIN_API_VIRTUAL_NAME} SIGNING_KEY=<base64-encoded 256-bit key> PAGING_OFFSET=0 PAGING_LIMIT=25 ADMIN_API_HEALTHCHECK_TEST="url -f http://${ADMIN_API_VIRTUAL_NAME}/health" API_INTERNAL_URL=http://${ODS_VIRTUAL_NAME} # Admin API - MultiTenancy SWAGGER_DEFAULT_TENANT=<Tenant-Id>

When running SQL Server on the same host computer as Docker Desktop - in other words, SQL Server is installed directly on the workstation, not running inside of Docker, set

SQLSERVER_ODS_DATASOURCE=host.docker.internal SQLSERVER_ADMIN_DATASOURCE=host.docker.internal

Step 4. Provide SSL certificate
The deployments require a valid SSL certificate to function. A self-signed certificate can be used for a Non-Production environment. The repository includes generate-cert.sh script that can be used to generate a self-signed certificate and place it in the ssl folder under the root directory to be used by the running Gateway container.
If deploying on a local Windows host, you will need Git Bash or WSL to run generate-cert.sh.

Git Bash
Start a Git Bash Session
Run the following commands:

export MSYS_NO_PATHCONV=1 cd '{your repo root}' ./generate-cert.sh

WSL
Enable WSL and install a Linux from the Microsoft Store.
Start a WSL session.
Run script generate-cert.sh (i.e., ./generate-cert.sh).
MacOSX
Tip from community members: Ensure that you have LibreSSL 3.1.0 or newer and add the new version of openssl to your path. Then, run the generation script from the command line.

brew install openssl export PATH=/usr/local/opt/openssl/bin:$PATH ./generate-cert.sh

Step 5. Run Docker Compose
In this step, you need to choose between deploying a Sandbox, SingleTenant, SingleTenant with ODS Context routes, MultiTenant, or MultiTenant with ODS Context routes environment, as described in Docker Compose Architecture: ConfigurationModes. The source code repository contains a Docker compose orchestration file for each scenario. The sample commands below show the use of the published images from Docker Hub. These commands can be run from any shell terminal that supports Docker commands (e.g., PowerShell in Windows).

Powershell scripts are also provided for each scenario; the up script will set up a new environment, and the -clean script will remove containers and volumes. These scripts should be executed from the repository root and require a .env file at the root level.
Using SQL Server Instead of PostgreSQL

The Docker Compose file does not create SQL Server databases; this means that EdFi_Admin and EdFi_Security databases should already exist within the server defined by SQLSERVER_ADMIN_DATASOURCE in the .env file. Similarly, the ODS databases for desired school years should already exist with the correct naming convention described for each configuration below.
Sandbox Environment

PostgreSQL
Docker command

docker compose -f ./Compose/pgsql/compose-sandbox-env.yml --env-file ./.env up -d

Powershell script

./sandbox-env-up.ps1

SQL Server
Docker command

docker compose -f ./Compose/mssql/compose-sandbox-env.yml --env-file ./.env up -d

Powershell script

./sandbox-env-up.ps1 -Engine SQLServer

SingleTenant Environment

PostgreSQL
Docker command

docker compose -f ./Compose/pgsql/SingleTenant/compose-single-tenant-env.yml --env-file ./.env up -d

Powershell script

./single-tenant-env-up.ps1

SQL Server
Docker command

docker compose -f ./Compose/mssql/SingleTenant/compose-single-tenant-env.yml --env-file ./.env up -d

Powershell script

./single-tenant-env-up.ps1 -Engine SQLServer

SingleTenant with ODS Context Routes Environments

PostgreSQL
Docker command

docker compose -f ./Compose/pgsql/SingleTenant-OdsContext/compose-single-tenant-odscontext-env.yml --env-file ./.env up -d

Powershell script

./single-tenant-odscontexts-up.ps1

Notice that the compose file provided in the repository is an example that defines ODS for the years 2022 and 2023. You can generate the compose file for the desired school years.
Generating Compose File...
To customize for desired school years, you can use the Mustache template provided in the repository.
Navigate to the folder where you cloned/downloaded the Ed-Fi-ODS-Docker repository and go to the sub-folder Compose-Generator/Alpine/templates/pgsql/SingleTenant-OdsContext; in there, you will find example configuration files that instruct the Mustache template on how to generate the SingleTenant with OdsContexts Docker Compose file.
Copy parameters.example.yml and update it as you see fit; the available options are:
odsContextRouteTemplate - Defines the route parameter template used to specify the school year. Configure it following route constraints.
contextKey - Key used to configure dbo.OdsInstanceContext table in EdFi_Admin database. Should match URI in odsContextRouteTemplate
tokens - A collection where you can define the desired years
tokens.token - The year
tokens.pgBouncerPort - (Optional) Exposed pgBouncer port that you can use to connect to the year or district Postgres server
To generate the final Docker Compose file:
Execute:

docker run --rm -v <full path to parameters file>:/parameters -v <full path to desired output folder>:/output edfialliance/ods-compose-generator

Note that the parameters file path must be fully qualified (Docker volumes don't support relative paths).
For example:

docker run --rm -v C:\Ed-Fi-ODS-Docker\Compose-Generator\Alpine\templates\pgsql\parameters-year-specific.yml:/parameters -v C:\Ed-Fi-ODS-Docker\Compose\pgsql\SingleTenant-OdsContext:/output edfialliance/ods-compose-generator

MultiTenant Environments

PostgreSQL
Docker command

docker compose -f ./Compose/pgsql/MultiTenant/compose-multi-tenant-env.yml --env-file ./.env up -d

Powershell script

./multi-tenant-env-up.ps1

Notice that the compose file provided in the repository is an example that defines Tenant1 and Tenant2 with one ODS database each. You can generate the compose file for desired tenants.
Generating Compose File...
You can use the Mustache template provided in the repository to customize for desired tenants.
You can just navigate to the folder where you cloned/downloaded the Ed-Fi-ODS-Docker repository and go to the sub-folder Compose-Generator/Alpine/templates/pgsql/MultiTenant; in the
Copy parameters.example.yml and update it as you see fit; the available options are:
tenants - A collection where you can define the desired tenants
tenants.tenantId - Id of the tenant
tenants.admin-pgBouncerPort - (Optional) Exposed pgBouncer port you can connect to the admin Postgres server
tenants.ods-pgBouncerPort - (Optional) Exposed pgBouncer port you can use to connect to the ODS Postgres server
tenants.isLast - Required to generate correctly a settings json file.
To generate the final Docker Compose file:
Execute:

docker run --rm -v <full path to parameters file>:/parameters -v <full path to desired output folder>:/output edfialliance/ods-compose-generator multiTenant.sh

Note that the parameters file path must be fully qualified (Docker volumes don't support relative paths).
For example:

docker run --rm -v C:\Ed-Fi-ODS-Docker\Compose-Generator\Alpine\templates\pgsql\parameters-year-specific.yml:/parameters -v C:\Ed-Fi-ODS-Docker\Compose\pgsql\SingleTenant-OdsContext:/output edfialliance/ods-compose-generator multiTenant.sh

MultiTenant with ODS Context Routes Environments

PostgreSQL
Docker command

docker compose -f ./Compose/pgsql/MultiTenant-OdsContext/compose-multi-tenant-odscontext-env.yml --env-file ./.env up -d

Powershell script

./multi-tenant-odscontexts-up.ps1

Notice that the compose file provided in the repository is an example that defines Tenant1 and Tenant2 with years 2022 and 2023 each. You can generate the compose file for desired tenants and school years.
Generating Compose File...
To customize for desired tenants and school years, you can use the Mustache template provided in the repository.
Navigate to the folder where you cloned/downloaded the Ed-Fi-ODS-Docker repository and go to the sub-folder Compose-Generator/Alpine/templates/pgsql/SingleTenant-OdsContext; in there, you will find example configuration files that instruct the Mustache template on how to generate the SingleTenant with OdsContexts Docker Compose file.
Copy parameters.example.yml and update it as you see fit; the available options are:
odsContextRouteTemplate - Defines the route parameter template used to specify the school year. Configure it following route constraints.
contextKey - Key used to configure dbo.OdsInstanceContext table in EdFi_Admin database. Should match URI in odsContextRouteTemplate
tenants - A collection where you can define the desired tenants
tenants.tenantId - Id of the tenant
tenants.admin-pgBouncerPort - (Optional) Exposed pgBouncer port you can connect to the admin Postgres server
tenants.isLast - Required to generate correctly a settings json file.
tenants.tokens - A collection where you can define the desired years
tenants.tokens.token - The year
tenants.tokens.pgBouncerPort - (Optional) Exposed pgBouncer port that you can use to connect to the tenant year Postgres server
To generate the final Docker Compose file:
Execute:

docker run --rm -v <full path to parameters file>:/parameters -v <full path to desired output folder>:/output edfialliance/ods-compose-generator multiTenant-OdsContext.sh

Note that the parameters file path must be fully qualified (Docker volumes don't support relative paths).
For example:

docker run --rm -v C:\Ed-Fi-ODS-Docker\Compose-Generator\Alpine\templates\pgsql\parameters-year-specific.yml:/parameters -v C:\Ed-Fi-ODS-Docker\Compose\pgsql\SingleTenant-OdsContext:/output edfialliance/ods-compose-generator multiTenant-OdsContext.sh

Step 6. Verify Your Deployments
You can just open your Docker Desktop instance and view running container instances.
The following image shows Sandbox Postgres deployment:
You can also verify deployed applications by browsing to:
Swagger UI: https://localhost/
ODS / API: https://localhost/api
Sandbox Admin: https://localhost/admin
The exact virtual name for each site after "https://localhost/" depends on your .env file settings.

The following image shows Single Tenant Postgres deployment:
You can also verify deployed applications by browsing to:
ODS / API: https://localhost/api
Admin Api : https://localhost/adminapi

The following image shows Single Tenant with Ods Context Postgres deployment:
You can also verify deployed applications by browsing to:
ODS / API: https://localhost/api
Admin Api : https://localhost/adminapi

The following image shows Multi Tenant Postgres deployment:
You can also verify deployed applications by browsing to:
ODS / API: https://localhost/api

The following image shows Multi Tenant with Ods Context Postgres deployment:
You can also verify deployed applications by browsing to:
ODS / API: https://localhost/api
Step 7. Accessing Application Log Files
The ODS/API, Admin App, and Sandbox Admin write their internal log entries to a mounted folder within their docker containers. The environment file described in Step 3 lets you configure the log location for easy access. For example, you could set the LOGS_FOLDER variable to c:/tmp/logs for Windows hosts or to ~/tmp/logs for Linux and MacOS hosts.
Tear Down
If deployed as a sandbox environment, navigate to the root directory of Docker deployment scripts and run the following command:

PostgreSQL
Docker command

docker compose -f ./compose/pgsql/compose-sandbox-env.yml --env-file ./.env down

Powershell script

./sandbox-env-clean.ps1

SQL Server
Docker command

docker compose -f ./compose/mssql/compose-sandbox-env.yml --env-file ./.env down

Powershell script

./sandbox-env-clean.ps1 -Engine SQLServer

If deployed as a single tenant environment, navigate to the root directory of Docker deployment scripts and run the following script:

PostgreSQL
Docker command

docker compose -f ./compose/pgsql/SingleTenant/compose-single-tenant-env.yml --env-file ./.env down

Powershell script

./single-tenant-env-clean.ps1

SQL Server
Docker command

docker compose -f ./compose/mssql/SingleTenant/compose-single-tenant-env.yml --env-file ./.env down

Powershell script

./single-tenant-env-clean.ps1 -Engine SQLServer

If deployed as a single tenant with an ods context environment, navigate to the root directory of Docker deployment scripts and run the following script:

PostgreSQL
Docker command

docker compose -f ./Compose/pgsql/SingleTenant-OdsContext/compose-single-tenant-odscontext-env.yml --env-file ./.env down

Powershell script

./single-tenant-odscontexts-clean.ps1

If deployed as a multi tenant environment, navigate to the root directory of Docker deployment scripts and run the following script:

PostgreSQL
Docker command

docker compose -f ./Compose/pgsql/MultiTenant/compose-multi-tenant-env.yml --env-file ./.env down

Powershell script

./multi-tenant-env-clean.ps1

If deployed as a multi tenant with an ods context environment, navigate to the root directory of Docker deployment scripts and run the following script:

PostgreSQL
Docker command

docker compose -f ./Compose/pgsql/MultiTenant-OdsContext/compose-multi-tenant-odscontext-env.yml --env-file ./.env down

Powershell script

./multi-tenant-odscontexts-clean.ps1

Additional Docker Tips

Add the argument --remove-orphans if the Docker Compose file has changed and you want to clear out old images that are no longer in use.
If you also wish to remove the local volumes used for persistent storage, add -v to the docker compose ... down command.
You can use the rmi command to clean up local copies of images if you don't want to use them again and to reclaim disk space.