Change Query Enhancements

Owned by Vinaya Mayya
Last updated: May 17, 2022

The goal of the changed record queries (or "change queries") feature is to surface information about the changes made to database tables to clients of Ed-Fi APIs. These changes include inserts, updates deletes and key changes. The change queries feature allows clients to request only data that has changed after a specified point in time thus avoiding time consuming full dataset overwrite. 

Previous implementations of change queries had the following limitations around deletes:

Lack of natural keys provided for tracked deletes: The Ed-Fi Data Standard uses natural keys for resource item identity, however in previous releases the tracked deletes API endpoints only exposed the host API's resource ids, which are not portable between ODS instances and thus could not be used to delete the corresponding item from a downstream ODS API. Change queries consumers would then need to record a mapping for each resource identifier in the source ODS that is known to the target system. This places an unnecessary burden on consumers, particularly if the data is being moved between ODS instances.
Lack of authorization-based filtering of tracked deletes: Since only the resource Id is provided for deletes in previous releases, no authorization filtering approach was designed and implemented. This lack of filtering means that every change queries consumer will be processing all of the tracked deletes for each resource, even if they didn't previously have access to the now-deleted resource item. Handling (ignoring) unknown resource Ids is a responsibility of the consumer.
Lack of support for tracking deletes of derived resources: Previous releases of the API currently has a bug that results in the failure to track deletes of derived resources (i.e. education organizations, student program associations, and descriptors).
Lack of support for tracking key changes: Several of the API resources allow API clients to update the key values using the PUT method (rather than deleting and recreating the resource and all its dependents). This means that when a natural key is changed in the API, Change queries consumers will only see that a new resource has been created and have no indication that the "old" version and its dependencies should be removed. The result will inevitably be that the downstream system will retain a stale copy of the old resource item and all its dependents.

A dev release has been published to fix the above issues and made available at:

The change queries feature is optional. The feature is turned on by default, and can be disabled by platform hosts via configuration. See the following documentation for details: