Date: Thu, 28 Mar 2024 06:21:53 -0500 (CDT) Message-ID: <1147580410.29660.1711624913829@PUBEDFIPRDWEB5.public.local> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_29659_10761589.1711624913828" ------=_Part_29659_10761589.1711624913828 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
These notes report on topics raised during the 2021 Summit's Tech Town H= all, and the subsequent "In the Weeds" WebEx call that was held for virtual= attendees. These notes attempt to summarize and synthesize the conversatio= ns, rather than represent specific voices. Please see the "Tech Town Hall 2= 021" tab of Ed-Fi In-The-Weeds-Meetings in Google Docs for the r= ealtime meeting notes.
When Ed-Fi Tracker ticket numbers and links are provided, you can commen= t on the ticket or upvote it to register interest or support for developmen= t of that new feature.
With education agencies learning how to mine value from their data= , and increasingly seeing that they can actually get access to their data, = come greater demands for large data sets. Two areas of particular concern a= re positive attendance and gradebooks; both of these have significant daily= volumes to process. Synchronizing these data from SIS to ODS/API, and from= ODS/API to ODS/API in the case of publishing to a state agency, can requir= e a significant amount of time and thus be prone to timeouts, network error= s, or simply not finishing in the time available. Another area of concern i= s startup synchronization: there may be a large amount of older data to wri= te when a client first starts publishing to an ODS/API.
Proposed solutions include:
The second approach - writing an array of data - has received the = most interest over the years. The simplest solution is to enable POSTing an= array of objects for an existing resource. For example, a POST request to = the /ed-fi/sections endpoint could contain an array of Sections instead of = just a single student. But what happens when one or more items has a proble= m - for example, one of the Section items references a Course Offering that= does not exist. The response could include a JSON response object with det= ails about the failed records. However, if the batch size is large, the API= client might not want to wait for the response. Supplying a webhook = URL, allowing the ODS/API to call back to the client system, is the typical= pattern for such asynchronous communication.
A more sophisticated approach was proposed in the Bulk Data Exchange Over REST API= special interest group of 2018. In this model, an API client uses a Bulk I= mport API instead of sending a collection of resources to a regular endpoin= t. This proposal is fully asynchronous and it gives the API client extensib= ility points for monitoring and even cancelling a bulk request.
Related tickets:
Key | Summary | T | Created | Status |
---|
The API in the ODS/API Platform is a RESTful one that responds to = requests as they are received, that is, it responds synchronously. For exam= ple, a client application submits a POST request to create a new resource i= n the API, and the client waits for the API to respond. The API immediately= validates the request and tries to save it in the ODS database, then respo= nds to the client. The client probably has a timeout setting, so that it gi= ves up on waiting for the API if there is an unusual delay.
While this kind of processing is easy to think about, it is not th=
e most performant. And it suffers from another significant drawback, this t=
ime in the opposite direction. When the API client wants to
Asynchronous processing has the potential for improving both of th= ese scenarios. Instead of thinking about resource requests, we can think ab= out events. Event architectures are asyn= chronous and operate on a =E2=80=9Cfire-and-forget=E2=80=9D model: submit a= request and do not expect a detailed response right away. When operating o= ver a Web API, this might mean that the request receives a typical HTTP res= ponse indicating that the request to create an event was accepted - but it responds to the client before it has act= ually processed the event. The processin= g happens separately.
How does the client learn about the result of the event? Two promi= nent options:
The first approach lets the API system push a response back to the client system, whereas the second one i= s an efficient approach for letting the client poll= for the new events on its own timetable. The second approach al= so came up at the Ed-Fi Summit in the data lake session.
Concrete use case examples:
It appears that no one has submitted a ticket requesting these types of = features. You can be the first! EDFI project in Tracker=
What can we do to improve synchronization of deleted data between = systems? Three different questions arose:
Key | Summary | T | Created | Status |
---|
The ODS/API Platform contains in-memory caching of descriptors, ed= ucation organizations, education organizations, and security claims. This i= mproves application performance by reducing the number of database calls. H= owever, it also increases the memory footprint. The memory usage is particu= larly noticeable when using the Docker solution, since Docker containers ar= e expected to run with a small footprint. Some have even started turning of= f the caching entirely to avoid this memory hit.
Furthermore, when running in load balanced mode with multiple inst= ances, whether using Docker or not, the separate caches are not in sync wit= h each other - a lost opportunity. The standard alternative is to implement= a distributed caching mechanism, using an external provider such as Redis = or Memcached.
Should the ODS/API be modified to have a native hook for connectin= g an external cache provider? And should the Docker solution provide an ext= ernal cache option out of the box?
Key | Summary | T | Created | Status |
---|
For scalability and performance, interest has been building in NoS= QL alternatives to SQL Server or PostgreSQL, while recognizing that there i= s too much entrenched investment of code and training on the SQL platforms = to abandon them. Project M= eadowlark has begun experimenting with key-value / document store; thou= gh it is far from being a production-ready system, it has taught the Allian= ce much with respect to thinking about referential integrity through applic= ation code instead of relying on the database. Project Meadowlark demonstra= tes how JSON objects received through the API can be stored "as-is" instead= of (partially) normalizing as done when storing in the ODS database. This = is the logical way to use a NoSQL database, and changing the existing ODS/A= PI to support this while continuing to normalize into SQL would be a large = undertaking.
Within the current tech stack, NHibernate (the Object Relational M= apper used by the ODS/API application) came up as a concern. It is old, dif= ficult to tune and understand, and would ideally be replaced with a more pe= rformant alternative such as Dapper.
Either of these changes is significant enough that they may not be= feasible with backward-compatible code in the ODS/API for Suite 3, version= 5. In other words, they may require such a substantial rewrite that the pl= atform would need to bump to version 6 or even to "Suite 4" status.<= /p>
Key | Summary | T | Created | Status |
---|
MappingEDU has proven immensely valuable... to a very small set of users= . Some of those users, or their representatives, spoke up to ask about upda= tes to the application: for example, improving the matchmaker functionality= , or auto-generating maps for Data Import.
The Alliance is currently evaluating options for the future of MappingED= U. Due to the high cost of maintenance relative to the usage, this includes= the possibility of retiring the service and/or opening the source code for= others.
Key | Summary | T | Created | Status |
---|
Wisconsin raised a couple of topics on minimizing network traffic. These= were acknowledged but little discussed real time. Stephen Fuqua, Ed-Fi's s= oftware architect, shared the following ideas with them after the fact, bas= ed on the knowledge that they are running their operations on a cloud provi= der:
There are no tickets specific to these topics.
Several other important questions came up, some of which are perennial:<= /p>
Key | Summary | T | Created | Status |
---|