📢 Tech Docs is going to the cloud! Stay tuned for more as we migrate to Atlassian’s Cloud.
This is the list of proposals, problems or opportunities submitted at the 2022 Technical Congress.Â
Ideation
Category/roll-up Title Details Support for event-based notifications from API Too many vendors still bulk processing instead of using an API for real-time integration Outside of our SIS vendors our vendors do a schedule bulk load to get all the changes, Instead it would be better if it was more of a change happens it get pushed to or pulled from vendors. Alert to notify of changes in data Something like a webhook where the system can just listen for changes for data out. Ability for the Ed-Fi ecosystem to publish data notifications to vendor systems As a part of the assessment roster initiate. one of the things that will challenged us to scale is the current solution requires the assessment vendor to poll for roster updates for each customer. Would it be possible for the Ed-Fi ecosystem to publish data updates to vendor systems, assuming the vendor provides Ed-Fi community base compliant exchanges. This could be a pub sub solution. Adopting newer design patterns 1) Adopting micro services architecture (breaking down the Ed-Fi API in smaller devices, one per domain). 2) implementing event-driven pattern to enable to “push” data use case for analytics and trigger external workflows. Improve support for graduation requirements capture and tracking Graduation Requirements Storage We do on-track tracking and we need a way to more easily store and organize graduation requirements. If it is part of ed-fi it is not clear. Transcript extensions for graduation tracking We do a lot of graduation and GPA tracking and to do so we had to create extra transcript columns. We'd love to hear how other folks have dealt with this challenge given that HS on-track tracking is so important. Graduation requirement tracking The ability to monitor student progress towards graduation, both with courses and standardized testing requirements Explore uses of blockchain in ecosystem Blockchain functionality Be able to provide blockchain functionality so that every student knows how it's data is used and when partner is using data back to source. Provide ODS to ODS replication via API native support for bi-directional ODS to ODS communication API publisher is good but is a workaround for true native comms features Develop solutions to validate ODS data External is on all tables Allow easy reconciliation to source system, yet support multiple Ed-Fi LEA vs State use cases Allow for API profiles to be used without recompile and redeployment Allow for profiles to be used without redeploying the application. Profiles seem to be very powerful, but operationally are difficult to use. Because profiles need to be compiled the whole application needs to be redeployed (when using docker). Improve L1 errors from the API Continue to better support vendors pushing more data into Ed-Fi Better errors and logging come to mind, i’m sure others could generate more ideas Better L1 messages- you may not be able to fix everything at once, but get started Tackle single slices at a time- missing descriptors, FK references, authorization errors Develop a multi-year ODS data model and drive vendor ecosystem to support multiple year submissions Multiyear ODS and changes in the SIS for the prior year due to state submissions We currently sync our ODS in support of our Charter schools and a topic came up that we need to still sync changes that are updated in the prior year's calendar. The issue is that our current SIS is not compatible with the process outlined in the Multi Year ODS document on techdocs. We also use the API Publisher to process a sync between our ODS and our Charter Schools so this would present issues in pulling changes from prior year calendars and how to connect them if there is a referential integrity issue with the data. Fully support multi year ODS The single year model creates too much overhead to maintain. I would have support 5 to 7 APIs. Of possibly 3 to 4 version of Ed-Fi. This means database wise I now have 5 to 7 times 5 database to manage. I would rather just manage a 100 or so more tables in one database. Add support for marking data as invalid and filtering based on that status Validation Framework for Tagging and Filtering Database Records Marked Invalid Use of a validation engine is great for creating reports of validation errors, there is no way to tag ODS records as "invalid" based on these errors. What I would like to see is (1) an Ed-Fi API for tagging records as "invalid", (2) an Ed-Fi API for querying invalid records for reporting, (3) an Ed-Fi API for resetting/clearing the "invalid" tag, and (3) an Ed-FI ODS copy/ETL process which would populate a second, duplicate Ed-Fi ODS structure with the contents of the first ODS but exclude the records marked invalid. This copy/ETL process would need to be smart enough to also skip records which, even if not marked invalid themselves, cannot be included due to a dependency on an invalid record. The lack of a built-in data validation / filtering framework is probably the #1 complaint MDE has with Ed-Fi, and one that 4 years later we still don't have a satisfactory workaround for. Define and implement canonical processes for assessment integration Ramping up a new Assessment (/other) Vendor who has not dealt with Ed-Fi before We are beginning work with our second Assessment vendor, and we talked with several others. In each case we replacing a roster/student-registration file transfer and receiving score data. I wonder if guidelines for spinning up an assessment vendor are worthwhile- perhaps even some documentation (assuming it doesn't exist and I don't know about it) around the recommended practices.
For example, should you download all the rostered students and all the student information you have access to and match them up or download all the rostered students go through endpoints one at a time? What is the process of establishing the mappings to the traditional file uploads, and how do we avoid a state-specific spec while allowing them access to the data we have? I wonder if a Roster composite would be worthwhile? Do we recommend change-data-query around rostering (Pearson is doing that during the rostering period)?
I have done some work with preliminary information for a vendor and I think that proves useful (here is the Alliance URL and where to register, here is our sandbox, to access UAT or Production you must request allow-list to our APIs, etc..). The vendors have also worked on what milestones are needed, and some of this information might be useful to new vendors.Create an X-prize-like competition for Ed-Fi API X prize like competition for Ed-Fi API based applications I don't think many developers know that Ed-Fi is a thing - let's encourage them to build killer apps. Create a clear Ed-Fi to CEDS data pipeline API based Ed-Fi to CEDS In theory I should be able to construct the CEDS ODS structure from the Ed-Fi one via API Create install and deployment processes that are less developer-centric Less Technical Installation We are strong at SQL and Power BI, but the instructions on how to install the full ODS and API had too steep of a learning curve. Instead of doing the full installation we found a script on your GitHub to just create all the tables in our SQL database with primary and foreign key constraints. We would like to look at and possible use the middle tier views but haven't figured out a less technical way to install them. It would be great to at least have a SQL script to create all the views. Even better would be very clearly differentiated levels for ed-fi implementation where folks who know SQL can get involved and start using ed-fi more quickly. Another idea would be to have more clear trainings on installing an ods/api solution, which we would be open to. Add support or define processes for snapshots Grade Snapshots In our graduation tracking we want to see students' grades at specific times in the past. This allows us to look at grade trends throughout the year, not just at the quarter, semester, and yearly posting times. To do this we take weekly snapshots of schools' gradebooks. We haven't seen a clear way how to handle this with ed-fi. Explore priorities for interop with IMS standards Explore the continued opportunities to align/connect IMS Global Standards with Ed-Fi. Providing connections to Ed-Fi that are also built on open standards will advance support of interoperability for many K12 school districts. Perhaps explore additional connections beyond the current One Roster work that will further acceleration to use of both the Ed-Fi ODS and the IMS Global/1 EdTech interoperability standards. Explore use cases for publication of state-calculated metrics back to districts Publishing or moving data from One to Many instance. We have a deployment where we have districts sitting in a district specific env, and the state in a shared instance. There is a use case for the State to publish down to the districts. We need help thinking about this and if it's possible and performant. Direct vendor-to-vendor student data transfer via API, e.g. for student transfers scenarios An ODS to ODS single student data transfer API This API could then be implemented by the SIS vendors to provide the user interface for the school administrators. Unlike the API Publisher, this data transfer should be driven by the schools themselves via their student information systems. The API should provide endpoints for request, approve, reject student transfers as well as support for specific entities that need transfer. Student Transfer API (part 2) More details about this:
https://docs.google.com/document/d/16bHqIxAlpbbnQhEoPJRZ3whnih3NJQIfYTcO6-gqb90/edit?usp=sharingProvide for student identification code translation Please bring the Student Identification Code translation back. We used it in Suite 2 and would to use it in suite 3 and any future version of Ed-Fi. Helps with Data Import and also enables districts to roster systems with IDs that are different from the ID that is the StudentUniqueId. Improve Ed-Fi ODS logging Would like API logs to specify which client caused the error. As a provider of multi district systems with vey active districts, it’s really hard to get usable data from the api logs since they can’t be filtered by the client id. Cloud watch can handle filtering and searching, but there needs to be something to search for to relate to which district caused the error. Better API logs It would be great to have better logs so that we can see what payload was submitted with a 400 bad data. Also if some required fields were not submitted it would be good to have a good error description. Better ODS error logging Contextual information
- which credentials were the source of the error?
- which ed-org(s), user, etc…?Continue to better support vendors pushing more data into Ed-Fi Better errors and logging come to mind, i’m sure others could generate more ideas Improve release documentation Don't do version GA release without full documentation For example, with each upgrade we have done there have been changes in Team City/Octopus configurations that aren't document. Make sure that all changes are documented and how to upgrade is document before GA release. Invest in multi-year data models and architectures in critical domains Capture Multi-year ODS data How can we address “improve support for graduation requirements…” without multi-year data?
State submissions are completed the following year December and require adjustments to comply with standards. These are lost without a multi-year ODS or ability to connect to multi- ODSs in a single SIS instance.Allows the /snapshots endpoint to refer to different database hosts. Allows the /snapshots endpoint to refer to different database hosts. The snapshots functionality was built for MSSQL. There is not an analogous feature in Postgres. To work around this and still provide isolation for the API Publisher we are using AWS Aurora to clone the rds instance. Because /snapshots can only refer to databases on the same database server, we have to run a separate instance of the web api to access this clone. Explore and implement alternative API security models Add record level security IDOE is adding multiple data sources for a given resource. This is causing conflicts in ownership and general concern of data management. Research and implement API performance improvements Make the default default page size not the max page size. And make the max page size much larger. (10,000) When reading a full ODS worth of data, one of the most effective things to do to make it faster is increase the page size. When we aren’t in control of the edfi system, using a larger page size isn’t always possible. Explore microservices architecture Adopting newer design patterns 1) Adopting micro services architecture (breaking down the Ed-Fi API in smaller devices, one per domain). 2) implementing event-driven pattern to enable to “push” data use case for analytics and trigger external workflows. Simplify Data Import interface to make more user-friendly Data Import more less tech user friendly It would be great that analyst with lower tech skills could easily create and manage templates. Decouple authentication and authorization from API I would like to see the authentication and authorization mechanism become less tightly coupled with the API so that it can be used by other applications (like a passthrough API) This came up in the context of talking about the data validation API. Make staff assignment and employment be 1:many Staff assignment and employment shouldn't be 1:1 We have several staff that have multiple assignments. When looking at the results set from employment it is hard to determine which employment record goes with which assignment record. In our case it is only one employment with multiple assignments. The issue is most noticeable with special education and nursing staff because they are entity within the district where one staff person can cover multiple schools. Support large text in the API and ODS Large text/blob options Sometimes we want to use the API to transfer data and that can require large text blocks of text or documents.
Existing Roadmap Items or Tickets Raised
| Action | Category | Title | Details |
|---|---|---|---|
| Coming in August v4.0 DS release - see https://tracker.ed-fi.org/browse/DATASTD-1723 and https://tracker.ed-fi.org/browse/DATASTD-1661 | Model and require vendors to provide in progress / current course grades | Include in-progress course grades. | We would like to suggest expanding the Ed-Fi SIS certification to include in-progress course grades. |
| We need access to in-progress student grades | At the current time Ed-Fi does not support in-progress grades, only posted grades. Therefore any entity using the Ed-Fi framework is not able to present a current snapshot of a students progress which reduces the value of the Ed-Fi framework when talking about pro-active student interventions and progress monitoring. | ||
| Already exists, but could be better documented - https://tracker.ed-fi.org/browse/DATASTD-1733 | Improve and document standards-based grading | Standards based grading | We have a lot of LEAs who uses non traditional grades in our SIS and growing to align to student services |
| https://tracker.ed-fi.org/browse/BIA-1064 | New AMT Views | Create a middle tier view that we call "Demo Long" which has each student repeated multiple times for each group they are in. It allows easier creating of reports using demographics to put equity front and center.   | See the equitable attendance page or the GroupType dropdown on the grades page on this dash for an example of how demo long enables strong equity views: https://app.powerbi.com/view?r=eyJrIjoiYTY4NzVlZDAtYWIzNS00ZTVhLTk3OTEtM2Y4ZDZkOGE0MmEyIiwidCI6ImI4NmUwZjQwLTFiNDQtNDk5MC1iOTA2LTQwYjFhMWNlMTExYSIsImMiOjF9&pageName=ReportSectionb4f30bd5e808b1956c19 |