Metadata Extractor
Workflow Diagram
Task
A user can train an AI model to extract metadata from text or documents. If the user is satisfied with the performance of the outputted model, then they can use it.
Intent
The main impact of this AI system is to save time for users who cannot afford to scan through hundreds of PDFs or paragraphs. The AI supports efficiency.
AI Workflow
Input
The user submits samples to the metadata UI, which then extracts the necessary information to train an AI model.
Process
The AI model generates its initial predictions based on the metadata structure given to the system.
Output
The user reviews the newly trained AI model’s predictions and either accepts (and finalizes the model) or rejects the model (and deletes or trains again with updated data).
Human Oversight Level
Human-Approved AI
Institutional Oversight Examples
- •Models should be pulled from a specific commit number (ad-hoc practice)
- •Services that use the AI should have a release version (ad-hoc practice)
- •All benchmarks should be saved in a public repository (organization best practice)
- •Test sets to assess that performance is maintained (organization best practice)
- •Code implemented is open-sourced (organization policy)
- •Services should be covered with unit tests, integration tests, and end-to-end tests (organization policy)
Risk
The trained AI model could make mistakes when the user uses it.