Checklist to evaluate machine learning systems - Open Source Initiative
This checklist is based on the paper
The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency and Usability in AI
DOI
published Mar 21, 2024. The Model Openness Framework (MOF) is implemented on the
Model Openness Tool website
(MOT).
Scope of this document
This Checklist was developed by volunteers during the co-design process to help reviewers of AI systems to identify and rank the components required to exercise the basic freedoms of Open Source AI. It’s been further refined via public comments, on the forum and on the public draft on hackmd.
This document should be seen as part of the definitional process, a learning tool:
The Checklist is not an operating manual to evaluate Open Source AI
Relationship to the Model Openness Framework
The MOF classifies systems in three degrees of availability of components, from some (Class III, Open Model) to all (Class I, Open Science). When using the MOF, one can think of the requirements of the “preferred form to make modifications to a ML system” as a bar overlayed on the MOF range of classes.
Known issues and limitations
Tied to generative AI
: Being based on the MOF, this Checklist appears to be tightly coupled to generative AI. The list of components is not generalized enough to be applied to all machine learning. More research is necessary to apply the principles of the Open Source AI Definition to other kinds of AI and different machine learning systems.
Subject to interpretation
: When the Datasets component is made available, the Data requirements should be satisfied. When AI systems don’t make the Datasets component available, one needs to extrapolate from the alternative Data components if they provide the requirements listed in the Open Source AI Definition. This is another area that requires further research as the practice of Open Source AI develops.
For more details, see also the
Open Source AI FAQ
Table of default required components
Required components
Legal frameworks
Data
See Known Issues. The requirements in the
Open Source AI Definition
must be satisfied.
– Datasets
Available under OSI-approved terms
– Research paper
Available under OSI-approved terms
– Technical report
Available under OSI-approved terms
– Data card
Available under OSI-approved terms
Code
All of these components are required
– Data pre-processing
Available under OSI-approved license
– Training, validation and testing
Available under OSI-approved license
– Inference
Available under OSI-approved license
– Supporting libraries and tools
Available under OSI-approved license
Model
All of these components are required
– Model architecture
Available under OSI-approved license
– Model parameters
Available under OSI-approved terms
Table of optional components
The other components listed in the Model Openness Framework are optional.
Optional components
Data
– Evaluation data
– Evaluation results
Code
– Code used to perform inference for benchmark tests
– Evaluation code
Model
– Model card
– Sample model outputs
– Model metadata
Available under OSI-approved terms
means that the OSI will review licenses and agreements to ensure that all materials are available under terms that conform with the Open Source Definition.
↩︎
Manage Cookie Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
Statistics
The technical storage or access that is used exclusively for statistical purposes.
The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
Manage options
Manage services
Manage {vendor_count} vendors
Read more about these purposes
View preferences
{title}
{title}
{title}