Multiple stages of validation help ensure our accuracy
Overview
This page explains how validation is embedded into each of the stages used to develop and populate the ontologies.
We have embedded multiple stages of validation through our development process:
This page explains how validation is embedded into each of the stages used to develop and populate the ontologies.
We have embedded multiple stages of validation through our development process:
- Qualitative development of the ontologies: Ensuring that the ontologies reflect the underlying material
- Validation of the classification: Ensuring that the material populates the developed ontologies correctly
- Manual oversight and face-validity: On-going checks to ensure that the overall population process is occurring as expected
Qualitative development of the ontologies
The first stage in classifying the texts involved developing a representation of the material. While each sentence in a particular communication medium (e.g., managerial backgrounds) is unique, there is substantial underlying similarity in the material discussed. For example, managerial backgrounds, typically discuss positions that a manager has worked in, experiences that they have gained, and qualifications and professional licenses that they have received.
Each of the ontologies was developed with substantial qualitative consideration of the underlying material, drawing from research on themes (e.g., Glaser and Strauss, 1967; Ryan and Bernard, 2003) to help ensure that the ontologies reflect the underlying text. After first identifying the primary dimensions of the text, each of these sentences were then dissected again into the components.
Each of the ontologies was developed with substantial qualitative consideration of the underlying material, drawing from research on themes (e.g., Glaser and Strauss, 1967; Ryan and Bernard, 2003) to help ensure that the ontologies reflect the underlying text. After first identifying the primary dimensions of the text, each of these sentences were then dissected again into the components.
Learn more about our qualitative development process
Validation of the classifications
The second stage in classifying the text is to ensure that the material is correctly populated into the ontologies. While this involved a combination of manual classification and machine-learned classification. Three primary approaches are used to ensure that the material is classified appropriately:
Validation by context
Checks that the context in which a term occurs in a sentence is appropriate; for example, while the concept sequencing PERSON_NAME IS MANAGEMENT_TITLE AT COMPANY_NAME is common, and appropriate, a concept sequencing such as PERSON_NAME RECEIVED COMPANY_NAME FROM UNIVERSITY_NAME is not common, and not likely to be correct (i.e., likely indicating that a degree acronym has incorrectly been classified as a company name). This validation includes three components:
Validation by dissection
By dissecting concepts to underlying properties, and manually verifying the much reduced number of terms in the sub-concepts, and the sequencing of the sub-concepts, it is possible to validate a much larger number of terms. For example, the validity of concepts comprised of separate parts (e.g., MANAGEMENT_TITLE) can be assessed, despite there being tens of thousands of unique titles at the overall level.
Validation of terms through external-data-checks
By connecting terms to external databases, it is possible to verify concepts underpinned by a large number of labels, such as location information, that are unfeasible to manually verify, and lack the repetition in underlying words to allow dissection.
- Validation by context: Ensuring that classifications are
Validation by context
Checks that the context in which a term occurs in a sentence is appropriate; for example, while the concept sequencing PERSON_NAME IS MANAGEMENT_TITLE AT COMPANY_NAME is common, and appropriate, a concept sequencing such as PERSON_NAME RECEIVED COMPANY_NAME FROM UNIVERSITY_NAME is not common, and not likely to be correct (i.e., likely indicating that a degree acronym has incorrectly been classified as a company name). This validation includes three components:
- Manual checks to identify unlikely concept sequencing.
- Machine-learned identification, where classifications through machine-learning are inconsistent with the classified concept.
- Identification as concept sequencing that does not conform to that expected in the ontology
Validation by dissection
By dissecting concepts to underlying properties, and manually verifying the much reduced number of terms in the sub-concepts, and the sequencing of the sub-concepts, it is possible to validate a much larger number of terms. For example, the validity of concepts comprised of separate parts (e.g., MANAGEMENT_TITLE) can be assessed, despite there being tens of thousands of unique titles at the overall level.
Validation of terms through external-data-checks
By connecting terms to external databases, it is possible to verify concepts underpinned by a large number of labels, such as location information, that are unfeasible to manually verify, and lack the repetition in underlying words to allow dissection.
Manual oversight and face-validity
Checks throughout the process to ensure that the ontologies are being populated in-line with those developed.
Beyond documenting the ontologies, the examples, and summary statistics included in Appendix D illustrate that the ontologies, properties, and classifications have a high correspondence to what would be expected.
Beyond documenting the ontologies, the examples, and summary statistics included in Appendix D illustrate that the ontologies, properties, and classifications have a high correspondence to what would be expected.
