Menai Insight
  • Home
  • Focus Areas
    • Managerial backgrounds
    • Committee activities
    • Performance evaluations
  • Articles
    • Theory
    • Our Approach
  • Help and Support
    • Using Menai Insight
    • Textual Structure Specifications
    • Manage Your Account
    • Contact Support

Perspective

News, updates, and articles to help you understand our approach to capturing and representing meaning

Coffee and Machine Learning: Embedding Validation

3/13/2019

 
Picture
Menai Insight combines qualitative approaches to textual analysis to ensure that nuanced relationships with sentences are captured, and machine learning to ensure that text is processed to a high degree of accuracy across millions of sentences. This article explains key stages in our approach to ensure high-accuracy large scale classifications of the text.
A key challenge with textual-constructs is capturing the desired information from the text. Nuanced, multi-dimensional constructs, common in organizational research, typically do not reduce to individual words, with traditional approaches based on word counts or term clusterings, often relatively coarse proxy of the desired construct. At Menai Insight, we take care at every stage to both ensure that the our textual structures capture the nuance of the sentences, and to ensure that the information is consistently extracted across texts. 

​Our ability to capture the information consistently is underpinned by multiple layers of qualitative and quantitative validations throughout the development and population process are summarized below.
Qualitative Analysis: Development of the Textual Structures
The first stage in classifying the texts involved developing a representation of the material. While each sentence in a particular communication medium, such as managerial backgrounds is unique, there is substantial underlying similarity in the material discussed. Thus, although specific details such as names, dates, and experiences inherently differ between managers, there are underlying similarities; backgrounds typically discuss the discussion positions that a manager has worked in, experiences that they have gained, and qualifications and professional licenses that they have received.   

​To identify the similarities, that provide the basis for our textual structures, we drew on qualitative on themes (e.g., Glaser and Strauss, 1967; Ryan and Bernard, 2003), reading substantial number of each texts in each medium to identify the underlying basis on which the texts were comparable, and the specifics on which they differed.   

Machine Learning: Populating the Textual Structures and Validating the Classifications
After the textual structures are developed, we then developed analytical approaches, drawing on information extraction research, to systematically populate the information from the text to the textual structures. Again, substantial qualitative and quantitative validation was implemented to ensure that the ontologies are populated as expected. For example, We give substantial, continued, attention to ensure that machine-learned classifications conform to plausible sequencing, and identify and correct classifications that that are not populating the textual structures as expected.

Since machine learning inherently relies on well classified data, our care to identify and correct mistakes leads to cumulative benefits: correcting mistakes increases the accuracy of our classifications, which in turn make it easier to identify and correct other errors. Indeed, while working with millions of sentences introduces its own challenges, the diversity of sentences previously classified helps ensure that the accuracy of classifying new material. 

Manual Oversight: Ensuring populated as expected
To ensure that mistakes are identified, our entire process has substantial manual oversight. This process, which spans our entire population process, helps ensure that classifications are continually improving, and not drifting from those expected. Moreover, the direct correspondence to the underlying text further illustrate the validity of the textual structures, while allowing more nuanced theorizing than feasible with distant proxies. ​

Transparency: Avoiding black-box analysis
Our overall process, and usage our the textual structures is designed to be transparent - with clear connections between the underlying text and the textual structures helping to illustrate the validity of extracted constructs. Being able to identify and verify at the specific sentence-level that desired terms are being extracted, gives confidence in the validity of aggregated constructs. Moreover, we extend this level of transparency, with clear documentation specifying the textual-structure and detailing the classification  process. 
Related Articles
Exploring the Research Possibilities Enabled with Rich-Textual Structures
​
This article explains some of the opportunities arising from being able to systemically capture within-sentence relationships, by transforming raw text into consistent textual structures.
​Read More
Our Approach to Text Analysis: Capturing the Relationships

​​This article explains the key basis of Menai Insight's tools in capturing a layer or meaning beyond the individual words on the page.
Read More

Gareth Keeves received a PhD in strategy from the Ross School of Business, University of Michigan. His research includes impression formation and management in the context of corporate leadership. As the CEO of Menai Insight he is developing approaches for capturing and representing meaning for corporate governance communications. 
Picture

References
Glaser, B., and A. Strauss
1967    The Discovery of Grounded Theory: Strategies for Qualitative Research. New Brunswick: Aldine Transaction.

Ryan, G.W., and H.R. Bernard
2003    Techniques to identify themes. Field Methods. 15. 85-109.

Comments are closed.

    Categories

    All
    Impression Management
    Our Approach
    Theory
    Top Management
    Upper Echelon

Our Focus

Managerial Backgrounds
Committee Oversight
Performance Evaluations

APPROACH

Overview
​Theoretical Possibilities 
Ensuring Accuracy
​
Validation

About

​Articles
​Our Story
​Developments
Contact Us
Help and Support

Picture
© 2019 Menai Insight, LLC      Terms, privacy, and other policies

  • Home
  • Focus Areas
    • Managerial backgrounds
    • Committee activities
    • Performance evaluations
  • Articles
    • Theory
    • Our Approach
  • Help and Support
    • Using Menai Insight
    • Textual Structure Specifications
    • Manage Your Account
    • Contact Support