Scholars conduct systematic literature reviews to summarize knowledge and identify gaps in understanding. Machine learning can assist researchers in carrying out these studies. This paper introduces a machine learning toolkit that employs Network Analysis and Natural Language Processing methods to extract textual features and categorize academic papers. The toolkit comprises two algorithms that enable

Content analysis has enabled organizational scholars to study constructs and relationships that were previously unattainable at scale. One particular area of focus has been on sentiment analysis, which scholars have implemented to examine myriad relationships pertinent to organizational research. This article addresses certain limitations in sentiment analysis. More specifically, we bring attention

Online data are constantly growing, providing a wide range of opportunities to explore social phenomena. Large Language Models (LLMs) capture the inherent structure, contextual meaning, and nuance of human language and are the base for state-of-the-art Natural Language Processing (NLP) algorithms. In this article, we describe a method to assist qualitative researchers in the theorization process by

The advent of machine learning and artificial intelligence has profoundly transformed organizational research, especially with the growing application of natural language processing (NLP). Despite these advances, managing long-sequence text input data remains a persistent and significant challenge in NLP analysis within organizational studies. This study introduces five different approaches for handling

Over the last two decades, much of management research has converged on the belief that one of its major aims is to identify the causal mechanisms that produce the phenomena that researchers seek to explain. In this paper, we review and synthesize the literature that has amassed around causal mechanisms. We do so by detailing the different methodological perspectives that are featured in management

In contemporary organizational research, when dealing with large heterogeneous datasets and complex relationships, statistical modeling focused on developing substantive explanations typically results in low predictive accuracy. In contrast, machine learning (ML) exhibits remarkable strength for prediction, but suffers from an unexplainable analytical process and output—thus ML is often known as a

Nearly 2 decades ago, Cycyota and Harrison (2006) documented a concerning trend of declining executive survey response rates and projected a continued decrease in the future. Their seminal work has significantly influenced the methodologies of upper echelons survey research. Our study examines the manner in which Cycyota and Harrison’s paper has impacted the existing upper echelons literature and replicates

While other applied sciences systematically distinguish between manipulation designs, organizational research does not. Herein, we disentangle distinct applications that differ in how the manipulation is deployed, analyzed, and interpreted in support of hypotheses. First, we define two archetypes: treatments, experimental designs that expose participants to different levels/types of a manipulation

In graded paired comparisons (GPCs), two items are compared using a multipoint rating scale. GPCs are expected to reduce faking compared with Likert-type scales and to produce more reliable, less ipsative trait scores than traditional binary forced-choice formats. To investigate the statistical properties of GPCs, we simulated 960 conditions in which we varied six independent factors and additionally

Websites represent a crucial avenue for organizations to reach customers, attract talent, and disseminate information to stakeholders. Despite their importance, strikingly little work in the domain of organization and management research has tapped into this source of longitudinal big data. In this paper, we highlight the unique nature and profound potential of longitudinal website data and present

When modeling responses to items measuring non-cognitive constructs that require introspection (e.g., personality, attitude), most studies have assumed that respondents follow the same item response process—either a dominance or an unfolding one. Nevertheless, the results are not equivocal, as some preliminary evidence suggests that some people use an unfolding response process, whereas others use

When assessing text, supervised natural language processing (NLP) models have traditionally been used to measure targeted constructs in the organizational sciences. However, these models require significant resources to develop. Emerging “off-the-shelf” large language models (LLM) offer a way to evaluate organizational constructs without building customized models. However, it is unclear whether off-the-shelf

While machine learning (ML) can validly score psychological constructs from behavior, several conditions often change across studies, making it difficult to understand why the psychometric properties of ML models differ across studies. We address this gap in the context of automatically scored interviews. Across multiple datasets, for interview- or question-level scoring of self-reported, tested, and

Computational modeling holds significant promise as a tool for improving how theory is developed, expressed, and used to inform empirical research and evaluation efforts. However, the knowledge and skillsets needed to build computational models are rarely developed in the training received by social and organizational scientists. The purpose of this manuscript is to provide an accessible introduction

Understanding the experiences of vulnerable workers is an important scientific pursuit. For example, research interest is often in quantifying the impacts of adverse exposures such as discrimination, exclusion, harassment, or job insecurity, among others. However, routine approaches have only focused on the average treatment effect, which encapsulates the impact of an exposure (e.g., discrimination)

This article proposes a framework for reflexive choice in qualitative research, centering on social interaction. Interaction, fundamental to social and organizational life, has been studied extensively. Yet, researchers can get lost in the plethora of methodological tools, hampering reflexive choice. Our proposed framework consists of four dimensions of interaction (content, communication patterns

Due to recent technological developments, vignette studies that have traditionally been done in text or video formats can now be done in immersive formats using virtual reality—but are such virtual reality video vignettes superior to traditional vignettes? To address this question, we examine participants’ experiences within a fictitious organization by comparing their responses to a relevant and particularly

Qualitative meta-studies (QMS) have emerged as a promising methodology for synthesizing qualitative research within organization and management studies. However, despite considerable progress, increasingly fragmented applications of QMS impede the advancement of the methodology. To address this issue, we review and analyze the expanding body of QMS in organization and management studies. We propose

In recent decades, multidimensional forced-choice (MFC) tests have gained widespread popularity in organizational settings due to their effectiveness in reducing response biases. Detecting differential item functioning (DIF) is crucial in developing MFC tests, as it relates to test fairness and validity. However, existing methods appear insufficient for detecting DIF induced by the interaction between

Forced-choice (FC) measures are becoming increasingly popular as an alternative to single-statement (SS) measures. However, to ensure the practical usefulness of an FC measure, it is crucial to address the tension between psychometric properties and faking resistance by balancing mixed keying and social desirability matching. It is currently unknown from an empirical perspective whether the two design

Employers commonly use cognitive ability tests in the personnel selection process. Although ability tests are excellent predictors of job performance, their validity may be compromised when test takers engage in careless responding. It is thus important for researchers to have access to effective careless responding measures, which allow researchers to screen for careless responding and to evaluate

Text analysis, particularly custom dictionaries and topic modeling, has helped advance management and organization theory. Custom dictionaries involve creating word lists to quantify patterns and infer constructs, while topic modeling extracts themes from textual documents to help understand a theoretical domain. Building on these two approaches, we propose another text analysis approach called word-text-topic

Insufficient effort responding (IER) to surveys, which occurs when participants provide responses in a haphazard, careless, or random fashion, has been identified as a threat to data quality in survey research because it can inflate observed relationships between self-reported measures. Building on this discovery, we propose two mechanisms that lead to IER exerting an unexpected confounding effect

Organizational research increasingly relies on online review data to gauge perceived valuation and reputation of organizations and products. Online review platforms typically collect ordinal ratings (e.g., 1 to 5 stars); however, researchers often treat them as a cardinal data, calculating aggregate statistics such as the average, the median, or the variance of ratings. In calculating these statistics

Mediation analysis tests X → M → Y processes in which an independent variable (X) exerts an indirect effect on a dependent variable (Y) through its influence on an intervening or mediator variable ...

Organizational researchers have increasingly noted the problems associated with nonnormal dependent variable distributions. Most of this scholarship focuses on variables with positive values and lo...

The attribute alignment approach to team composition allows researchers to assess variation in team member attributes, which occurs simultaneously within and across individual team members. This ap...

The impact and relevance of our discipline's research is determined by its ability to engage the big questions of the grand challenges we face today. Our central argument is that we need innovative...

Natural language processing (NLP) techniques are becoming increasingly popular in industrial and organizational psychology. One promising area for NLP-based applications is scale development; yet, ...

Personality assessments help identify qualified job applicants when making hiring decisions and are used broadly in the organizational sciences. However, many existing personality measures are quit...

This article and the related Feature Topic at Organizational Research Methods upcoming were motivated by the concern that despite the bourgeoning number and diversity of review articles, there was ...

Many topics in organizational research involve examining the interpersonal perceptions and behaviors of group members. The resulting data can be analyzed using the social relations model (SRM). Thi...

Qualitative researchers often work with texts transcribed from social interactions such as interviews, meetings, and presentations. However, how we make sense of such data to generate promising cue...

We review the development of path model fit measures for latent variable models and highlight how they are different from global fit measures. Next, we consider findings from two published simulati...

Organizational research increasingly tests moderated relationships using multiple regression with interaction terms. Most research does so with little concern regarding curvilinear relationships. B...

We know very little about the performance measures executives use to make decisions. To fill this void, we investigate the performance variables that executives reference in corporate filings with ...

Machine learning (ML) enables the analysis of large datasets for pattern discovery. ML methods and the standards for their use have recently attracted increasing attention in organizational researc...

Confirmatory factor analyses (CFA) are widely used in the organizational literature. As a result, understanding how to properly conduct these analyses, report the results, and interpret their impli...

Marshall, J. D., Yammarino, F. J., Parameswaran, S., & Cheong, M. (2022). Using CATA and machine learning to operationalize old constructs in new ways: An illustration using U.S. governors’ COVID-19 press briefings. Organizational Research Methods. Advance Online Publication. https://doi.org/10.1177/10944281221098607

We review contemporary best practice for developing and validating measures of constructs in the organizational sciences. The three basic steps in scale development are: (a) construct definition, (...

This paper investigates the ethnographic researcher's positionality and its role in sensemaking within the research process. Using autoethnographic data of the first author - a black female West Af...

Increased computing power and greater access to online data have led to rapid growth in the use of computer-aided text analysis (CATA) and machine learning methods. Using “big data”, researchers ha...

Relational event models expand the analytical possibilities of existing statistical models for interorganizational networks by: (i) making efficient use of information contained in the sequential o...

The use of logit and probit models when examining binary dependent variables including those in the form 0/1 (i.e., dummy variables), yes/no, and true/false (hereafter binary DVs) is commonplace. Y...

Interdependence is a defining characteristic of groups and teams. However, a vast range of constructs and conceptualizations for interdependence has left researchers with a dizzying array of framew...

Researchers often turn to post hoc statistical techniques to identify common method variance (CMV) in same source data and one viable option is to use a marker variable. The choice of marker variab...

This article addresses the question of how to achieve immersion in organizational ethnography. Working through a broad set of ethnographies in organization studies, sociology, and anthropology, I develop a multi-faceted conceptualization of immersion and offer a framework that integrates four methodological principles—involvement, engagement, duration, and sites—to help organizational ethnographers

Models that combine moderation and mediation are increasingly common. One such model is that in which one variable causes another variable that, in turn, moderates the relationship between two other variables. There are many recent examples of these Endogenous Moderator Models (EMMs). They bear little superficial resemblance to second-stage moderation models, and they are almost never conceptualized

Multilevel methods allow researchers to investigate relationships that expand across levels (e.g., individuals, teams, and organizations). The popularity of these methods for studying organizational phenomena has increased in recent decades. Methodologists have examined how these methods work under different conditions, providing an empirical base for making sound decisions when using these methods

The study of measurement equivalence has important implications for organizational research. Nonequivalence across groups or over time can affect the results of a study and the conclusions that are drawn from it. As a result, the review paper by Vandenberg & Lance (2000) has been highly cited and has played an important role in understanding the measurement of organizational constructs. However, that

In this feature topic, we explore the burgeoning trend to employ templates in qualitative research. To understand authors’ motivations to use templates and perceptions regarding template use in the scholarly community, we conducted an interview study with 21 interviewees who had published qualitative research in one of nine premier management journals between 2014 and 2018. Our analysis identified

Several recent studies have examined the prevention, causes, and consequences of insufficient effort responding (IER) to surveys. Scientific progress in this area, however, rests on the availability of construct-valid IER measures. In the current paper we describe the potential merits of the page time index, which is computed by counting the number of questionnaire pages to which a participant has

Ozkok, O., Vaulont, M. J., Zyphur, M. J., Zhang, Z., Preacher, K. J., Koval, P., & Zheng, Y. (2021). Interaction effects in cross-lagged panel models: SEM with latent interactions applied to work-family conflict, job satisfaction, and gender. Organizational Research Methods. Advance online publication. https://doi.org/10.1177/10944281211043733

Inflection points, kinks, and jumps identify places where the relationship between dependent and independent variables switches in some important way. Although these switch points are often mentioned in management research, their presence in the data is either ignored, or postulated ad hoc by testing arbitrarily specified functional forms (e.g., U or inverted U-shaped relationships). This is problematic

This study explores how researchers in the organizational sciences use and/or cite methodological ‘best practice’ (BP) articles. Namely, are scholars adhering fully to the prescribed practices they cite, or are they cherry picking from recommended practices without disclosing? Or worse yet, are scholars inaccurately following the methodological best practices they cite? To answer these questions, we

This article contributes to the practice of coding in meta-analyses by offering direction and advice for experienced and novice meta-analysts on the “how” of coding. The coding process, the invisible architecture of any meta-analysis, has received comparably little attention in methodological resources, leaving the research community with insufficient guidance on “how” it should be rigorously planned

Researchers often combine longitudinal panel data analysis with tests of interactions (i.e., moderation). A popular example is the cross-lagged panel model (CLPM). However, interaction tests in CLPMs and related models require caution because stable (i.e., between-level, B) and dynamic (i.e., within-level, W) sources of variation are present in longitudinal data, which can conflate estimates of interaction

We are honored to be the next co-Editors of ORM. Under the previous editorial teams, led by Larry Williams, Herman Aguinis, Bob Vandenberg, José Cortina, James LeBreton, and Paul Bliese, ORM has been succeeding by every available metric. ORM is widely recognized as the premier outlet for methodological scholarship in the organizational sciences, and this success is due to the collaboration between

Research methods for studying team composition tend to employ either a variable-centered or person-centered approach. The variable-centered approach allows scholars to consider how patterns of attributes between team members influence teams, while the person-centered approach allows scholars to consider how variation in multiple attributes within team members influences subgroup formation and its effects