Researcher
Mark Whiting
Senior Computational Social Scientist
Mark Whiting is a Senior Computational Social Scientist at the CSSLab, affiliated with the Department of Computer & Information Science at Penn Engineering and the Department of Applied Science and Operations, Information and Decisions at Wharton. He builds systems to study how people behave and coordinate at scale. At the Lab, he leads the High-Throughput Experiments on Group Dynamics, COVID-Philadelphia, and Common Sense Projects.
Mark was previously a post-doctoral researcher under Michael S. Bernstein in the HCI group in Computer Science at Stanford. He holds bachelor’s and master’s degrees in Industrial Design from RMIT and KAIST respectively, and a PhD in Mechanical Engineering from CMU.
Publications +
Whiting, Mark E.; Watts, Duncan J.
A framework for quantifying individual and collective common sense Journal Article
In: PNAS, vol. 121, iss. 4, 2024.
@article{nokey,
title = {A framework for quantifying individual and collective common sense },
author = {Whiting, Mark E. and Watts, Duncan J. },
doi = {https://doi.org/10.1073/pnas.2309535121},
year = {2024},
date = {2024-01-16},
urldate = {2024-01-16},
journal = {PNAS},
volume = {121},
issue = {4},
abstract = {The notion of common sense is invoked so frequently in contexts as diverse as everyday
conversation, political debates, and evaluations of artificial intelligence that its meaning might
be surmised to be unproblematic. Surprisingly, however, neither the intrinsic properties of
common sense knowledge (what makes a claim commonsensical) nor the degree to which it
is shared by people (it’s “commonness”) have been characterized empirically. In this paper,
we introduce an analytical framework for quantifying both these elements of common sense.
First, we define the commonsensicality of individual claims and people in terms of latter’s
propensity to agree on the former and their awareness of one another’s agreement. Second,
we formalize the commonness of common sense as a clique detection problem on a bipartite
belief graph of people and claims, defining pq common sense as the fraction q of claims shared
by a fraction p of people. Evaluating our framework on a dataset of 2,046 raters evaluating
4,407 diverse claims, we find that commonsensicality aligns most closely with plainly-worded,
fact-like statements about everyday physical reality. Psychometric attributes such as social
perceptiveness influence individual common sense, but surprisingly demographic factors
such as age or gender do not. Finally, we find that collective common sense is rare: at most a
small fraction p of people agree on more than a small fraction q of claims. Together, these
results undercut universalistic beliefs about common sense but also open new questions
about its variability of that are relevant both to human and artificial intelligence. },
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The notion of common sense is invoked so frequently in contexts as diverse as everyday
conversation, political debates, and evaluations of artificial intelligence that its meaning might
be surmised to be unproblematic. Surprisingly, however, neither the intrinsic properties of
common sense knowledge (what makes a claim commonsensical) nor the degree to which it
is shared by people (it’s “commonness”) have been characterized empirically. In this paper,
we introduce an analytical framework for quantifying both these elements of common sense.
First, we define the commonsensicality of individual claims and people in terms of latter’s
propensity to agree on the former and their awareness of one another’s agreement. Second,
we formalize the commonness of common sense as a clique detection problem on a bipartite
belief graph of people and claims, defining pq common sense as the fraction q of claims shared
by a fraction p of people. Evaluating our framework on a dataset of 2,046 raters evaluating
4,407 diverse claims, we find that commonsensicality aligns most closely with plainly-worded,
fact-like statements about everyday physical reality. Psychometric attributes such as social
perceptiveness influence individual common sense, but surprisingly demographic factors
such as age or gender do not. Finally, we find that collective common sense is rare: at most a
small fraction p of people agree on more than a small fraction q of claims. Together, these
results undercut universalistic beliefs about common sense but also open new questions
about its variability of that are relevant both to human and artificial intelligence.
conversation, political debates, and evaluations of artificial intelligence that its meaning might
be surmised to be unproblematic. Surprisingly, however, neither the intrinsic properties of
common sense knowledge (what makes a claim commonsensical) nor the degree to which it
is shared by people (it’s “commonness”) have been characterized empirically. In this paper,
we introduce an analytical framework for quantifying both these elements of common sense.
First, we define the commonsensicality of individual claims and people in terms of latter’s
propensity to agree on the former and their awareness of one another’s agreement. Second,
we formalize the commonness of common sense as a clique detection problem on a bipartite
belief graph of people and claims, defining pq common sense as the fraction q of claims shared
by a fraction p of people. Evaluating our framework on a dataset of 2,046 raters evaluating
4,407 diverse claims, we find that commonsensicality aligns most closely with plainly-worded,
fact-like statements about everyday physical reality. Psychometric attributes such as social
perceptiveness influence individual common sense, but surprisingly demographic factors
such as age or gender do not. Finally, we find that collective common sense is rare: at most a
small fraction p of people agree on more than a small fraction q of claims. Together, these
results undercut universalistic beliefs about common sense but also open new questions
about its variability of that are relevant both to human and artificial intelligence.
Almaatouq, Abdullah; Griffiths, Thomas L.; Suchow, Jordan W.; Whiting, Mark E.; Evans, James; Watts, Duncan J.
Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences Journal Article
In: Behavioral and Brain Sciences, 2022.
@article{nokeyb,
title = {Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences },
author = {Almaatouq, Abdullah and Griffiths, Thomas L. and Suchow, Jordan W. and Whiting, Mark E. and Evans, James and Watts, Duncan J. },
url = {https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/beyond-playing-20-questions-with-nature-integrative-experiment-design-in-the-social-and-behavioral-sciences/7E0D34D5AE2EFB9C0902414C23E0C292#article},
doi = {10.1017/S0140525X22002874},
year = {2022},
date = {2022-12-21},
urldate = {2022-12-21},
journal = {Behavioral and Brain Sciences},
abstract = {The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. The researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm—and with far greater efficiency.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. The researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm—and with far greater efficiency.
Almaatouq, Abdullah; Becker, Joshua; Bernstein, Michael S.; Botto, Robert; Bradlow, Eric T.; Damer, Ekaterina; Duckworth, Angela; Griffiths, Tom; Hartshorne, Joshua K.; Lazer, David; Law, Edith; Liu, Min; Matias, J. Nathan; Rand, David; Salganik, Matthew; Emma Satlof-Bedrick, Maurice Schweitzer; Shirado, Hirokazu; Suchow, Jordan W.; Suri, Siddharth; Tsvetkova, Milena; Watts, Duncan J.; Whiting, Mark E.; Yin., Ming
Scaling up experimental social, behavioral, and economic science Technical Report
2021.
@techreport{almaatouq2021scaling,
title = {Scaling up experimental social, behavioral, and economic science},
author = {Abdullah Almaatouq and Joshua Becker and Michael S. Bernstein and Robert Botto and Eric T. Bradlow and Ekaterina Damer and Angela Duckworth and Tom Griffiths and Joshua K. Hartshorne and David Lazer and Edith Law and Min Liu and J. Nathan Matias and David Rand and Matthew Salganik and Emma Satlof-Bedrick, Maurice Schweitzer and Hirokazu Shirado and Jordan W. Suchow and Siddharth Suri and Milena Tsvetkova and Duncan J. Watts and Mark E. Whiting and Ming Yin.},
url = {https://drive.google.com/file/d/1-4kcD8yn4dTikxrbbm5oaGnaEvj3XQ5B/view?usp=sharing},
doi = {10.17605/OSF.IO/KNVJS},
year = {2021},
date = {2021-06-29},
urldate = {2021-06-29},
pages = {40},
abstract = {The standard experimental paradigm in the social, behavioral, and economic sciences is extremely limited. Although recent advances in digital technologies and crowdsourcing services allow individual experiments to be deployed and run faster than in traditional physical labs, a majority of experiments still focus on one-off results that do not generalize easily to real-world contexts or even to other variations of the same experiment. As a result, there exist few universally acknowledged findings, and even those are occasionally overturned by new data. We argue that to achieve replicable, generalizable, scalable and ultimately useful social and behavioral science, a fundamental rethinking of the model of virtual-laboratory style experiments is required. Not only is it possible to design and run experiments that are radically different in scale and scope than was possible in an era of physical labs; this ability allows us to ask fundamentally different types of questions than have been asked historically of lab studies. We argue, however, that taking full advantage of this new and exciting potential will require four major changes to the infrastructure, methodology, and culture of experimental science: (1) significant investments in software design and participant recruitment, (2) innovations in experimental design and analysis of experimental data, (3) adoption of new models of collaboration, and (4) a new understanding of the nature and role of theory in experimental social and behavioral science. We conclude that the path we outline, although ambitious, is well within the power of current technology and has the potential to facilitate a new class of scientific advances in social, behavioral and economic studies.},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
The standard experimental paradigm in the social, behavioral, and economic sciences is extremely limited. Although recent advances in digital technologies and crowdsourcing services allow individual experiments to be deployed and run faster than in traditional physical labs, a majority of experiments still focus on one-off results that do not generalize easily to real-world contexts or even to other variations of the same experiment. As a result, there exist few universally acknowledged findings, and even those are occasionally overturned by new data. We argue that to achieve replicable, generalizable, scalable and ultimately useful social and behavioral science, a fundamental rethinking of the model of virtual-laboratory style experiments is required. Not only is it possible to design and run experiments that are radically different in scale and scope than was possible in an era of physical labs; this ability allows us to ask fundamentally different types of questions than have been asked historically of lab studies. We argue, however, that taking full advantage of this new and exciting potential will require four major changes to the infrastructure, methodology, and culture of experimental science: (1) significant investments in software design and participant recruitment, (2) innovations in experimental design and analysis of experimental data, (3) adoption of new models of collaboration, and (4) a new understanding of the nature and role of theory in experimental social and behavioral science. We conclude that the path we outline, although ambitious, is well within the power of current technology and has the potential to facilitate a new class of scientific advances in social, behavioral and economic studies.
Hu, Xinlan Emily; Whiting, Mark; Bernstein, Michael
Can Online Juries Make Consistent, Repeatable Decisions? Journal Article
In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-16, 2021.
@article{hu2021can,
title = {Can Online Juries Make Consistent, Repeatable Decisions?},
author = {Xinlan Emily Hu and Mark Whiting and Michael Bernstein},
url = {https://www.researchgate.net/profile/Mark-Whiting-2/publication/351417423_Can_Online_Juries_Make_Consistent_Repeatable_Decisions/links/60a63ff7a6fdcc731d3ecc47/Can-Online-Juries-Make-Consistent-Repeatable-Decisions.pdf},
doi = {10.1145/3411764.3445433},
year = {2021},
date = {2021-05-06},
journal = {Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems},
pages = {1-16},
abstract = {A jury of one’s peers is a prominent way to adjudicate disputes and is increasingly used in participatory governance online. The fairness of this approach rests on the assumption that juries are consistent: that the same jury would hand down similar judgments to similar cases. However, prior literature suggests that social influence would instead cause early interactions to cascade into different judgments for similar cases. In this paper, we report an online experiment that changes participants’ pseudonyms as they appear to collaborators, temporarily masking a jury’s awareness that they have deliberated together before. This technique allows us to measure consistency by reconvening the same jury on similar cases. Counter to expectation, juries are equally consistent as individuals, a result that is “good for democracy.” But this consistency arises in part due to group polarization, as consensus develops by hardening initial majority opinions. Furthermore, we find that aggregating groups’ perspectives without deliberation erodes consistency.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
A jury of one’s peers is a prominent way to adjudicate disputes and is increasingly used in participatory governance online. The fairness of this approach rests on the assumption that juries are consistent: that the same jury would hand down similar judgments to similar cases. However, prior literature suggests that social influence would instead cause early interactions to cascade into different judgments for similar cases. In this paper, we report an online experiment that changes participants’ pseudonyms as they appear to collaborators, temporarily masking a jury’s awareness that they have deliberated together before. This technique allows us to measure consistency by reconvening the same jury on similar cases. Counter to expectation, juries are equally consistent as individuals, a result that is “good for democracy.” But this consistency arises in part due to group polarization, as consensus develops by hardening initial majority opinions. Furthermore, we find that aggregating groups’ perspectives without deliberation erodes consistency.