Demystifying deliberation, while reinventing social science research on the side

Depending on the group, the discussion topic, or the context of a negotiation, interventions meant to streamline a deliberation process can have varying degrees of success. This summer, student researchers at Penn’s Computational Social Science Lab set out to demystify the intervention process.

The goal? Make tailor-made deliberation advice simple—and chart a new path for social science.

How do you get a group of people to deliberate a contentious topic effectively?

While it may seem intuitive in some situations, there is no one-size-fits-all approach to smooth decision making. When debating a technical topic, for example, deferring to experts may be best. On the other hand, when all group members must be bound by the group’s decision, we try to make sure all voices are heard. So what do we do if a topic is technical, and the decision affects all members?

James Houghton, a researcher at Penn’s Computational Social Science Lab (CSSLab), is tackling this conundrum at a massive scale. This summer, he onboarded a cohort of five Penn undergraduates to help drive his Experiments in Deliberation, a project aiming to understand how to depolarize deliberations where group members disagree. Their work explores the context-dependency of interventions systematically—and, in the process, helps redefine what social science research looks like.

How do you reinvent traditional social science?

The Deliberation project is built around high-throughput experimentation, an approach that leverages new technologies to get the most out of social science. Whereas other approaches may focus on one-off results that do not generalize easily to real-world contexts, high-throughput research uses computational methods to run a massive array of experiments.

Built on this principle, the Deliberation project has three key stages. The first is mapping the design space of potential experiments. Using multiple factors—such as group demographics, size, and polarization towards the deliberation topic—researchers can create a high-dimensional “map” of possible deliberation scenarios. This allows them to determine whether two scenarios are similar based on how far apart they are in the design space—and, by extension, lets them see whether practices that work well in one case can map neatly onto another.

After creating the map, Houghton and his team can design experiments corresponding to different locations in the design space. They first choose sample experiments already in the literature, then use machine learning methods to “learn” the overall landscape of the design space. These models use all of the collected data to make statistical predictions for outcomes at points in the map they have yet to test.

Finally, the team runs a virtual deliberation experiment using real participants. These are designed and administered on the open-source Empirica platform, a new tool for managing experiments and recruiting participants with particular characteristics.

“It’s a highly iterative process,” Houghton explains, “and the beauty of it is that we get to constantly course-correct as we go.” Each iteration of the cycle—data collection, statistical modeling, and testing—generates fresh data that can be used to hone the team’s models and choose the next set of experiments to run. And as a core member of Empirica’s build team, Houghton’s yields from the process are twofold: Empirica enables the Deliberation project, while the project stress-tests Empirica and highlights room for improvement on the platform.

Assembling the team

Houghton selected five rising third-year students from across disciplines as summer 2022 research assistants.

Kimberly Epstein was introduced to the Deliberation project through the Penn Undergraduate Research Mentoring program (PURM), an initiative by the Center for Undergraduate Research & Fellowships (CURF). Over the course of one summer, PURM gives rising second- and third- year students the opportunity to conduct cutting-edge research with Penn faculty, supplemented with a research and living stipend.

As a double-major in cognitive science and computer science, Epstein saw Houghton’s work as an opportunity to explore tech’s applications across seemingly disparate fields. “It was interesting to see computer science being used more broadly than in, say, pure web development,” Epstein notes. “It can be applied to many different situations, and it’s great to see it being used to accomplish something that honestly produces good in the world.”

Looking to bridge her studies in philosophy and computer science, Becca Smith was also drawn to the Deliberation project through PURM because of its interdisciplinary core.

“I was initially interested in the Lab because I’ve struggled to find anything at the intersection of my two majors,” she recounts. “I’m especially interested in political philosophy, the idea of ethical political discourse, and the ways that we can engage with our moral obligations politically. So I love coding, but I also love the motivation and social scientific purpose behind the project and the Lab.”

Alan Qiao saw Houghton’s PURM opening as a chance to explore avenues within and beyond his computer science major. “What really interested me was the breadth of things that might happen in the project,” he notes. “It sounded like a little bit of web work, some natural language processing, and other tasks that I thought would be great ways to get to know the field.”

Qiao grew more excited after speaking with Houghton, who outlined the project’s potential to revolutionize the way researchers conduct social science. “What made me want to get involved in the end was that, while [Houghton’s] work is technical, it scales up very nicely,” Qiao explains. “It’s very easy to take this platform, give it to many researchers at once, and build up a database,” providing a solution for investigating hard-to-test concepts in fields where it is difficult to control experiments. “If you know your environment well enough, you can take this model and get a list of recommended treatments. There’s so much potential here.”

Pursuing political science with a minor in survey research & data analytics, Isaac Gateno saw parallels between the CSSLab’s research and his own interest in political polarization. “I opened up Penn Today one day over lunch, and I saw that the headline article was Duncan [Watts]’s research on polarization,” he explains. “At the time, I had been waiting for a specific professor—basically the lead on polarization in the political science department—to come back from sabbatical. I really wanted to work with him on some independent research and didn’t really see other people in the department taking on the sorts of things that I was interested in.”

After reading about Watts’ work at the CSSLab, however, Gateno viewed it as a natural complement to the research he hopes to pursue—and a stroke of luck that he could find mentorship beyond his home department. “When I saw that [Penn Today] article, I thought it was amazing. I wasn’t familiar with the CSSLab, but I decided to reach out to [Watts].” From there, Watts put him in touch with Houghton to discuss involvement through PURM. “Of course, I was happy to come on board.”

Skylar Rearick, the only non-PURM member of the Deliberation cohort, became involved with the CSSLab through the A. James Clark Scholars Program. The program is aimed at Penn Engineering students from all fields—many of them first-gen or low-income—and encourages exploration in business and leadership through a sophomore year research stipend.

Having recently switched from mechanical engineering to a computer science major, Rearick “knew that [she] had more of an interest in research that has an impact on people.” Houghton’s work appeared on her radar through a friend in the Penn Media Accountability Project (PennMAP), a separate initiative within the CSSLab dedicated to enhancing media transparency across the information ecosystem. “After talking to him about the week-to-week process in his group, I thought the Lab sounded like a really cool place to be,” Rearick notes. “I was drawn to the fact that I’d be working in a larger group, rather than doing individualized research or one-on-one work with a professor. I reached out to [Houghton], and here I am. And I’ve had the best time so far.”

…And, action: Inside a week at the CSSLab

With the summer cohort assembled, Houghton designed a rigorous research schedule. A typical week starts “bright and early on Monday morning” with a hybrid meeting, where the team discusses objectives for the week by cross-referencing notes from previous experiments. “We go through each issue one by one and see if anyone is interested in taking it on—either because it’s something they’re more comfortable with, or because it’s something they want to learn more about,” Rearick says.

With the stage set, the team works through their objectives until Thursday in preparation for their weekly experiment. On Friday, they convene to observe their experiment group and ensure that the Empirica platform runs smoothly, and are left with data that they use to debrief and set the agenda for the following week.

This rapid research cadence makes it easy to see results and incorporate feedback. “We’re able to do things like run a quick pilot on Friday morning, spend the next two hours pushing out edits, and redeploy the whole code two minutes before the actual experiment starts on Friday afternoon,” Qiao adds. “We’re only able to do these things because we can get quick feedback, our team has a good workflow, and we trust each other to get things done.”

Houghton was impressed with the summer cohort’s success. “We tried to run an experiment as early as the third week of the summer, and successfully ran one on the fourth. And from then on, we ran one every week,” he recounts. “The fact that we were able to do that is quite remarkable—in previous research, I didn’t run any pilots for years, in some cases.”

Summer 2022 as a springboard

The cohort’s work is the central piece of what will become a vast experimental infrastructure. From statistical models for analyzing data and predicting what to study next, to dispatch tools for tailor-making experimental groups, to a research network dedicated to understanding the growing dataset, the project is poised to expand quickly in the coming months and years.

With enough of this infrastructure in place, the goal is to “slowly remove [themselves] from the loop” through an increasingly automated research process. The end result, Houghton describes, is a “huge experimental engine which is able to explore the design space efficiently and quickly. That fundamentally changes the way we do social science.”

As for the summer cohort, their work on the Deliberation project has helped them define their trajectories within and beyond Penn. Gateno is currently implementing independent research on partisan labeling with Houghton as his advisor, funded by the Andrea Mitchell Center for the Study of Democracy. Smith notes that her work has solidified her interest in academic research; she is continuing her research assistantship at the CSSLab in preparation for a Ph.D.

While Epstein and Rearick are taking time to explore avenues of study, the summer affirmed their interests in software and web development, and they both hope to extend their involvement with the Lab. Finally, Qiao found the technical aspects of Houghton’s project most rewarding, and plans to remain with the Lab for the duration of his time at Penn. “There are so many things I’d like to try out that we haven’t even gotten to yet,” he explains. “There’s a lot more that can happen with the project that I want to be a part of.”

—

The cohort presented their work at the CSSLab’s Student Research Conference in July, where students from across the Lab’s projects convened to present their summer research. They will also be presenting at CURF’s 2022 Fall Research Expo on September 19.

To learn more about the project, visit the High-Throughput Experiments on Group Dynamics info page on the CSSLab’s website, or read about Houghton’s work in the CSSLab’s Spring 2022 Researcher Spotlight.

Research was conducted under the mentorship of James Houghton, post-doctoral researcher at the CSSLab; and Duncan Watts, Stevens University Professor, Penn Integrates Knowledge Professor, and director of the CSSLab.