Penn Ph.D. student Xinlan Emily Hu leads a group of budding engineers and social scientists who study communication across teams. The group has developed a new toolkit aimed at helping researchers analyze and measure teamwork.

Originally published in
Penn Today

by Nathi Magubane

August 30, 2024

Hu works in PIK Professor Duncan Watts’s Computational Social Science Laboratory (CSSLab) and examines the complexities that make or break a team’s success. Hu and her team move beyond platitudes to explore the underlying mechanics of collaboration, which she notes is a “ubiquitous part of life in various settings, from laboratories where controlled experiments probe team dynamics to real-world environments like the military, health care, corporate governance, and even our personal lives, where the stakes of teamwork are incredibly high.”

“Whether we’re solving complex problems, making critical decisions, or executing precise operations, teams are integral to how we operate across different domains,” Hu says.

Her current project, which she and her team have just released, focuses on developing an open-source toolkit designed to help social science researchers quantify and analyze the dynamics of team text-based communications throughout task execution by using 164 (and counting) quantified, research-backed dimensions. The aim? To turn what she describes as the “often fuzzy concept of teamwork into something measurable, understandable, and, ultimately, optimizable.”

Building a team to study how teams talk

Hu explains that the project bears some origins and inspiration from her foray in working in industry as a product manager, where “the name of the game is teamwork.”

“That sort of work demanded seamless integration across diverse teams, diverse domains, and yet when you ask a group of people if they’ve had a bad experience with a team, most of them would cite instances where teammates didn’t show up, where things just fell apart, or the overall experience was just plain awful,” she says.

When Hu joined Penn in 2021, she started brainstorming ways to more systematically understand some of the observations she had made on team dynamics. For the first couple of semesters, it was just Hu and one volunteer research assistant, an undergraduate from Cornell who’d reached out expressing interest in the project. The two of them spent months combing through the literature, “documenting everything in this space in a giant spreadsheet.”

This early work was a slog, Hu recalls, but was crucial in allowing them to identify, label, categorize, and encode the various dimensions of team communications, and it eventually culminated in Hu receiving funding from the Wharton AI & Analytics Initiative to support more student researchers.

Among the first to join in the fall of 2022 was Priya D’Costa, now a recent master’s graduate from the School of Engineering and Applied Science. D’Costa entered Penn from a career in finance, with little experience in computer science.

“I spent five years in the industry, where I felt this huge void in my skill set, particularly in coding,” says.

Seeking to bridge that gap, D’Costa decided to pursue a master’s in computer science at Penn, which eventually led her to Hu’s project. “I told Emily and Eric Shapiro, the CSSLab research operations manager, that I’d worked with a lot of teams in my career and was pretty eager to learn more about the nuts and bolts of a team’s dynamics. But at that point, I could barely code ‘Hello world’ in Python,” she says, “so, I started with organizing the literature in the spreadsheet, which I loved because of my finance background.”

As she grew more comfortable, D’Costa transitioned into developing methods to computationally quantify conversational attributes “features,” beginning with simple text analysis, such as counting the use of words that suggest hesitation. During two years on the team, she also eventually built her own spinoff project on gauging conflict in teams, which laid the foundation for a master’s thesis she defended last May.

By then, the project was growing. Yashveer Singh Sohi, a data science master’s graduate from Penn Engineering, joined the following semester and describes his role as “infrastructure optimizer with a bird’s-eye view of all operations,” where he was “tasked with making the entire pipeline more seamless, quicker,” and sliced the runtime of an inefficient sequence of interacting features from around 15 hours to under a minute.

Next came Nikhil Kumar, a third-year undergraduate in the College of Arts and Sciences and Penn Engineering, whose research interests were fueled by his background as an international master in chess. “I previously did some work on things like eye tracking and how complexity in society relates to chess,” says Kumar, who is from South Bend, Indiana. “So, Emily’s work in bridging theoretical, societal ideas to measurable computational tools like this really drew me in and got me excited about what I was doing. It started out as literature review and turned into feature extraction, coding, and actual experimentation.”

And more recently many others have joined, including Penn Engineering third-year Shruti Agarwal from Bridgewater, New Jersey, who came on to help build various features in the toolkit, along with putting together an automated testing pipeline, and Amy Zheng, a second-year engineering student from Princeton, New Jersey, who was onboarded earlier this year and works on developing and building the website, as well as implementing features such as “named entity recognition and turn-taking analysis,” she says.

Solving team problem-solving

Expanding on her work with teams, Emily Hu collaborates with others in the CSSLab to develop a framework for assessing how teams tackle tasks. “It’s often really difficult to generalize the findings from one task or context to another because tasks vary widely in their characteristics,” Hu says. This variation creates challenges in building a cohesive science of teamwork, as the outcomes observed in one scenario may not hold true in another.

Third-year in the School of Engineering and Applied Science Shruti Agarwal leads a discussion at the whiteboard as the team collaborates on developing new features for their open-source toolkit.

To address this, Hu’s approach to studying team dynamics breaks down teamwork into three major components: who is in the team, what they have to do, and how they communicate while tackling that task. “We realized that some of the conventional ways of categorizing tasks, like labeling them as ‘creative,’ ‘physical,’ or ‘logic-based,’ weren’t really giving us the granularity we needed to identify what was going on,” Hu says.

In response, her team developed a more nuanced understanding by quantifying multiple task attributes. These include factors like the level of creativity or physical effort required, the degree of interdependence among team members, and the amount of communication necessary for success. This multidimensional approach provides a more detailed and practical framework for analyzing and improving teamwork across various contexts.

This approach is detailed in a paper, in which Hu and her colleagues argue that traditional taxonomies fail to account for the diverse and multidimensional nature of tasks, which can lead to incomplete or even misleading conclusions about team performance. Their work introduces the “Task Space,” a representation of team tasks in 24 theoretically inspired dimensions.

“By quantifying these tasks across multiple dimensions, we could see patterns that weren’t all that visible before,” Hu says. “For example, tasks that require high creativity but low physical effort, like brainstorming sessions, cluster together in one part of the space, while tasks that are more physically demanding or have clear, demonstrable outcomes are in another part.”

The Team Communication Toolkit extends this multidimensional approach into the domain of conversations. As with tasks, the aim is to shift from viewing conversations in terms of just a few dimensions at a time towards a systematic view of conversations across many dimensions at once. Hu hopes that, by pairing her research with the development of open-sourced tools, she can make a nuanced understanding of teamwork more accessible, both within academic research and for practitioners.

Building these open-sourced tools started small, but it has grown into a more sophisticated and robust workflow. Agarwal, who worked on extracting quantifiable metrics features from the conversations, explains that, in taking the conversational data, she was at first in charge of simpler metrics, like “assigning a score to a conversation’s overall level of positivity.”

But eventually, as the project became increasingly complex, she started to help identify how diverse speakers were in were in their communication styles and eventually, helped develop a testing pipeline “so that, as soon as anyone makes a change to the toolkit code base, we automatically generate all the features and subsequently validates the output via unit and relative test cases to make sure it all works directly for the users, and that the feature outputs align with their original interpretation,” Agarwal says.

Emily Hu and her team gather around to explore new possibilities for their toolkits, exchanging ideas and considering innovative directions for their next phase of research and development.

Becoming better scientists and better conversationalists

Zheng, who took the lead in developing the toolkit’s turn-taking feature, which analyzes how conversations shift between speakers, says that working on this project and developing these tools has made her reexamine everyday conversations.

“I started noticing these back-and-forth patterns more and more. I started noticing how often one person holds court in conversation or how smoothly or awkwardly a group takes turns speaking,” she says. “It made me more conscious of how I engage in conversations, both in the lab and outside of it.”

For D’Costa, the project made her more empathetic yet analytical when working with others, noting how she’s “learned to think more granularly about why people behave the way they do,” something she feels has made her “a better team player and a more thoughtful researcher.”

And Hu shares how excited she is about this project’s potential as a tool that can “help make collaborations like these all the more fruitful.

“The toolkit and its announcement are just the beginning, and there is so much more to come,” Hu says. “We have a couple papers in the works, one looking at the nitty gritty of the infrastructure we’ve created here, and then from there we plan to explore a range of applications, like studying conflict, online debates, negotiations, and so on. We’re very excited to brainstorm new ideas with folks in our lab and external collaborators.”