AI tools help people with speech disabilities make timely jokes

A team of researchers from Cornell Tech is reimagining how technology can support users with speech disabilities—not just in functional speech, but also in making real-time jokes during conversations.

The research explores how artificial intelligence interfaces can help users of augmentative and alternative communication (AAC) technology deliver witty remarks while talking with others—a way of communicating that’s often made impossible by the slow pace of traditional AAC systems.

“Humor is a crucial form of social interaction and is usually taken as a trivial thing. But for someone with a speech impairment, it is not trivial to match the timing of the conversation,” said the paper’s lead author, Cornell Tech Ph.D. student Tobias Weinberg, who lost his ability to speak at 15 and now uses AAC technology.

“Losing my ability to speak from one month to the next, I had to learn to reshape my humor to this new form of communication. This research tries to channel that experience.”

The research, titled “Why So Serious? Exploring Timely Humorous Comments in AAC Through AI-Powered Interfaces,” won a Best Paper Honorable Mention award and Jury Best Demo award at the Association for Computing Machinery’s Conference on Human Factors in Computing Systems (CHI ’25), April 26–May 1 in Yokohama, Japan.

The project included researchers at Cornell Tech and the Cornell Ann S. Bowers College of Computing and Information Science, as well as assistant professor Stephanie Valencia of the University of Maryland.

“There’s a trade-off between agency and efficiency. A button that gives you jokes is super-efficient but they are likely not your jokes. On the other hand, typing out the whole message gives you full control, but it’s really slow, and you will likely miss the timing,” Weinberg said.

“We explored the space in between: How can we guide the AI towards the joke you want to make?”

The study introduced participants with speech disabilities to four AI-assisted interfaces: “Full-auto,” “Keywords,” “Context Bubble Selection” and “Wizard.” Each interface offered participants a different balance of speed and control, allowing them to either quickly generate jokes or fine-tune them to better reflect their personal voice.

The most hands-off option, Full-auto, allows the AI to generate content with minimal input. The “Keywords” option adds a layer of interaction by letting users choose from a set of guiding terms.

For those wanting more agency, the Context Bubble interface enables users to highlight specific parts of a conversation for the AI to focus on, followed by keyword refinement. The most customizable interface, Wizard, breaks down conversations into keywords and prompts users to select related ideas, offering the highest level of AI input but requiring more time and effort.

“The trade-off between agency and timing obviously plays a role in all AI-based interfaces,” said project advisor Thijs Roumen, assistant professor at Cornell Tech. “Most of us would not be willing to surrender agency at all. It is interesting to see this trade-off playing out differently for different groups.”

Before trying the tools, participants expressed a preference for wanting more agency to hand-craft a joke before trying the tools. However, after using the tools, they favored more time-efficient interfaces.

“The time window where the humorous comment can land is so short,” Weinberg said. “Even if the joke is not fully mine, it would still serve its purpose—which is to make other people laugh.”

The study also revealed some challenges, including difficulties with scrolling for users relying on eye-gaze or joystick input, and the lack of emotional nuance in text-to-speech voices.

There is no one-size-fits-all solution for improving AAC devices, Weinberg said. “Each user has different preferences, and those preferences can even change depending on the conversation partner.”

“So much work is focused on the direct ‘needs’ of people with disabilities, but empowering self-expression, humor and emotional connections are often overlooked,” said Roumen, who’s also the director of the Matter of Tech Lab at Cornell Tech and affiliated with Cornell Bowers CIS.

“I hope we pave a path toward more research in this direction.”

The paper also includes a call to expand the scope of accessibility research. Weinberg said, “We need to enhance expressivity in AAC communication, and more generally, enable people with speech impairments to fully engage in conversations and flourish in society.”