BBB: Reimagining Interpreting with AI and People

Period2023.03 – 2023.05
RoleService concept ideation; UI/UX design for multi-party and sign language interpreting
AffiliationEXPC Lab
CollaboratorsJuhyuk Yoon, Sangyun Lee, Yeji Park
Project TypeUI/UX Design
TopicsCommunication, AI

1 Overview

  • Following our redesign of BBB Korea’s user and volunteer apps, this concept project explored how the interpreting service could evolve with advances in AI. We considered the different roles of AI and volunteer interpreters, and proposed a new way for people to communicate across languages and modes of communication.
  • We developed four core experiences: sharing the context of an interpreting request, multi-party and sign language interpreting, language guides based on past sessions, and visualizing help given and received. The project was named a Winner in the Apps category of the 2023 Red Dot Award: Brands & Communication Design.
  • Role: Core concept ideation (ways to thank volunteers, multi-party interpreting with personal audio devices, and sign language interpreting); UI design for multi-party and sign language interpreting

2 Context

BBB Korea connects people who need language support with volunteer interpreters over the phone.

While our previous app redesign focused on the structure and usability of the existing service, this follow-up project explored what role a human-centered interpreting service could play as machine translation and conversational AI became more widespread.

We reviewed feedback from users of the existing service and relevant literature, and focused on four issues:

  • Difficulty sharing context: Volunteer interpreters may not learn a user’s location, purpose, or level of urgency until they connect, leading to repeated explanations.
  • Missing cultural context: Even an accurate translation can miss slang, idioms, accents, and cultural norms.
  • Limits of one-to-one calls: Phone interpreting alone makes it difficult to support conversations across several languages or include people who use sign language.
  • A one-off experience: Language and cultural information from one session often goes unused afterward, while volunteers’ contributions can fade from view once the call ends.

With these issues in mind, we envisioned a communication service where AI and people could work together as needed, support different ways of communicating, and turn a single interpreting experience into an opportunity for ongoing learning and participation.


3 Approach

3-1 Defining the Roles of AI and Human Interpreters

We first considered what AI and volunteer interpreters should each be responsible for.

AI would handle tasks where speed and scale matter, such as instant translation, conversation records, separating speakers, and suggesting frequently used expressions. People, meanwhile, would interpret cultural nuances and ambiguous language, or step in when someone needed emotional support or a judgment they could trust.

From there, we proposed different kinds of help depending on the urgency and complexity of a request. AI could support everyday conversations right away, while urgent situations or those requiring particularly accurate interpretation would be connected to a volunteer interpreter.

3-2 Beyond Translation: Understanding Cultural Context

We developed hypothetical personas and scenarios centered on people living abroad and international students. We imagined a student struggling to join a conversation because they did not understand Australian slang, and another who could not make sense of a disagreement involving bus customs in Korea through literal translation alone.

These scenarios drew our attention to the gap between knowing what someone said and understanding what they meant in context. We expanded the project’s direction from interpreting conversations to interpreting culture. Users could select something they did not understand, ask AI about its context, or turn to a person for an explanation when needed.

We also considered a separate cultural exchange community where users and volunteers could ask each other questions. In the final concept, though, we narrowed the scope to language guides based on interpreting history and activity records, keeping the emphasis on continued learning and connection.

3-3 Beyond One-to-One Voice Interpreting

We explored how to move beyond one-to-one voice interpreting so that several people could join a conversation in different ways.

Multi-party interpreting assumed a setting where personal audio devices such as AirPods were commonplace. People speaking different languages in the same space could hear translations through their own devices, while the screen showed participants, their languages, and each speaker’s contributions.

I designed the interface to keep conversations between several people easy to follow, clearly separating participant information and speaker-by-speaker records. Users could replay a particular utterance, check its translation, or ask for more cultural context.

Sign language interpreting was proposed to address the limits of a voice-centered service. The concept involved using a camera to recognize signs and convert them into sentences, while showing the other person’s speech as text.

I designed an interface that brings together the camera view, recognition status, converted sentences, and conversation history. We did not implement a working sign language recognition model. This was a concept prototype intended to illustrate a possible future experience.

3-4 Designing the Moments Before and After

We considered what happens both before and after an interpreting session.

Before a session, users could share their location and situation using tags, while AI suggested additional tags based on the context. This was intended to reduce the burden on volunteer interpreters of having to piece together the situation after connecting.

Afterward, we envisioned recommendations for useful expressions and cultural information based on a user’s location and past sessions. We also proposed recording and visualizing the help users received and the contributions volunteers made, so a one-off encounter could lead to further learning and volunteer participation.


4 Outcome

4-1 Four Core Service Experiences

Request Interpretation: Sharing context before a session

Before calling a volunteer interpreter, users can describe their current location and purpose with tags. AI suggests additional tags based on the location and context, helping the interpreter understand the request before connecting.

Conversation Bridge: Connecting languages and ways of communicating

The service distinguishes speakers and delivers translations in each participant’s language. Users can review conversation records by speaker or ask for further explanations. A camera-based sign language interpreting interface also imagines a future conversation that includes people who do not use spoken language.

Language Guides: Bringing past experiences into future conversations

Based on earlier interpreting sessions and the user’s current location, the Language Guides feature suggests useful expressions and cultural information. It helps people draw on what they learned during interpreting when communicating in the future.

Activity History: Making help and contributions visible

Activity History visualizes the number, duration, and location of interpreting sessions, along with records of help given and received. Volunteers can reflect on their contributions, while users can thank the people who helped them.

4-2 Red Dot Award

The project was named a Winner in the Apps category of the 2023 Red Dot Award: Brands & Communication Design.

Red Dot Design Award: BBB: Revolutionising Cross-Language Communication Services in a Multilingual World


5 Reflection

In this project, we let AI take on the parts of interpreting where speed and repetition mattered, while keeping people involved when cultural context and trust were essential. We wanted to draw on what technology could offer without losing the value of a human-centered interpreting service.

Many AI-based services aim to automate as much as possible. But I don’t think more automation always creates more value. What matters is carefully deciding which roles belong to technology and which belong to people, then designing the service around what each situation calls for.