Posts Tagged ‘natural language’

Topic-based Evaluation for Conversational Bots

Dialog evaluation is a challenging problem, especially for non task-oriented dialogs where conversational success is not well-defined. We propose to evaluate dialog quality using topic-based metrics that describe the ability of a conversational bot to sustain coherent and engaging conversations on a topic, and the diversity of topics that a bot can handle. To detect conversation topics per utterance, we adopt Deep Average Networks (DAN) and train a topic classifier on a variety of question and query data categorized into multiple topics. We propose a novel extension to DAN by adding a topic-word attention table that allows the system to jointly capture topic keywords in an utterance and perform topic classification. We compare our proposed topic based metrics with the ratings provided by users and show that our metrics both correlate with and complement human judgment. Our analysis is performed on tens of thousands of real human-bot dialogs from the Alexa Prize competition and highlights user expectations for conversational bots.

READ THE PAPER:
Topic-based Evaluation for Conversational Bots
by F Guo, A Metallinou, C Khatri, A Raju, A Venkatesh, A Ram
NIPS-2017 Workshop on Conversational AI

arxiv.org/abs/1801.03622

Announcing Winners of 2017 Alexa Prize

Earlier today, Rohit Prasad, vice president and head scientist, Alexa Machine Learning, and I had the pleasure of announcing the winner of the inaugural Alexa Prize competition for university students dedicated to accelerating the field of conversational artificial intelligence (AI).

Congratulations to team Sounding Board, an inspiring group of students from the University of Washington, whose socialbot earned an average score of 3.17 on a 5-point scale from our panel of independent judges and achieved an average conversation duration of 10:22. As the winner of our inaugural competition, team Sounding Board earned our $500,000 first-place prize, which will be shared among the students.

We also had the privilege of honoring and surprising our other finalists on stage. Our runner up was team Alquist from Czech Technical University in Prague. We presented them with a $100,000 prize for their efforts. We also awarded our third-place winner, team What’s Up Bot from Heriot-Watt University in Edinburgh, Scotland, with a $50,000 prize.

CHAT WITH THE WINNERS:

Just say “Alexa, let’s chat” to any Alexa-enabled device. (If you’re outside the U.S., set your Amazon Preferred Marketplace (PFM) to U.S. or use a U.S. based Amazon account.)


VIEW THE KEYNOTE:

youtu.be/HXtjdXjpJwI?t=32m18s


VIEW A SHORT FILM SHOWCASING THE WINNING SOCIALBOTS AND HOW THEY WERE SELECTED:

youtu.be/WTGuOg7GXYU


READ MORE:

developer.amazon.com/blogs/alexa/post/1a6a19d8-e45d-4b3b-981d-776a378ba625/university-of-washington-students-win-inaugural-alexa-prize

Conversational AI: The Science behind the Alexa Prize

Conversational agents are exploding in popularity. However, much work remains in the area of social conversation as well as free-form conversation over a broad range of domains and topics. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million-dollar university competition where sixteen selected university teams were challenged to build conversational agents, known as “socialbots”, to converse coherently and engagingly with humans on popular topics such as Sports, Politics, Entertainment, Fashion and Technology for 20 minutes.

The Alexa Prize offered the academic community a unique opportunity to perform research with a live system used by millions of users. The competition provided university teams with real user conversational data at scale, along with the user-provided ratings and feedback augmented with annotations by the Alexa team. This enabled teams to effectively iterate and make improvements throughout the competition while being evaluated in real-time through live user interactions.

To build their socialbots, university teams combined state-of-the-art techniques with novel strategies in the areas of Natural Language Understanding, Context Modeling, Dialog Management, Response Generation, and Knowledge Acquisition. To support the teams’ efforts, the Alexa Prize team made significant scientific and engineering investments to build and improve Conversational Speech Recognition, Topic Tracking, Dialog Evaluation, Voice User Experience, and tools for traffic management and scalability.

This paper outlines the advances created by the university teams as well as the Alexa Prize team to achieve the common goal of solving the problem of Conversational AI.

Conversational AI: The Science behind the Alexa Prize

by Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Yi Pan, Han Song, Sk Jayadevan, Gene Hwang, Art Pettigrue

Proceedings of the 2017 Alexa Prize
Invited talk at NIPS-2017 Workshop on Conversational AI
Invited talk at re:Invent 2017 (with Spyros Matsoukas)

READ THE PAPER:

arxiv.org/abs/1801.03604

WATCH THE TALK:

youtu.be/pn5QJQZjGpM

			

Conversational News Experiences

News consumption is a passive experience—reading print or online newspapers, listening to radio shows and podcasts, watching television broadcasts. News producers create, curate, and organize  content which consumers absorb passively. With the advent of interactive conversational technologies ranging from chatbots to voice-based conversational assistants such as Amazon Alexa, there is an opportunity to engage consumers in more interactive experiences around news.

At the Computation+Journalism symposium held at Northwestern University this year, Emily Withrow, editor at Quartz Bot Studio and assistant professor at Northwestern’s Medill School of Journalism and I had a fireside chat to share recent technological developments in this area and explore what kinds of conversational news experiences these technologies might enable.

Panel at the 2017 Computation+Journalism Symposium, Northwestern University, Evanston, IL. #cj2017 

Announcing the 2017 Alexa Prize Finalists

We’ve hit another milestone in the Alexa Prize, a $2.5 million university competition to advance conversational AI. University teams from around the world have been hard at work to create a socialbot, an AI capable of conversing coherently and engagingly with humans on popular topics and news events for 20 minutes.

I am now excited to announce the university teams that will be competing in the finals! After hundreds of thousands of conversations, the two socialbots with the highest average customer ratings during the semifinal period are Alquist from the Czech Technical University in Prague and Sounding Board from the University of Washington in Seattle. The wildcard team is What’s Up Bot from Heriot-Watt University in Edinburgh, Scotland.

READ MORE:

developer.amazon.com/blogs/alexa/post/783df492-4770-4b11-81ac-59e009669d56/announcing-the-2017-alexa-prize-finalists

 

Conversational AI: Voice-Based Intelligent Agents

As we moved from the age of the keyboard, to the age of touch, and now to the age of voice, natural conversation in everyday language continues to be one of the ultimate challenges for AI. This is a difficult scientific problem involving knowledge acquisition, natural language understanding, natural language generation, context modeling, commonsense reasoning and dialog planning, as well as a complex product design problem involving user experience and conversational engagement.

I will talk about why Conversational AI is hard, how conversational agents like Amazon Alexa understand and respond to voice interactions, how you can leverage these technologies for your own applications, and the challenges that still remain.

Variants of this talk presented (click links for video):
 
Keynote talks at The AI Conference (2017), O’Reilly AI Conference (2017), The AI Summit (2017), Stanford ASES Summit (2017), MLconf AI Conference (2017), Global AI Conference (2016).
 
Distinguished lectures at Georgia Tech/GVU (2017), Northwestern University (2017).
 
Keynote panel at Conversational Interaction Conference (2016).
 
Lightning TED-style talks at IIT Bay Area Conference (2017), Intersect (2017).
 

Join the Alexa Prize Journey and Test the Socialbots

On September 29, 2016, Amazon announced the Alexa Prize, a $2.5 million university competition to advance conversational AI through voice. In April, university teams from around the world assembled at the appropriately named Day 1 building in Seattle for the Alexa Prize Summit. The event was a base camp for teams to share learnings and make preparations for the most challenging leg of their journey: to build and scale an AI capable of conversing coherently and engagingly with humans for 20 minutes.

As they build their “socialbots,” they will encounter esoteric problems like context modeling and dialog planning as well as exoteric problems like user experience and conversational engagement. And they will need all the help they can get.

We invite you to join the students on their journey and help them along the way. You can interact with their socialbots simply by saying, “Alexa, let’s chat” on any device with Alexa.

READ MORE:
developer.amazon.com/blogs/alexa/post/e4cc64d1-f334-4d2d-8609-5627939f9bf7/join-the-alexa-prize-journey-and-test-the-socialbots