🗣️Lesson 7: Start a conversation

Aim

In this lessons you will learn how to work with Natural Language Processing (NLP) concepts and Speech Recongition in Python and create your own conversations with Misty. By the end of lesson you will be able to master the full range of Misty's speech and conversational abilities.

Waking up

Challenge 1: Misty Wake-up

You may recall from the Blockly Lessons that before you start a conversation with Misty you can let her know that you want to get her attention with her default wake-up phrase "Hey, Misty". To do this, you will need to set up the start key-phrase recognition event. The code structure of the key-phrase recognized event is identical to any other event.

Once you have imported the libraries required, you can call the KeyPhraseRecognized event by building a callback function that will determine what you want Misty to do when the key-phrase is recognized. How would you choose to design your wake up sequence?

from mistyPy.Robot import Robot
from mistyPy.Events import Events
import time

misty = Robot()

misty.display_image("e_SleepingZZZ.jpg", 1)
misty.play_audio("e_Sleepy.wav", 50)
misty.move_head(60, 0, 0, 80)
misty.move_arms(85, 85, 80, 80)
misty.change_led(0, 0, 255)
misty.start_key_phrase_recognition()

def Key_Phrase_Recognized(data):
    print(data)
    misty.display_image("e_Surprise.jpg", 1)
    misty.play_audio("s_PhraseHello.wav", 50)
    time.sleep(2)
    misty.speak("Good morning")
    misty.move_head(0, 0, 0, 80)
    misty.move_arms(-85, -85, 80, 80)

misty.register_event(event_name='Key_Phrase_Recognized', event_type=Events.KeyPhraseRecognized, callback_function=Key_Phrase_Recognized, keep_alive=True)
misty.keep_alive()

Building NLP conversations

Now that you've created your wake-up sequence in Python you can start building a conversation. In Blockly Lesson 8: NLP we used the conversation tree concept to understand the relationships between different elements of the conversation such as context, intents, sample, etc. In Python, you can use the same concepts to build your conversation. Now let's try rebuilding the YesNoQuestion conversation tree example from Blockly in Python.

The first step in order to build an NLP Conversation in Python is to create the conversation with the following command:

from mistyPy.Robot import Robot

misty = Robot()

misty.create_conversation(name="name_of_your_new_conversation", startingState="new_start_state", overwrite=True)

It includes the conversation name and the starting state.

Later on, you can think about the states that Misty will assume in the flow of your conversation. To do so you can use the misty.create_state command.

This state will be the one called when you will trigger its intent.

There are two types of states:

• Start States

• Flow States

The start states contain the context and so the “question” that you will answer, when the code runs, your answers will be recorded and the flow state associated with that intent will play.

You can find several properties for each state, like:

• State name: the unique name for the state

• Speak: what you would like Misty to say during the state

• StartAction: A particular action associated with that specific state, you can find the list of Misty's pre-built actions here NLP Actions or build your own.

The Start state must contain an extra necessary parameter: the context.

It’s necessary to insert the context because Misty has in her memory several contexts and you can choose which one will be used in that specific scenario. The way that you can include a context in the start_state is "['context_name']", as you can see in the example:

misty.create_state(name= "new_start_state", speak="new question", contexts="['furhat.context.en.yes-no']", listen=True, overwrite=True)

For the flow states, instead, you’re free to use all the parameters you like and here is an example of working flow states:

from mistyPy.Robot import Robot

misty = Robot()

misty.create_state(name="flow_state_1", speak="Yay, that's awesome!")
misty.create_state(name="flow_state_2", speak="Wow, it's so obvious that humans will eventually be replaced by robots.")

Important: If you want Misty to listen to you speaking during a state, it's important to select 'listen' in the action field. This way Misty will recognize your speech and reference it with the intent. Of course a flow state can be the start state for a next question.

There are many more parameters you can modify while creating a conversation state:

Last but not least is to map the conversation. You will need to determine and filter how you want Misty to transit between one state and another. To achieve this you can use the misty.map_state() API call.

In the map state, you need to insert which is the conversation you want to map, which is the starting state for that transition and the next one and lastly which is the trigger and the trigger filter.

Important: The triggerFilter name should match the intent name.

from mistyPy.Robot import Robot

misty = Robot()

misty.map_state(conversation= "name_of_your_new_conversation", state= "new_start_state", trigger= "SpeechHeard", nextState= "flow_state_1", triggerFilter="trigger1", reEntry=False, overwrite=True)
misty.map_state(conversation= "name_of_your_new_conversation", state= "new_start_state", trigger= "SpeechHeard", nextState= "flow_state_2", triggerFilter="trigger2", reEntry=False, overwrite=True)

Once you have built your conversation you can call it with the command:

from mistyPy.Robot import Robot

misty = Robot()

misty.start_conversation("name_of_your_new_conversation")
misty.keep_alive()

Important: You don’t need to always re-build your conversation because it will be stored in Misty’s memory. So feel free to delete the block of code that creates it after feeling satisfied about it.

To check your states, your contexts and your conversation you can use Misty’s API links:

If you insert the following links in your browser you will be able to see your contexts, your conversations and your states in Misty’s current storage.

(When you are working on them remember to refresh the page to see the current results)

• http://<your_robot_ip_address>/api/dialogs/contexts

• http://<your_robot_ip_address>/api/conversations

• http://<your_robot_ip_address>/api/states

Important: Misty may not distinguish "really well" capital and not-capital letters, so for your names we suggest to use something different each time (es not Start/start but StartConv1/StartConv2).

You can use pre-built actions and contexts for your conversations with Misty.

But if you want to create your actions you can use the misty.create_action API. In the parameters, you will need to define the unique name of the action and a script, which contains all of the actions you want Misty to perform, you can find a list of them in Action Commands.

misty.create_action(name="question_action", script= "LED-PATTERN:0,0,255,40,0,112,1200,breathe;IMAGE:e_ApprehensionConcerned.jpg;ARMS:29,29,1000;HEAD:10,0,0,1000;", overwrite= True)

You can check your action at:

• http://<your_robot_ip_address>/api/actions

At the moment we’re facing some issues with the misty.train_nlp_engine. So if you want to use your contexts we would suggest to use Blockly to build it.

Example of creation of Misty NLP conversation :

You will need to run first the blockly workflow. In the blockly workflow remember to run first the context block and then the start_state, in this case, you’ll have available the context you just created. After, you will be able to use the Python Studio with the new elements you built.

from mistyPy.Robot import Robot

misty = Robot()

misty.create_conversation(name="ColorConversation", startingState="StartColor", overwrite=True)
misty.create_state(name= "StartColor", speak=" Which color do you prefer between green and blue?", contexts="['color_contexs']", listen=True, noMatchSpeech= "Could you please repeat that?",  repeatMaxCount= 3, startAction="None", overwrite=True)
misty.create_state(name= "GreenState", speak= "I love green", listen=True, noMatchSpeech="could you please repeat that?", repeatMaxCount= 3, startAction="admire", overwrite=True)
misty.create_state(name= "BlueState", speak= "I love blue", listen=True, noMatchSpeech="could you please repeat that?", repeatMaxCount= 3, startAction="admire2", overwrite=True)

misty.map_state(conversation= "ColorConversation", state= "StartColor", trigger= "SpeechHeard", nextState= "GreenState", triggerFilter="green", reEntry=False, overwrite=True)
misty.map_state(conversation= "ColorConversation", state= "StartColor", trigger= "SpeechHeard", nextState= "BlueState", triggerFilter="blue", reEntry=False, overwrite=True)

Example of using NLP conversation:

from mistyPy.Robot import Robot

misty = Robot()

misty.start_conversation("ColorConversation")
misty.keep_alive()

Congratulations on completing your own conversation tree! You are ready to combine all that you've learned so far to create amazing human-robot interactions using event skills together with physical touch, face recognition, object recognition, audio localization, speech recognition and natural language processing.

Advanced conversation:

In this example, you'll be able to see how you can create a conversation with more interactions. The key concept is that you're building one conversation, so you need only one, that has all the states and the map that can control the flow between them. Code explanation: As usual, you can import the required libraries, create the robot and start the event that will start the conversation (key_phrase_recognized). The main steps to build a conversation are:

• Initialization of one new conversation

• Creation of states

• States' mapping

In this case, the conversation's name is ColorConversation1, and the starting state will be StartColorConverasation1. When the state is a Start State it will need the context as well, as, in this case, it is in StartColorConverasation1 and YesColorConversation1. After, you can map the different states how you prefer. In the last code lines, you can find the function that will be called when "hey, Misty" is heard and you'll be able to start your conversation. In this case, there are three possibilities after the YesColorConversation1.

For the moment you would still need to create your context in Blockly. Python code:

from mistyPy.Robot import Robot
from mistyPy.Events import Events
import time

misty = Robot()
misty.start_key_phrase_recognition()
misty.change_led(255, 0, 0)

misty.create_conversation(name="ColorConversation1", startingState="StartColorConverasation1", overwrite=True)

misty.create_state(name= "StartColorConversation1", speak="Do you like colors?", contexts="['furhat.context.en.yes-no']", listen=True, noMatchSpeech= "Could you please repeat that?",  repeatMaxCount= 3, startAction="listen", overwrite=True)
misty.create_state(name= "YesColorConversation1", speak= "I like that one, which color do you prefer between red, green or blue?", listen=True, noMatchSpeech="could you please repeat that?", repeatMaxCount= 3, startAction="admire", overwrite=True, contexts="['color_context_3_colors']")
misty.create_state(name= "NoColorConversation1", speak= "Okay, don't play with me then", listen=True, noMatchSpeech="could you please repeat that?", repeatMaxCount= 3, startAction="mad2", overwrite=True)
misty.create_state(name= "RedStateConversation1", speak= "I'm so excited, I love red, it's love's color", listen=True, noMatchSpeech="could you please repeat that?", repeatMaxCount= 3, startAction="love", overwrite=True)
misty.create_state(name= "GreenStateConversation1", speak= "I love green, it's hope's color", listen=True, noMatchSpeech="could you please repeat that?", repeatMaxCount= 3, startAction="hi", overwrite=True)
misty.create_state(name= "BlueStateConversation1", speak= "Blue! Wow! I love to look at the blue sky", listen=True, noMatchSpeech="could you please repeat that?", repeatMaxCount= 3, startAction="hug", overwrite=True)

misty.map_state(conversation= "ColorConversation1", state= "StartColorConversation1", trigger= "SpeechHeard", nextState= "YesColorConversation1", triggerFilter="yes", reEntry=False, overwrite=True)
misty.map_state(conversation= "ColorConversation1", state= "StartColorConversation1", trigger= "SpeechHeard", nextState= "NoColorConversation1", triggerFilter="no", reEntry=False, overwrite=True)
misty.map_state(conversation= "ColorConversation1", state= "YesColorConversation1", trigger= "SpeechHeard", nextState= "RedStateConversation1", triggerFilter="red", reEntry=False, overwrite=True)
misty.map_state(conversation= "ColorConversation1", state= "YesColorConversation1", trigger= "SpeechHeard", nextState= "GreenStateConversation1", triggerFilter="green", reEntry=False, overwrite=True)
misty.map_state(conversation= "ColorConversation1", state= "YesColorConversation1", trigger= "SpeechHeard", nextState= "BlueStateConversation1", triggerFilter="blue", reEntry=False, overwrite=True)

def Key_Phrase_Recognized(data):
    misty.change_led(0, 255, 0)
    misty.play_audio("s_PhraseHello.wav")
    time.sleep(2)
    misty.start_conversation("ColorConversation1")

misty.register_event(event_name='KPR_event', event_type=Events.KeyPhraseRecognized, callback_function=Key_Phrase_Recognized, keep_alive=True)
misty.keep_alive()

Coming Soon 👉 Lesson 9: Connecting External APIs