💬Lesson 8: NLP

Aim

Natural Language Processing (NLP) is an incredibly powerful tool that allows you to verbally communicate with robots and have them understand you. Here you will discover the basics of how to work with Misty's NLP, and by the end of lesson you will be able to create conversations about any topic. If you want to know more about NLP before you start, check out NLP.

Wake Misty up

Challenge 1: Start with a wake-up word

Before you start a conversation with Misty you can let her know that you want to get her attention. To do this, you will need to use the start key-phrase recognition block found under the 'Speech' tab in Misty blocks. If for Alexa for example, the wake-up word is "Alexa", and for Google it's "Hey, Google", for Misty the wake-up word is "Hey, Misty". The next step is to decide what you want Misty to do when she recognizes the wake-up word and starts waking up. For this you will need the key-phrase recognized event block found in the 'Event' tab. Here you can nest your unique wake-up sequence. Once you start your program, you will see that Misty's tally light turns blue, this means that she's listening to you. How would you build your wake-up sequence?

How to talk to Misty

Challenge 2: Building your first conversation

Now that you've created your wake-up sequence to get Misty's attention, you can start familiarizing yourself with how a 'conversation tree' is built by using the YesNoQuestion conversation tree example below. A 'conversation tree' is similar to the 'skill tree' concept we explored in previous lessons. In a conversation tree the context of the conversation is the main body of the tree, and the various topics are branches that grow and branch off into subtopics. For this example we will build a very simple conversation tree with Yes and No branches.

The first step in building your conversation tree in Blockly is to select the conversation-starting state block from the 'NLP' tab. This is the block where you will nest your whole conversation tree. In the first text field you can define the name of your conversation, in this example we will use the 'YesNoQuestion' preset to create a conversation tree where your Misty will ask you questions and understand if you say 'Yes' or 'No'.

The next step is to nest your starting state block, this block will determine what Misty will say and do when the conversation starts. After you have chosen what Misty should ask you, you will need to select a context. For this example we will use the 'furhat.context.en.yes-no' found in the drop-down field. A context in a conversation tree contains all the topics and words associated with them that Misty can understand. These are commonly referred to as 'intents'. The context in this example contains only two intents: Yes and No.

Important: If you want Misty to listen to you speaking during a state, it's important to select 'listen' in the action field. This way Misty will recognize your speech and reference it with the intents.

You will also notice that this block has no match and no match count fields. In the event that Misty does not understand you, you can decide what she will say and how many times she will say it.

To let Misty know how the conversation should flow after you have initiated the start state, you need to use the state transition block and place it after all of your states.

To trigger a state transition, Misty will need to reference your speech with the intents in the context. To let her know which intent should trigger a transition you will need to write the intent name in the trigger field. Remember that in this case we only have two intents: Yes and No. Important: Your state names in the state transition block have to match the state block names, otherwise the program won't work. Finally, you will need to add a start conversation block and write the name of the conversation tree you have created to get it going. Remember that just like with events, you will need to use a run until stopped block for Misty to be able to monito input.

Now that you have explored all the blocks you need to construct a conversation tree, let's put it all together using the conversation tree example below.

Challenge 3: Build your own context

Great job on building your first conversation tree with Misty! Let's try building a new conversation from the very beginning, but this time with your own context.

To get started, you will need to clear your Blockly Workspace and insert the context block from the 'NLP' tab in Misty Blocks. Choose a name for your context and think about what you want the intents to be. To add an intent you can use the gear icon in the top left corner and drag a new intent block into your context block.

To name your intents and choose the words you want to associate with them you will need to connect a name-sample block.

In the example below we've named the context samplefood and created two intents. The first intent is pizza and the second is taco. In the sample fields you can write as many words as you like that you want to be associated with your intents.

Once you build your new context you can run the code and your context will be uploaded to Misty's memory. Important: Context names should not be re-used since Misty will not know which version you want to use. Every new context needs to have a unique name.

Challenge 4: Build your own conversation tree

Congratulations! You are ready to start building your very own conversation tree. Just like in the example shown in Challenge 2, you will need to use the intents you just created in your context to trigger states and state transitions. If you want to continue building on the context from the previous example, you can use the following conversation tree.

Challenge 5: Animate your conversation

Now that you've built your conversation tree, let's take it to the next level! You can also include all that you've learned about Misty in the previous lessons and include it in your conversation. Using the action block in the NLP tab, you can build new actions for your states. To select them in your action drop-down in the state block you will have to upload them by running the program first. Try it out using the example below!

Unfortunately, there are some issues with the action blocks.

• It seems that you can't use negative parameters in the movement blocks.

The solution is: instead of utilizing the Value Block, you can opt for the Operation Block and calculate your number. This should address the problem effectively.

• There are some blocks who generate an "Invalid diagram" in Blockly. The reasons are unclear, but Misty's actions are stored in the same space in her API. Indistinctly if they come from Blockly or from Python.

So you can create new actions in Misty by writing in Python a code built in this way:

from mistyPy.Robot import Robot
from mistyPy.Events import Events

misty = Robot()

misty.create_action(name="move", script = "HEAD:0,0,0,1750;ARMS:0,0,1750;HEADING:0,0.5,3000,false;", overwrite=True)

For example, this one leads Misty to move her head, move her arms and to a front movement of 0.5 meters.

create_action has 3 parameters : -name (the name that you'll use to recall your action) -script (the list of commands that Misty will use) -overwrite (we use this to always have the latest upload)

You can find the complete list to customize your Misty's actions here : Action Commands