Misty Lessons
  • Misty Lessons
    • 📖Welcome to Misty Lessons
    • 📚Get to know your Misty
    • 📲Connect to Misty
    • 👩‍💻Misty Studio
    • 🖥️Desktop Environment
    • ⬆️Update your Misty
    • 👥Projects
  • Blockly
    • 🧩Blockly Lessons
      • 🤸Lesson 1: Movement
      • 🎶Lesson 2: Voice and Sound
      • 🎭Lesson 3: Expressions
      • 🎥Lesson 4: Robot Cinema
      • 🛠️Lesson 5: Events
      • ☺️Lesson 6: Face recognition
      • 🔢Lesson 7: Variables and Functions
      • 💬Lesson 8: NLP
    • 🏫Blockly projects
  • Blockly Elements
    • ⚛️Misty Blocks
      • Movement
      • Speech
      • Audio
      • Vision
      • Events
      • Miscellaneous
      • NLP
      • System
    • 🔁Basic Blocks
      • Logic
      • Loops
      • Math
      • Text
      • Lists
    • 🅰️Advanced Blocks
      • Variables
      • Functions
  • Python
    • 🐍Python Lessons
      • 🦿Lesson 1: Loco-motion
      • 🤖Lesson 2 : Build a character
      • 🧠Lesson 3: Create memories
      • ⚒️Lesson 4: Event skills
      • 👀Lesson 5: Expand awareness
      • 🔗Lesson 6: Compact code
      • 🗣️Lesson 7: Start a conversation
  • Python Elements
    • 🐸Misty Python API
      • Motion and Mobility
      • Display and LED
      • Record Assets
      • Change/Remove Assets
      • Stream Assets
      • Get Assets
      • Events
      • Sensor Events
      • Speech and NLP
      • Arduino Backpack
      • System
      • Slam
    • 📗Python Definitions
  • Python Projects
    • 🔮MistyGPT
    • 🚨Misty Intruder Alert
    • 📺Conference Assistant
    • 🏷️QR code detector
    • 🕵️‍♂️Misty follow human
    • 👋Misty wave back
    • 🖲️Misty OA
    • 🌐Get weather
    • 🚚Misty Delivery
    • 🫂Motivational Misty
    • 🖼️Misty Museum Guide
    • 🎃Who for Halloween
  • ARDUINO
    • ♾️Arduino Backpack
    • 🦎Arduino
    • 🔧Arduino Lessons
      • 🔌Arduino to Misty
      • ➕Misty to Arduino
  • ARDUINO PROJECTS
    • 🛠️Misty Tracker
    • 🦾Misty Arm
  • Community Projects
    • 🌤️Misty weather forecaster
  • HARDWARE EXTENSION
    • ⚙️Arduino breadboard support
    • 🦾Misty arm
    • 🥤Tin holder
  • Resource Database
    • 📁Image files
    • 📁Audio files
    • 📁Languages
    • 📁Known objects
    • 📁NLP Actions
    • 📁Action Commands
    • 📁ChatGPT PDF files
    • 📁AR Tag Dictionary
    • ⚙️Technical Specifications
Powered by GitBook
On this page
  • Aim
  • Face Recognition
  • Challenge 1: Remember me?
  • Challenge 2: Recognize more people
  • Object Detection
  • Challenge 3: What's that over there?
  • Audio Localization
  • Challenge 8: Marco Polo
  • Up Next 👇
  1. Python
  2. Python Lessons

Lesson 5: Expand awareness

Aim

The aim of this lesson is to learn how to make Misty more aware of her surroundings using event-based skills with face and object recognition, as well as audio localization. By the end of the lesson, you will be able to build more complex skills in Python for your robot character.

Face Recognition

Challenge 1: Remember me?

Now that you've explored how to make Misty more aware of her environment and respond to human touch in Python, you can also make her recognize faces using her face recognition capabilities just like in Blockly. If you've trained Misty to recognize your face in Lesson 6: Face recognition you don't need to repeat the process for Python, your FaceID is already in her memory.

The amazing thing about programming is that once you have learned a way to solve a problem if you you can use the same solution for similiar problems. So even for the face recognition event the structure of the code for the function and for the event is the same as for the bump and touch sensors events with one exception. Some event functions require you to first enable the service you are using, in this case we need to use the misty.start_face_recognition API call to enable the service for the face recognition event.

In order to distinguish faces in Python you can write the FaceID name in the “Label” parameter and build your conditions with sequences as shown in the example below. Give it a try!

from mistyPy.Robot import Robot
from mistyPy.Events import Events
import time

misty = Robot()
misty.start_face_recognition()

def recognized(data):
    print(data)  
    if data["message"]["label"] == 'Simone':
        misty.speak("Hi Simone!", 1)
        misty.play_audio("s_Awe.wav", 50)
        misty.transition_led(0, 255, 0, 255, 127, 0, "TransitOnce", 1000)
    for i in range(2):
        misty.move_arms(80, -80, 50, 50)
        time.sleep(1)
        misty.move_arms(80, 0, 50, 50)
        time.sleep(1)
    elif data["message"]["label"] == 'Denis':
        misty.speak("Hi Denis!", 1)
        misty.play_audio("s_Awe2.wav", 50)
        misty.transition_led(255, 0, 0, 0, 0, 255, "TransitOnce", 1000)
    for i in range(2):
        misty.move_arms(80, -80, 50, 50)
        time.sleep(1)
        misty.move_arms(80, 0, 50, 50)
        time.sleep(1)
    else :
        misty.change_led(0, 0, 0)

misty.register_event(event_name='face_recognition_event', event_type=Events.FaceRecognition, callback_function=recognized, keep_alive=False)
misty.keep_alive()

Challenge 2: Recognize more people

If you want Misty say hello to everyone she already recognizes you can generalize the code using event messages that we covered in the previous lesson. This will not only compact your code, but open up a new realm of interactions that you can create. Try editing the previous code using the example below.

from mistyPy.Robot import Robot
from mistyPy.Events import Events
import time

misty = Robot()
misty.start_face_recognition()
def recognized(data):
    print(data)  
    misty.speak("Hi " + data["message"]["label"], 1)
    if data["message"]["label"] == 'Simone':
        time.sleep(1)
        misty.play_audio("s_Awe.wav", 50)
        misty.transition_led(0, 255, 0, 255, 127, 0, "TransitOnce", 1000)
    elif data["message"]["label"] == 'Denis':
        time.sleep(1)
        misty.play_audio("s_Awe2.wav", 50)
        misty.transition_led(255, 0, 0, 0, 0, 255, "TransitOnce", 1000)
    else :
        misty.change_led(0, 0, 0)
    for i in range(2):
        misty.move_arms(80, -80, 50, 50)
        time.sleep(1)
        misty.move_arms(80, 0, 50, 50)
        time.sleep(1)

misty.register_event(event_name='face_recognition_event', event_type=Events.FaceRecognition, callback_function=recognized, keep_alive=False)
misty.keep_alive()

Object Detection

Challenge 3: What's that over there?

Well done so far! You can now make Misty recognize you and your friends, this is an essential element for robots to be able to establish and remember relationships just like us humans. But it doesn't stop there, you can even have her recognize objects like a TV, Toothbrush, Bicycle and much more. See Known objectsfor more information. Using the same event skill structure as for face recognition and the example below, try building a skill where Misty tells you what object she sees.

from mistyPy.Robot import Robot
from mistyPy.Events import Events
import time

misty = Robot()
misty.start_object_detector()
def recognized(data):
    print(data)  
    misty.speak("Oh sweet, I think I see a" + data["message"]["description"], 1)
    if data["message"]["description"] == 'person':
        time.sleep(2)
        misty.play_audio("s_Awe.wav", 20)
        time.sleep(1)
        misty.speak("I love humans, they are my best friends")
        misty.transition_led(0, 255, 0, 255, 255, 0, "TransitOnce", 1000)
    elif data["message"]["description"] == 'tv':
        misty.speak("Lets watch some sci-fi robot movies together")
        misty.transition_led(255, 0, 0, 255, 127, 0, "TransitOnce", 1000)

misty.register_event(event_name='object_detection_event', event_type=Events.ObjectDetection, callback_function=recognized, keep_alive=False)
misty.keep_alive()

Audio Localization

Challenge 8: Marco Polo

Besides recognizing faces and objects, you can also teach Misty to understand where your voice or a sound is coming from using audio localization. This is set up like any other event using the source tracking event together with the misty.start_recording_audio API. The tricky part about setting up an audio localization event is using the audio data to tell Misty in which direction she should move her head. To solve this you can set a variable called doa (degreeofarrival). Since Misty will create an audio recording, you will need to delete it once the event is complete. To do this, you can set up sensor event with one of her bumpers. Let's try playing the game Marco Polo using audio localization together with a face recognition event!

from mistyPy.Events import Events
from mistyPy.Robot import Robot

misty = Robot()
audio_file_name = "deleteme.wav"

def bumper_press(event):
    if(event["message"]["sensorId"] == "bfr" and event["message"]["isContacted"] == True):
        misty.stop_recording_audio()
        misty.delete_audio(audio_file_name)
        misty.unregister_all_events()
        print("finished")

def source_tracking(event):
    doa = event["message"]["degreeOfArrivalSpeech"]
    # Calculate the head angle based on the DOA (adjust the scaling factor as needed)
    head_angle = 90 - doa  # Subtract 90 degrees to center the head
    # Set the head movement duration and units as needed
    movement_duration = 1.0  # You can adjust the duration as needed
    movement_units = "degrees"
    # Move the head towards the sound source
    misty.move_head(0, 0, head_angle, duration=movement_duration, units=movement_units)
    print(f"DOA: {doa}, Head Angle: {head_angle} degrees")

def recognized(data):
    print(data)  
    if data["message"]["label"] == 'yourname':
        misty.speak("Polo")
        misty.transition_led(0, 255, 0, 255, 127, 0, "TransitOnce", 1000)

# Register for the FaceRecognition event
misty.register_event(event_name='face_recognition_event', event_type=Events.FaceRecognition, callback_function=recognized, keep_alive=True)

misty.register_event(event_type=Events.BumpSensor, event_name="bump pressed", keep_alive=True, callback_function=bumper_press)
misty.register_event(event_type=Events.SourceTrackDataMessage, event_name="some name", keep_alive=True, callback_function=source_tracking, debounce=1000)
misty.start_recording_audio(audio_file_name)
misty.start_face_recognition()
misty.keep_alive()

Last updated 1 year ago

Up Next

🐍
👀
👇
🔗Lesson 6: Compact code