Vision

One of the most awesome features in Misty is the customizability of her display. This allows her to show different facial expressions and visual information in the form of images, videos and text. Misty's display resolution is 480x272 pixels. The size of the image is calculated from the top right point, so in case the image is larger in size, Misty will try to resize the image to fit the screen, if the image is too narrow or too wide it will be cut. The display can reproduce a wide range of default Misty's eyes and actions. You can find a library of the default images in Image files

There are a few limitations to consider when it comes to using new images and video files:

• The maximum size of a video file is 6 MB

• The video file types supported are .MP4 or .wmv

• The maximum size of an image is 3 MB

• The images file types supported are .jpg, .jpeg, .gif, .png

In the 'Vision' tab in Misty Blocks, you will be able to explore all of her blocks related to vision such as taking images and videos, displaying them and recognizing faces and objects.

visual blocks

visual blocks

Take picture

take picture block

With this block, Misty can take a picture with her RGB camera.

In order to download your image and re-upload it with a custom file name, we suggest to use the Take Photo section in Vision under the Explore tab in Misty Studio.

take photo

There are two different options: RGB camera and Wide-Angle camera.

The former takes a color photo in 4K. You can either choose to download it or display it.

The latter is a feature in Misty II Pro that allows you take a black-and-white photo with the wide-angle camera on Misty's Occipital Structure Core depth sensor. You can also download it or display it.

Images

images blocks

With the display image block you can explore the wide range of Misty's expressions found in Image files. You can also choose the opacity of the image. Its range is 0 to 1, where 0 means all black and 1 means picture with no opacity.

The upload image block allows you to add new custom expressions that you've designed or regular images. To save an image to Misty's memory and access it in the display image block, you will first have to run the upload image block.

With the delete image block you can easily remove any file you've uploaded from Misty's memory. Default images can't be deleted.

Text

text block

Misty has also the possibility to display text. Due to the size of Misty's display you can put on average 80 characters. In order to remove the text you can run this block with an empty field.

Video recording

video recording blocks

Misty can record high-quality videos with her cameras.

With the start video recording block, you can write the name of your video in the text field, select the duration and choose the quality. Misty has a wide range of video quality, depending on your needs: 3840x2160, 1920x1080, 1280x960, 640x480, 320x240.

The second block stops Misty's video recording.

The third block works in the same way as the delete image block.

The upload video block also works in the same way as a upload image block. It allows you to access your computer and choose video files to upload, remember that you have to run this block first for the video to show up your list of videos.

Video asset

video asset blocks

In order to play or delete the video you uploaded you will need to use these blocks.

Face detection

face detection blocks

When Misty uses face detection, she uses algorithms to identify and locate human faces in front of her. This block is useful when your skill is supposed to count how many people there are in front of Misty and interact with her. The first block starts face detection, the second block stops it.

You can also try out the face detection feature in the Vision tab under Explore.

Face training

face training blocks

This is a very popular group of blocks in Misty because they allows her to learn the faces and names of the individual she interacts with. During face training, Misty takes several pictures of your face with her RGB camera. She then attempts to create a face matrix that she can use to recognize you in the future.

With the start face training block you can train Misty to recognize a face and associate it with a FaceID. To create your Face ID all you need to do is write your name in the text field, run the block and wait 30 seconds for her to learn your face. The chest LED will turn green when your face is memorized.

The second block stops the face training, if you don't use it Misty will continue to look for more faces to learn.

The third block gives you access to all the learned faces and the last block clears them.

You can also use the face training button in the Vision tab under Explore. During face training Misty will guide you with messages at the bottom. For instance, if it's too dark you are asked to move into the light.

Face recognition

face recognition blocks

After Misty has learned your face, she can recognize you any time you run the start face recognition block. To get information about the face that Misty is currently detecting you can go to the Face Recognition section in the Vision tab under Explore. There you you have several labels tall tell you the FaceID she identified and the position of the face.

If face recognition is running on the robot, and a previously trained face is recognized, the label value is the name previously assigned to that face. The label value is unknown_person if an untrained/unknown face is detected.

Object detector

object detector blocks

The start object detector can give information about the type and position of objects.

Misty has a list of pre-trained objects that she can recognize:

person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic_light, fire_hydrant, stop_sign, parking_meter, bench, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports_ball, kite, baseball_bat, baseball_glove, skateboard, surfboard, tennis_racket, bottle, wine_glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot_dog, pizza, donut, cake, chair, couch, potted_plant, bed, dining_table, toilet, tv, laptop, mouse, remote, keyboard, cell_phone, microwave, oven, toaster, sink, refrigerator, book, clock, vase, scissors, teddy_bear, hair_drier, and toothbrush.

Please Note: All the functions related to object detection and face training must be used together with their associated events. If we just start the object detection and don't use that information in another block the robot will learn it and forget it the moment after, and we can't build anything on it.