User Makeblock Computer Vision Extension for App Inventor

MIT App Inventor is a drag-and-drop block-based editor which can make Android Apps. Makeblock provides a simple computer vision plugin for color detection, face detection, and feature analysis (using online API).

Please also consider our mBot extension which allows you to control mBot robots.

Add the mBot extension to App Inventor

You need an App Inventor website that supports extensions. One example is Login and start a new project.


In the “Extension” group of the left panel, click “Import Extension” link, then select “URL”. Paste the following link to the text box:

Then click “import”.

Color Detection

Here is a sample program for color detection:

Screen Shot 2016-06-03 at 10.15.10 AM

The common process of color detection is: “Open the camera (at the first time)” -> “Take a picture” -> Analyze Color of the Picture” -> “Get RGB Values” -> “Take a picture again”.

When “Button 1” is clicked, the program opens the camera, then take a picture immediately.

Since taking the picture needs some time, the following steps are put into “After Picture Taken” blocks. In that block, “Analyze Color of the Picture” is firstly called  – this is needed before getting any color values. “Analyze Color of the Picture” calculates the mean color of the whole image, while you can get the color around a certain point using this command:

Screen Shot 2016-06-03 at 10.59.51 AM

Here the range of x, y is between 0-1. That means (0,0) is the left top corner of the image; (0.5, 0.5) is the image center, and (1,1) is the bottom right corner.

After analyzing the picture, I displayed the red, green, blue values of the picture to three text labels.

After all that, I took a picture again, which means “After Picture Taken” blocks will be called over and over.

Face Detection

Here is a sample program:

Screen Shot 2016-06-03 at 10.15.37 AM

When Button1 is clicked, I open the camera and start face detection. This is needed for all operations involving face detection.

“After face detection” block is called every time there’s a result about face detection. In this block, you can get number of faces using “Number of Faces” block; you may get the center and size of the faces detected.

Topic Analysis using Microsoft Cognitive Service

This plugin uses Microsoft’s Cognitive Service for topic analysis. Before using any of the following functions, you need to login and register an API account in Microsoft’s website:

After logged in, click “Request new trials”:

Screen_Shot_2016-06-03_at_11_18_29_AM 2

Then select “Computer Vision – Preview” to add computer vision API to your subscription list.


After that, you can check your API key by clicking “show” link. Copy that key, because it is needed in the following steps.


Here is a sample program of topic analysis:

Screen Shot 2016-06-03 at 10.30.40 AM

First, when Button1 is clicked, open the camera and Prepare computer vision with the API key mentioned before.

After Button2 (take the picture and analysis) is clicked, I take a picture. After the picture is taken, I use “Submit Picture for Computer Vision” to submit the picture to the Microsoft server.

When the result is returned from the server, “After Computer Vision Result” block will be called. Then I can get the following information:

  • DescriptionFromCV: a description in a line of text, such as “a man sitting in front of a chair”
  • Foreground/BackgroundColorNameFromCV: the name of the dominant color of the picture foreground/background. For example “blue”.
  • TagsFromCV: a list of tags describing the picture, such as “person”, “cat”, “desk”.


Posted on