How to control your computer using hand gestures

Published by DevicePlus Editorial Team at July 1, 2021

How Controlling Your Computer With an Air Gesture Works

For this guide, we’re going to build on two of our previous guides covering ultrasonic sensors and the HID-project library. The HC-SR05 we’ll be using is a sensor that uses ultrasonic sound waves to detect how close an object is. We can combine this with the HID-project library which allows you to use your Arduino as a mouse and keyboard and send basic commands like pausing media, opening apps like the calculator, or, as we’ll be doing in this guide, muting your system audio.

In order to use this library without an external shield, you’ll need an Arduino that’s based on the ATmega32u4 chipset. For our purposes, we’re going to use an Arduino Leonardo, instead of the usual Arduino Uno, as this board has the necessary chipset, natively supports USB communication, and has a micro USB port which makes it a little easier to find a cable you can use to connect to your computer.

You’ll also need to enable the HID library in order to use it in your project. To do so, open your Arduino IDE and navigate to Sketch > Include library > Manage libraries… then search for “hoodloader”. When you find it, click “Install.” Now, you can use a single line of code in order to include the HID library in your projects, which we’ll show you how to do in the code section below.

What You’ll Need

Since we’re focusing on the Arduino Leonardo, the set up for this project should be relatively simple. However, if you only have an Arduino Uno, you can look into the Hoodloader2 library so you can keep using your existing board. If you want to follow this guide verbatim, however, you’ll need the following:

Arduino Leonardo
Arduino IDE
A micro USB cable
An HC-SR05 sensor: The older HC-SR04 can also work for this project, but it will behave slightly differently so be sure to take a look at the specifics for the sensor you’re using.
A Windows computer: This project can work on other operating systems, but for this guide we’ll be focusing on Windows. If you want to use this device on other platforms, you’ll need to tweak the sketch.

If you don’t already have the Arduino IDE set up on your machine, you can get started with our previous guide here.

The Code

First up, here’s the full code for the sketch that we’ll be using on this project:

#include &lt;HID-Project.h&gt;

void setup() {
  pinMode (6,OUTPUT); //Insert HC-SR-05 with VCC in pin 6
  pinMode (5, INPUT); // Assign pin 5 to Echo
  pinMode (4, OUTPUT);// Assign pin 4 to Trig
  pinMode (2,OUTPUT); // Assign pin 2 to GND.
  Serial.begin(9600); // This will allow you to read the current value of the dial
}

void loop() {
  long duration, cm; // Initialize variables for duration and cm
  
  digitalWrite(6, HIGH); // Power the sensor
  digitalWrite(4, LOW);  // Clear pulse before sending a 10 microsecond ping,
  delayMicroseconds(2); 
  digitalWrite(4, HIGH);
  delayMicroseconds(10);
  digitalWrite(4, LOW); 
  
  duration = pulseIn(5, HIGH); // Detect pulse length from the Echo pin, measured in microseconds
  cm = (duration/2)/29.155;    // Divide duration in half (due to round trip), then convert distance to centimeters (1cm per 29.155 microseconds), assign to cm variable
 
  Serial.print(cm);   // Print distance in cm to serial monitor
  Serial.print("cm");
  Serial.println();

  if (cm &lt; 10 &amp;&amp; cm &gt; 0) {
    Consumer.write(MEDIA_VOLUME_MUTE);
    delay(500);
  }
}

#include <HID-Project.h>

void setup() {

pinMode (6,OUTPUT); //Insert HC-SR-05 with VCC in pin 6

pinMode (5, INPUT); // Assign pin 5 to Echo

pinMode (4, OUTPUT);// Assign pin 4 to Trig

pinMode (2,OUTPUT); // Assign pin 2 to GND.

Serial.begin(9600); // This will allow you to read the current value of the dial

}

void loop() {

long duration, cm; // Initialize variables for duration and cm

digitalWrite(6, HIGH); // Power the sensor

digitalWrite(4, LOW); // Clear pulse before sending a 10 microsecond ping,

delayMicroseconds(2);

digitalWrite(4, HIGH);

delayMicroseconds(10);

digitalWrite(4, LOW);

duration = pulseIn(5, HIGH); // Detect pulse length from the Echo pin, measured in microseconds

cm = (duration/2)/29.155; // Divide duration in half (due to round trip), then convert distance to centimeters (1cm per 29.155 microseconds), assign to cm variable

Serial.print(cm); // Print distance in cm to serial monitor

Serial.print("cm");

Serial.println();

if (cm < 10 && cm > 0) {

Consumer.write(MEDIA_VOLUME_MUTE);

delay(500);

}

Now, let’s go through a few key parts of this code so we can learn how it works. First, and critically important, is the first line.

#include &lt;HID-Project.h&gt;

1	#include <HID-Project.h>

This line tells the Arduino IDE to include the HID library that you added earlier. This library includes a collection of commands that you can now include in your sketches. Without this line, the IDE won’t recognize the commands that we’re going to use to control your computer.

void setup() {
  pinMode (6,OUTPUT); //Insert HC-SR-05 with VCC in pin 6
  pinMode (5, INPUT); // Assign pin 5 to Echo
  pinMode (4, OUTPUT);// Assign pin 4 to Trig
  pinMode (2,OUTPUT); // Assign pin 2 to GND.
  Serial.begin(9600); // This will allow you to read the current value of the dial
}

void setup() {

pinMode (6,OUTPUT); //Insert HC-SR-05 with VCC in pin 6

pinMode (5, INPUT); // Assign pin 5 to Echo

pinMode (4, OUTPUT);// Assign pin 4 to Trig

pinMode (2,OUTPUT); // Assign pin 2 to GND.

Serial.begin(9600); // This will allow you to read the current value of the dial

}

In this section, we’ll assign the pins for the ultrasonic sensor. Pin 6 will supply power to the sensor, so it should be set to output. Pin 4 is the trigger that will send a signal so it’s an output, and pin 5 will receive the reflected signal, so it should be an input. Finally, pin 2 is ground so it’s another output. We’ll also activate the Serial monitor in this step so that you can see what values the sensor is returning.

  long duration, cm; // Initialize variables for duration and cm
  
  digitalWrite(6, HIGH); // Power the sensor
  digitalWrite(4, LOW); // Clear pulse before sending a 10 microsecond ping,
  delayMicroseconds(2); 
  digitalWrite(4, HIGH);
  delayMicroseconds(10);
  digitalWrite(4, LOW); 
  
  duration = pulseIn(5, HIGH); // Detect pulse length from the Echo pin, measured in microseconds
  cm = (duration/2)/29.155; // Divide duration in half (due to round trip), then convert distance to centimeters (1cm per 29.155 microseconds), assign to cm variable

long duration, cm; // Initialize variables for duration and cm

digitalWrite(6, HIGH); // Power the sensor

digitalWrite(4, LOW); // Clear pulse before sending a 10 microsecond ping,

delayMicroseconds(2);

digitalWrite(4, HIGH);

delayMicroseconds(10);

digitalWrite(4, LOW);

duration = pulseIn(5, HIGH); // Detect pulse length from the Echo pin, measured in microseconds

cm = (duration/2)/29.155; // Divide duration in half (due to round trip), then convert distance to centimeters (1cm per 29.155 microseconds), assign to cm variable

In this section of code, we’ll activate the ultrasonic sensor and send out a 10 microsecond ping. Next, we’ll detect how long it takes for the signal to return. With this information, the sensor can determine how far away an object is.

  if (cm &lt; 10 &amp;&amp; cm &gt; 0) {
    Consumer.write(MEDIA_VOLUME_MUTE);
    delay(500);
  }

if (cm < 10 && cm > 0) {

Consumer.write(MEDIA_VOLUME_MUTE);

delay(500);

}

With the distance information we got from the ultrasonic sensor, we can now use a very simple command to mute our computer’s audio. In this code, if the object that the sensor detects is less than 10cm away, the sketch will mute the system audio. This function works as a toggle, so if the audio is already muted, this command will unmute it. With this, you can place your hand roughly over the ultrasonic sensor to mute your computer’s audio.

You might also notice that the if() statement includes a second condition of “cm > 0”. In my testing, I found that occasionally the sensor could get tripped up and return a value of 0. This condition helps unsure that the command doesn’t accidentally get activated. You can also adjust the first factor of 10cm if you need to make your gesture work closer or further away from the sensor.

The Wiring

For this project, the wiring is extremely simple. However, you should also consider how you want to use your gesture in its final form. You can easily mount the ultrasonic sensor on the edge of your monitor or place it on your desk somewhere. I placed mine on the side of my desk with my mouse. To use the gesture, I simply slide my arm towards the right. You can be creative in figuring out how to mount your sensor and that will determine how you wire it up.

For the wiring, you have two main options:

Plug the ultrasonic sensor directly into the Arduino itself. The five pins can be plugged into pins 2 through 6, with VCC plugged into pin 6 all the way to GND plugged into pin 2.
If you want to mount your sensor somewhere further away from your Arduino, use wires to connect the pins on the sensor to the board. The pin configuration shouldn’t change.

Now, your project should be ready to go! You can also try modifying the code to add more gestures at different distance levels, or even activate the gesture based on how long an object is detecting in a certain place. Experiment with the code and explore the full HID library to create your own gestures.