This VR/Leap Motion project is the continuation of some previous research I’ve done into accessibility in games, and specifically how to apply universal design principles to game interfaces. In other words, how do we make controllers and other user devices that are designed for gamers of all abilities?

​

Too often there’s a division between standard/norm but inaccessible vs. accessible but non-standard. This is largely where we’re at in games right now. Standard controllers are everywhere and mostly inexpensive, but can be really difficult to use if you don’t have peak fine motor control in your hands and fingers. Specialized controllers exist for gamers who have difficulties with standard controllers, but tend to be in one of two categories; prototypes and custom builds that are not widely available or commercially available at a price tag closer to most game systems than game controllers.

​

Universal design (an architectural principle – read about it here, here, and here) seeks to address this by designing products that are accessibility to people of all (or at least many) abilities. Put very simply, accessibility is a lot more affordable when the non-disabled are lining up to buy in.

​

For this specific project, I used the Leap Motion sensor and the Oculus DK2. The Leap Motion senses hands and fingers so you can use them as inputs. I was specifically using the Leap Motion Orion software which allows you to develop and use the Leap Motion in a VR environment. Development was in Unity (and a special thank you to Bright Shining Star who made the beautiful and free “3 Skyboxes” asset package.)

 

Control Schemes

​

I decided early that I wanted to create a simple game that could be quickly understood and required few actions to use. This would make is easier for players to pick up the game when faced with a new means of input. Also, I had never developed in VR or for the Leap Motion, so simplicity served me just fine! I decided on a wave of enemies, arcade-style game. Think Space Invaders in 3D, but it fell over 90 degrees. The player’s ship needs to blast the incoming baddies with its laser, which I set to fire constantly. That way the player only worries about moving forward, back, and side-to-side.

​

When you enter the level and hold your hands in front of your face (aka in front of your VR headset with mounted Leap Motion) you’ll see polygonal/skeletal hand and wrist avatars appear in front of you. (Honestly, the first time, it’s take your breath away cool.) From here, there are two ways of using the Leap Motion that I decided to focus on. First, you can use the in-game hands like you use real world hands and directly manipulate objects by touching them. Second, you can tell the game to look for specific hand gestures, and to use those as the means of control. Let’s start with the former – using your in-game hands like real hands.

​

I started by using a panel with four buttons (taken from the Leap Motion UI example and tweaked) to control the four directions the ship can move in. This was actually my starting point for this whole project after I saw the Leap Motion for the first time; I wanted to know if I could use an oversized digital version of a game controller to shift the required skills from fine motor finger control to gross motor hand and arm control.

​

I originally had the panel as a giant digital version of a physical controller. Press button and thing happens, release button and thing stops. However, unlike with a physical controller, holding down a button isn’t a matter of pushing against a solid object. Instead you have to hover your hand in just the right position of empty air. So I switched from buttons to toggles. So now its press button thing happens, press button again and thing stops.

I originally had speed control as well, controlled through a slider. However, this wasn’t a feature I incorporated into any other control schemes. I scrapped it for now, but will include it again if I can incorporate it in all control schemes.

​

The panel control worked pretty well. You can even combine to non-opposite directions like you do on a controller (press “Left” and “Forward” to go to the front-left corner). Because of the way I wrote the script, toggling off either direction stops the motion completely, instead of just stopping the direction you turned off. In other words, if the ship is going to the front-left corner and I turn off the left toggle, the ship stops completely instead of still moving forward. However, this can be fixed by reworking how the script tells the ship to stop moving. That will be one of my next updates.

​

The second control method I explored is gesture controls. Leap Motion has some very cool options here, though not all of them are suitable for a universally designed control scheme.  And, wow, there are a lot of potential gestures. Finger extension, finger and palm orientation, pinching actions, distance between the two hands… all of these can be combined in interesting ways. 

​

I kept the four gestures simple. I avoided any sort of finger movement and instead focused on the palm direction of an open hand. Left hand facing forward moves the ship forward, left hand to the side moves left, right hand facing forward moves the ship backwards, and right hand turned to the side moves the ship right. When a gesture stops, or if the hand leaves the field of view, the ship stops moving.

​

Gestures ended up feeling very good. Even though it’s different from other control method I’ve used in the past the four simple gross-motor movements were easy to pick up. There are some gesture recognition issues, but again, the four simple movements do better with this issue than many of the more complicated gestures. As with the control panel, directions can be combined, though this time only in two ways. You can move forward and right, or backwards and left. You cannot move forward and left because they use the same hand, and ditto for backwards and right. If this diagonal motion is something people find useful, reciprocal motions could be moved to the same hands (one hand controls front/back and the other right/left). My current set-up feels more natural than this (to me at least!) but does limit combination opportunities.

​

So, that’s where this project is at! If you have any questions, suggestions, or want to talk about collaboration with your own accessibility work, feel free to reach out to me through the contact info on the home page.

-Ffion