Free floating VR menu in jMonkeyEngine with Oculus Rift

The recommended way of displaying GUI and menus in VR is to not have it static, tied to the ”screen”. This despite the first thing you're being greeted by when using the Oculus Rift is a big warning screen pasted on your face.

The idea behind a free-floating GUI is that it's more natural and adding to the comfort of the user (which is a big thing in VR!). When it comes to menus this also has the benefit of making the user able to do selections without any other controller than the HMD itself.

In this tutorial we'll create a menu that lets the user select menu items simply by looking at them for a certain amount of time. I've chosen to write it similar to Packt's Cookbooks since it's something I'm familiar with and it will hopefully make owners of the jMonkeyEngine 3.0 Cookbook feel right at home!

Disclaimer: The API for the Oculus Rift plugin is in no way final and it might change after this tutorial is written.

Preparation

We'll need the Oculus Rift plugin for jMonkeyEngine. How to download and set it up can be found here: https://code.google.com/p/jmonkeyengine-oculus-rift/

You can also find the full code for this tutorial in the "examples" folder.
This tutorial won't explain how the OVRApplication works, for this it's recommended to check out the project site's wiki. The tutorial will be implemented in two classes. One application class containing the basics and an AppState which will contain all the menu specific code.

Implementation

We'll start with the application class which is implemented in 5 steps.

1. For this example we start by creating a class that extends OVRApplication.
2. In the simpleInit method we start by removing the default GUI elements like this:

setDisplayFps(false);

setDisplayStatView(false);

3. Next we set up the GUI node for manual positioning and scale it somewhat.

ovrAppState.getGuiNode().setPositioningMode(OculusGuiNode.POSITIONING_MODE.MANUAL);

ovrAppState.getGuiNode().setGuiDistance(0.5f);

ovrAppState.getGuiNode().setGuiScale(0.5f);

4. Like in the example application for OVRApplication we create an observer node and assign the StereoCameraControl to it.

Node observer = new Node("Observer");

observer.setLocalTranslation(new Vector3f(0.0f, 0.0f, 0.0f));

observer.addControl(ovrAppState.getCameraControl());

rootNode.attachChild(observer);

5. Now we add two lines to the simpleUpdate method to keep the GUI in place.

guiNode.setLocalTranslation(cam.getLocation().add(0, 0, 0.5f));

guiNode.lookAt(cam.getLocation(), Vector3f.UNIT_Y);

Apart from revisiting to add the AppState, that's it for the application class! The basics for the menu will be done in the following 8 steps.

6. We start by creating a class called MenuAppState that extends AbstractAppState.
7. We define a number of colors we'll use to apply to the menu items to indicate their status.

private static ColorRGBA DEFAULT_COLOR = ColorRGBA.DarkGray;

private static ColorRGBA SELECT_COLOR = ColorRGBA.Gray;

private static ColorRGBA ACTIVATE_COLOR = ColorRGBA.White;

8. Its constructor should receive a width, height and the OculusGuiNode, all which it stores in private fields.
9. We also create and store a Vector2f called screenCenter which should be initialized with width * 0.5f and height * 0.5f;
10. In the initialize method we start by creating and storing a Node called menuNode and a Ray called sightRay.
11. We set up a simple unshaded material to apply on the menu items and give it the DEFAULT_COLOR.

Material mat = new Material(app.getAssetManager(), "Common/MatDefs/Misc/Unshaded.j3md");

mat.setColor("Color", DEFAULT_COLOR);

12. Next we create a number of menu items in the form of Quad's. They're set up in a grid and each gets its own clone of the material before being attached to the menuNode.

for(int x = -2; x < 2; x++){

for(int y = -2; y < 2; y++){

Geometry menuQuad = new Geometry("Test ", new Quad(width * 0.25f, height * 0.25f));

menuQuad.setMaterial(mat.clone());

menuQuad.setLocalTranslation(x * width * 0.25f, y * height * 0.25f, 0);

menuNode.attachChild(menuQuad);

}

}

13. Then we attach the menuNode to the guiNode.
14. Now we override the setEnabled method and add some logic so that the menuNode is added to the scenegraph when the AppState is enabled and removed when it's disabled.

if(enabled && !this.isEnabled()){

guiNode.attachChild(menuNode);

} else if (!enabled && this.isEnabled()){

guiNode.detachChild(menuNode);

}

Now we have a basic menu showing but no way of selecting anything. Adding this functionality will consist of an additional 16 steps.

15. First of all we'll add another static field called ACTIVATE_TIME which is a float and set to 5f.
16. In addition we need another float field called timer, a CollisionResults called collisionResults and a Geometry called selectedGeometry.
17. We create a new method called selectItem which takes a Geometry called g as input.
18. Inside, we set selectedGeometry to g and the color of its material to a clone of SELECT_COLOR.

selectedGeometry.getMaterial().setColor("Color", SELECT_COLOR.clone());

19. Lastly we set timer to 0.
20. Now we create another method called unselectItem.
21. Inside, we check if selectedGeometry is not null and if so we set the color of its material to a clone of DEFAULT_COLOR before setting selectedGeometry to null.
22. Next we override the update method and start by setting up the Ray we created in the initialize method. It should originate from the camera.

sightRay.setOrigin(app.getCamera().getWorldCoordinates(screenCenter, 0f));

sightRay.setDirection(app.getCamera().getDirection());

23. We do a collision check between the Ray and menuNode and see if any of the Geometries inside are being looked at. If any of them are we perform the following piece of code to set the Geometry to the selected one.

Geometry g = collisionResults.getClosestCollision().getGeometry();

if(g != selectedGeometry){

unselectItem();

selectItem(g);

}

24. If instead none is being looked at we call unselectItem to null a potentially selectedGeometry.
25. After performing this check we call the clear method on collisionResults.
26. Before doing the last bit in this method we create another (empty for now) method called activateItem.
27. Continuing in the update method we check if selectedGeometry is not null.
28. If it is assigned we increase timer by tpf.
29. Then we perform the following piece of code to make the color brighter of the selectedGeometry.

float interpolateValue = timer / ACTIVATE_TIME;

((ColorRGBA)selectedGeometry.getMaterial().getParam("Color").getValue()).interpolateLocal(SELECT_COLOR, ACTIVATE_COLOR, interpolateValue);

30. Lastly, we check if timer is greater than ACTIVATE_TIME in which case we call the activateItem method.
31. The final thing we need to do is to return to the application class and create an instance of the MenuAppState class.

MenuAppState menuAppState = new MenuAppState(settings.getWidth(), settings.getHeight(), (OculusGuiNode) guiNode);

stateManager.attach(menuAppState);

Explanation

In the beginning of the application class, we telling the OculusGuiNode that we'll manage the positioning of the GUI ourselves. AUTO mode will otherwise always place it in front of the camera and rotate with it. We still want it to be in front of the camera which is why we manually place it there. The difference is that the cameras rotation won't be propagated to the guiNode's translation. We also want the GUI to always face the camera even if it's not strictly necessary for this example.

In the MenuAppState we create a bunch of placeholder menu items in the form of Quads which we align in a grid. In this example we attach and detach the menuNode via the setEnabled method but depending on the rest of the application it might be neater to do it using the stateAttached and stateDetached methods.

In the update method we use raycasting originating from the camera to detect if any of the menu items is in the center of the screen. When this occurs we change the color of the item to indicate this to the user and also begin a counter. We also start interpolating the color of the selected item to indicate that something is happening. When the timer exceeds 5 seconds it triggers an activation of the item. It's not as quick as if one would have used a mouse but controlling methods are different for VR and this enables the user to navigate menus without having an additional controller.

The one thing the example leaves out is what to do when an item is selected. Since the example is using instances of Geometry there's not much to do. A simple approach would be to create a MenuItem class that extends Geometry or even better; add a custom Control to the Geometry which is called through the activateItem method.