CS 291A,
Assignment 4, Option: Interactive teapots

Change Log:
...

For this assignment, you will program a "magic lens" application that augments the view from a mobile device or camera with 3D graphics. You can use a MixedReality library of your choice for this assignment.

Starting with a simple Image Target Tracker sample application, you will add functionality step by step to create an interactive mobile AR application. The ultimate result of this assignment will be an AR application that overlays 3D teapots and allows a user to interact with the AR environment to "pick up" and move the teapots on the marker surface.

1. Starting with Planar Image Target tracking

Start with basic planar image target tracking. In the case of OpenCV, you can use the checkerboard pattern as your image target, all other libraries that you'll likely be using have their own image target support.

Here are a few starting points for various platforms (but start with whatever code base you can find and find suitable):

ARKit: You can start with the built-in "Augmented Reality App" template and then follow for example this video for setting up basic image target tracking.

ARCore: Here is some information on image targets in AR Core:
https://developers.google.com/ar/develop/java/augmented-images/guide
And here is a step-by-step codelab:
https://codelabs.developers.google.com/codelabs/augimg-intro/index.html

Vuforia: We already did a simple example in Vuforia and Unity in our very first assignment exercise. Here is a youtube video that shows how to implement raycasting on various objects. Vuforia native on Android or IOS should be equally straightforward.

8th Wall: Here is a starter example for Image Targets: https://www.8thwall.com/8thwall/flyer-aframe

If you found other resources for your particular platform, please post them on Slack!

2. Adding Multiple Teapots

Your image targets establishes a coordinate system. For example, you may have (0,0,0) at the center of the image target (i.e. the picture). Your object is rendered at this position by default. We would like to make the object location dynamic, and have it based on some sort of user input.

You should test out this step by creating objects at various positions as parameter. Make sure that your objects move appropriately when you change the position.

Use this functionality to create at least four objects (e.g. teapots, maybe in different colors) on the marker. Place them at various locations along one side of the marker, as shown in the image below:

3. Creating a UI

Now that we can dynamically change the position of an object, we can create a UI experience to interact with the object. Our goal of this step is to allow to user to "pick up" an object and "put it down" in a different position. The method for "picking up" objects is to move the phone/camera closer and closer to the object until an invisible cursor in front of the image plane intersects with the object. Once the object gets picked up it "sticks" to the phone (as if it was velcroed to the camera) independently of the viewing angle until it is placed back down on the marker (such as in the image below)

To "put down" an object, you simply move the phone sufficiently close to the image marker. If that space is unoccupied by another object then the object you are holding will be placed there.

To do this, we can use a very common strategy in graphics and AR and assume a "bounding box" around the object. By specifying a space around the object (usually a cube or a sphere) that tightly encompasses the entire object, we can deem anything that intersects with this bounding box as something that hits the object. This simplifies attempting to find intersections with complex objects (such as a teapot). Once a bounding box is determined, it is very simple to determine if a point lies inside (i.e. intersects the object) or outside the bounding area. Given the location of an object and the location of the camera, you have all the information needed to determine intersections with bounding boxes.

For OpenCV solutions: Your camera may not be mobile, and your tracking target (i.e. checkerboard pattern) might not be tracked robustly enough to get close to the teapots. Instead, it's ok if you only implement the simple ray-based selection techniques from 4. below.

4. Enhancing the UI

Take into account orientation: When you drop the teapot down on the ground make it sit flat on the ground but leave its orientation determined by your camera pose, i.e. you can put down teapots in various rotational arrangements. When picking up the teapot, stick the teapot to your mobile device or camera so that you see it directly from above (with it's base parallel to the image plane), but also keep it's original yaw orientation.

The idea is to avoid teapot discontinuities/jumps but, via the snap to the parallel plane, give some feedback on if you actually 'caught' the teapot or not.

Also, implement picking up and dropping off teapots on mouse clicks / finger taps. You will need to also determine whether the point you are touching intersects an object. You will have to relate your 2D screen point (where you are touching) to the 3D coordinates of the ground plane and the bounding volume to determine if you are "touching" the object (i.e. if you are intersecting the ground within the bounding box).

Optional: Implement moving teapots around on the ground plane with your finger: You can create an action from touching the screen with an onTouchEvent function, which is likely provided by your toolkit.

Submission