19. Augmented Reality

Augmented Reality (AR) overlays digital content over the physical world. This typically makes use of the camera of the smartphone or desktop computer.

To create an AR experience on A-Frame we most often use the AR.js library.

AR.js

AR.js works by tracking a real-world object and then overlaying digital content over the position of that object in the camera of the device.

To use AR.js in A-Frame, we must include its script. To use the latest version:

<script src="https://raw.githack.com/AR-js-org/AR.js/master/aframe/build/aframe-ar.js"></script>

We also need to use the arjs component in the <a-scene>. For example, for using AR.js to detect barcode markers:

<a-scene 
    embedded
    arjs="sourceType: webcam; 
          detectionMode: mono_and_matrix; 
          matrixCodeType: 3x3;">

AR.js can track three types of real-world objects:

Markers
Images
(Geo) locations

Marker-based AR

Markers are images that follow a pre-defined structure that makes it easier for the detection algorithm to recognize and identify them.
AR.js supports three types of markers

Default markers (Hiro and Kanji) are pre-defined shapes that AR.js recognizes.
These are useful mostly for quickly testing something. Figure 1 shows the images associated with these markers.
Pattern markers are custom images that we can create. These still follow a predefined structure but are customizable. The AR.js documentation includes a link to an online app that allows us to create these customized shapes. Figure 2 shows an example of a marker that has been customized with the shape of an alligator.
Barcode markers are pre-defined sets of matrix markers. Each of these markers has a number associated with it (the most common set has values from 0 to 63). The images associated with these makers can be found at https://github.com/nicolocarpignoli/artoolkit-barcode-markers-collection. Figure 3 shows an example of the barcode marker with value 0.

Figure 2 - Custom marker with the shape of an alligator (by Maria Duarte - https://github.com/MdCDuarte).

How to use markers

In an A-Frame scene, markers are invisible entities that show their inner content when the corresponding marker image is visible on the device's camera.

For example, to overlay an image on the hiro marker we would add the following to our scene:

<a-marker preset="hiro">
    <a-image src="#tron1" width="1" height="1" rotation="-90 0 0"></a-image>
</a-marker>

The <a-marker> primitive represents the marker to be detected by AR.js. By using the preset attribute we are configuring this marker as a default marker -- in this case the hiro shape.
The contents (<a-image>) will not be visible, unless the device is able to "see" the hiro shape through the camera. The contents can be any 3D content, including images, video, 3D models, other primitives, etc.

If you load the following example on your smartphone (2000-ar-marker-01-simple) and point it to the Hiro marker above you should see an image overlaying the marker.

Example `2000-ar-marker-01-simple` [new tab ] [source ]

The other types of markers (pattern and barcode) are used in a similar manner. For the pattern type, we need to specify the type="pattern" attribute and the URL of the file that represents the pattern (generated by the web application for creating the customized shapes).

<a-marker        
    type="pattern"
    url="https://raw.githubusercontent.com/MdCDuarte/ar-3d-modles/main/Pattern-Silhouetts/pattern-aligator.patt"
>
    <a-image src="#tron3" width="1" height="1" rotation="-90 0 0"></a-image>
</a-marker>

Image tracking AR

TBD

Location-based AR

TBD

Tangible Interaction

AR.js can be used to create Tangible User Interfaces (TUIs) in different ways. One approach is to create a smartphone-based tangible VR experience where you put the smartphone into a VR enclosure and use the camera of the smartphone to detect physical objects that have markers on them. These physical objects can then be virtually represented inside the virtual scene.

Smartphone-based Tangible VR/AR

Example `2000-tui-book-ar` [new tab ] [source ]

Table-based TUI

Example `2000-ar-marker-table` [new tab ] [source ]

:::

References

AR.js - Augmented Reality on the Web
- https://ar-js-org.github.io/AR.js-Docs/

Web-based Virtual Reality Environments