Build your first Android AR app with ARCore and Sceneform in 5 minutes*

In this tutorial, I’ll walk through how to start building an Augmented Reality Android application using Google’s new Sceneform SDK. First off, I’ll cover some quick concepts and then jump into project setup and some code. If you just want to dive in straight away, you can check out my GitHub repo below!

The screenshot below shows what we’ll build today!

Core Concepts

ARCore is Google’s platform for building Augmented Reality experiences. It uses your mobile device for motion tracking, environmental understanding, and light estimation. With these three capabilities, ARCore can build an understanding of the real world around the device. With the mobile device’s position, we as developers can then augment the real world with information, 3D objects, or anything else!

Sceneform is a new SDK built to empower developers to work with ARCore without having to learn the intricacies of 3D graphics programming and OpenGL. ARCore was initially released with OpenGL samples, and as someone with very basic graphics programming knowledge, I found the learning curve to be super steep. With Sceneform, you can easily write in Java (or Kotlin) just like you would when building a normal Android app. The official samples for ARCore and Sceneform further illustrate this.

Getting Started

To get started, you’ll need Android Studio 3.1 or higher. We want to create a new project with Kotlin support targeting API 24: Android 7.0, so let’s start with an empty activity for now.

In our app/build.gradle file, we can add the dependency for Sceneform— remember to click “Sync Now” to update the project. Although we’ll be writing this tutorial in Kotlin, Sceneform uses some language constructs from Java 8, so we’ll need to explicitly add support since our minimum API is less than 26.

Since this is an augmented reality app, we’re going to have to ask the user for access to the camera. We also need an AR permission, which designates that this app requires ARCore. Finally, there’s a specific AR meta tag used by the Play Store to filter out applications for users who do not have an ARCore-enabled device.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.gathergood.calum.ardemo">

    <uses-permission android:name="android.permission.CAMERA" />
    <uses-feature
        android:name="android.hardware.camera.ar"
        android:required="true" />

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:roundIcon="@mipmap/ic_launcher_round"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">

        <meta-data
            android:name="com.google.ar.core"
            android:value="required" />

        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>

</manifest>

Adding our Model

To make life easier, and to allow us to import our own 3D assets, we’ll also include the Sceneform plugin for Android Studio. You can install this by selecting preferences, plugins, browse repositories, and search for Google Sceneform Tools (Beta)

The first thing we want to do is get a 3D model to use—if you don’t want to make your own, you can head on over to Poly and grab one for free (just remember to credit the creator!). The Sceneform plugin supports OBJ, FBX and glTF file formats. Then, it’s just a case of creating a sampledata/ directory within your app and adding your object file.

Note that the purpose of this folder is for design time only, and these resources won’t be added to the final APK. The next step is right clicking your downloaded object and selecting “Import Sceneform Assert”. For this app, I’m going to pick the Earth asset created by Google.

After this gradle task has finished, you’ll find a reference to your object in the app level gradle file and a .sfb file inside the assets directory. This Sceneform Binary asset is what we’ll use inside our app. The Sceneform Android Studio plugin lets us view a realtime preview of our newly created asset.

Time to Code

The next step is to add the Sceneform fragment to our layout. You can just replace the default textview with this element. We’ll also add a Floating Action Button (FAB) to enable us to add our model to the real word.

<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <fragment
        android:id="@+id/sceneform_fragment"
        android:name="com.google.ar.sceneform.ux.ArFragment"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toTopOf="parent" />

    <android.support.design.widget.FloatingActionButton
        android:id="@+id/floatingActionButton"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginBottom="8dp"
        android:layout_marginEnd="8dp"
        android:layout_marginStart="8dp"
        app:layout_constraintBottom_toBottomOf="@+id/sceneform_fragment"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:srcCompat="@drawable/ic_explore_white_24dp" />

</android.support.constraint.ConstraintLayout>

We want our users to be able to add a model only when ARCore has detected a scene. In order to do this, we’ll only display the FAB when a suitable plane has been detected. In the official Google Codelab, they achieve something similar by drawing a custom Pointer on the screen when a model can be added to the scene.

The structure of our main activity is as follows:

  1. Detect a suitable plane using ARCore onUpdate
  2. Show the user the Floating Action Button (FAB)showFab
  3. On click of the button load in our model addObject
  4. Use the .sfb file to create a renderable object placeModel
  5. Attach our renderable to the scene and display to the user addNodeToScene
class MainActivity : AppCompatActivity() {

    private lateinit var arFragment: ArFragment

    private var isTracking: Boolean = false
    private var isHitting: Boolean = false

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        arFragment = sceneform_fragment as ArFragment

        // Adds a listener to the ARSceneView
        // Called before processing each frame
        arFragment.arSceneView.scene.addOnUpdateListener { frameTime ->
            arFragment.onUpdate(frameTime)
            onUpdate()
        }

        // Set the onclick lister for our button
        // Change this string to point to the .sfb file of your choice :)
        floatingActionButton.setOnClickListener { addObject(Uri.parse("NOVELO_EARTH.sfb")) }
        showFab(false)

    }

    // Simple function to show/hide our FAB
    private fun showFab(enabled: Boolean) {
        if (enabled) {
            floatingActionButton.isEnabled = true
            floatingActionButton.visibility = View.VISIBLE
        } else {
            floatingActionButton.isEnabled = false
            floatingActionButton.visibility = View.GONE
        }
    }

    // Updates the tracking state
    private fun onUpdate() {
        updateTracking()
        // Check if the devices gaze is hitting a plane detected by ARCore
        if (isTracking) {
            val hitTestChanged = updateHitTest()
            if (hitTestChanged) {
                showFab(isHitting)
            }
        }

    }

    // Performs frame.HitTest and returns if a hit is detected
    private fun updateHitTest(): Boolean {
        val frame = arFragment.arSceneView.arFrame
        val point = getScreenCenter()
        val hits: List<HitResult>
        val wasHitting = isHitting
        isHitting = false
        if (frame != null) {
            hits = frame.hitTest(point.x.toFloat(), point.y.toFloat())
            for (hit in hits) {
                val trackable = hit.trackable
                if (trackable is Plane && trackable.isPoseInPolygon(hit.hitPose)) {
                    isHitting = true
                    break
                }
            }
        }
        return wasHitting != isHitting
    }

    // Makes use of ARCore's camera state and returns true if the tracking state has changed
    private fun updateTracking(): Boolean {
        val frame = arFragment.arSceneView.arFrame
        val wasTracking = isTracking
        isTracking = frame.camera.trackingState == TrackingState.TRACKING
        return isTracking != wasTracking
    }

    // Simply returns the center of the screen
    private fun getScreenCenter(): Point {
        val view = findViewById<View>(android.R.id.content)
        return Point(view.width / 2, view.height / 2)
    }

    /**
     * @param model The Uri of our 3D sfb file
     *
     * This method takes in our 3D model and performs a hit test to determine where to place it
     */
    private fun addObject(model: Uri) {
        val frame = arFragment.arSceneView.arFrame
        val point = getScreenCenter()
        if (frame != null) {
            val hits = frame.hitTest(point.x.toFloat(), point.y.toFloat())
            for (hit in hits) {
                val trackable = hit.trackable
                if (trackable is Plane && trackable.isPoseInPolygon(hit.hitPose)) {
                    placeObject(arFragment, hit.createAnchor(), model)
                    break
                }
            }
        }
    }

    /**
     * @param fragment our fragment
     * @param anchor ARCore anchor from the hit test
     * @param model our 3D model of choice
     *
     * Uses the ARCore anchor from the hitTest result and builds the Sceneform nodes.
     * It starts the asynchronous loading of the 3D model using the ModelRenderable builder.
     */
    private fun placeObject(fragment: ArFragment, anchor: Anchor, model: Uri) {
        ModelRenderable.builder()
                .setSource(fragment.context, model)
                .build()
                .thenAccept {
                    addNodeToScene(fragment, anchor, it)
                }
                .exceptionally {
                    Toast.makeText(this@MainActivity, "Error", Toast.LENGTH_SHORT).show()
                    return@exceptionally null
                }
    }

    /**
     * @param fragment our fragment
     * @param anchor ARCore anchor
     * @param renderable our model created as a Sceneform Renderable
     *
     * This method builds two nodes and attaches them to our scene
     * The Anchor nodes is positioned based on the pose of an ARCore Anchor. They stay positioned in the sample place relative to the real world.
     * The Transformable node is our Model
     * Once the nodes are connected we select the TransformableNode so it is available for interactions
     */
    private fun addNodeToScene(fragment: ArFragment, anchor: Anchor, renderable: ModelRenderable) {
        val anchorNode = AnchorNode(anchor)
        // TransformableNode means the user to move, scale and rotate the model
        val transformableNode = TransformableNode(fragment.transformationSystem)
        transformableNode.renderable = renderable
        transformableNode.setParent(anchorNode)
        fragment.arSceneView.scene.addChild(anchorNode)
        transformableNode.select()
    }
}

Final Product

And that’s it! In less than 150 lines of code, we have our very first augmented reality app. I’ve added comments where applicable in the MainActivity.kt. Please feel free to get in touch with me if you have any questions whatsoever. Hopefully you can use this as a starting point for your AR experience on Android. Thanks for reading!

Further reading

If this quick tutorial has piqued your interest, I’d recommend reading the below links to find out more about Android and AR.

Announcing ARCore 1.0 and new updates to Google Lens

ARCore Overvew

Sceneform Overview

Building ARCore apps using Sceneform – Hari Vignesh Jayapalan

Augmented Images with ARCore and Sceneform – Calum Gathergood

Discuss this post on Hacker News and Reddit.

Avatar photo

Fritz

Our team has been at the forefront of Artificial Intelligence and Machine Learning research for more than 15 years and we're using our collective intelligence to help others learn, understand and grow using these new technologies in ethical and sustainable ways.

Comments 0 Responses

Leave a Reply

Your email address will not be published. Required fields are marked *