Eliminating the guesswork in VR/AR apps with Observer Analytics

Observer Analytics have a platform coming soon to help VR/AR devs improve their apps using aggregated data on how people are interacting with their experiences. Lucas Toohey, CEO and co-founder, gives the lowdown on analytics in VR/AR and how their platform works.

Observer Analytics charts and heatmaps showing VR activity

Rather than trying to explain the platform myself, Lucas explains it perfectly,

“Observer Analytics is an analytics platform for VR and AR content, allowing developers and content creators alike to easily track and analyze data points around where users look, what they interact with, and how they physically move, just to name a few. We’ve focused on ease of integration, automating the data collection process for core XR metrics to make the integration process easy and save development time.”

Like many good platforms, Observer Analytics sprang out of necessity. In 2016, their founding team was building a social VR app that let eSports enthusiasts watch streams in an interactive environment. Their vision was to “bring the massive amount of 2D content that existed and enhance the viewing experience by adding an interactive 3D layer”. The one difficulty that stood out to their team throughout development was not knowing how users were spending time in their app.

“Because designing for three-dimensions is a new frontier, our design and development decisions were rooted in assumption and we just hoped that our design was intuitive.” — Lucas Toohey on their time before Observer Analytics

It turns out, not having data on how users are spending their time in a VR experience can make it really hard to diagnose issues and improve the experience, as Lucas recalls,

“After launching the app to Steam, we watched our usage and retention metrics stagnate, without any data as to why. This frustration became more prevalent as we spoke with our peers and continued to hear that this was a recurring problem for all developers building VR and AR content. This is when we decided to stop building our app and transition to building Observer Analytics.”

“We’ve spent the last year building a platform specifically for immersive content, with the goal of eliminating the guesswork for all VR and AR content creators via data.” — Lucas Toohey

Their goal is to help developers build more intuitive and engaging experiences by better understanding the behaviour of people in a VR environment.

How it all works

Observer Analytics tracks metrics through drag-and-drop SDKs for Unity and Unreal Engine:

• Unity — they use a single prefab that contains numerous scripts to handle the automatically-tracked metrics
• Unreal Engine — they use a Blueprint class that also contains scripts to handle the built-in events.

They’ve thought about things from a cross-platform perspective which is really promising to see:

“We have SDKs optimized for different hardware/platforms including: Rift, Vive, Windows Mixed Reality, Oculus Go, Daydream, ARKit, etc.” — Lucas Toohey

While they don’t currently do not have open APIs to connect to other data platforms like Google Analytics, Lucas says that feature is coming to their platform soon.

Once you’ve got usage data, you can see that data through graphs and can dig deeper into the data with custom segments and more detailed graphs:

“Our current dashboards have a few dozen predefined visualizations to save teams the time of having to create every graph from scratch, and includes the ability to further dig into data via custom segments and graphing tools.”

The Observer Analytics dashboard

What metrics should VR devs track?

There are three categories of data that Lucas thinks are extremely valuable for VR/AR developers:

• Usage Metrics — “Usage metrics are the standard metrics to track application usage including growth, user retention, session length, etc. With Observer Analytics, developers can now track cross-platform as they push to all major app stores and get these stats delivered right to their inbox.”
• Engagement — “User engagement encompasses the new three-dimensional metrics now present in VR and AR. Observer automatically tracks where users look, physical and virtual movement, and object interactions. These metrics give insight into what objects draw your users attention, where they are spending a majority of their time, and how they use their playspace, for example.”
• Performance — “Optimizing and testing your experience across various headsets and hardware configurations is time intensive. Observer automatically tracks FPS throughout each session and helps identify objects that are potentially causing framerate issues. You can analyze performance by headset, GPU, CPU, RAM, and OS.”

An example of Observer Analytics’ heatmap

Physical movement data

One fascinating thing that developers have been finding value in tracking is physical movement data. Devs have been able to use this to improve how they encourage people to move around in the space:

“Developers are using our play space heatmap to see if users are actively moving around their play space or if they are standing still and relying on a locomotion method, such as teleportation, to move around within the experience. Developers who have noticed that their users are not actively utilizing their play space have begun to make changes to their onboarding flow to encourage more physical movement.”

Onboarding is key and easy to fix with analytics

Introducing people to a VR experience and getting them oriented towards your virtual world and what they can do is definitely a really challenging task. Lucas has found that many developers have been struggling with this very thing but finds that analytics can provide some easy info to help developers improve this part of their app:

“One of the biggest issues we see across all application types is onboarding confusion. Because there isn’t a set standard for controls or UX layout, applications need to find a balance between spending the time to educate users and making sure they don’t get bored or restless in a tutorial.”

“Your onboarding process is one of the most important elements of your experience and is also one of the easiest areas to optimize with analytics.” — Lucas Toohey

The big privacy question

There have been some pretty big uproars around tracking people in VR environments in the past! Oculus got into hot water for tracking users on their platform a while back, so one question I had for Lucas that I was particularly curious about was — is there a right and a wrong way to approach analytics in VR?

I was pretty glad to hear that Lucas and his team have thought about this and are designing their platform with a particular focus that doesn’t focus on targeting individuals for marketing purposes. He explains,

“Privacy is important across any medium and we believe that the inputs now present in VR open up a new discussion around how data should be collected, analyzed, and stored. There is a distinct difference between product teams using data to improve design and iterate upon an experience vs marketers using data to target individuals.”

Their current focus is on aggregating data to provide the dev team with a look at trends, rather than identifying individuals,

“We’ve built Observer for product teams that are looking at usage and engagement metrics, rather than retargeting stats. This allows developers to find value in aggregated data where they don’t necessary need to correlate a user with an account.”

“Teams will find more value in underlying trends that they’re seeing amongst a larger user group compared to actions performed by one particular user.” — Lucas Toohey

“I think that the term “analytics” has gotten a bad rap in tech over the last year because users were unaware that actions they performed in an app were tied back to their account and to unique identifiers that could be used outside that particular experience.”

Lucas also sees the community building up standards around keeping people in the loop around how their data is being used in VR environments,

“In VR, we aren’t seeing a ton of applications that start by having a user create an account, but as this becomes the standard (similar to mobile and web), then users should have a clear understanding into how their data is being utilized. This is something that we are building into the Observer platform as the market matures to help developers with data transparency and usage. I don’t think there is a definitive right vs. wrong approach to data privacy in XR; the community will develop standards and set precedent as the technology progresses.”

“Currently, there is a ton of value in collecting usage data in a way that doesn’t affect the individual end users in a malicious way, and that is our focus here at Observer.” — Lucas Toohey

The future is bright

Lucas says they’re looking to automate things much more in the future to bring even more value,

“Looking into the future, our goal is to make analytics more prescriptive, rather than reactive, to automate the analysis process for teams to extract more value from immersive data.”

How to try it out!

If you’re keen to give their platform a go, it’s currently in beta, with a public release on the way on September 12th. Head to ObserverAnalytics.com for more details:

“We are currently in beta working with some amazing development teams including FitXR, TribeXR, and Alvios Games. We are in the final stretch towards launch, with our public release scheduled for September 12th. Our platform is free to sign up, just visit our website to get started!”

Thank you to Lucas Toohey for sharing so much info about the Observer Analytics platform. It is going to be a hugely helpful tool for improving VR/AR applications!