Rob Manson and his team at awe.media have managed a fantastic achievement this year — image recognition and image tracking working in web AR! I wanted to ask about how they did it and how developers can try out their mixed reality creation tool.
Rob has been all about the augmented web for quite a while, I spoke to him about this very topic back in 2015 on Dev Diner! Since then, his team has put together a whole platform for building web based mixed reality called awe.media. He explains it best:
“awe.media is a self-service Mixed Reality creation tool that gives you the power to both “create” & “view” Mixed Reality using the browser in your pocket, or on your computer. This includes making 360° photos & videos interactive, creating VR scenes, creating location based AR (like Pokemon Go) and now image based AR (e.g. bringing printed images to life).” — Rob Manson
In October 2017, they released an impressive new feature for the web AR part of their platform — image recognition and image tracking working in the browser! It was no small feat — Rob says they’d been working on this for five years! He says the new features are the “final big piece” in the WebAR puzzle:
“It’s taken a lot of effort in R&D and we’ve also had to wait for the web standards and Moore’s law to move everything along. But now we’re here, so it’s pretty exciting. And this is just the beginning. Our platform now includes “image recognition” for visual search (e.g. point your phone at a poster and it looks up the content for you) and “image tracking” for placing realtime overlays on the scene you’re looking at through your web browser/camera. This is the final big piece in the WebAR puzzle.”
Here’s a look at their web-based AR in action:
Bringing augmented reality to the web was already a pretty big task back in 2015, so bringing in this image tracking within web-based AR is even bigger! I was keen to hear what the challenges were around getting this to work and Rob was happy to give some of the background details,
“Computer vision requires a lot of refined computational power. It’s taken a hell of a lot of work to get this to a point where it’s stable and runs fast enough on all the mainstream browsers and devices. The speed of the tracking is very much based on the type of device you’re using and we’re squeezing every last bit of performance we can out of them. For the best experience we’d currently recommend using modern Android devices (e.g. a Pixel) or a new iPhone (e.g. iPhone8+). However, it does run on older devices – even Androids as old as 2013 and back to the iPhone6 – but the fps can vary a lot.
For this initial launch, we’ve focused on making sure it runs on as wide a range of devices as possible with the most robust tracking as possible. In fact, another release went out today that improves that a little further. From here we’ll be working to optimise the fps and we expect upcoming changes in the browsers to speed this up over the next few months too.
Another one of the big challenges was just timing. It’s taken quite a long time for all the major OS vendors (we’re looking at you Apple) to adopt the required web standards like WebRTC. But it’s great they’ve jumped on board and everything works now.”
With iOS 11, all of the wonderful augmented reality potential was opened up on the web thanks to Apple adopting the WebRTC standard. Rob points out that this is a good example of why open web standards are so important.
“All of our developments are purely a side-effect of them adopting WebRTC. This is one of the massive benefits of open web standards – it unleashes opportunities for whole new innovations like this.” — Rob Manson
So does this mean that Apple might be behind the concept of web-based AR and ready to push it to the masses? Rob isn’t so sure just yet:
“You never know exactly what Apple will do, but I honestly don’t think they will adopt any web-based AR in the near term future. Before they can adopt any of the current ARKit/ARCore wrapper APIs they’ll need to adopt WebVR on iOS first. And I’m pretty sure they don’t want to blur their ARKit message right now either.”
Where are we headed? Rob hopes that next we’ll manage to get ARKit/ARCore and the like available to browsers so that we’ll use those too:
“Now we’re waiting for the next generation of APIs like ARKit/ARCore to be exposed to the web so we can use them as well… and that’s all on its way. But in the meantime we can create a lot of interesting and exciting WebAR experiences.” — Rob Manson
Thanks Rob for taking the time to chat about the new image tracking, it’s a really exciting time for the augmented web! You can find out more about their platform at awe.media.