Hydrix and IOTA joined forces to develop the OneBox, an IoT device that provides water authorities the ability to control and monitor pumps, pressure sewer units, and other remote assets. Filip Zalio from Hydrix will be speaking at YOW! Connected next week about the OneBox, so I wanted to get in early and get tips on setting up IoT devices like this effectively and reliably.
To start with, it’s always fascinating to see how someone got into the space. Filip, as the Solution Architect at Hydrix, develops systems at huge scale that are safe and reliable, a mammoth task! Yet, it all began when he was a teenager, tinkering with a newfound obsession — electronics. He spent his time building radios, antennas and the like.
“I used to fall asleep reading through electronics component catalogues and trying to understand how the 8080 processor worked.” – Filip Zalio
He moved to Australia in 1996, working as an electronics design engineer for a small radio equipment manufacturer, followed by some time at NEC Australia working on a few mobile phone chipsets. (Filip points out that it was “before the mobile phone industry consolidated to a few large companies”). That experience was invaluable — “The stuff I learned there about embedded product development, and the in-depth knowledge of the 3G and 4G mobile systems is still very valuable in my day to day working life.”
Many of the projects Hydrix works on have two things in common — “built-in smarts and reliable communications”. Both suit the skills that Filip had built up throughout his career.
“Working as a solutions architect / system engineer / gadget designer is a privilege. I consider myself lucky that I get to work on things that genuinely improve the world in small ways.” – Filip Zalio
What is behind that mysterious title? Filip explains, “If you were to draw a Venn diagram of my responsibilities, it would have project leadership, system engineering, and geeking out on various specialties — communications, embedded software, electronics, and a few others. […] Mostly, I stress out about designing things that work well for our clients.” It seems like a pretty diverse day!
The OneBox concept was to have a connected device that allows people to to monitor their pressure sewer units in real time and operate them remotely. Integrate it into real time data analysis systems and you can get instant info on blockages, faults and more – so issues can be identified and fixed before anyone even notices there was something wrong. That’s a pretty great use of the Internet of Things!
IOTA, a South East Water subsidiary, came to Hydrix with a concept they’d already architected themselves as a proof-of-concept. Hydrix took the hand-made prototype and redesigned it into a product that “meets a cost target, is manufacturable and reliable”. Filip was the embedded software team lead on the project — he architected the OneBox firmware and wrote parts of it.
Creating a mobile network connected IoT device is a massive endeavour. I can’t even begin to imagine just how much is involved at something at this scale… especially when reliability is crucial to this being of any use. So where do you even start with something like this? I put the question to Filip, who gave the following advice:
“If you look at the bigger picture, when building an IoT system, you will need a range of expertise that will typically only come from a multi-disciplinary team.” – Filip Zalio
“For example, Hydrix has people that are great at business development, industrial design, system engineering, electronics, mechanical engineering, communications, embedded software, verification, testing, medical devices and industrial standards and regulations, project management and manufacturing. I might have forgotten a few disciplines.”
Basically, if you don’t have a number of these disciplines on your IoT project and weren’t aware you need them, Filip says “you’re likely to be in trouble already”.
From a first-hand engineering perspective, Filip says it’s important for all stakeholders to really understand the problem they’re trying to solve with a focus on potential risks that might cause you trouble:
“On the down-to-earth level, as an engineer, which is what I know personally, all the stakeholders need to understand the problem being solved really well. What are the needs of the end users or the “product requirements”? For example, when building a network-connected thermostat, it seems like a good idea to design it in a way that it virtually always works in its basic function: being able to control the room temperature even when there is no network connectivity, or even when a firmware upgrade fails or in a number of other failure scenarios. This is an example of a reliability requirement that seems obvious. Yet it is easy to miss such an “obvious” requirement during the early analysis and design, and at the same time, quite hard to fix after the system has already been implemented and you are getting horrible failures in-field.”
“In other words, I like to use a risk-driven approach. It helps to constantly keep an eye on risks, throughout the life cycle of a project. From the concept/idea stage all the way to the system being deployed and running. Knowing the risks that are likely to cause trouble is why you need the multi-disciplinary team. Then all you have to do is to gradually remove the risks and unknowns until you have the system deployed and running. To be cheeky: there is a quote somewhere in the Hitchhikers Guide to the Galaxy about learning to fly by basically just avoiding the ground.”
With something at this scale, how do you ensure that connectivity stays up? What happens when the connection goes down?
“The connectivity doesn’t stay up, that’s a fact of life. In Onebox, the core functionality was designed to keep working no matter what happens with the connectivity.” – Filip Zalio
Here’s how OneBox specifically keeps itself alive and well, despite inevitable connectivity issues:
“[OneBox’s] basic function was simple: execute the “rules” for controlling the output (such as a pump) based on a few inputs (for example, the water level in a tank) and log some data at the same time. This function had to keep working even without connectivity. We also included a software process that monitored the network and reset the whole modem and network stack whenever the connectivity goes down. It’s the “Have you tried turning it off and on again?” solution.”
Even with all of that experience, every project comes with its difficulties and lessons learned. Filip and the team went through three different TCP/IP stacks for the service before finally finding the right one for their needs. Each one took weeks to test out to see whether it was the right solution. That’s a lot of time spent but in the end, it was worth the reward of a solid choice. In the end, for those keen to know, they went with the open source LWIP stack.
The main lesson that Filip wants readers to take from this is that you need to be very familiar with the tech you plan to use in your solution:
“If you don’t know the tech inside-out, you can’t estimate the effort and time required to include it in your design and more importantly, you don’t know what the risks (gotcha’s) are. Going into projects naively, without understanding these risks, can kill a project.” – Filip Zalio
A lot of people ask me for advice on getting into the Internet of Things — so I thought I’d ask Filip if he has any tips. In particular, I asked, “How would you recommend developers get into creating reliable IoT devices like you do as a solution architect? Where should they start? What advice would you give someone who came to you uncertain of how to enter the industry?” His response shows just how much is involved in building connected devices… there’s a LOT out there:
“I find it interesting that you ask about “devices” specifically. Those are actually just a part of the picture. Building gadgets is, of course, the most fun and fulfilling part. But, what about building whole systems or solutions? What about the back end software? What about usability? What about security? Reliability? Etc. Where I am coming from — I sometimes fall into the trap of thinking, ‘Hey, this is easy, let’s just get an Arduino, write some code over the weekend, hook up some sensors to it, and I’m done.’ Because that’s the fun part. But, over the years, I gradually found out what my blind zones were, and that it takes a lot of work to get to the point where you can sit down and design some circuits, write some code and be reasonably certain that it will actually work.
That was just a tangential observation, so to actually answer your question: First of all, I am an engineer, so I can only speak in that capacity. If I was to tell a younger crowd how to “get into IoT”, I would, first of all, say not to worry about IoT. That’s just an acronym that may be soon replaced by something else. The underlying technology, however, is staying. I personally would not be able to work on these types of projects if I hadn’t been lucky enough to be given the opportunity to learn how to develop (somewhat) reliable embedded code, how the communications systems work, electronics fundamentals and so on. Sometimes it is back-breaking hard work. I was debugging the some of the Onebox firmware over long hours and weekends while at the same helping at home with my newborn.”
“I think you have to really enjoy doing this type of work to the point where you spend many hours, days and years to continuously keep learning. Running entirely on your own motivation, even if it was for free. If you are crazy like that, you’ll do well.” – Filip Zalio
It’s also valuable to be able to pair engineering skills with a commercially-minded mentality:
“On a less philosophical note, you can get involved with IoT if you are a great electronics engineer, embedded software engineer, and so on. You also need to be commercially minded – do the task required in a minimum amount of time. We engineers fall into the trap of enjoying the “tinkering” or reinventing the wheel too much. Usually, it’s choosing a solution that is too low level: a chipset rather than a pre-built module, for example. Learn how to avoid that.”
Thank you to Filip for sharing his experiences in the ever expanding world of connected devices! Filip will be speaking at YOW! Connected 2016 in Melbourne on the 5th-6th October, that’s next week! If you’re around Melbourne, grab a ticket and come along! For the companies and investors out there — If you have an idea for an electronics, mechanical or IoT style product or system and are struggling to find people who can actually develop it for you, Hydrix might be your guys!