What if you could sit across a table and talk appearing virtually in the living room of a relative in California while staying at home in Thiruvananthapuram? Simon Scaria, a Keralite, is Director of Product for Microsoft Mesh, which ushers in 'Holoportation' technology. Two weeks ago, Microsoft officially released 'Mesh', which radically changes the structure of video calls. The famous filmmaker James Cameron joined the presentation at the Microsoft Ignite conference remotely through Mesh. The tech world believes that the mesh will bring about a major revolution in video conferencing. 'Mesh' is based on mixed reality technology that incorporates another artificially created view or object into a real view. Simon Scaria. Who hails from Ernakulam's Maneedu, speaks to Malayala Manorama on this revolutionary leap in 3D virtual meet technology.
How would this happen?
Suppose three people from three countries are attending a meeting. The first step of the 3D meeting is for the three of them to wear HoloLens glasses, which they experience as a mixed reality. At this point, each of the other two figures will appear in three-dimensional form in the form of a cartoon (avatar). If a person raises his hand in it, the avatar also raises his hand. Even the sound can be felt from the side where they appear. In short the other two can give the feeling of being really close. The movement is detected by sensors in the HoloLens and the phone. In the next phase, Microsoft plans to replace the avatar with a three-dimensional hologram of the person who participates in the video call. The service will also be available on the Microsoft Teams platform.
The possibilities
With this, the limitations of time and distance will be eliminated. Children abroad can attend their parents' wedding anniversary virtually without having to return home. An automobile company can easily discuss the design of a new car among team members from different countries. If there is a mechanical problem in a factory, the technician will go there virtually and find out the issue and provide fixes.
One of the easiest ways to think about it is that Microsoft Mesh connects the physical and the digital worlds allowing us to transcend the traditional boundaries of space and time. You can feel the true presence, you can create together from any device of your choice, whether that’s AR, VR, PC or Phones. And for our developers, Microsoft Mesh can build immersive, multi-user, cross platform Mixed Reality applications leveraging Al-powered tools for Spatial Maps, Immersive Presence, Holographic Rendering, and Multi-user Sync. And you can leverage Microsoft Mesh to conduct virtual design sessions, learn together virtually, enhance virtual meetings, host virtual social gatherings and meet-ups and much more with a geographically diverse set of participants.
Your current role at Microsoft?
I am currently the Director of Product for Microsoft Mesh. My role is to seek customer signals, build partnerships inside and outside Microsoft, collaborate with various engineering and research teams to build the product and take it to the market, but most importantly ensure it is making a true difference to our customers. Prior to this role, I led product teams for Microsoft Vision AI and Office 365.
How is Microsoft Mesh going to transform the world?
In simple terms, Mesh will eliminate the constraints of distance and time. If you and I need to work together, we can be virtually together in a more engaging experience compared to a video chat. If we need to join our parents’ wedding anniversary, we can do that as a Holoportation or an Avatar and it will be the next best thing to being physically together. If a team wants to look at the design of a car, instead of looking at 2D pictures OR spending the time and money to build clay models of the car, we can visualize the design with 3-dimensional fidelity. Another example would be, creating a time capsule where employees can look back in time and see exactly what went into the construction of a building – day by day as it happened. My kids can relive their school graduation as it happened. When done right, it gives us super-powers, we just need to do it thoughtfully and responsibly . Sounds exciting?
We have seen the demo of virtual 3D meetings. What are the other common use cases?
There are several scenarios where the power of Mesh platform will be differentiated:
Remote Assistance: A remote technician in the field could be assisted by an expert from the field who can virtually teleport themselves into the scene among the collaborators.
Train/Learn together: With COVID training and learning has fundamentally changed and that change is a one-way shift where hybrid will be the new norm where some will be local and some will be remote. Never miss out an opportunity to learn and train together just because you cannot be physically together.
In-situ Information: By definition mixed reality is about placing 3D digital content in the physical world. Imagine how a factory supervisor can see last known service records etc by simply looking at the assembly line through a goggle? Imagine seeing entire building – inside and outside – before a single drop of concrete is poured? Imagine walking around the real world and playing your favourite game? That is in-situ information. In-situ is the Latin word for “in context”
Do you think Mixed reality is the future?
Mixed reality liberates screen bound experiences into instinctual interactions, in your space, among your things, with your people. When we look back at technology, we would distinctly remember these 200 years as a time when we oddly used 2D screens and papers to communicate about a world that is naturally 3D. Mixed Reality does not necessarily require you to have a headset to experience it. Mixed Reality has already gone mainstream. 100's of millions of Niantic explorers around the world have experienced MR through the devices in your pocket with games like Pokemon Go. As we miss in-person social connections, meet-ups like concerts and fitness training are moving to the virtual world using apps like AltspaceVR. Powered by the sensors already on phones, consumers are trying before they buy using Mixed Reality experiences in Amazon, IKEA and similar shopping apps. Salvatore Ferragamo is revolutionizing how its customers design, try and buy their iconic shoes. With Mixed Reality, Mercedes Benz is delivering material ROI to their business by reducing cost and eliminating errors. Knowledge is captured and transferred from industry experts to the new employees making the workforce confident and trained using apps like Dynamics 365. More than 50% of the Fortune 500 organizations have deployed HoloLens and other Mixed Reality solutions to drive material RoI to their business. So, clearly, Mixed Reality has gone mainstream.
Cost is a major barrier in the Mixed Reality adoption. Do you think the price will come down soon?
Cost certainly is a factor, but cost factor is seldom considered in isolation. Cost is always reconciled with a “return on investment” equation, and the question is whether there is sufficient ROI behind your investment. Lockheed Martin reduced their touch labour by 90% – what used to take an 8-hour shift to complete can now be done error-free in 45-minutes. Toyota is reducing the time spent doing inspections by as much as 20% a significant return on investment that has them rolling out this HoloLens 2 solution to all their dealerships across Japan. Bentley uses Mixed Reality to enable real-time high-fidelity rendering allowing multiple experts to view large sections of the bridge all at the same time saving 40% of the labor costs. If I can invest in Mixed Reality, and if I see 10X, 50X RoI on that investment, that’s what matters. Also note Mixed Reality devices is a spectrum – most phones today are ARKit or ARCore enabled and can deliver Mixed Reality experiences without any additional investments. Some VR Goggles cost less than 100$. Microsoft Mesh is designed to deliver collaborative MR experiences whether you are in 2D or 3D devices.
The ultimate aim of Mesh is to have holoportation. When do you think it will be a reality?
On March 25, 2016, Alex Kipman, Technical Fellow at Microsoft, demonstrated 360-degree holoportation live on stage his TED Talk. In the summer of 2020, Microsoft hosted Burning Man show as a holoportation. On March 2nd, 2021, at Microsoft’s Ignite conference, we had the largest Mixed Reality show with multi presenter holoportation between Alex Kipman and James Cameron on stage. In 2021, Holoportation will be coming to concerts, tradeshows and executive townhalls. Our vision is to democratize it and make holoportation accessible with the sensors on the back of your phones – that will transition such experiences from novelty to the norm. We are already working on this and you will see some of the more accessible holoportation options in Microsoft Teams soon.
Do you think we will soon see a world in which people wear an all-day MR glass?
Like Mark Weiser said, “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.” We have some ways to go, but that shift is durably under way. We will not be burying our head and locking our hands on a 6-inch piece of glass forever. We will be heads up and hands free. Input and output for computing will be more natural, engaging and in context. When – and not if – we provide a comfortable form factor with socially acceptable interaction models, the world will embrace Mixed Reality as pervasively as other consumer technologies. With the breakthroughs in chip technology, micro-displays and network technologies, you will see adoption accelerating in the future.
After Mixed Reality, what should we expect next? Anything with sensory aspect?
Haptics technologies are already here. With haptics, it is possible for me to feel the weight of a hologram. It is possible to let your thoughts control a computer and you see its applications in physically impaired accessibility scenarios. When I play Forza on Xbox and when my car crashes in the game, the controller provides a sensor feedback that makes the experience very real. There are far more realistic adaptations of such technologies already under way.
How can developers use MS Mesh?
Simple. Sign up for our Mesh SDK at www.mesh.com
Internet bandwidth is crucial in ensuring the quality of Extended reality experiences. How do you foresee the adoption of Extended reality in India, especially with regard to the bandwidth aspect?
Internet bandwidth is crucial, and I see the situation steadily improving. More importantly, Internet penetration rate in India went up to nearly around 50% in 2020, from just about four per cent in 2007. For a populous nation like ours, 50% hits an inflection point where it becomes the launchpad of new technologies. There is more economic opportunity both for telcos and for users for new technologies to be introduced in countries like India versus other less populous nations. India remains the 3rd largest economy in purchasing power parity. India has the world’s largest diaspora population making it one of the most ideal nations to rapidly achieve escape velocity with technologies like Mixed Reality.
What is you advice to the students who would like to pursue career in Mixed Reality?
First, follow your passion and take risks. To quote Michelangelo, “The greater danger for most of us lies not in setting our aim too high and falling short; but in setting our aim too low, and achieving our mark.” Set your aim high. You can take the most risk with the least exposure as a young student. If your passion is in MR, put premium on getting started right now. Get a device that you can afford and write an app to learn about the challenges and opportunities in MR. Think about a scenario where MR can disrupt and go make that happen. Starting a company while you are still completing school/college is possible. I started a company fresh out of college and I wish I stuck with it longer. Starting a company while still at college/schools will be the best thing you do to kick start your career. There are plenty of free courses on the science and research of Mixed Reality – seek them out.
Tell us about your family, education and journey to Microsoft?
I was born and brought up in a village named Maneed, in Ernakulam district and raised by two primary school teachers. I attended Malayalam medium government schools in my village. I am grateful for the amazing & dedicated teachers there that I had the good fortune to learn under. . I attended Maharaja’s college for my Pre-Degree, College of Applied Sciences, Calicut for my BS in Computer Science, IIT Bombay for my MS in Computer Science and came to the US to attend University of California, Irvine, for graduate research in Internet Communication and Security. Prior to joining Microsoft, I founded a company - Albits Inc. - for hosing Virtual Tradeshows on the Internet. The company was not a financial success, but I learned a lot and this really helped me identify and personally relate to the use cases even in the early days of Microsoft Mesh.
