data.bythebay.io: Bo Huang, Towards a virtual reality meta-Earth
Recording: data.bythebay.io: Bo Huang, Towards a virtual reality meta-Earth
Great. Okay. Hi everyone. Uh my name is Bo Bhuan. It's an honor to be here today to talk talk to you guys about subjects I'm very passionate about. So uh a little bit background about me. It's like uh my background is in uh computer graphics and games. Some of the things um I do include for example I do things like uh atmospheric scattering map visualization rate tracing uh games material simulation and uh everything that's like say material capture and photoggramometry architecture etc
Some of the companies I worked for in the past include say X-right which is a color science company which I was engineering scientist and other companies uh I was mo mostly developing games. Okay. So today I'm going to be talking about uh data. I mean every everyone here is very knowledgeable about data so I won't go into too much detail but uh just to summarize that there is basically a lot of data go in in our current uh world right now and most of them are invisible you know and a lot of them are generated by say IoT data your typical data from your cloud sources and etc and those data are spatial tempor temporal and scattered all over the world and all of them have are in different format and different organizational structures. So you may not be able to to work with all of them and but they need to be really displayed to be meaningful to people. I mean otherwise you don't know what they are. So one thing that uh I think the best is to start with a with a demo of sort. So to to to let people know what uh what is uh going on with with data that I'm working with
So here is um I'm using the Unity game engine and I'm going to show you here is a a section of San Francisco. It's very slow right now because uh obviously there's a lot of data, but uh we could zoom in to Oh, it's slow because um I'm not plugged into to power. So So the So the laptop goes into goes into, you know, the low CPU cycle mode. Uh okay. But so you have to kind of pretend that uh it's in real time because usually it is. You just have to believe me. Maybe you can tweak the power to the maximum zoom to the maximum. You said no the power mode the control panel setting
Maybe you could change it on battery. Ah, okay. Or we could plug it in. How about this? I mean, I could continue with the talk and at the end we'll we'll plug it in and see how it works like. Excellent. Thank you. Okay. Okay
So you got a rough pre preview but uh I think one thing to talk about is like you know we have all sorts of data and in this specific case it's a map and geology I mean geoscience related no not geocience uh geodata related and uh a lot of this data is uh want to we know we want to display them in real time and for bandwidth considerations that we need a compact data format um and uh some infrastructure to to catalog modify, delete and of course retrieve those kind of data for all kinds of devices. So what's the data source? I mean this talk is mostly about mapping data. So open street map is a very obvious choice. Open Street Map for those who don't know is a opensource uh data format data collection that's a crowdsourced by users all over the world in which everybody crowdsource mapping data and contribute to this master source. So unlike uh Google's or others which is closed. So what that means you could you could download the data and uh do whatever you want with them. So, so in our case, you know, we we retrieve them from uh from the data and make a make a 3D world uh to play around with. So, some applications include just like visualization, your typical virtual reality session settings
For example, here is a is an image of uh when we were demoing at the Nvidia's GPU technology conference. You know, here's the booth and we were demoing the this 3D world in a u HTC Live setting. Okay, for this part I'm going to be relatively simple because like um you could look it up look it up online. You know, it's has a long history over 10 years. Uh a lot of users you can modify, download and if you go to the website, it approximately looks like this. looks like your typical map and you can zoom in out but uh if you follow the the pertinent links you're going to be able to to download the data sources and those data sources are in usually in XML format so you could parse I mean other data sources I mean other data formats are also available so why is it good because like there are lots of building buildings lots of roads lots of amenities amenities being uh say restaurants your loco bars or your loco water fountain, parks in the bench, etc. And as a you know, you could you could also add your own stuff to it. I mean there are lots of uh for data you could input in
I mean as long as you agree to certain uh standardizations that's agreed upon by the com by the community and of course like saying IoT setting like you know if you're if you are able to get your IoT devices onto this map then you automatically have a have a in infrastructural platform for your device to to register itself and also discover other IoT devices uh in its proximity or in your local neighborhood or the city. Some examples um I think I mentioned about just a lot of different things that you could imagine from our real world. And here are some uh say rendering of u what what it looks like from our San Francisco data source. Now um you know we we have this data um and we have uh we're looking for areas of application. I say one area is like is your your IoT devices like why do you want to you know talk with your IoT devices because like uh internet connected that's the the trend to go. So in terms of say data set um I think that this is talking about that OSM data is reversible. It's moderated by the community and uh you you should have fairly accurate data and it's constantly being improved. How to add your devices or how to add your data? You just sign in and there there are editors that you could learn to use and add your data and how to modify
You know again again there are APIs and editors all available and other other devices um and APIs. So say for example the overpass API is a very useful one that you could query in real time a particular section of data that you you are of some some study squares. So, so let's talk about some use case like say for example you know smart build is an obvious thing because like increasingly we're having a lot of different uh IoT devices and just devices in general available from your robots your vacuum cleaners or your security cameras etc. all of them need to know what what and where they are and of course I mean they they probably will increasingly have some functionality of knowing using say computer vision techniques to know where they are what kind of obstacles that uh you you are uh they're facing but uh for the most part they're limited to very high-end devices which are now available to us now but so one use case of using this mapping data especially in a 3D fashion is that you could use those data and uh create a 3D 3D world in which uh the the devices your your vacuum cleaners and the robots they could collide against or know where where is a is a piece of wall where is like uh the staircase so that they don't make a fool of themselves another use case is your say disaster I mean open street map is already used by uh by volunteers and professional rescuers to to you know to to rescue people and uh one thing they do is like say because like when when disaster strike it's uh the the terrain and ge geography changes like so people need to rescue workers need to know accurate accurate knowledge of you know what is the most up-to-date uh ge map condition for example in your in your first 72 hours of of disaster mobile devices are very valuable because uh for the most part they they still remain uh maintain a charge for people trapping rubbles for example so that it's uh it's relevant to have those u to detect where the devices are. So so that's where your uh the previously mentioned IoT device uh networking and infrastructure comes comes into play. So if you are a rescuer and you're equipped with devices uh in a in a virtual map to know hey there here's this uh this floor or this subway and I need to go down and and there are people trapping inside whereas your traditional map is a traditional is a 2D flat plane and you're not going to be able to to do so much especially say people who are trapped under the subway. Then of course uh for for the post construction stage for disasters you'll be able to to track say oh where's your um say water leaks where is your I don't know nuclear hazard for those extreme cases and you could synchronize with uh with satellite imagery sources. Another use case is with drones
I mean drones are increasingly ubiquitous nowadays but most of them are not so smart like say they require users the controller to to control where they fly to. So I mean we we we would use the maps to to n to to make the drones smarter like say the drones will be able to store say map of the the location that they where they are flying to including in this case 3D maps multi-level stories of buildings so that they they know approximately where they are say how how where's the in in how many meters they will be able to collide they will hit or hit a wall and destroy themselves and of course you know they they will supplement ment the accuracy with um their own own sensors when when they get to that technology level and another thing I want to mention is like say why do you not want to use Google Earth or other similar mapping mapping technologies because for Google Earth is uh there's a lot of visual noise I mean there are cars there are people there are trees they're all scanned and baked inside uh the map even though it's a 3D structure And uh for for say multi-level stories for for many many buildings you're not able to go inside. And of course uh another downside is that it's updated once every few month perhaps and maybe for hotspots it's updated quicker but uh you know compared to your open source maps you know it's much slower. Oh, and another thing I should mention is that uh for for Google Earth, you're also limited to to to what your your satellite could see. So if you are so your subways and underground say water pipelines or whatnot would not be seen by by the satellite and you are missing those type of information. Whereas with with open mapping data it will be increasingly available. Now I think th this is talking about uh this slide is talking about what I mentioned earlier that drones are not smart. They need uh human assistance
So all to deduce like what is around them. So with uh and especially with say computer vision they it's um modified or it's impacted by the lighting conditions. For example in this in this current room there's a very bright light shining on my face right now. You know, if you have a drone, like even if it's very smart and using computer vision to detect where is the, you know, where's the next wall, but uh the the light the strong light will will will influence uh will will make the will be a bright splotchy spot for for the computer vision camera. And so that the drone essentially what the drone sees is a very overexposed imagery that it may not be able to to deduce. um what is uh say where the light is compared to say if the light is uh were were to be off and not emitting any u photographic sources. Yeah. So this this is also the same thing like so with um with these kind of data sources that we increasingly have access to and are able to make a interactive use of we could create 3D meshes to help with the collision detection and uh you could you could plan a custom zigzapping path of where the drone is to fly to and you have very high levels of confidence that you you'll be able to succeed
Now finally um I think I'll show some another another few sets of pictures. This is a uh Stanford, Connecticut, okay, where we had a which we were part of a hackathon that was held over there for smart cities in which um the Stanford supplied their smart city sources which is which is like because like Stanford is a high high-tech city just like San Francisco and it's very keen on presenting itself as high-tech uh data friendly and also smart and uh We use their their data source and constructed constructed three 3D 3D simulations I mean 3D 3D models so that it will be usable for the hackathon. Okay. And um I think one I'll just say that um for for that hackathon the the data sources basically the 3D models are already available on GitHub. We put it online here. So you go to go to github slsense earth you will be able to download uh all kinds of data source I mean uh the 3D models I mean they're they're in OBJ format which is a very standard um 3D format so you'll be you'll be able to use them in your typical computer uh your game game engines or WebGL. Okay. Uh that concludes my talk and here's my contact information and I will be happy to assist if you have questions
Thank you.