scale.bythebay.io: Keita Broadwater, IoT, Wavelets and Machine Learning in the Smart Home

scale.bythebay.io: Keita Broadwater, IoT, Wavelets and Machine Learning in the Smart Home

Recording: scale.bythebay.io: Keita Broadwater, IoT, Wavelets and Machine Learning in the Smart Home

you thanks everyone for coming one note before I begin the in the abstract for this talk I said that I will give a live demo due to some glitches I'm not able to do that so this will just be a talk only that said the agenda for today's I'm going to talk about motivation I'm going to give you a high-level overview of a data pipeline I'm using I'm gonna talk about hardware for a bit and then I'm going to do a little deep dive into the pre-processing that I'm using for this system so I'm Kecia Broadwater as he said I've been in tech for about 15 years I'm currently a co-founder of oxigen AI oxygen AI is a consulting company that specializes in helping small to medium businesses and organizations leverage data science and AI to drive the business results I'm also the co-founder of PB Tech that's the solar energy company that I sit on the board of that does has operations in East Africa in Southern California so what are we talking about it a high-level the system that I'm developing is one that uses sensors specifically accelerometers to do an interrogation of footsteps these accelerometers are placed on the floor of a room or in a building and taking the signals from these footsteps we're able to use analyses and machine learning to draw out information like what is the identity of a person in a room what is how many people are in a room and other information so why is this worth pursuing a lot of times when I talk about this to people they ask me well why not use a video camera well I think that this technology fits a niche that balances what I would call intrusion versus Intel versus intelligence so think about the types of technologies that can give you some information about what's going on instead of an area on one in would be something like an IR sensor used for motion detection but not much else on the other end would be a video camera if you combine that with recognition technology you can get lots of information about what's going on in a room but in some cases particularly when you have concerns about privacy a lot of information could be too much information so for example a video camera can tell you that with recognition technology could probably tell you that I walked into a room if it can recognize my face it can also tell you that I'm picking my nose so in cases where you want to have privacy for people having a system that can draw out good information with but not not put people in compromising positions so to speak or is compelling so an overview of my pipeline is so I'm using sensors based on right now I'm prototyping this so I'm using sensors based on Adafruit and Arduino those sensors string data into Kafka that goes into spark spark streaming specifically structure streaming data frame I use a C or I use a series of transformations to transform a signal from an accelerometer into an image that I can use the machine learning model so a bit about the hardware that I'm using using a accelerometer that's on an Adafruit chip board and I'm also using a wireless a wireless microcontroller so for this microcontroller you have two chips one is a controller chip that can you can use to control the sensor the other is a is a Wi-Fi chip with the Linux operating system on it and this is what I use to create when data is coming through from the sensor I use the Linux operating system chip to create a TCP socket and then stream that to a server which places the data on to CAFTA producer the next step I want to go with this and it's easy to do with this type of board is since one of the chips on this microcontroller board has a Linux operating system I can create a Kefka client directly on the board so the next step for me for this part of the pipeline is to completely overstep the the mid server that I'm using and place a kafka client producer on this on this board directly so pre-processing consists of a few steps one is taking a raw data signal from the sensor that I just described breaking it into overlapping overlapping frames and on those frames performing a wavelet transformation which I'll go into the detail in the next slide this produces this transform is a problem from time series problem to an image problem so once we have the image we flatten it and we we do dimension reduction on on those on that image and then we can use it to train or to create get prediction from a machine learning model so what a wavelets wavelets are short-lived wave waves that have a mean of 0 they're perfect for this application because you have footsteps or when you have activity in a room it's really very localized in its non stationary you can see an example here one piece of data shows a person walking into a room getting very close to the sensor and then going away from the sensor so it's much better to use transforms to interrogate these types of signals than to use to use a Fourier transform so when we use the when we do a wavelet transform we take a wavelet function and we we break we decompose a signal into two spaces one a space one its translation and a lot of times people use these as analogs for frequency or pseudo free yes reading out a better cancer because it's more efficient because I just I I think um so what could you repeat the question because it reads not a better answer and I it reads not a better answer because the the signals that we're interrogating or are very localized in a non-stationary so so you so there are some there are some ways that you can use Fourier transforms there are some ways even though Fourier transforms for sure yeah I think it gives a better answer I think it gets a better answer Thanks yeah thanks for the question yeah so once I Paloma Fourier transform I I create an image by creating something called a scale gram so we're dealing with scales not frequency and if we were doing something else we will call this excuse me we have frequencies we would have a spectrogram but right now we're dealing with a scale gram so this scale gram or something like it becomes the image that I used to train a model and to interrogate a model so there are there are several packages that exist for this to create wavelets and to do wavelet transformation i use pi wavelet i think it's a very comprehensive way it has lots of types of wavelets about discrete and continuous there are also lots of packages that deal with seismic data ABB's phi is one of those packages it has a pretty good wavelet functionality there are also wavelet packages and modules in Java and Scala so what I like to do in the end is have everything in Scala at this point though I'm using PI wavelet to patch on some custom functions using PI wavelets and some other modules but I want to use the scale module eventually to have everything within Scala for the transformations and so the code that you see is just an example of using a wavelet packet transform using a specific one called a double Shea 10 to 20 excuse me 20 tap wavelet on a piece of data so I can use this code to create a wavelet which will be a matrix of coefficients and then I can use those coefficients to create the scalar gram image that you saw so the second part of pre-processing for me is to take a scale of gram in this case the image that you see is a scale of gram for a discrete wavelet transform take that in a lot of cases you have an image with hundreds to thousands of features so you want to create you want to do some dimensionality reduction how I accomplish this is just to brute force take quite a few methodologies and so I'm using scikit-learn for this I'm using quite a few methodologies to interrogate I mean excuse me to find the right parameters find the right function - that will give me the best f1 score the best precision recall so this is a scale gram so just considered an image with pixels I flattened it so each pixel becomes a feature no no I found that I thought it may be necessary but I found that some of the results the results aren't getting as far as from the women classification models I'm using I've been pretty good without going you mean the name is like this so for this example this is a eleven hundred and twenty features so for this example I take the eleven hundred and twenty features and reduce it down to eight features using a non-negative matrix decomposition and how I got to that method non-negative matrix factorization and eight features is by as I said before just kind of doing a brute force kind of a grid search type of way testing several types of decomposition methods and a range of feature numbers so I think I I went from five to 100 to see what would be the best what will give me the best results in the Train model so at this point I was going to go into the live demo but as I said I don't have I don't I don't have that demo available now to do some glitches so I'm just going to close out and say that you know using wavelets I found is a very good way to get results that have high precision I higher f1 scores with class when you do classification on gates and footsteps and that I found that is very possible now to have an end-to-end very low-cost system that can do some height that can do a very accurate characterization of activity within an area within the building and so the next steps for me are to have everything in scala have it in the end and increase the scale of the number of sensors that I'm using and increase the the sampling rate of the sensors that I'm using so right now I'm driving sensors at about 100 samples 100 excuse me a thousand Hertz and I want to kind of I want to ramp that up to a greater number in that thousands of Hertz so that's that's all I have for you thank you very much yeah [Applause] I can IEP using an outdoor environment they can but you have to it just depends on the packaging the ones I'm using now I'm using indoors so I don't use any kind of hermetic packaging for the sensors but you can you can create yourself especially now with a lot of the the 3d printing technology you can create a hermetically sealed package that you could deploy outside so sorry you oh I see yeah so I think it also it depends on the packaging and I pinned it depends on the type of sensor so most of my work has been inside indoors so I use I can get away with using a very low-cost sensor like you see here and getting good results still outside usually when people are doing seismic analysis they use seismic sensors they're a little the design is a little more hefty and the range of the sensors are much greater and using some techniques you can isolate footsteps you can isolate cars driving by and things of that nature so you need a different type of sensor and you need more robust packaging to them to do that but it can be done can you give us any comments on you know physically what's happening with the sensor is it just taped to the floor or I mean is there anything that you found it's like you know do this do this kind of if you do this this problems a second question would be you know which of this processing that you eventually hope to run at the edge on the new people or you push it to your cloud the problem so what it's doing so I'm using basically an adhesive just to put it on the floor not a non permanent adhesive so it's not really doing anything just is sitting there and it's getting the vibrations from from whatever activities going on so it could be a human walking it would be a Roomba traversing the floor to be a pet just you know what got me into this project was a year ago I was woken up in the middle of the night by something bumping furniture downstairs so I went downstairs I thought I was gonna have my moment of truth protect my family and homestead but it turned out to be a room but that was misprogrammed instead of 4 p.m. it went on at 4 a.m. in the morning so then I was thinking okay it would be great to have an app on my phone that could just tell me don't have to worry this is a room but just turn it off so it's a it can it can it's just sitting there it's just a passive sensor for the most part and your other question was I mean I would I would love to be able to I don't I don't think that that young controller can accommodate this but I'd love to just be able to do the transformation directly on the board but I think what's more realistic is that I did Lisa be able to put the CAFTA client on the on the board instead of having a third an unneeded computer running and running a server so yeah yep I guess my question is more theoretical which is do you have any sense of like what the limits of sort of Union is like deep it's like you know they say a fingerprint there's it's not like completely so I'm curious like have you done research on how thank you I don't think that I think there has to be limits I haven't reached him yet one of the good things about this type of system of this type of project is that if I need data I just put a sensor down and walk past it or put a sensor in the hall and a crowded building and have people walk past it or or finagle my wife into walking for a few minutes to give me more data so but I haven't done those studies to do that I think is going to require some large-scale labeling to efforts so well I can go and put it lay a sensor down in a crowded hallway and do some unsupervised clustering to figure out if there are some there's some limits of uniqueness I think probably they get the answer I need like a statistically significant number of people to to determine that so and I haven't haven't done that work [Applause]