EH-Safe Work: Putting the ‘A’ in Safety and ‘I’ in Streamline with AI
Combining Artificial Intelligence and Augmented Reality
Introduction
Welcome to the go-to online resource that will guide you on how to use Artificial Intelligence (AI) to streamline workplace safety processes, more specifically to allow for robust augmented reality capabilities that can be used for an array of use cases from recognizing the use of Personal Protective Equipment (PPE) to identifying potential hazardous materials that can lead to injuries; the potential and possibilities are endless.
With the continued advancements in technology and the ever-changing pace of society as a whole, the need to streamline processes related to workplace safety has been critical, which has been made more apparent since the pandemic. By taking advantage of machine learning, organizations can direct their focus on being innovative and delivering value without having to balance the act of managing what can otherwise be streamlined with the help of AI.
What is MindsDB?
MindsDB is an open-source platform that uses automated machine learning (AutoML) to provide a simple interface that allows users to train and deploy predictive models with ease. Not only does MindsDB provide a range of features such as data analysis and visualization, natural language processing, and time-series forecasting, it has a user-friendly interface that makes it easy for people to leverage the power of machine learning to improve anything from customer experiences to an organization’s operations.
What inspired this initiative?
In 2009, during my summer internship in the Mechanical Engineering Technology department at my college, I was entrusted with the vital responsibility of migrating a hazardous tool and machine testing web page. The existing manual system was prone to errors and mismanagement, which became a major concern after a machinist lost their thumb in the department while I was working there. To address these issues, I leveraged modern technologies to create an automated system for test results that significantly enhanced safety and efficiency. This system received overwhelmingly positive feedback from the Mechanical Engineering Technology instructors and helped me earn recognition as the Co-op student of the year. This experience ignited my passion for integrating technology to improve safety.
Throughout my career, I have worked as a Database Administrator, Analyst, and Developer, continually finding interest in emerging technologies and their potential to enhance safety. I provided technical guidance for the Virtual Reality (VR) and testing aspects of the Level Up Manitoba and DRIVR-X campaign. I developed an application for an Augmented Reality (AR) device called the Vuzix Blade. In 2021, I delved into Artificial Intelligence (AI) and Open Source development contributions with MindsDB thanks to the best Facebook Ad I’ve seen along with the help of an extremely supportive and helpful community. I am eager to continue leveraging emerging technologies like AI and AR, using MindsDB to promote safety.
How does this all work?
This section provides a step-by-step guide to help anyone interested, be it an organization or individual, to streamline their workplace safety requirements using AI machine learning and augmented reality. The following series of steps will include examples and tips on how to use what I’ve created.
So, let’s dive in and discover how this all works.
Prerequisites:
Basic knowledge of the following is required to properly execute the steps outlined in this article.
Machine Learning
Git
SQL
Data Analysis
Linux
Python
Steps to Execute:
Step 1: Install Minds DB locally
Good News: Following the steps outlined below will allow you to take advantage of some new and exciting experimental features that as of April 29, 2023, haven't been introduced into an official MindsDB release.
These features include expanding on MindDB's NLP Hugging Face integration with the addition of using Hugging Face Computer Vision to help predict image classification(s) and object detection(s). Stay tuned for integrations of this functionality being included in a MindsDB release to cloud.mindsdb.com based on the pull requests created for Object Detection in the hugging-py-face and mindsdb repositories.
Bad News: You won't be able to use the Cloud, Docker or pip versions of MindsDB to test these exciting new experimental features...yet.
Here, we follow the steps of MindsDB installation from the source code using Linux. You can follow the steps below. These steps are almost identical to what's outlined in the existing documentation Contributing Guide: MindsDB Installation for Development with the following changes:
Removing step to fork repository
Checking out a new code branch
Installing the necessary handler before running MindsDB
Before proceeding, make sure you have Git and Python 3.8.x installed.
Clone the following repository locally:
git clone https://github.com/MichaelLantz/mindsdb.gitCheckout the hf_inference_api_playground branch:
cd mindsdb && git checkout ml_integration/hf_inference_api_playground && cd ..Create a new virtual environment:
python -m venv mindsdb-venvActivate the virtual environment:
source mindsdb-venv/bin/activateInstall dependencies:
cd mindsdb && pip install -r requirements.txtInstall MindsDB:
python setup.py developInstall Hugging Face Inference API handler:
python -m mindsdb --install-handlers huggingface_api && cd mindsdb/integrations/handlers/huggingface_api_handler && pip install -r requirements.txt && cd ../../../../Start MindsDB:
python -m mindsdb
If everything works as expected, you should see the following message in the console:
...
http API: starting...
mysql API: starting...
jobs API: starting...
2023-04-29 11:56:03,224 - INFO - Scheduler starts
2023-04-29 11:56:03,748 - INFO - Starting MindsDB Mysql proxy server on tcp://127.0.0.1:47335
2023-04-29 11:56:03,749 - INFO - Waiting for incoming connections...
mysql API: started on 47335
2023-04-29 11:56:07,174 - INFO - - GUI available at http://127.0.0.1:47334/
http API: started on 47334
Step 2: Retrieve the Hugging Face API token
Log into or create your Hugging Face account and obtain your API token.
Step 3: Test Computer Vision Functionality in MindsDB
Navigate to your MindsDB Web interface (Should be http://127.0.0.1:47334/editor if running locally)
Run the following SQL by pressing Shift + Enter (replacing "hf_.." with your Hugging Face API Token):
CREATE MODEL mindsdb.ppe_detection
PREDICT detection
USING
task = 'object-detection',
engine = 'hf_api_engine',
api_key = 'hf_..',
column = 'image',
model = 'devonho/detr-resnet-50_finetuned_cppe5';
The command will generate a Predictive Model using Object Recognition specifically aimed at identifying Personal Protective Equipment (PPE). Give the command a couple of seconds to generate.
Run the following SQL by pressing Shift + Enter:
SELECT image, detection
FROM mindsdb.ppe_detection
WHERE image = 'https://datasets-server.huggingface.co/assets/cppe-5/--/default/train/1/image/image.jpg';
You should see results listing an image location along with the object detection result(s). In this case, a mask has been detected in the image.
The captured result lines up exactly with the Hugging Face Inference API widget located on the Hugging Face fine-tuned cppe5 model.
Conclusion
I'm currently working on infusing these brand new experimental MindsDB features into my favourite Open Source Camera app for Android (Open Camera) for use with Phones, along with adding support for the Vuzix Blade. The aim will be to introduce image classification and object detection capabilities with the objective to streamline safety inspections powered by MindsDB. Below is a demo demonstrating a basic integration of Open Camera with PPE Object Detection using MindsDB and Hugging Face.
By infusing AI to allow augmented reality capabilities, organizations can reduce accidents and injuries, minimize downtime, and improve productivity. By freeing up time and resources from having to oversee the environmental aspect of workplace safety (i.e., the external factors), organizations can now better focus on employees and their well-being/mental health (i.e., the internal factors) by providing the necessary support in whatever capacity that is required for employees to carry out their duties without it leading to burn-out or employee turnover.
PS: Big thank you to hashnode.com for hosting this blog and spreading awareness about MindsDB along with information to help democratize Machine Learning!
Reference(s)
Source Code
MindsDB (with experimental Object detection)
Open Camera (with experimental Computer Vision capabilities)