A back arrow icon.
RDFox Blog

Machine Learning and Semantic Reasoning, the Perfect Union—Using Object Detection with RDFox

Machine Learning and Semantic Reasoning, the Perfect Union—Using Object Detection with RDFox
Gabriel Lindman

The two branches of AI, machine learning and semantic reasoning, are often discussed as opposing forces—one a probabilistic black-box and the other reliant upon provable logical inferences. In reality, their strengths can be combined to create a solution greater than the sum of its parts. While exploring RDFox, our in-house experts at Volvo Cars set out to do just that, pairing knowledge graph reasoning with object detection.

With this in mind, there were a few features of RDFox in particular that we wanted to explore:

·       OWL2 to build ontologies

·       Semantic reasoning for a real-time decision engine

·       In-memory data storage and management well suited for edge applications

We wanted to create a project utilizing these features to learn more about RDFox and to create something visually engaging to demonstrate the potential to our colleagues. So, after some brainstorming, we decided to try and feed the database with object detection information.

Detecting objects in an image using YOLOv5

Object detection is a growing field within computer vision related to identifying objects as well as their relative size and position. In this case, we used an open-source pre-trained model with 80 classes from the MS coco dataset as this suited our purposes for the project. These computer vision models can use various types of input data to extract features for classification, like point cloud data from LiDAR or RADAR. We opted for a trusty web camera to keep things simple. The model we used is the YOLOv5, an open-source PyTorch implementation and further developed version of the YOLO algorithm by Joseph Redmond. Like Doctor Frankenstein, he became scared of his own creation and decided to halt his research when he saw its use in military applications and because of privacy concerns. He did an interesting TED talk about the potential benefits this technology has, as well as the dangers it unlocks. With this in mind, we decided to approach the project carefully... First adding the direct information that the model provides.

Object information fed to RDFox

The detected car and cup from the first figure are passed to RDFox together with a frame. Each object is also given a confidence, as well as information about their relative bounding boxes. If the bounding box of one object overlaps another, it :isOverlapping. If an object bounding box were to completely surround another, that object :isContaining the relatively smaller second object. Each frame is also time stamped and given a count integer from the start of the video stream. After that we decided to add a simple ontology to the graph to give the objects some basic classes. To build on this, one could add all sorts of wonderful meaning to the detected objects using the OWL2 web ontology language.

Objects with classes in RDFox

Our graph is growing. Next, we wanted to explore reasoning, so using rules we implemented a little made up scenario. If a vehicle of any kind is detected in the graph, we mark it with a heads up: 🙂. If there is more than one vehicle in the graph, we can guess that there is a bit more traffic, so we mark this with a somewhat more concerned alert: 😯. Finally, if bounding boxes are overlapping, we guess that this is an immediate traffic situation, like an overtake: 😱.

Three scenarios

You can see me acting out another scenario below, this time of a person using a cell phone. The assumption is that a person bounding box containing a technology bounding box of any kind makes the person distracted. Notice that our ontology deducts that our cell phone node should be classified as a technology node, and we use that in our rules to decide about the person node.

A cell phone distraction scenario

So, what we have done is look at a simple use case of graph technology operating in real-time, along with the basic use of ontologies and reasoning with RDFox.

Some things we could do to take the project further:

·       Train the object detection model to a more applied use case

·       Use the frames and/or time information to give the graph a "memory" that spans more than one frame.

·       Run the project on an edge device, like a raspberry PI, using an edge TPU to offload the inference of the object detection model.

This has been a fun project, learning RDFox while thinking about how this type of technology can be used in the future. Graphs can lead to an abundance of useful insights for the user, offering personalization and functionality without even having to share any private data from their devices. The market pull may even increase moving forward as consumers become more aware of their digital footprint and concerned with data privacy, creating an ever greater need for solutions like this one.

About the Author:

Hello there! My name is Gabriel Lindman, and I joined the Volvo Cars graduate program last year. A few months back, I gained an interest in graph technologies. I did not know much about it back then, but started talking to a more senior colleague at Volvo Cars who specializes in the field. I did not major in computer science, or work with graphs prior to this, but lack of experience has never stopped my enthusiasm before.

Take your first steps towards a solution.

Start with a free RDFox demo!

Take your first steps towards a solution.

Get started with RDFox for free!

Team and Resources

The team behind Oxford Semantic Technologies started working on RDFox in 2011 at the Computer Science Department of the University of Oxford with the conviction that flexible and high-performance reasoning was a possibility for data-intensive applications without jeopardising the correctness of the results. RDFox is the first market-ready knowledge graph designed from the ground up with reasoning in mind. Oxford Semantic Technologies is a spin-out of the University of Oxford and is backed by leading investors including Samsung Venture Investment Corporation (SVIC), Oxford Sciences Enterprises (OSE) and Oxford University Innovation (OUI).