Dr Bakos Receives NSF Grant Award

Dr. Jason Bakos has received and NSF grant award for his project "A Unified Approach for Scheduling Computer Vision Dataflow Graphs".


Advances in computer vision and machine learning are enabling continuous development of new capabilities in autonomous systems and knowledge extraction from images and video. These advances are possible in large part to increased computational capability brought about by a new class of processor designed for visual and neural processing. This includes Google's Visual Core and Tensor Processor Unit, Qualcomm's Hexagon DSP, Nvidia's Programmable Vision Accelerator and Tensor Core, and Intel's Crest processors. In these processors, the program code exerts far greater control of the underlying processing and memory resources than in general purpose and graphical processors. This serves to greatly reduce the uncertainties that would otherwise restrict accurate performance prediction and open the door for new approaches for automatic and portable performance optimization. Despite the growing market for embedded vision processors, their associated compiler tools are nascent, relying on hand-tuning program code using trial-and-error methods. This project will build upon existing technologies to build a "universal" front-end for a broad class of vision processors. As part of this project the researchers will also develop an database of OpenVX 1.2 kernels (https://www.khronos.org/openvx/) and an OpenVX benchmark suite. It is expected that developing both the tools and benchmarks in the open source will facilitate community involvement. The artifacts developed in this project will serve as a pedagogical infrastructure for a new course in Internet-of-Things and machine learning at the edge of the computer network.

This project is developing a new program compilation flow, in which a program written in the platform-independent OpenVX representation is converted into a low-level representation and explicitly scheduled onto the underlying hardware resources. The conversion is governed by a mathematical model of the hardware behavior trained from mining performance data generated from executing and instrumenting a series of benchmarks. The compiler framework developed using this approach has the potential to replace the ad hoc methods currently in use and become a powerful tool for developing software for visual and neural processors, which will power a future generation of cognizant smart devices.