The purpose of the project is to develop an AI camera that can do detection and recognition of objects in real time. The researcher will work with a video processing library called Gstreamer, Linux Debian, OpenCV, TensorflowLite, and C++ with gcc compilers.

This will involve:

• Setup the Gstreamer pipeline to collect frames of video data
• Use CUDA and GPU acceleration to resize the frame to the appropriate size required by the model
• Store the frames in a vector object with datetime and video segment references
• Invoke Tensorflowlite - tflite detection model on frame to extract feature information, class scores, and bounding boxes
• Invoke Tensorflowlite - tflite recognition model to extract object signatures
• Process polygon intersection algorithm to identify the camera zones
• Generate and save the metadata

Additional activities related to firmware and C++ development may include:

• Use Valgrind to debug memory leaks
• Debug socket, threading, and networking issues
• Create new SOAP/XML methods

By the end of the project, the student will provide the following

• Sample code to process fixed frame images to obtain required metadata, with performance benchmarking of results.
• Integrate sample code into main code body
• Verify performance and accuracy of the model by checking metadata output
• Documentation of the code

Bonus steps in the process would also include:

• Develop SOAP/XML for server communication
• Develop socket enabled python client for firmware updates management.

Students will join a daily check in over Google hangouts with a senior developer to discuss their progress and review any challenges. They will be given code snippets and working code, overall program architecture, algorithms and strategies, and have to work with snippets to navigate their way through C++ development.

They will have access to the remote development environment or can work onsite.