This testbed belongs to our work SoK: Security of the Image Processing Pipeline in Autonomous Vehicles, where we bridge the research on security and robustness of the image processing pipeline for autonomous vehicles. This pipeline consists of multiple layers (see below), which can be configured and analyzed with this testbed.
This image sensor testbed contains multiple tools to capture, analyze and work with raw images (DNG format) and processed images. This test bed was tested with the following setup:
The following sections highlight the individual tools
This tool allows to captur images with manual settings. It acts as a wrapper of the rpicam toolchain.
- Python 3.11
- Tkinter (
pip install tk) - Pillow (
pip install pillow) - ImageTk (
sudo apt-get install python-imaging-tk) - tkinter-tooltip (
pip install tkinter-tooltip)
- Tkinter (
rpicamapps
- Adjust the parameters with GUI: Start the camera wrapper via
python camWrapper.py- Adjust you camera settings (lower right corner)
- Click "Capture Preview" to capture a (jpeg-only) preview with the current settings
- Click "Capture measurement" to capture both a raw and a high-quality JPEG image with the current settings. This will additionally store a txt file with the specified parameters.
- Run the tool automated w/o GUI but providing of a text-based config file:
python camWrapper.py <path/to/config.txt>- Define the camera parameters as specified online
- Execute the tool to create a measurement
- How can I reset the parameters in the GUI?
- For text fields / sliders: Move your mouse over the desired field and follow the instructions of the tool tip
- For selection fields: Select the first entry in the list
- Please note: The text fields must not be emtpy
This tool allows to compare two images. It will both calculate comparison metrics and provide a visualization of the results. Addtionally, it allows cropping to investigate only the region of interest (ROI). For JPEG images only: Since the analysis is done for each color channel, you can toggle the different viewing options with a double right-click to a blank area in the displayed graphs.
- Python 3.11
- Rawpy (
pip install rawpy) - Matplotlib (
pip install matplotlib) - Numpy (
pip install numpy) - Pillow (
pip install pillow)
- Rawpy (
General syntax of the tool:
python imageComparison.py <image 1> <image 2> [<x1,y1> <x2,y2>]image 1is a file path to the first image (=reference image). It can be a raw image in the DNG format or a JPEG image. Example:measurements/image1.dngimage 2is a file path to the second image (=comparison image). It can be a raw image in the DNG format or a JPEG image. Example:measurements/image2.dng- Optionally the tool can directly analyze only the ROI
<x1,y1>is the upper left corner for the ROI. Example:100,100<x2,y2>is the lower right corner for the ROI. Example:201,201
- The tool complains, that my ROI is not working
- Make sure, that the ROI follows the Bayer pattern, e.g., the upper left corner will have only even coordinates and the lower right corner will have only odd coordinates.
This tool represents the functionality of a object detection algorithm in the application layer. Addtionally, it allows cropping to investigate only the region of interest (ROI)
- Python 3.11
- PyTorch (see https://pytorch.org/get-started/locally/)
- Rawpy (
pip install rawpy)
General syntax of the tool:
python imageDetection.py <image> [<x1,y1> <x2,y2>]imageis a file path to the image to analyze. Example:measurements/image1.jpg- Optionally the tool can directly analyze only the ROI
<x1,y1>is the upper left corner for the ROI. Example:100,100<x2,y2>is the lower right corner for the ROI. Example:201,201
- The tool complains, that my ROI is not working
- Make sure, that the ROI follows the Bayer pattern, e.g., the upper left corner will have only even coordinates and the lower right corner will have only odd coordinates.