JPEG Decoding on Arduino Tutorial

arduino jpeg

arduino jpeg

When most people hear the term “JPEG decoding,” they will usually assume that it’s something really difficult, something that requires lots of processing power and complicated mathematics, something that is impossible – or at least impractical – on relatively cheap and slow 8-bit microcontroller platforms like the Arduino. In this article, we’ll learn how to take JPEG photo using an Arduino-controlled camera, turn the photo into lots and lots of pixels, and to transmit all of them via serial port to our PC – or wherever we like!

Hardware

  • Arduino Mega
  • VC0706 Serial camera
  • SD card module with SPI interface

Software

  • Arduino IDE
  • Processing (version 3.3.2 or higher)
  • Adafruit VC0706 library (available on GitHub)
  • Bodmer’s JPEGDecoder library (also on GitHub)

Even though what is described above is entirely possible, it is worth mentioning why exactly are we going into all the trouble of decoding a JPEG photo. After all, there’s an SD module listed in the hardware requirements above, and you’d ask “Can we just store the photo on the SD card as a photo.jpeg file?” Sure, that is actually an important part of the entire process, but try to look at this from a different perspective: What if we want to send that photo somewhere using a slow, somewhat unreliable connection? If we simply chopped up the JPEG photo into packages and send them via a slow connection, we risk that some of them might get corrupted, while others may get lost entirely. When that happens, we most likely won’t be able to restore the original photo from the corrupted data.

However, when we decode the JPEG into bitmap, and then send the actual pixels, we risk nothing. If some of the data gets corrupted or lost during the transmission, we will still have an image, only with the corrupted data somehow discolored, misplaced, or simply missing. Granted, it’s not the same picture we originally started with, but it still carries most of the original information and is still “human-readable”. Now that we know why we’re doing this, let’s take a look at how we can approach this method.

 

Taking pictures

Before we start decoding JPEG photos, first we have to take the photos. Since our ultimate goal is to take a photo, store it on a SD card and then send it somewhere. Let’s start with a simple setup that will allow us to do this.

arduino jpeg

Figure 1. The setup that will allow us to take and store photos using Arduino.

Since we need quite a bit of RAM to decode the photos, we’ll be using Arduino Mega. Also, there’s an added bonus in the form of four separate hardware serial ports on Mega, so we can use port Serial1 to communicate with the camera, and port Serial to communicate with our PC.

You probably noticed there’s a simple resistor voltage divider on the camera RX line. This is because the logic level of the VC0706 chip is 3.3 V (even though the supply voltage is 5 V), but the logic level of Arduino Mega is 5 V. Here’s a friendly advice: always use at least a voltage divider on the RX line when interfacing 5 V Arduino with 3.3 V modules. It’s much quicker than waiting until a new module arrives. The SD card reader is connected directly by SPI interface.

Now that the hardware is set up, we need to get the code sorted out. Since the library for SD cards is already a part of the standard Arduino IDE installation, we can check the SD card off the list.

The other device we have to control is the VC0706 camera. The controls are relatively simple, we just have to send some commands using the serial line and receive the JPEG photo using the same line. We could write a library to do this, but since we don’t really care about the overall sketch size at this point, we are going to use a VC0706 library developed by Adafruit. To take a picture and save it on the SD card, we’re going to use the following code, which is a slightly modified Snapshot example provided with the library.

// Include all the libraries
#include <Adafruit_VC0706.h>
#include <SPI.h>
#include <SD.h>

// Define Slave Select pin
#define SD_CS  53

// Create an instance of Adafruit_VC0706 class
// We will use Serial1 for communication with the camera
Adafruit_VC0706 cam = Adafruit_VC0706(&Serial1);

void setup() {
  // Begin Serial port for communication with PC
  Serial.begin(115200);

  // Start the SD
  if(!SD.begin(SD_CS)) {
    // If the SD can't be started, loop forever
    Serial.println("SD failed or not present!");
    while(1);
  }

  // Start the camera
  if(!cam.begin()) {
    // If the camera can't be started, loop forever
    Serial.println("Camera failed or not present!");
    while(1);
  }

  // Set the image size to 640x480
  cam.setImageSize(VC0706_640x480);
}

void loop() {
  Serial.print("Taking picture in 3 seconds ... ");
  delay(3000);
  
  // Take a picture
  if(cam.takePicture()) {
    Serial.println("done!");
  } else {
    Serial.println("failed!");
  }

  // Create a name for the new file in the format IMAGExy.JPG
  char filename[13];
  strcpy(filename, "IMAGE00.JPG");
  for(int i = 0; i < 100; i++) {
    filename[5] = '0' + i/10;
    filename[6] = '0' + i%10;
    if(!SD.exists(filename)) {
      break;
    }
  }

  // Create a file with the name we created above and open it
  File imgFile = SD.open(filename, FILE_WRITE);

  // Get the size of the image
  uint16_t jpglen = cam.frameLength();
  Serial.print("Writing ");
  Serial.print(jpglen, DEC);
  Serial.print(" bytes into ");
  Serial.print(filename);
  Serial.print(" ... ");

  // Read all the image data
  while(jpglen > 0) {
    // Load the JPEG-encoded image data from the camera into a buffer
    uint8_t *buff;
    uint8_t bytesToRead = min(32, jpglen);
    buff = cam.readPicture(bytesToRead);

    // Write the image data to the file
    imgFile.write(buff, bytesToRead);
    jpglen -= bytesToRead;
  }

  // Safely close the file
  imgFile.close();
  
  Serial.println("done!");
  delay(3000);
}

Now, the Arduino will take a picture every 10 seconds or so until we run out of space on the SD card. But since the photos are typically around 48 kB, and I’m currently using 2 GB SD card, we have enough space for more than 43,000 photos. It seems reasonable to say that we don’t need that many. But now that we have some photos taken, we can now move on to the fun stuff: turning them from JPEG-compressed hard-to-manage gibberish into a simple array of pixels!

 

Decoding & Sending pictures

Before we start decoding, let’s take a quick look at how exactly the picture data are stored inside a JPEG file. If you don’t really care about this, please feel free to skip the next three paragraphs. If you actually know a thing or two about graphics and compressions – unlike me – you could also skip this part, as well. The following text is simplified to an extent.

When we talk about storing any sort of picture data, there are two basic approaches: lossless or lossy compression. The difference between the two is clear: when image is encoded using lossless compression, PNG for example, every pixel is exactly the same as when you started at the end of the process. This is great for things like computer graphics, but unfortunately, it comes at a cost of increased file size. On the other hand, with lossy compression like JPEG, we lose some details but the resulting file size is much smaller.

The way this is achieved in JPEG can be somewhat challenging to grasp since it involves a little something called “discrete cosine transformation”, but the main principle is actually pretty simple. First, the picture is converted from RGB color space into YCbCr. We all know RGB color space – it stores colors as values of red (R), green (G) and blue (B). YCbCr is quite different – it uses luminance (Y – basically the original image in grayscale), blue-difference chroma component (Cb – “blueness” of the picture) and red-difference chroma component (Cr – “redness” of the picture).

arduino jpeg

Figure 2. JPEG photo with the components separated. Top left is the original image, bottom left is the Y component, top right is the Cb component and the bottom right is the Cr component.

The way JPEG achieves the reduction in file size is actually closely related to the way human eyes process colors. Take a look at the three pictures of the Y, Cb and Cr components in the above picture. Which one looks more like the original picture? That’s right, the grayscale one! This is because the human eye is much more sensitive to luminance than to the other two components. JPEG compression uses this in a very clever way that allows it to reduce the amount of information in the Cb and Cr components while keeping the original Y component. This leads to a picture that is much smaller than the original file, and because most the compressed information was in the components human eyes aren’t too sensitive towards, you can barely notice the difference of a compressed picture in comparison to an uncompressed one.

Now let’s run a code that does the actual magic of turning JPEG into an array of pixels. Fortunately, there is a library that does exactly that – Bodmer’s JPEGDecoder (available on GitHub) which is based on an excellent picojpeg library by Rich Geldreich (also on GitHub). Even though JPEGDecoder was originally written to display images on TFT display, with a few minor tweaks it will work just fine for us.

Using the library is fairly simple: we give it the JPEG file, and the library will start generating arrays of pixels – so called Minimum Coded Units, or MCUs for short. The MCU is a block of 16 by 8 pixels. The functions in the library will return the color value for each pixel as 16-bit color value. The upper 5 bits are the red value, the middle 6 are green and the lower 5 are blue. Now we can send these values by any sort of communication channel we like. I’m going to use Serial port so that we can easily receive the data later. The following Arduino sketch decodes an image, then sends the 16-bit RGB value for each pixel in the MCU and repeats this for all the MCUs in the image file.

// Include the library
#include <JPEGDecoder.h>

// Define Slave Select pin
#define SD_CS  53

void setup() {
  // Set pin 13 to output, otherwise SPI might hang
  pinMode(13, OUTPUT);

  // Begin Serial port for communication with PC
  Serial.begin(115200);

  // Start the SD
  if(!SD.begin(SD_CS)) {
    // If the SD can't be started, loop forever
    Serial.println("SD failed or not present!");
    while(1);
  }

  // Open the root directory
  File root = SD.open("/");
  
  // Wait for the PC to signal
  while(!Serial.available());

  // Send all files on the SD card
  while(true) {
    // Open the next file
    File jpgFile = root.openNextFile();

    // We have sent all files
    if(!jpgFile) {
      break;
    }

    // Decode the JPEG file
    JpegDec.decodeSdFile(jpgFile);

    // Create a buffer for the packet
    char dataBuff[240];

    // Fill the buffer with zeros
    initBuff(dataBuff);

    // Create a header packet with info about the image
    String header = "$ITHDR,";
    header += JpegDec.width;
    header += ",";
    header += JpegDec.height;
    header += ",";
    header += JpegDec.MCUSPerRow;
    header += ",";
    header += JpegDec.MCUSPerCol;
    header += ",";
    header += jpgFile.name();
    header += ",";
    header.toCharArray(dataBuff, 240);

    // Send the header packet
    for(int j=0; j<240; j++) {
      Serial.write(dataBuff[j]);
    }

    // Pointer to the current pixel
    uint16_t *pImg;

    // Color of the current pixel
    uint16_t color;

    // Create a data packet with the actual pixel colors
    strcpy(dataBuff, "$ITDAT");
    uint8_t i = 6;

    // Repeat for all MCUs in the image
    while(JpegDec.read()) {
      // Save pointer the current pixel
      pImg = JpegDec.pImage;

      // Get the coordinates of the MCU we are currently processing
      int mcuXCoord = JpegDec.MCUx;
      int mcuYCoord = JpegDec.MCUy;

      // Get the number of pixels in the current MCU
      uint32_t mcuPixels = JpegDec.MCUWidth * JpegDec.MCUHeight;

      // Repeat for all pixels in the current MCU
      while(mcuPixels--) {
        // Read the color of the pixel as 16-bit integer
        color = *pImg++;
        
        // Split it into two 8-bit integers
        dataBuff[i] = color >> 8;
        dataBuff[i+1] = color;
        i += 2;

        // If the packet is full, send it
        if(i == 240) {
          for(int j=0; j<240; j++) {
            Serial.write(dataBuff[j]);
          }
          i = 6;
        }

        // If we reach the end of the image, send a packet
        if((mcuXCoord == JpegDec.MCUSPerRow - 1) && 
          (mcuYCoord == JpegDec.MCUSPerCol - 1) && 
          (mcuPixels == 1)) {
          
          // Send the pixel values
          for(int j=0; j<i; j++) {
            Serial.write(dataBuff[j]);
          }
          
          // Fill the rest of the packet with zeros
          for(int k=i; k<240; k++) {
            Serial.write(0);
          }
        }
      }
    }
  }

  // Safely close the root directory
  root.close(); 
}

// Function to fill the packet buffer with zeros
void initBuff(char* buff) {
  for(int i = 0; i < 240; i++) {
    buff[i] = 0;
  }
}

void loop() {
  // Nothing here
  // We don't need to send the same images over and over again
}

Most of the code is explained in the comments, but I want to take some time to explain the “packets”. To keep the data transfer well organized, everything is transferred in packets, which are at most 240 bytes long. There are two possible types of packets:

  1. Header packet: This packet starts with the string “$ITHDR” and contains basic information about the image we will be sending: height and width in pixels, number of MCUs pre row and column and finally the original filename. For each image we want to send, we will send one header packet.
  2. Data packet: This packet starts with “$ITDAT” and contains all the color data. Every two bytes in this packet represent one 16-bit pixel.

The packet length might seem random at first glance. Why exactly 240 bytes? Why not 256 so we can send two MCUs in each separate packet? That’s a mystery we will solve another day, but I can assure you that there is nothing random about the number 240. Here’s a little hint: If I had 256 bytes of data in the packets, where would I store the source and destination addresses?

Now we have a code that will decode and send picture files, but there’s still one core feature missing: right now, there’s nothing listening for those data at the other end. This means it’s time to start up Processing again!

 

Receiving pictures

I covered a little bit of Processing in Arduino Hexapod PART 3: REMOTE CONTROL to write an app that allowed us to easily control the hexapod. For a quick refresher: Processing is a Java-based language that is primarily focused on drawing stuff. This makes it perfect for what we need to do, which is displaying pixels! This program does just that.

// Import the library
import processing.serial.*;

Serial port;

void setup() {
  // Set the default window size to 200 by 200 pixels
  size(200, 200);
  
  // Set the background to grey
  background(#888888);
  
  // Set as high framerate as we can
  frameRate(1000000);
  
  // Start the COM port communication
  // You will have to replace "COM30" with the Arduino COM port number
  port = new Serial(this, "COM30", 115200);
  
  // Read 240 bytes at a time
  port.buffer(240);
}

// String to save the trimmed input
String trimmed;

// Buffer to save data incoming from Serial port
byte[] byteBuffer = new byte[240];

// The coordinate variables
int x, y, mcuX, mcuY;

// A variable to measure how long it takes to receive the image
long startTime;

// A variable to save the current time
long currentTime;

// Flag to signal end of transmission
boolean received = false;

// Flag to signal reception of header packet
boolean headerRead = false;

// The color of the current pixel
int inColor, r, g, b;

// Image information variables
int jpegWidth, jpegHeight, jpegMCUSPerRow, jpegMCUSPerCol, mcuWidth, mcuHeight, mcuPixels;

// This function will be called every time any key is pressed
void keyPressed() {
  // Send something to Arduino to signal the start
  port.write('s');
}

// This function will be called every time the Serial port receives 240 bytes
void serialEvent(Serial port) {
  // Read the data into buffer
  port.readBytes(byteBuffer);
  
  // Make a String out of the buffer
  String inString = new String(byteBuffer);
  
  // Detect the packet type
  if(inString.indexOf("$ITHDR") == 0) {
    // Header packet
    
    // Remove all whitespace characters
    trimmed = inString.trim();
    
    // Split the header by comma
    String[] list = split(trimmed, ',');
    
    // Check for completeness
    if(list.length != 7) {
      println("Incomplete header, terminated");
      while(true);
    } else {
      // Parse the image information
      jpegWidth = Integer.parseInt(list[1]);
      jpegHeight = Integer.parseInt(list[2]);
      jpegMCUSPerRow = Integer.parseInt(list[3]);
      jpegMCUSPerCol = Integer.parseInt(list[4]);
      
      // Print the info to console
      println("Filename: " + list[5]);
      println("Parsed JPEG width: " + jpegWidth);
      println("Parsed JPEG height: " + jpegHeight);
      println("Parsed JPEG MCUs/row: " + jpegMCUSPerRow);
      println("Parsed JPEG MCUs/column: " + jpegMCUSPerCol);
      
      // Start the timer
      startTime = millis();
    }
    
    // Set the window size according to the received information
    surface.setSize(jpegWidth, jpegHeight);
    
    // Get the MCU information
    mcuWidth = jpegWidth / jpegMCUSPerRow;
    mcuHeight = jpegHeight / jpegMCUSPerCol;
    mcuPixels = mcuWidth * mcuHeight;
    
  } else if(inString.indexOf("$ITDAT") == 0) {
    // Data packet
    
    // Repeat for every two bytes received
    for(int i = 6; i < 240; i += 2) {
      // Combine two 8-bit values into a single 16-bit color
      inColor = ((byteBuffer[i] & 0xFF) << 8) | (byteBuffer[i+1] & 0xFF);
      
      // Convert 16-bit color into RGB values
      r = ((inColor & 0xF800) >> 11) * 8;
      g = ((inColor & 0x07E0) >> 5) * 4;
      b = ((inColor & 0x001F) >> 0) * 8;
      
      // Paint the current pixel with that color
      set(x + mcuWidth*mcuX, y + mcuHeight*mcuY, color(r, g, b));
      
      // Move onto the next pixel
      x++;
      
      if(x == mcuWidth) {
        // MCU row is complete, move onto the next one
        x = 0;
        y++;
      }
      
      if(y == mcuHeight) {
        // MCU is complete, move onto the next one
        x = 0;
        y = 0;
        mcuX++;
      }
      
      if(mcuX == jpegMCUSPerRow) {
        // Line of MCUs is complete, move onto the next one
        x = 0;
        y = 0;
        mcuX = 0;
        mcuY++;
      }
      
      if(mcuY == jpegMCUSPerCol) {
        // The entire image is complete
        received = true;
      }
    }
  }
}

void draw() {
  // If we received a full image, start the whole process again
  if(received) {
    
    // Reset coordinates
    x = 0;
    y = 0;
    mcuX = 0;
    mcuY = 0;
    
    // Reset the flag
    received = false;
    
    // Measure how long the whole thing took
    long timeTook = millis() - startTime;
    println("Image receiving took: " + timeTook + " ms");
    println();
  }
}

When you run this program with the Arduino connected, and then press any key on your keyboard, you will (hopefully) see the dull, boring gray background being gradually replaced by the image that was originally stored on the SD card. And since the replacement is done pixel by pixel, the entire process has a sort of old-school, dial-up-modem style of loading the image!

arduino jpeg

Figure 3. Photo being loaded from Arduino into PC using the Processing app.

Even though we are running the serial port at a fairly high baud rate – 115200 to be exact – receiving one image takes about 60 seconds. We can use that to calculate the real transfer speed.

The original image is 640 pixels wide and 480 pixels tall for a total of 307,200 pixels. Each of these pixels is represented by 2-byte color value, that is a total of 614,400 bytes – or 600 kilobytes – to transfer. This leaves us with the final speed of about 10 kB/s. That’s not that terrible for a “protocol” that we just made up on the go, isn’t it? Also, it shows you why image compression is so useful. The original JPEG file was only around 48 kB, while the decoded bitmap takes 600 kB. If we were to transfer the JPEG file, we would be done with it in less than 5 seconds, even when using our extremely simple “protocol.” Of course, we most likely wouldn’t be able to retrieve any data in case the transfer fails – which is something that cannot happen now.

 

Conclusion

Finally, we have proven what this article started with: processing images on Arduino is possible and can be even useful in certain situations. We can now snap pictures using a serial camera, decode them, send them over a serial port and then receive them on the other side! Consider this article your short intro into image processing on Arduino.

As usual, there is a lot of things that can be improved. One major addition could be to encrypt our messages using AES, which is fairly easy to implement, even on Arduino. Security is usually dangerously overlooked on Arduino, so we might focus a bit more on that in another project.

Thank you for reading the article and stay tuned for other exciting projects!  Maybe some projects that will use all that we have just learned here!

Jan Gromes
Jan Gromes
Jan is currently studying Electrical Engineering at Brno University of Technology. He has many years of experience building projects using Arduino and other microcontrollers. His special interest lies in mechanical design of robotic systems.

Check us out on Social Media