Originally published by Jun 23, 2017
When most people hear the term “JPEG decoding,” they will usually assume that it’s something really difficult, something that requires lots of processing power and complicated mathematics, something that is impossible – or at least impractical – on relatively cheap and slow 8-bit microcontroller platforms like the Arduino. In this article, we’ll learn how to take JPEG photo using an Arduino-controlled camera, turn the photo into lots and lots of pixels, and to transmit all of them via serial port to our PC – or wherever we like!
Even though what is described above is entirely possible, it is worth mentioning why exactly are we going into all the trouble of decoding a JPEG photo. After all, there’s an SD module listed in the hardware requirements above, and you’d ask “Can we just store the photo on the SD card as a photo.jpeg file?” Sure, that is actually an important part of the entire process, but try to look at this from a different perspective: What if we want to send that photo somewhere using a slow, somewhat unreliable connection? If we simply chopped up the JPEG photo into packages and send them via a slow connection, we risk that some of them might get corrupted, while others may get lost entirely. When that happens, we most likely won’t be able to restore the original photo from the corrupted data.
However, when we decode the JPEG into bitmap, and then send the actual pixels, we risk nothing. If some of the data gets corrupted or lost during the transmission, we will still have an image, only with the corrupted data somehow discolored, misplaced, or simply missing. Granted, it’s not the same picture we originally started with, but it still carries most of the original information and is still “human-readable”. Now that we know why we’re doing this, let’s take a look at how we can approach this method.
Before we start decoding JPEG photos, first we have to take the photos. Since our ultimate goal is to take a photo, store it on a SD card and then send it somewhere. Let’s start with a simple setup that will allow us to do this.
Figure 1. The setup that will allow us to take and store photos using Arduino.
Since we need quite a bit of RAM to decode the photos, we’ll be using Arduino Mega. Also, there’s an added bonus in the form of four separate hardware serial ports on Mega, so we can use port Serial1 to communicate with the camera, and port Serial to communicate with our PC.
You probably noticed there’s a simple resistor voltage divider on the camera RX line. This is because the logic level of the VC0706 chip is 3.3 V (even though the supply voltage is 5 V), but the logic level of Arduino Mega is 5 V. Here’s a friendly advice: always use at least a voltage divider on the RX line when interfacing 5 V Arduino with 3.3 V modules. It’s much quicker than waiting until a new module arrives. The SD card reader is connected directly by SPI interface.
Now that the hardware is set up, we need to get the code sorted out. Since the library for SD cards is already a part of the standard Arduino IDE installation, we can check the SD card off the list.
The other device we have to control is the VC0706 camera. The controls are relatively simple, we just have to send some commands using the serial line and receive the JPEG photo using the same line. We could write a library to do this, but since we don’t really care about the overall sketch size at this point, we are going to use a VC0706 library developed by Adafruit. To take a picture and save it on the SD card, we’re going to use the following code, which is a slightly modified Snapshot example provided with the library.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 |
// Include all the libraries #include <Adafruit_VC0706.h> #include <SPI.h> #include <SD.h> // Define Slave Select pin #define SD_CS 53 // Create an instance of Adafruit_VC0706 class // We will use Serial1 for communication with the camera Adafruit_VC0706 cam = Adafruit_VC0706(&Serial1); void setup() { // Begin Serial port for communication with PC Serial.begin(115200); // Start the SD if(!SD.begin(SD_CS)) { // If the SD can't be started, loop forever Serial.println("SD failed or not present!"); while(1); } // Start the camera if(!cam.begin()) { // If the camera can't be started, loop forever Serial.println("Camera failed or not present!"); while(1); } // Set the image size to 640x480 cam.setImageSize(VC0706_640x480); } void loop() { Serial.print("Taking picture in 3 seconds ... "); delay(3000); // Take a picture if(cam.takePicture()) { Serial.println("done!"); } else { Serial.println("failed!"); } // Create a name for the new file in the format IMAGExy.JPG char filename[13]; strcpy(filename, "IMAGE00.JPG"); for(int i = 0; i < 100; i++) { filename[5] = '0' + i/10; filename[6] = '0' + i%10; if(!SD.exists(filename)) { break; } } // Create a file with the name we created above and open it File imgFile = SD.open(filename, FILE_WRITE); // Get the size of the image uint16_t jpglen = cam.frameLength(); Serial.print("Writing "); Serial.print(jpglen, DEC); Serial.print(" bytes into "); Serial.print(filename); Serial.print(" ... "); // Read all the image data while(jpglen > 0) { // Load the JPEG-encoded image data from the camera into a buffer uint8_t *buff; uint8_t bytesToRead = min(32, jpglen); buff = cam.readPicture(bytesToRead); // Write the image data to the file imgFile.write(buff, bytesToRead); jpglen -= bytesToRead; } // Safely close the file imgFile.close(); Serial.println("done!"); delay(3000); } |
Now, the Arduino will take a picture every 10 seconds or so until we run out of space on the SD card. But since the photos are typically around 48 kB, and I’m currently using 2 GB SD card, we have enough space for more than 43,000 photos. It seems reasonable to say that we don’t need that many. But now that we have some photos taken, we can now move on to the fun stuff: turning them from JPEG-compressed hard-to-manage gibberish into a simple array of pixels!
Before we start decoding, let’s take a quick look at how exactly the picture data are stored inside a JPEG file. If you don’t really care about this, please feel free to skip the next three paragraphs. If you actually know a thing or two about graphics and compressions – unlike me – you could also skip this part, as well. The following text is simplified to an extent.
When we talk about storing any sort of picture data, there are two basic approaches: lossless or lossy compression. The difference between the two is clear: when image is encoded using lossless compression, PNG for example, every pixel is exactly the same as when you started at the end of the process. This is great for things like computer graphics, but unfortunately, it comes at a cost of increased file size. On the other hand, with lossy compression like JPEG, we lose some details but the resulting file size is much smaller.
The way this is achieved in JPEG can be somewhat challenging to grasp since it involves a little something called “discrete cosine transformation”, but the main principle is actually pretty simple. First, the picture is converted from RGB color space into YCbCr. We all know RGB color space – it stores colors as values of red (R), green (G) and blue (B). YCbCr is quite different – it uses luminance (Y – basically the original image in grayscale), blue-difference chroma component (Cb – “blueness” of the picture) and red-difference chroma component (Cr – “redness” of the picture).
Figure 2. JPEG photo with the components separated. Top left is the original image, bottom left is the Y component, top right is the Cb component and the bottom right is the Cr component.
The way JPEG achieves the reduction in file size is actually closely related to the way human eyes process colors. Take a look at the three pictures of the Y, Cb and Cr components in the above picture. Which one looks more like the original picture? That’s right, the grayscale one! This is because the human eye is much more sensitive to luminance than to the other two components. JPEG compression uses this in a very clever way that allows it to reduce the amount of information in the Cb and Cr components while keeping the original Y component. This leads to a picture that is much smaller than the original file, and because most the compressed information was in the components human eyes aren’t too sensitive towards, you can barely notice the difference of a compressed picture in comparison to an uncompressed one.
Now let’s run a code that does the actual magic of turning JPEG into an array of pixels. Fortunately, there is a library that does exactly that – Bodmer’s JPEGDecoder (available on GitHub) which is based on an excellent picojpeg library by Rich Geldreich (also on GitHub). Even though JPEGDecoder was originally written to display images on TFT display, with a few minor tweaks it will work just fine for us.
Using the library is fairly simple: we give it the JPEG file, and the library will start generating arrays of pixels – so called Minimum Coded Units, or MCUs for short. The MCU is a block of 16 by 8 pixels. The functions in the library will return the color value for each pixel as 16-bit color value. The upper 5 bits are the red value, the middle 6 are green and the lower 5 are blue. Now we can send these values by any sort of communication channel we like. I’m going to use Serial port so that we can easily receive the data later. The following Arduino sketch decodes an image, then sends the 16-bit RGB value for each pixel in the MCU and repeats this for all the MCUs in the image file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
// Include the library #include <JPEGDecoder.h> // Define Slave Select pin #define SD_CS 53 void setup() { // Set pin 13 to output, otherwise SPI might hang pinMode(13, OUTPUT); // Begin Serial port for communication with PC Serial.begin(115200); // Start the SD if(!SD.begin(SD_CS)) { // If the SD can't be started, loop forever Serial.println("SD failed or not present!"); while(1); } // Open the root directory File root = SD.open("/"); // Wait for the PC to signal while(!Serial.available()); // Send all files on the SD card while(true) { // Open the next file File jpgFile = root.openNextFile(); // We have sent all files if(!jpgFile) { break; } // Decode the JPEG file JpegDec.decodeSdFile(jpgFile); // Create a buffer for the packet char dataBuff[240]; // Fill the buffer with zeros initBuff(dataBuff); // Create a header packet with info about the image String header = "$ITHDR,"; header += JpegDec.width; header += ","; header += JpegDec.height; header += ","; header += JpegDec.MCUSPerRow; header += ","; header += JpegDec.MCUSPerCol; header += ","; header += jpgFile.name(); header += ","; header.toCharArray(dataBuff, 240); // Send the header packet for(int j=0; j<240; j++) { Serial.write(dataBuff[j]); } // Pointer to the current pixel uint16_t *pImg; // Color of the current pixel uint16_t color; // Create a data packet with the actual pixel colors strcpy(dataBuff, "$ITDAT"); uint8_t i = 6; // Repeat for all MCUs in the image while(JpegDec.read()) { // Save pointer the current pixel pImg = JpegDec.pImage; // Get the coordinates of the MCU we are currently processing int mcuXCoord = JpegDec.MCUx; int mcuYCoord = JpegDec.MCUy; // Get the number of pixels in the current MCU uint32_t mcuPixels = JpegDec.MCUWidth * JpegDec.MCUHeight; // Repeat for all pixels in the current MCU while(mcuPixels--) { // Read the color of the pixel as 16-bit integer color = *pImg++; // Split it into two 8-bit integers dataBuff[i] = color >> 8; dataBuff[i+1] = color; i += 2; // If the packet is full, send it if(i == 240) { for(int j=0; j<240; j++) { Serial.write(dataBuff[j]); } i = 6; } // If we reach the end of the image, send a packet if((mcuXCoord == JpegDec.MCUSPerRow - 1) && (mcuYCoord == JpegDec.MCUSPerCol - 1) && (mcuPixels == 1)) { // Send the pixel values for(int j=0; j<i; j++) { Serial.write(dataBuff[j]); } // Fill the rest of the packet with zeros for(int k=i; k<240; k++) { Serial.write(0); } } } } } // Safely close the root directory root.close(); } // Function to fill the packet buffer with zeros void initBuff(char* buff) { for(int i = 0; i < 240; i++) { buff[i] = 0; } } void loop() { // Nothing here // We don't need to send the same images over and over again } |
Most of the code is explained in the comments, but I want to take some time to explain the “packets”. To keep the data transfer well organized, everything is transferred in packets, which are at most 240 bytes long. There are two possible types of packets:
The packet length might seem random at first glance. Why exactly 240 bytes? Why not 256 so we can send two MCUs in each separate packet? That’s a mystery we will solve another day, but I can assure you that there is nothing random about the number 240. Here’s a little hint: If I had 256 bytes of data in the packets, where would I store the source and destination addresses?
Now we have a code that will decode and send picture files, but there’s still one core feature missing: right now, there’s nothing listening for those data at the other end. This means it’s time to start up Processing again!
I covered a little bit of Processing in Arduino Hexapod PART 3: REMOTE CONTROL to write an app that allowed us to easily control the hexapod. For a quick refresher: Processing is a Java-based language that is primarily focused on drawing stuff. This makes it perfect for what we need to do, which is displaying pixels! This program does just that.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 |
// Import the library import processing.serial.*; Serial port; void setup() { // Set the default window size to 200 by 200 pixels size(200, 200); // Set the background to grey background(#888888); // Set as high framerate as we can frameRate(1000000); // Start the COM port communication // You will have to replace "COM30" with the Arduino COM port number port = new Serial(this, "COM30", 115200); // Read 240 bytes at a time port.buffer(240); } // String to save the trimmed input String trimmed; // Buffer to save data incoming from Serial port byte[] byteBuffer = new byte[240]; // The coordinate variables int x, y, mcuX, mcuY; // A variable to measure how long it takes to receive the image long startTime; // A variable to save the current time long currentTime; // Flag to signal end of transmission boolean received = false; // Flag to signal reception of header packet boolean headerRead = false; // The color of the current pixel int inColor, r, g, b; // Image information variables int jpegWidth, jpegHeight, jpegMCUSPerRow, jpegMCUSPerCol, mcuWidth, mcuHeight, mcuPixels; // This function will be called every time any key is pressed void keyPressed() { // Send something to Arduino to signal the start port.write('s'); } // This function will be called every time the Serial port receives 240 bytes void serialEvent(Serial port) { // Read the data into buffer port.readBytes(byteBuffer); // Make a String out of the buffer String inString = new String(byteBuffer); // Detect the packet type if(inString.indexOf("$ITHDR") == 0) { // Header packet // Remove all whitespace characters trimmed = inString.trim(); // Split the header by comma String[] list = split(trimmed, ','); // Check for completeness if(list.length != 7) { println("Incomplete header, terminated"); while(true); } else { // Parse the image information jpegWidth = Integer.parseInt(list[1]); jpegHeight = Integer.parseInt(list[2]); jpegMCUSPerRow = Integer.parseInt(list[3]); jpegMCUSPerCol = Integer.parseInt(list[4]); // Print the info to console println("Filename: " + list[5]); println("Parsed JPEG width: " + jpegWidth); println("Parsed JPEG height: " + jpegHeight); println("Parsed JPEG MCUs/row: " + jpegMCUSPerRow); println("Parsed JPEG MCUs/column: " + jpegMCUSPerCol); // Start the timer startTime = millis(); } // Set the window size according to the received information surface.setSize(jpegWidth, jpegHeight); // Get the MCU information mcuWidth = jpegWidth / jpegMCUSPerRow; mcuHeight = jpegHeight / jpegMCUSPerCol; mcuPixels = mcuWidth * mcuHeight; } else if(inString.indexOf("$ITDAT") == 0) { // Data packet // Repeat for every two bytes received for(int i = 6; i < 240; i += 2) { // Combine two 8-bit values into a single 16-bit color inColor = ((byteBuffer[i] & 0xFF) << 8) | (byteBuffer[i+1] & 0xFF); // Convert 16-bit color into RGB values r = ((inColor & 0xF800) >> 11) * 8; g = ((inColor & 0x07E0) >> 5) * 4; b = ((inColor & 0x001F) >> 0) * 8; // Paint the current pixel with that color set(x + mcuWidth*mcuX, y + mcuHeight*mcuY, color(r, g, b)); // Move onto the next pixel x++; if(x == mcuWidth) { // MCU row is complete, move onto the next one x = 0; y++; } if(y == mcuHeight) { // MCU is complete, move onto the next one x = 0; y = 0; mcuX++; } if(mcuX == jpegMCUSPerRow) { // Line of MCUs is complete, move onto the next one x = 0; y = 0; mcuX = 0; mcuY++; } if(mcuY == jpegMCUSPerCol) { // The entire image is complete received = true; } } } } void draw() { // If we received a full image, start the whole process again if(received) { // Reset coordinates x = 0; y = 0; mcuX = 0; mcuY = 0; // Reset the flag received = false; // Measure how long the whole thing took long timeTook = millis() - startTime; println("Image receiving took: " + timeTook + " ms"); println(); } } |
When you run this program with the Arduino connected, and then press any key on your keyboard, you will (hopefully) see the dull, boring gray background being gradually replaced by the image that was originally stored on the SD card. And since the replacement is done pixel by pixel, the entire process has a sort of old-school, dial-up-modem style of loading the image!
Figure 3. Photo being loaded from Arduino into PC using the Processing app.
Even though we are running the serial port at a fairly high baud rate – 115200 to be exact – receiving one image takes about 60 seconds. We can use that to calculate the real transfer speed.
The original image is 640 pixels wide and 480 pixels tall for a total of 307,200 pixels. Each of these pixels is represented by 2-byte color value, that is a total of 614,400 bytes – or 600 kilobytes – to transfer. This leaves us with the final speed of about 10 kB/s. That’s not that terrible for a “protocol” that we just made up on the go, isn’t it? Also, it shows you why image compression is so useful. The original JPEG file was only around 48 kB, while the decoded bitmap takes 600 kB. If we were to transfer the JPEG file, we would be done with it in less than 5 seconds, even when using our extremely simple “protocol.” Of course, we most likely wouldn’t be able to retrieve any data in case the transfer fails – which is something that cannot happen now.
Finally, we have proven what this article started with: processing images on Arduino is possible and can be even useful in certain situations. We can now snap pictures using a serial camera, decode them, send them over a serial port and then receive them on the other side! Consider this article your short intro into image processing on Arduino.
As usual, there is a lot of things that can be improved. One major addition could be to encrypt our messages using AES, which is fairly easy to implement, even on Arduino. Security is usually dangerously overlooked on Arduino, so we might focus a bit more on that in another project.
Thank you for reading the article and stay tuned for other exciting projects! Maybe some projects that will use all that we have just learned here!
Discover all of the powerful capabilities of Arduino with more of our comprehensive guides:
