Using Lepton JPG compression on Linux

How to download, compile and use Dropbox’s Lepton compression software on Linux, reducing your disk storage requirements for JPEG images.


Reducing your cloud storage bills

Like most people who treasures their families digital photos I backup all my photo albums off-site to the cloud. I pay to have my entire photo collection archived on Amazon’s S3 simple storage service. Amazon is my preferred off-site storage solution which offers a good trade off between price, availability and redundancy.

While it’s relatively cheap to store your files on S3, especially when you use the infrequent access storage class, your monthly bills can start to mount up when you’re storing many thousands of photographs. Any compression software which allows you to reduce your ongoing monthly bills is certainly worth investigating.

Dropbox developed Lepton to compress the billions of JPEG images they have stored on their infrastructure. Since its release in July 2016 it has saved many multiple petabytes of disk space for the company amounting to huge monetary savings.

Thankfully, the developers at Dropbox released the software as open source, so we can all take benefit of the 22% savings in disk space it provides. Lepton preserves the original file bit-for-bit which makes it ideal for archiving and backup purposes. The software achieves its impressive compression ratio by predicting coefficients in JPEG images and feeding those predictions as context into an arithmetic coder. If you’re interested in reading how the coder actually works there is a great blog post which explains how the compression and decompression works.

Now we’ll look at how to download, compile and use Lepton on our photo collections.

Installing the required dependencies

Before you start you’ll need to make sure your Linux set-up has the required tools and dependencies to allow you to compile the Lepton source code into an executable program. The tools you’ll need to compile and build the software are automake, autoconf and a C compiler like GCC. You will also need to install the version control system Git so that you can download the source files from the projects GitHub page.

To install these tools on an Ubuntu based Linux system you would issue the following commands from a terminal session:

sudo apt-get update
sudo apt-get install automake autoconf git g++

Compiling Lepton

Before you can compile the Lepton source code into an executable program, you’ll need to download the latest version of the source files from the projects GitHub page. To do this, you can use the Git version control system to clone the files to your computer.

git clone https://github.com/dropbox/lepton.git

This will download all the appropriate files to a sub folder on your computer called lepton. To successfully compile the software you need to change into the lepton folder.

cd lepton

Finally you’re ready to compile the software, which is done by running the four commands shown below in order.

./autogen.sh
./configure
make
make check

The final command runs a variety of checks to ensure that the resulting lepton program has been correctly compiled. You should see a final readout something like:

=================================
Testsuite summary for lepton 0.01
=================================
# TOTAL: 39
# PASS:  39
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
=================================

If the number of passed tests matches the total number of tests performed, then you’re good to go. You can now copy the lepton program file to the bin folder within your home folder so that the program is accessible from any location on your computer. In the copy command shown below the tilde character is a bash shortcut for your home folder.

cp lepton ~/bin/

Compressing a JPEG image

To compress a given image file you need to specify the input and output file names. It’s a good idea to change the output file names so they have an extension of .lep - like:

lepton - < skiing2016.jpg > skiing2016.lep
Skiing photograph used in Lepton compression test

The results for compressing the above image are shown below. The saving was an impressive 20.91%

skiing2016.jpg      1,967,613 bytes
skiing2016.lep      1,556,322 bytes

Saving = 20.91 percent

To get the original file back from the compressed lepton version you just change the parameters around.

lepton - < skiing2016.lep > skiing2016.jpg

Writing a BASH script to compress a folder full of images

The current version of lepton doesn’t support the use of wildcards so you can’t compress all the JPEG images in a given folder by default. However, you can use the bash script shown below to process and compress all the files in the current folder. Create a file called lep.sh in your bin folder like:

nano ~/bin/lep.sh

And then paste the following script into the text editor and save the file. The script works by assigning the current working directory / folder name to the variable folder. It then assigns a search pattern which is the folder name followed by the wild card which will find all the .jpg files. It then uses the ls (list directory contents) command to get a list of files which match the search pattern.

Working through each file found it assigns a suitable output file name by getting the basename of the original file and appending the .lep extension. The bash script then calls the lepton command on each JPEG found and creates the compressed output file in the same folder as the original image.

#!/bin/bash

folder=${PWD}/
pattern=$folder*.jpg

for file in `ls $pattern`
do
  if [ -f $file ];then
    outputfile=$(basename "$file" .jpg)
    outputfile=$outputfile.lep

    lepton - < $file > $folder$outputfile
  fi
done
exit

You’ll need to make the bash script executable which you can do by using the chmod command:

chmod +x ~/bin/lep.sh

Running the script lep.sh from within a folder full of images will compress each image in turn and create the associated .lep file. It would be these compressed .lep files that you upload to your cloud account for backup. Once you had synced all the .lep files to your cloud account then you could remove them from your local computer to reclaim the space.


Book sample

This article has been a short extract from my upcoming book. Archiving Digital Photographs on Linux covers many Linux commands to help you manage and protect your digital photographs ensuring they will be available for future generations to enjoy.

The book is available for pre-order on Smashwords.