Decompressing data with cc65

Colin Leroy-Mira

1. Overview

2. LZ4

3. LZSA

4. ZX02

5. In-place decompression

6. Which decompressor to choose

1. Overview

cc65 ships with multiple decompressors, each having pros and cons. This page will detail each of them, and how to use them.

2. LZ4

The LZ4 format is widely known. It has a number of drawbacks, though:

There are many LZ4 subformats available, and generating LZ4 data compatible with the cc65 decompressor requires some work. The cc65 decompressor works on raw LZ4 data with no header, which makes it difficult to generate compressed data simply using the lz4 command-line utility.

This also means that the function needs to be passed the expected decompressed size as argument. This makes generating compressed LZ4 data even harder.

The simplest way to generate "correct" LZ4 data for the cc65 decompressor is to write a small C utility, like this one (example stripped of any error checking):

  FILE *fp;
  size_t read_bytes, wrote_bytes;
  char header[2];
  char in_buf[MAX_COMPRESSED_DATA_SIZE*16];
  char out_buf[MAX_COMPRESSED_DATA_SIZE];

  fp = fopen(argv[1], "rb");
  read_bytes = fread(in_buf, 1, sizeof(in_buf), fp);
  fclose(fp);

  wrote_bytes = LZ4_compress_HC(in_buf, out_buf, read_bytes, sizeof(out_buf), 16);
  header[0] = (read_bytes & 0xFF);
  header[1] = (read_bytes & 0xFFFF) >> 8;

  fp = fopen("DATA.LZ4", "wb");
  fwrite(header, 1, sizeof(header), fp);
  fwrite(out_buf, 1, wrote_bytes, fp);
  fclose(fp);

Decompressing in a cc65 program then looks like:

  int fd;
  int decompressed_size;
  char compressed[MAX_COMPRESSED_DATA_SIZE];
  char destination[MAX_COMPRESSED_DATA_SIZE*16];

  fd = open("DATA.LZ4", O_RDONLY);
  read(fd, &decompressed_size, sizeof(int));
  read(fd, compressed, MAX_COMPRESSED_DATA_SIZE);
  close(fd);

  decompress_lz4(compressed, destination, decompressed_size);

3. LZSA

The LZSA formats come from Emmanuel Marty and has its code hosted in the Github LZSA repository.

Compressing data is simple, from a command-line or shell:

  lzsa -r -f 1 input.bin DATA.LZSA #For lzsa1 format
  lzsa -r -f 2 input.bin DATA.LZSA #For lzsa2 format

Decompressing is then as simple as possible:

  int fd;
  char compressed[MAX_COMPRESSED_DATA_SIZE];
  char destination[MAX_COMPRESSED_DATA_SIZE*16];

  fd = open("DATA.LZSA", O_RDONLY);
  read(fd, compressed, MAX_COMPRESSED_DATA_SIZE);
  close(fd);

  decompress_lzsa1(compressed, destination);
  // or
  // decompress_lzsa2(compressed, destination);

4. ZX02

The ZX02 formats come from dmsc and has its code hosted in the Github ZX02 repository.

Compressing data is simple, from a command-line or shell:

  zx02 -f input.bin DATA.ZX02

Decompressing is then as simple as possible:

  int fd;
  char compressed[MAX_COMPRESSED_DATA_SIZE];
  char destination[MAX_COMPRESSED_DATA_SIZE*16];

  fd = open("DATA.ZX02", O_RDONLY);
  read(fd, compressed, MAX_COMPRESSED_DATA_SIZE);
  close(fd);

  decompress_zx02(compressed, destination);

5. In-place decompression

As memory is often sparse in the cc65 targets, it is often possible to decompress data "in-place", requiring only the destination buffer and no compressed data buffer. But as the cc65 decompressors do not support backwards compressed data, it is necessary to have the compressed data at the end of the destination buffer, and that the destination buffer has a few extra bytes (8 are enough) so that decompressing does not overwrite the end of the compressed data too soon:

  #define BUFFER_SIZE = (MAX_UNCOMPRESSED_DATA_SIZE) + 8;

  int fd;
  int compressed_size;
  char dest_buf[BUFFER_SIZE];
  char *end_of_buf = dest_buf + BUFFER_SIZE;

  fd = open("DATA.ZX02", O_RDONLY);
  compressed_size = read(fd, dest_buf, MAX_UNCOMPRESSED_DATA_SIZE);
  close(fd);
  memmove(end_of_buf - compressed_size, dest_buf, compressed_size);
  decompress_zx02(end_of_buf - compressed_size, dest_buf);

6. Which decompressor to choose

The best decompressor depends on your use-case and whether you favor size or speed. This table allows for a simple comparison. Decompression speed is the number of uncompressed bytes per second at 1MHz.

In short, if you want to have the smallest amount of data, go with ZX02. If you want the fastest decompression speed, go with LZSA1.