{
"id": "brianpugh/tamp",
"name": "tamp",
"prettyName": "tamp",
"description": "Tamp is a low-memory, DEFLATE-inspired lossless compression library.",
"version": "",
"author": "brianpugh",
"license": "Apache-2.0",
"tags": [
"c",
"compression",
"deflate",
"embedded",
"heatshrink",
"javascript",
"lossless",
"low-memory",
"lz77",
"lzss",
"microcontroller",
"micropython",
"python",
"zlib"
],
"status": "VALID",
"official": false,
"source": "https://github.com/brianpugh/tamp",
"updatedAt": "2025-08-08T07:00:07.152Z",
"page": "https://checkmim.com/packages/brianpugh+tamp",
"createdAt": "2025-05-27T04:28:11.120Z",
"submittedBy": "oliverr",
"install": {
"mpremote": "mpremote mip install github:brianpugh/tamp",
"mip": "import mip; mip.install(\"github:brianpugh/tamp\")"
},
"readme": "\n
![](\"https://raw.githubusercontent.com/BrianPugh/tamp/main/assets/logo_300w.png\")
\n
\n\n\n[](https://pypi.python.org/pypi/tamp)\n[](https://github.com/BrianPugh/tamp/actions?query=workflow%3Atests)\n[](https://codecov.io/github/BrianPugh/tamp?branch=main)\n[](https://tamp.readthedocs.io/en/latest/?badge=latest)\n\n
\n\n---\n\n**Documentation:** \n\n**Source Code:** \n\n**Online Demo:** \n\nTamp is a low-memory, DEFLATE-inspired lossless compression library optimized\nfor embedded and resource-constrained environments.\n\nTamp delivers the highest data compression ratios, while using the least amount\nof RAM and firmware storage.\n\n# Features\n\n- Various language implementations available:\n - Pure Python reference:\n - `tamp/__init__.py`, `tamp/compressor.py`, `tamp/decompressor.py`\n - `pip install tamp` will use a python-bound C implementation optimized for\n speed.\n - Micropython:\n - Native Module.\n - `mpy_bindings/`\n - C library:\n - `tamp/_c_src/`\n - Javascript/Typescript via Emscripten WASM.\n - `wasm/`\n - Unofficial [rust bindings](https://github.com/tmpfs/tamp-rs).\n - See documentation [here](https://docs.rs/tamp/latest/tamp/index.html).\n- High compression ratios, low memory use, and fast.\n- Compact compression and decompression implementations.\n - Compiled C library is <5KB (compressor + decompressor).\n- Mid-stream flushing.\n - Allows for submission of messages while continuing to compress subsequent\n data.\n- Customizable dictionary for greater compression of small messages.\n- Fuzz tested with libFuzzer + AddressSanitizer/UBSan.\n- Convenient CLI interface.\n\n# Installation\n\nTamp contains several implementations:\n\n1. A reference desktop CPython implementation that is optimized for readability\n (and **not** speed).\n2. A Micropython Native Module implementation (fast).\n3. A C implementation (with python bindings) for accelerated desktop use and to\n be used in C projects (very fast).\n4. A JavaScript/TypeScript implementation via Emscripten WASM (see `wasm/`).\n\nThis section instructs how to install each implementation.\n\n## Desktop Python\n\nThe Tamp library requires Python `>=3.9` and can be installed via:\n\n```bash\npip install tamp\n```\n\nTo also install the `tamp` command line tool:\n\n```bash\npip install tamp[cli]\n```\n\n## MicroPython\n\n### MicroPython Native Module\n\nTamp provides pre-compiled [native modules]{.title-ref} that are easy to\ninstall, are small, and are incredibly fast.\n\nDownload the appropriate `.mpy` file from the\n[release page](https://github.com/BrianPugh/tamp/releases).\n\n- Match the `.mpy` version to your micropython version: `mpy6.3` for v1.23+,\n `mpy6.2` for v1.22, `mpy6.1` for v1.20-v1.21.\n- Match the architecture to the microcontroller (e.g. `armv6m` for a pi pico).\n\nRename the file to `tamp.mpy` and transfer it to your board. If using\n[Belay](https://github.com/BrianPugh/belay), tamp can be installed by adding the\nfollowing to `pyproject.toml`.\n\n```toml\n[tool.belay.dependencies]\ntamp = \"https://github.com/BrianPugh/tamp/releases/download/v2.6.0/tamp-2.6.0-mpy6.3-armv6m.mpy\"\n```\n\n## C\n\nCopy the `tamp/_c_src/tamp` folder into your project. For more information, see\n[the documentation](https://tamp.readthedocs.io/en/latest/c_library.html).\n\n### Zephyr\n\nTamp is a\n[Zephyr module](https://docs.zephyrproject.org/latest/develop/modules.html). Add\nit to your `west.yml`:\n\n```yaml\nmanifest:\n remotes:\n - name: brianpugh\n url-base: https://github.com/BrianPugh\n projects:\n - name: tamp\n remote: brianpugh\n revision: main # pin to a release tag or commit\n path: modules/lib/tamp\n```\n\nThen enable it in `prj.conf`:\n\n```\nCONFIG_TAMP=y\n```\n\n`CONFIG_TAMP_COMPRESSOR` and `CONFIG_TAMP_DECOMPRESSOR` (both default `y`)\nselect which halves get compiled. Include the headers as `\"tamp/compressor.h\"`\nand `\"tamp/decompressor.h\"`.\n\n# Usage\n\nTamp works on desktop python and micropython. On desktop, Tamp can be bundled\nwith the `tamp` command line tool for compressing and decompressing tamp files.\nInstall with `pip install tamp[cli]`.\n\n## CLI\n\n### Compression\n\nUse `tamp compress` to compress a file or stream. If no input file is specified,\ndata from stdin will be read. If no output is specified, the compressed output\nstream will be written to stdout.\n\n```bash\n$ tamp compress --help\nUsage: tamp compress [ARGS] [OPTIONS]\n\nCompress an input file or stream.\n\n╭─ Parameters ───────────────────────────────────────────────────────────────────────────────╮\n│ INPUT,--input -i Input file to compress. Defaults to stdin. │\n│ OUTPUT,--output -o Output compressed file. Defaults to stdout. │\n│ --window -w Number of bits used to represent the dictionary window. [default: 10] │\n│ --literal -l Number of bits used to represent a literal. [default: 8] │\n╰────────────────────────────────────────────────────────────────────────────────────────────╯\n```\n\nExample usage:\n\n```bash\ntamp compress enwik8 -o enwik8.tamp # Compress a file\necho \"hello world\" | tamp compress | wc -c # Compress a stream and print the compressed size.\n```\n\nThe following options can impact compression ratios and memory usage:\n\n- `window` - `2^window` plaintext bytes to look back to try and find a pattern.\n A larger window size will increase the chance of finding a longer pattern\n match, but will use more memory, increase compression time, and cause each\n pattern-token to take up more space. Try smaller window values if compressing\n highly repetitive data, or short messages.\n- `literal` - Number of bits used in each plaintext byte. For example, if all\n input data is 7-bit ASCII, then setting this to 7 will improve literal\n compression ratios by 11.1%. The default, 8-bits, can encode any binary data.\n\n### Decompression\n\nUse `tamp decompress` to decompress a file or stream. If no input file is\nspecified, data from stdin will be read. If no output is specified, the\ncompressed output stream will be written to stdout.\n\n```bash\n$ tamp decompress --help\nUsage: tamp decompress [ARGS] [OPTIONS]\n\nDecompress an input file or stream.\n\n╭─ Parameters ───────────────────────────────────────────────────────────────────────────────╮\n│ INPUT,--input -i Input file to decompress. Defaults to stdin. │\n│ OUTPUT,--output -o Output decompressed file. Defaults to stdout. │\n╰────────────────────────────────────────────────────────────────────────────────────────────╯\n```\n\nExample usage:\n\n```bash\ntamp decompress enwik8.tamp -o enwik8\necho \"hello world\" | tamp compress | tamp decompress\n```\n\n## Python\n\nThe python library can perform one-shot compression, as well as operate on\nfiles/streams.\n\n```python\nimport tamp\n\n# One-shot compression\nstring = b\"I scream, you scream, we all scream for ice cream.\"\ncompressed_data = tamp.compress(string)\nreconstructed = tamp.decompress(compressed_data)\nassert reconstructed == string\n\n# Streaming compression\nwith tamp.open(\"output.tamp\", \"wb\") as f:\n for _ in range(10):\n f.write(string)\n\n# Streaming decompression\nwith tamp.open(\"output.tamp\", \"rb\") as f:\n reconstructed = f.read()\n```\n\n# Benchmark\n\nIn the following section, we compare Tamp against:\n\n- [zlib](https://docs.python.org/3/library/zlib.html), a python builtin\n gzip-compatible DEFLATE compression library.\n- [heatshrink](https://github.com/atomicobject/heatshrink), a data compression\n library for embedded/real-time systems. Heatshrink has similar goals as Tamp.\n\nAll of these are LZ-based compression algorithms, and tests were performed using\na 1KB (10 bit) window. Since zlib already uses significantly more memory by\ndefault, the lowest memory level (`memLevel=1`) was used in these benchmarks. It\nshould be noted that higher zlib memory levels will having greater compression\nratios than Tamp. Currently, there is no micropython-compatible zlib or\nheatshrink compression implementation, so these numbers are provided simply as a\nreference.\n\nTamp is also available in [TurboBench](https://github.com/powturbo/TurboBench),\nan all-in-one compression benchmark executable, for comparing against many other\ncompressors on your own data.\n\n## Compression Ratio\n\nThe following table shows compression algorithm performance over a variety of\ninput data sourced from the\n[Silesia Corpus](https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia) and\n[Enwik8](https://mattmahoney.net/dc/textdata.html). This should give a general\nidea of how these algorithms perform over a variety of input data types.\n\n| dataset | raw | tamp | tamp (LazyMatching) | zlib | heatshrink |\n| --------------- | ----------- | ----------- | ------------------- | ------------- | ---------- |\n| enwik8 | 100,000,000 | 51,016,917 | **50,625,930** | 56,205,166 | 56,110,394 |\n| RPI_PICO (.uf2) | 667,648 | **289,454** | 290,577 | 303,763 | - |\n| silesia/dickens | 10,192,446 | 5,538,353 | **5,502,834** | 6,049,169 | 6,155,768 |\n| silesia/mozilla | 51,220,480 | 24,413,362 | **24,229,925** | 25,104,966 | 25,435,908 |\n| silesia/mr | 9,970,564 | 4,520,091 | **4,391,864** | 4,864,734 | 5,442,180 |\n| silesia/nci | 33,553,445 | 6,824,403 | 6,772,307 | **5,765,521** | 8,247,487 |\n| silesia/ooffice | 6,152,192 | 3,773,003 | **3,755,046** | 4,077,277 | 3,994,589 |\n| silesia/osdb | 10,085,684 | 8,466,875 | **8,464,328** | 8,625,159 | 8,747,527 |\n| silesia/reymont | 6,627,202 | 2,818,554 | **2,788,774** | 2,897,661 | 2,910,251 |\n| silesia/samba | 21,606,400 | 8,383,534 | **8,346,076** | 8,862,423 | 9,223,827 |\n| silesia/sao | 7,251,944 | 6,136,077 | **6,100,061** | 6,506,417 | 6,400,926 |\n| silesia/webster | 41,458,703 | 18,146,641 | **18,010,981** | 20,212,235 | 19,942,817 |\n| silesia/x-ray | 8,474,240 | 7,509,449 | 7,404,794 | **7,351,750** | 8,059,723 |\n| silesia/xml | 5,345,280 | 1,472,562 | **1,455,641** | 1,586,985 | 1,665,179 |\n\nTamp outperforms both heatshrink and zlib on most datasets, winning 12 out of 14\nbenchmarks. This is while using around 10x less memory than zlib during both\ncompression and decompression (see next section).\n\nLazy Matching is a simple technique to improve compression ratios at the expense\nof CPU while requiring very little code. One can expect **50-75%** more CPU\nusage for modest compression gains (around 0.5 - 2.0%). Because of this\ntrade-off, it is disabled by default; however, in applications where we want to\ncompress once on a powerful machine (like a desktop/server) and decompress on an\nembedded device, it may be worth it to spend a bit more compute. Lazy matched\ncompressed data is the exact same format; it appears no different to the tamp\ndecoder.\n\n### Ablation Study\n\nThe following table shows the effect of the `extended` and `lazy_matching`\ncompression parameters across all benchmark datasets (`window=10`, `literal=8`).\n\n| dataset | raw | Baseline | +lazy | +extended | +lazy +extended |\n| --------------- | ----------- | ---------- | ------------------ | ------------------ | ------------------ |\n| enwik8 | 100,000,000 | 51,635,633 | 51,252,694 (−0.7%) | 51,016,917 (−1.2%) | 50,625,930 (−2.0%) |\n| RPI_PICO (.uf2) | 667,648 | 331,310 | 329,893 (−0.4%) | 289,454 (−12.6%) | 290,577 (−12.3%) |\n| silesia/dickens | 10,192,446 | 5,546,761 | 5,511,681 (−0.6%) | 5,538,353 (−0.2%) | 5,502,834 (−0.8%) |\n| silesia/mozilla | 51,220,480 | 25,121,385 | 24,937,036 (−0.7%) | 24,413,362 (−2.8%) | 24,229,925 (−3.5%) |\n| silesia/mr | 9,970,564 | 5,027,032 | 4,888,930 (−2.7%) | 4,520,091 (−10.1%) | 4,391,864 (−12.6%) |\n| silesia/nci | 33,553,445 | 8,643,610 | 8,645,399 (+0.0%) | 6,824,403 (−21.0%) | 6,772,307 (−21.6%) |\n| silesia/ooffice | 6,152,192 | 3,814,938 | 3,798,393 (−0.4%) | 3,773,003 (−1.1%) | 3,755,046 (−1.6%) |\n| silesia/osdb | 10,085,684 | 8,520,835 | 8,518,502 (−0.0%) | 8,466,875 (−0.6%) | 8,464,328 (−0.7%) |\n| silesia/reymont | 6,627,202 | 2,847,981 | 2,820,948 (−0.9%) | 2,818,554 (−1.0%) | 2,788,774 (−2.1%) |\n| silesia/samba | 21,606,400 | 9,102,594 | 9,061,143 (−0.5%) | 8,383,534 (−7.9%) | 8,346,076 (−8.3%) |\n| silesia/sao | 7,251,944 | 6,137,755 | 6,101,747 (−0.6%) | 6,136,077 (−0.0%) | 6,100,061 (−0.6%) |\n| silesia/webster | 41,458,703 | 18,694,172 | 18,567,618 (−0.7%) | 18,146,641 (−2.9%) | 18,010,981 (−3.7%) |\n| silesia/x-ray | 8,474,240 | 7,510,606 | 7,406,001 (−1.4%) | 7,509,449 (−0.0%) | 7,404,794 (−1.4%) |\n| silesia/xml | 5,345,280 | 1,681,687 | 1,672,827 (−0.5%) | 1,472,562 (−12.4%) | 1,455,641 (−13.4%) |\n\nThe `extended` parameter enables additional Huffman codes for longer pattern\nmatches, which significantly improves compression on datasets with many long\nrepeating patterns (e.g., nci, samba, xml). Extended support was added in\nv2.0.0.\n\n## Memory Usage\n\nThe following table shows approximately how much memory each algorithm uses\nduring compression and decompression.\n\n| | Compression | Decompression |\n| --------------------- | ----------------------------- | ----------------------- |\n| Tamp | (1 << windowBits) | (1 << windowBits) |\n| ZLib | (1 << (windowBits + 2)) + 7KB | (1 << windowBits) + 7KB |\n| Heatshrink | (1 << (windowBits + 1)) | (1 << (windowBits + 1)) |\n| Deflate (micropython) | (1 << windowBits) | (1 << windowBits) |\n\nAll libraries have a few dozen bytes of overhead in addition to the primary\nwindow buffer, but are implementation-specific and ignored for clarity here.\nTamp uses significantly less memory than ZLib, and half the memory of\nHeatshrink.\n\nOn 64-bit x86_64/aarch64, the Python package and CLI additionally allocate a\ncompression match index (about 128 KB plus 6 bytes per window byte) for faster\ncompression; output is unchanged. Embedded builds don't use it.\n\n## Runtime\n\nAs a rough benchmark, here is the performance (in seconds) of these different\ncompression algorithms on the 100MB enwik8 dataset. These tests were performed\non an M3 Macbook Air.\n\n| | Compression (s) | Decompression (s) |\n| ---------------------------- | --------------- | ----------------- |\n| Tamp (Pure Python Reference) | 136.2 | 105.0 |\n| Tamp (C bindings) | 5.45 | 0.544 |\n| ZLib | 3.65 | 0.578 |\n| Heatshrink (with index) | 4.42 | 0.67 |\n| Heatshrink (without index) | 27.40 | 0.67 |\n\nHeatshrink v0.4.1 was used in these benchmarks. When heathshrink uses an index,\nan additional `(1 << (windowBits + 1))` bytes of memory are used, resulting in\n4x more memory-usage than Tamp. Tamp could use a similar indexing to increase\ncompression speed, but has chosen not to to focus on the primary goal of a\nlow-memory compressor.\n\nTo give an idea of Tamp's speed on an embedded device, the following table shows\ncompression/decompression in **bytes/second of the first 100KB of enwik8 on a pi\npico (rp2040)** at the default 125MHz clock rate, with native-module\nmeasurements taken on MicroPython v1.26.1 firmware. The C benchmark **does not**\nuse a filesystem nor dynamic memory allocation, so it represents the maximum\nspeed Tamp can achieve. In all tests, a 1KB window (10 bit) was used.\n\n| | Compression (bytes/s) | Decompression (bytes/s) |\n| -------------------------------- | --------------------- | ----------------------- |\n| Tamp (Micropython Native Module) | 34,510 | 980,392 |\n| Tamp (C) | 36,127 | 1,400,600 |\n| Deflate (micropython builtin) | 6,885 | 294,985 |\n\nTamp resulted in a **50841** byte archive, while Micropython's builtin `deflate`\nresulted in a larger, **59442** byte archive.\n\nSee\n[devices/BENCHMARKS.md](https://github.com/BrianPugh/tamp/blob/main/devices/BENCHMARKS.md)\nfor this benchmark across various hardware.\n\n## Binary Size\n\nTo give an idea on the resulting binary sizes, Tamp and other libraries were\ncompiled for the Pi Pico (`armv6m`). All libraries were compiled with `-O3`.\nNumbers reported in bytes. Tamp sizes were measured using `arm-none-eabi-gcc`\n15.2.1 and MicroPython v1.27, and can be regenerated with `make binary-size`.\n\n| | Compressor | Decompressor | Compressor + Decompressor |\n| -------------------------------- | ---------- | ------------ | ------------------------- |\n| Tamp (MicroPython Native) | 5469 | 4725 | 9125 |\n| Tamp (C, no extended, no stream) | 1932 | 1634 | 3328 |\n| Tamp (C, no extended) | 2214 | 1872 | 3848 |\n| Tamp (C, extended, no stream) | 3120 | 2452 | 5334 |\n| Tamp (C, extended) | 3402 | 2690 | 5854 |\n| Heatshrink (C) | 2956 | 3876 | 6832 |\n| uzlib (C) | 2355 | 3963 | 6318 |\n\nTamp C \"extended\" includes `tamp_compressor_compress_and_flush`. Tamp C includes\na high-level stream API by default. Even with `no stream`, Tamp includes\nbuffer-looping functions (like `tamp_compressor_compress`) that Heatshrink lacks\n(Heatshrink only provides poll/sink primitives).\n\n## Acknowledgement\n\n- Thanks @BitsForPeople for the esp32-optimized compressor implementation.\n\n## License\n\nTamp is dual-licensed under either the\n[Apache License 2.0](https://github.com/BrianPugh/tamp/blob/main/LICENSE-APACHE)\nor the [MIT License](https://github.com/BrianPugh/tamp/blob/main/LICENSE-MIT),\nat your option.\n"
}