Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: make Compressor::train 2x faster with bitmap index #16

Merged
merged 6 commits into from
Aug 20, 2024
Merged

feat: make Compressor::train 2x faster with bitmap index #16

merged 6 commits into from
Aug 20, 2024

Commits on Aug 20, 2024

  1. feat: make Compressor::train 2x faster with bitmap index 

    The slowest part of Compressor::train is the double-nested loops
over codes.

Now compress_count when it records code pairs will also populate
a bitmap index, where `pairs_index[code1].set(code2)` will indicate
that code2 followed code1 in compressed output.

In the `optimize` loop, we can eliminate tight loop iterations by
accessing `pairse_index[code1].second_codes()` which yields the value
code2 values.

This results in a speedup from ~1ms -> 500micros.
    @a10y
    a10y committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    b374b5d View commit details
    Browse the repository at this point in the history

  2. add miri action

    @a10y
    a10y committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    720d506 View commit details
    Browse the repository at this point in the history

  3. final cleanups

    @a10y
    a10y committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    fecde16 View commit details
    Browse the repository at this point in the history

  4. only run miri on develop 

    i don't want to lose my 30s CI checks
    @a10y
    a10y committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    17ac1be View commit details
    Browse the repository at this point in the history

  5. turn miri back on for CI

    @a10y
    a10y committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    aea4ae3 View commit details
    Browse the repository at this point in the history

  6. fix small bug in iterator, more tests

    @a10y
    a10y committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    f74d185 View commit details
    Browse the repository at this point in the history