Dan
vilse.bsky.social
Dan
@vilse.bsky.social
If you are using longtail as a lib you need to switch from `Longtail_ChangeVersion()` to `Longtail_ChangeVersion2()` and use `Longtail_CreateConcurrentChunkWriteAPI`. It is that you drop the LRU block store layer is becomes redundant and only adds memory overhead.
Very happy with how this turned out
January 3, 2024 at 11:22 PM
Turns out the new strategy is significantly simpler, uses less memory, reduces file I/O, cuts down on CPU usage and drastically reduces wall time to write a new version when it contains files larger than a single block.
January 3, 2024 at 11:21 PM
The implementation was focused sequential writes of files that caused huge amount of extra I/O, memory and CPU work to be done.
Now, realizing that I don't *need* to write big files from start to finish, but instead I can write the parts in any order, I got to work implementing a new strategy.
January 3, 2024 at 11:20 PM
Just before the holiday break at work I noticed that writing a version from an archive containing files larger than a single block was **very** slow - like, slow as in "this can't be right".
But it was.
January 3, 2024 at 11:20 PM