if I were in your position, I'd start with cascaded grids or "clip maps", in which you allocate cells sparsely. My first time seeing that was in Kajiya's irradiance cache.
user-images.githubusercontent.com/16522064/170...

github.com/EmbarkStudio...

Here's a video of it in action so far. Thanks to Autumn on the Teardown Discord for their wonderful "Lock-n-Load Framework" which helps immensely with animating

not nolram commenting on patricks posts 🫣

The Half-Life tril... 💀