We also release ckpts for normal and depth estimation where the model should translate an input image into a normal or depth map.

🤗 Ckpt: huggingface.co/jasperai/LBM...
🤗 Ckpt: huggingface.co/jasperai/LBM...

Today, we release on @hf.co, the model for object relighting where the model should relight a foreground object so that it blends perfectly with a target background.

🤗Ckpt: huggingface.co/jasperai/LBM...
🤗Demo: huggingface.co/spaces/jaspe...

In the proposed LBM method, we propose to encode the source and target image into a latent space and then build a stochastic path called a Brownian Bridge between them. In particular, the stochasticity of these paths makes the method differ from flow matching.

Latent Bridge Matching (LBM) is a method that aims at transporting a source distribution of images (e.g. pasted images) to a target distribution (e.g. relighted images).

🤗Demo: huggingface.co/spaces/jaspe...

@onurxtasar.bsky.social @jeev2.bsky.social @benjaminaubin.bsky.social @damien-henry.bsky.social

… finally, we also consider common tasks such as normal and depth estimation where the model should translate an input image into a normal or depth map

... we also derive a conditional framework of LBM and demonstrate its effectiveness by tackling the tasks of controllable image relighting and shadow generation 🕹️ …

... and image restoration 🧹 where the model must transport the distribution of the degraded images to the distribution of the clean images …

… but also object removal ✂️ where the model is trained to find a transport map from the masked images to the images without the objects …

During training, we also introduce a pixel loss that consists of decoding the estimated target latent and comparing it to the real target image. We found that LPIPS works well in practice and speeds up domain shift.

We validate the method for object relighting 🔦...

We encode paired images into the latent space and bridge the latents 🌉. A timestep is drawn from a well chosen distribution. The latent on the trajectory at this timestep is then passed to the denoiser which predicts the drift of the associated Stochastic Differential Equation.

In the proposed LBM method, we propose to encode the source and target image into a latent space and then build a stochastic path called a Brownian Bridge 🌉 between them. In particular, the stochasticity of these paths makes the method differ from flow matching.

💻 Code: github.com/gojasper/fla...

🤗 @hf.co models and @gradio-hf.bsky.social demo:
huggingface.co/collections/...

@onurxtasar.bsky.social @benjaminaubin.bsky.social @damien-henry.bsky.social