And we also show improvement in many tactile perception tasks such as force estimation, pose estimation and full-hand joystick state estimation.
5/6

With this we see a 75% improvement in real-world tactile plug insertion over end-to-end using vision and tactile:
4/6

We pretrain Sparsh-skin with 4 hours of unlabeled data via self-distillation, and make several changes to get highly performant reps:

Decorrelate signals by tokenizing 1s window of tactile data.
Condition the encoder on robot hand configurations via sensor positions as input
3/6

Sparsh-skin is an approach to pretrain encoders for magnetic skin sensors on a dexterous robot hand.

It improves tactile tasks by over 56% in end-to-end methods and by over 41% in prior work.
It is trained via self-supervision for the Xela sensor, so no labeled data needed!
2/6

Robots need touch for human-like hands to reach the goal of general manipulation. However, approaches today don’t use tactile sensing or use specific architectures per tactile task.

Can 1 model improve many tactile tasks?
🌟Introducing Sparsh-skin: tinyurl.com/y935wz5c

1/6

SSL methods that learn in latent space are the most competitive.
On cross-sensor transfer, e.g. for textile classification: decoders for GelSight can transfer to DIGIT with only 10 samples using Sparsh (9→61%) vs end-to-end learning (3→10%)
5/6

For tactile tasks, it’s expensive (needs hardware) to collect labeled data, and often impractical!
With Sparsh, we extend self-supervised learning (SSL) to the tactile domain showing that SSL features allow for sample efficient learning of downstream tasks.
2/6

The robotics community has built many vision-tactile sensors such as GelSight and DIGIT resulting in significant progress in robot tactile perception. How can one reconcile these sensors to work across many tactile tasks without collecting a lot of labeled data?
More on 🎯Sparsh 🧵⬇
1/6