And we also show improvement in many tactile perception tasks such as force estimation, pose estimation and full-hand joystick state estimation.
5/6

With this we see a 75% improvement in real-world tactile plug insertion over end-to-end using vision and tactile:
4/6

We pretrain Sparsh-skin with 4 hours of unlabeled data via self-distillation, and make several changes to get highly performant reps:

Decorrelate signals by tokenizing 1s window of tactile data.
Condition the encoder on robot hand configurations via sensor positions as input
3/6

Sparsh-skin is an approach to pretrain encoders for magnetic skin sensors on a dexterous robot hand.

It improves tactile tasks by over 56% in end-to-end methods and by over 41% in prior work.
It is trained via self-supervision for the Xela sensor, so no labeled data needed!
2/6

Robots need touch for human-like hands to reach the goal of general manipulation. However, approaches today don’t use tactile sensing or use specific architectures per tactile task.

Can 1 model improve many tactile tasks?
🌟Introducing Sparsh-skin: tinyurl.com/y935wz5c

1/6

Last week I passed my thesis proposal, and I'm now officially a Ph.D. candidate!

I'm grateful to my committee, and everyone who supported me.

My proposed thesis "Self supervised perception for tactile dexterity" will explore ways to improve dexterous manipulation using tactile reps.

Some pictures of the Pittsburgh spring to reduce the spiciness of the bsky feed!

a6700 w/ 17-70mm Tamron lens

SSL methods that learn in latent space are the most competitive.
On cross-sensor transfer, e.g. for textile classification: decoders for GelSight can transfer to DIGIT with only 10 samples using Sparsh (9→61%) vs end-to-end learning (3→10%)
5/6

For standardization, and to nudge the community toward physical tasks, we introduce TacBench w/ 6 touch-centric tasks designed to probe tactile properties, perception & planning capabilities.
We find that Sparsh outperforms task & sensor specific models by an average of 95.1%
4/6

Sparsh is a family of SSL models trained with 460k+ tactile images from GelSight, GelSight mini and DIGIT.
Sparsh is general-purpose and shows strong performance in many tasks. One could decode representations from Sparsh into normal and shear fields to train downstream policies.
3/6

For tactile tasks, it’s expensive (needs hardware) to collect labeled data, and often impractical!
With Sparsh, we extend self-supervised learning (SSL) to the tactile domain showing that SSL features allow for sample efficient learning of downstream tasks.
2/6

The robotics community has built many vision-tactile sensors such as GelSight and DIGIT resulting in significant progress in robot tactile perception. How can one reconcile these sensors to work across many tactile tasks without collecting a lot of labeled data?
More on 🎯Sparsh 🧵⬇
1/6