Giving Robots a Sense of Touch

Touch is central to human dexterity. The skin of each hand is served by approximately 17,000 tactile sensory units, providing fine force sensitivity with high spatial and temporal resolution [2]. Our fingertips can detect forces below one millinewton (1 mN)—equivalent to the weight of a grain of rice [3]—and perceive vibrations at frequencies approaching 1 kHz [4]. Without tactile feedback, even familiar tasks such as picking up and lighting a match become difficult [5]. Despite its importance to human dexterity, most industrial robotic manipulation systems operate without tactile sensing.

Achieving human-level tactile sensing in robots remains an open problem. Existing tactile sensing technologies offer different strengths in force sensitivity, spatiotemporal resolution, form factor, reliability, and cost, and each has its own limitations [6, 7] (Table 1). At Rhoda, we developed a robust tactile sensor that avoids limitations of other approaches while enabling light-contact sensitivity, fast response, and normal- and shear-force sensing in a thin package. In our experiments below, we compare its contact sensitivity with that of existing sensors and demonstrate its response to rapid contact events and heavy loads. We then integrate touch into a multimodal version of our Direct Video-Action (DVA) model [1]. By generating future observations of touch, we enable our robots to perform manipulation tasks even when vision is unavailable.

We developed our tactile sensing technology to meet a demanding combination of requirements: detecting light touches, distinguishing normal and shear forces, and tracking rapid changes while handling substantial loads—all in a compact, reliable, and low-cost sensor. Existing commercial approaches offer individual strengths (Table 1), but their tradeoffs motivated us to develop a solution that brings these capabilities together [6, 7]. In the following, we provide a brief overview of the most common tactile sensing technologies and their tradeoffs.

Camera-based optical tactile sensors use high-resolution images of a deformable contact surface to estimate normal and shear forces. However, accommodating the camera, lens, and illumination complicates thin packaging and increases hardware cost. Their temporal response depends on the camera’s frame rate and the processing required to convert images into force estimates [6, 7].

Capacitive sensors measure pressure-induced deformation through changes in capacitance. Conventional designs primarily measure normal pressure, while shear force sensing requires specialized electrode and mechanical structures. Hysteresis, electromagnetic interference, and crosstalk can affect measurement accuracy [6].

Magnetic tactile sensors track the motion of magnets or magnetic particles embedded in a deformable material and provide sensitive, fast measurements of normal and shear forces in a compact, inexpensive package. However, their sensing elements also respond to magnetic fields unrelated to contact. Nearby magnets, ferromagnetic objects, and neighboring tactile sensors can therefore disturb the measurements, making it difficult to separate actual contact forces from interference without additional shielding or compensation [6, 8].

Resistive films are thin and inexpensive and can be used to detect contact through changes in electrical resistance. Conventional designs primarily measure normal pressure, however, and do not independently resolve shear forces. Force estimation is also complicated by drift, where the signal changes under a sustained load or as temperature varies, and hysteresis [6].

Strain-gauge force sensors provide precise multi-axis force measurements, but conventionally measure aggregate forces and torques rather than a spatial map of contact [6]. Integrating miniature strain gauges into dense tactile arrays that combine high sensitivity, substantial load capacity, and low crosstalk remains an active and promising area of research [9, 10].

Rhoda tactile sensor. We developed compact, low-cost tactile sensors that navigate the tradeoffs of existing technologies and perform well across all metrics considered in Table 1. We co-designed algorithms with the sensor hardware to simplify calibration and mitigate drift and hysteresis. Our experiments demonstrate the sensor’s responsiveness to light contact and its wide dynamic range.

8 mm

Here, we demonstrate a specific implementation of our tactile sensing technology packaged to fit a robotic gripper (Fig. 1). Using this implementation, we examine the sensor’s response to light contact, shear forces, rapid changes in contact, and temperature. Our sensor detects very light contact, remains sensitive after heavy loading, and detects shear forces (Figs. 2–5).

Sensitivity and robustness. Brushing the sensor with a feather produces a measurable response to only a few millinewtons of force. Under the same test conditions, the off-the-shelf optical and magnetic sensors we tested show no distinguishable response (Fig. 2). We repeat the feather test after placing a heavy weight on the sensor, demonstrating that it remains sensitive to light contact after heavy loading. The sensor measures forces spanning millinewtons to tens of newtons (Fig. 3). The sensor also detects the small force variations associated with a pulse when pressed against the wrist (Fig. 4).

Shear force. In Fig. 5, we press a finger on the sensor surface and observe how the signal changes as the direction of force changes. This demonstrates the sensor’s response to shear forces.

We next demonstrate how the high temporal resolution of the sensor captures vibrations and how we can recover temperature changes at the contact surface (Figs. 6–8).

Temporal resolution. Our sensor can sample contact-induced vibrations at approximately 500 Hz, capturing differences between surface textures. In Fig. 6, we play these signals back as audio, revealing distinct responses as the sensor brushes across a notebook’s spiral binding, ridged plastic tubing, and a canvas bag and zipper. The sensor can also record vibrations from music when placed against a computer speaker (Fig. 7).

Temperature. Finally, we demonstrate the ability of the sensor to measure temperature (Fig. 8). We track the sensor’s temperature response as we alternate between immersing it in warm water and placing an ice cube on its surface. Temperature feedback could help a robot adapt how it handles objects—for example, by distinguishing frozen from thawed items or recognizing contact with a hot surface.

We integrate our tactile sensors into a handheld Universal Manipulation Interface (UMI), recording contact forces as a person manipulates everyday objects (Fig. 9). UMI enables demonstrations to be collected in diverse environments without bringing a robot to each location [11]. Standard UMI demonstrations capture vision and motion, but omit the forces the demonstrator feels and applies. Recording those forces is critical for learning how firmly to grasp an object and how to modulate contact during manipulation [12]. Our tactile-enabled UMI captures normal and shear forces at the contact surface alongside visual demonstrations, preserving this information for policy learning.

In our previous post on DVA models [1], we established the paradigm of video-model-as-robot-policy, in which a web-scale, pre-trained video model makes the core decisions for robot manipulation. We extend this approach to tactile signals: our multimodal DVA model additionally takes a history of tactile videos as input and generates future tactile videos (Fig. 10). An inverse dynamics model then translates the generated tactile videos into robot actions. As the robot acts, new tactile observations guide the next predictions, closing the feedback loop. This allows DVA to complement vision with touch and could improve manipulation when poor lighting or occlusion limits visual feedback. In an extreme case described below, it can even perform a complex manipulation task entirely without vision.

Task: Find the textured ball inside a box entirely by touch.

The box contains both a textured ball and a smooth ball of similar diameter. We blindfolded the robot by blocking its wrist and head cameras. Like a person reaching into a box without looking, the robot must feel around to learn what is inside.

This takes more than just detecting contact. The robot must use touch alone to guide its search, distinguish the balls’ textures through contact with its tactile sensors, and adjust the gripper after contact to get a secure grip.

For this task, we train with one smooth blue ball and one green textured ball. Fig. 11 shows an extended autonomous run in this two-ball setup. We then test the same model with four smooth balls and three textured balls, including two new textured balls that differ in diameter, firmness, and texture. The model still identifies and retrieves the textured balls. Fig. 12 shows an extended autonomous run with this seven-ball setup.

Our tactile sensing technology and multimodal DVA model give robots a sense of touch, using predictions of future tactile signals to guide their actions. Our sensor detects pressure, texture, vibration, shear, and temperature while combining sensitivity, responsiveness, and robustness in a compact, low-cost package. Together, the sensor and model enable a robot to identify and grasp objects by touch.

This capability opens a path toward tasks such as picking tiny parts from a bin when the gripper blocks the camera’s view, adjusting a grasp as an object begins to slip, or manipulating objects in poor lighting. In assembly, touch could help a robot align a connector and recognize when it is fully seated. When handling delicate or unfamiliar objects, it could help the robot apply enough force to maintain a secure grip without causing damage. Ultimately, just as touch helps people handle objects with confidence and precision, we expect tactile sensing to make robots more capable, efficient, and reliable.

Key Contributors: Andrew Xie*, Leo Dong, David B. Lindell

*Work done during an internship at Rhoda AI.