Multimodal Tactile Perception of Objects in a Real Home

Tapomayukh Bhattacharjee1, Henry M. Clever2, Joshua Wade3, Charlie Kemp2

  • 1University of Washington
  • 2Georgia Institute of Technology
  • 3Georgia Tech



Interactive Session


10:30 - 13:00 | Tue 22 May | podI | TuA@I

Haptics and Interfaces

Full Text


When operating in human environments, such as homes, robots could use tactile sensing to better perceive objects. A challenge that has not been sufficiently addressed is the influence of an object's surroundings on tactile perception. Prior research has focused on perception of objects in laboratory settings. Yet, a number of factors found in homes can affect multimodal tactile sensing. For example, the time-varying thermal characteristics of an object's surroundings, such as sunlight, HVAC, and refrigeration, can affect thermal sensing. Likewise, the placement of an object with respect to other objects and surfaces will affect force sensing and alter the way an object moves when pushed. In order to investigate these and other issues, we had a mobile robot reach out and push 47 different objects found in a real home over a three day period resulting in 1340 pushing episodes. We then characterized the performance of data-driven methods for a variety of tactile perception problems using the first two seconds of force, thermal, and motion sensing data collected by the robot. We paid particular attention to the ability of these methods to generalize what they have learned to different robot velocities, times of day, and object instances. Our results demonstrate the value of multimodal tactile sensing and data-driven methods for tactile perception from short-duration contact, and also illustrate the great diversity of real-world phenomena relevant to tactile sensing.

Additional Information

No information added


No videos found


A robot with a multimodal sensor module touches objects in a real home

  • We use multimodal sensing modalities for inferring properties of objects in a real home
  • We present methods that generalize performance across different robot velocities, times of day and object instances
  • Force, motion and active thermal sensing are important for material recognition and object identification
  • Passive thermal sensing is not very informative