RGB and 3D-Segmentation Data Combination for the Autonomous Object Manipulation in Personal Care Robotics

This paper proposes the re-design of the functionalities of the personal care robot (PCR) Pepper by SoftBank Robotics. Pepper is mainly designed for verbal interaction with patients and despite the presence of two upper arms, it lacks object manipulation capabilities. In the proposed re-design, a combination of data from Pepper RGB camera and 3D depth sensor is used to identify and localize a specific pharmaceutical envelope in a dedicated repository. In this context, the semantic segmentation of the RGB image has been entrusted to a dedicated pretrained YOLOv3 object detector, while a dedicated algorithm has been realized for the hand (gripper) positioning. Basing on this 3D positional information, the PCR operates an ad-hoc designed routine to grasp the object and to scan it. Once the scanning procedure confirms that the grasping has been successfully completed and that the grasped package matches with the needed drug, the PCR must be able to safely navigate towards the user (e.g., physician, patient), delivering the drug. The proposed procedure is fully automatic, and no internet connection is needed for the nominal use case, preserving -in this way- sensitive data like home/hospital maps, patient’s data and so on. Experimental results on the here proposed object manipulation routine demonstrated a grasping success rate up to 96 %, even if the objects are not properly positioned in the dedicated repository. Finally, a proof of concept that implements a sequential pick-up, object recognition and delivery operation is also provided demonstrating real-life scenario applicability.