Research work on cloth recognition and manipulation in the Autonomous Intelligence and Systems (AIS) laboratory at Shinshu University

Abstract

The Autonomous Intelligence and Systems (AIS) laboratory at Shinshu University, Japan, is working on an ambitious project to programme a two-armed robot to manipulate cloth and other deformable materials – a particularly challenging problem for autonomous machines. Techniques such as deep neural networks and point cloud imaging have been applied in other experiments. To enable a dual-arm robot to translate a known cloth state to a different required shape or state, the robot was trained to associate different manipulations with an expected cloth state. By using machine learning on this scale, the robot was able to determine the manipulation or series of manipulations most likely to achieve the desired cloth state. Yamazaki explains: ‘By closely coupling each manipulation with an expected cloth shape in the learning architecture and training the robot on repeated operations, the computer can predict the likely outcome of manipulations and quickly determine the fastest and most effective operations to achieve a desired cloth shape.’ The team focused on identifying a range of different features including wrinkles, cuffs and cloth overlaps. The way in which wrinkles and cloth overlaps form is often dependent on the type of fabric, which helps determine the type of garment. Only a greyscale image of the heap of clothing is required. Complex image processing is undertaken to characterise the visible features and identify the type of garments present. A 3D range image sensor mounted in the robot’s head looking down onto the cloth can distinguish colour and shadow. Despite this, mistakes are common. The nature of cloth means it can fall into many different shapes and states, such that folds and wrinkles might easily be mistaken for a corner. AIS has developed imaging processing methods that can minimise uncertainty, as well as inline checking to ensure each manipulation results in the required shape. Simple intermediate actions have been derived by repeating different manipulations many times over and selecting those that most often result in a useful cloth arrangement.