Transparent objects are widely used in our daily lives, making teaching robots to interact with them important. However, it's not easy because the reflective and refractive effects can make RGB-D cameras fail to give accurate geometry measurements. To solve this problem, this paper presents RFTrans, an RGB-D-based method for surface normal estimation and manipulation of transparent objects. By leveraging refractive flow as an intermediate representation, RFTrans circumvents the drawbacks of directly predicting the geometry (\eg surface normal) from RGB images and helps bridge the sim-to-real gap. RFTrans integrates the RFNet, which predicts refractive flow, object mask, and boundaries, followed by the F2Net, which estimates surface normal from the refractive flow. To make manipulation possible, a global optimization module will take in the predictions and refine the raw depth, from which we can construct the point cloud with normal. An analytical grasp planning algorithm, ISF, is followed to generate the grasp poses. We build a synthetic dataset with physically plausible ray-tracing rendering techniques to train the networks. Results show that the RFTrans trained on the synthetic dataset can consistently outperform the baseline ClearGrasp in both synthetic and real-world benchmarks by a large margin. Finally, a real-world robot grasping task witnesses an 83\% success rate, proving that refractive flow can help enable direct sim-to-real transfer.

Given an RGB-D image, RFNet first predicts the mask, the boundary, and the refractive flow of transparent objects. Next, F2Net will predict the surface normal based on the refractive flow. The global optimization will generate the singulated point cloud with normal. Finally, we apply the off-the-shelf manipulation algorithm, ISF, to generate grasp poses. The black points represent the fingers of the Franka Emika Panda robot.