Deep convolutional neural networks have been investigated for atmospheric particle removal and accomplished the state-of-the-art performance. Most of the previous studies however focus on removing the effects of atmospheric particles but not on glares caused by direct or reflected sunlight on images. In this paper, we propose a decompose-refine network for single image glare removal. Specifically, our network is composed of a glare detection subnetwork and a glare removal subnetwork, which are respectively in charge of glare detection and removal. Experimental results show that our network outperforms the state-of-the-art network baselines on testing dataset.