Buildings are one of the most important urban structures that are used for various applications and urban mapping. In recent years, with the development of the high resolution data acquisition technologies, many different approaches and algorithms have been presented to extract the accurate and timely updated 3D models of buildings as a key element of city structures for numerous applications in urban mapping. In this paper, a novel and model-based approach is proposed for automatic recognition of buildings’ roof models such as flat, gable, hip, and pyramid hip roof models based on deep structures for hierarchical learning of features that are extracted from both LiDAR and aerial orthophotos. The main steps of this approach include building segmentation, feature extraction and learning, and finally building roof labelling in a supervised pre-trained Convolutional Neural Network (CNN) framework to have an automatic recognition system for various types of buildings over an urban area. In this framework, the height information provides invariant geometric features for CNN network to localize the boundary of each individual roofs. CNN is a kind of feed-forward neural network with the multilayer perceptron concept that consists of a number of convolutional and sub-sampling layers in an adaptable structure which is widely used in pattern recognition and object detection applications. Since the training dataset is a small library of labelled models for different shapes of roofs, via using the pre-trained models, the computation time of learning can decrease significantly. The experimental results highlight the effectiveness of the deep learning approach to detect and extract the pattern of buildings’ roofs automatically considering the complementary nature of height and RGB information. Based on the training results, the top 1 error and accuracy of training are about 0.05 and 95 %, respectively. Moreover, the average of correctness and completeness rates are about 97 % and 69 %, respectively.