Highlight and Specular reflection removal in photogrammetric techniques applied to Architectural Heritage 3D modeling
DOI:
https://doi.org/10.23738/ccsj.i72017.04Keywords:
3D modelling, Structure from Motion, Automatic photogrammetry, Specular removal, Image processing, Color mapping, Reflection component separationAbstract
In this paper, we present a new technique to remove specular effects from the photo- grammetric results in a automatic photogrammetric workflow for Architectural Heritage (AH) 3D model construction. Our solution provides a new reconstruction pipeline completely integrated in the automatic photogrammetric pipeline re-using existing data to arrange new results. The process of acquisition of the images to get the finished 3D model is therefore unique and the process for acquiring and visualizing the correct perceived color is fully integrated with the process of shape capture. Overall, the method does not require specific technical knowledge, being therefore relatively easy to use, and it can be used over many different urban settings and contexts. The proposed methodology is a high-level image-processing algorithm. As such, it uses several lower-level methods for its building blocks. We consider these methods as black boxes, and we explain below their input, output and purpose.
We demonstrated the efficiency of our method using case study of our work in many cases of the ca 43 km of historical porticoes system in Bologna, Italy, a superset of the family of AH objects that it belongs to.
References
N. Snavely, S.M. Seitz, R. Szeliski, “Modeling the world from internet photo collections”, IJCV, Vol. 80, N. 2, 2008, pp. 189-210.
D. Lowe, “Distinctive image features from scale- invariant keypoints”, IJCV, Vol. 60, N. 2, 2004, pp. 91-110.
C. Wu, et al., “Multicore bundle adjustment”, CVPR Proceedings, pp. 3057-3064.
Y. Furukawa, J. Ponce, “Accurate, Dense, and Robust Multi-View Stereopsis”, IEEE Transactions on PAMI, Vol. 32, N. 8, 2010, pp. 1362-1376.
F. Remondino, et al., “Low-Cost and Open-Source Solutions for Automated Image Orientation – A Critical Overview”, Euromed 2012 Proceedings, Springer, 2012, pp. 40-54.
P. Musialski, et al., “A Survey of Urban Reconstruction”, Computer Graphics Forum, Vol. 32, N. 6, 2013, pp. 146– 177.
F.I. Apollonio, et al., “A color digital survey of arcades in Bologna”, Colour and Colorimetry Multidisciplinay Contributions, Vol. IXB, 2013, pp. 58-68.
H.C. Lee, D.J. Breneman, C.O. Schulte, “Modeling light reflection for computer color vision”, IEEE Transactions on PAMI, 1990, Vol. 12, pp. 402-409.
H. Barrow, J. Tanenbaum, “Recovering intrinsic scene characteristic from images”, Proceedings of the Computer Vision System, 1978, pp. 3-26.
R. Tan, K. Ikeuchi, “Intrinsic properties of an image with highlights”, Meeting on Image Recognition and Understanding (MIRU), 2004.
B. K. Horn, B.G. Schunck, “Determining optical flow”, 1981TechnicalSymposiumEastProceedings,1981,pp. 319-331.
S. Baker, et al., “A database and evaluation methodology for optical flow, IJCV, Vol. 92, N. 1, 2011, pp. 1-31.
D. Sun, S. Roth, M.J. Black, “Secrets of optical flow estimation and their principles”, CVPR Proceedings, 2010, pp. 2432-2439.
C. Vogel, S. Roth, K. Schindler, “An evaluation of data costs for optical flow”, GCPR proceedings 2013, pp. 343- 353.
L. Ce, J. Yuen, A. Torralba, “SIFT Flow: Dense Correspondence across Scenes and Its Applications”, IEEE Transaction on PAMI, Vol. 33, N. 5, 2011, pp. 978-994.
L. Yu, M.S. Brown, “Exploiting Reflection Change for Automatic Reflection Removal”, IEEE ICCV Proceedings, 2013, pp. 2432-2439.
M. Ebner, C. Herrmann, “On Determining the Color of the Illuminant Using the Dichromatic Reflection Model”. In: W.G. Kropatsch, R. Sablatnig, A. Hanbury (Eds), Pattern Recognition. DAGM 2005. Lecture Notes in Computer Science, vol. 3663. Berlin, Heidelberg: Springer, 2005.
A. Artusi, F. Banterle, D. Chetverikov, “A Survey of Specularity Removal Methods”, Computer Graphics Forum, Vol. 30, N. 8, 2011, pp. 2208-2230.
T. Chen, et al., “Mesostructure from specularity”,
CVPR Proceedings, 2006, pp. 1825-1832.
B. Lamond, P. Peers, P. Debevec, “Fast image-based separation of diffuse and specular reflections”, ACM SIGGRAPH 2007 sketches, ACM, New York, 2007, art. 74.
S. Lin, et al., “Diffuse-specular separation and depth recovery from image sequences”, ECCV Proceedings, 2003, pp. 210-224.
Y. Weiss, “Deriving intrinsic images from image sequences”, ICCV Proceedings, 2001, Vol. 2, pp. 68-75.
W.-C. Ma, et al., “Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination”, Proceedings of the 18th Eurographics conference on Rendering Techniques, 2007, pp. 183-194.
A. Agrawal, et al., “Removing Photography Artifacts Using Gradient Projection and Flash-Exposure Sampling,” ACM Trans. on Graphics, Vol. 24, N. 3, 2005, pp. 828-835.
G.J. Klinker, et al., “The measurement of highlights in color images”, IJCV, 1988, Vol. 2, N. 1, pp 7-32.
K. Schluns, A. Koschan, “Global and local highlight analysis in color images”, CGIP ‘2000 Proceedings, 2000, pp. 300-304.
H.L. Shen, Q.Y. Cai, “Simple and efficient method for specularity removal in an image”, Applied Optics, Vol. 48, N. 14, 2009, pp. 2711-2719.
S.P. Mallick, et al., “Specularity removal in images and videos: A PDE approach”, ECCV Proceedings, 2006, pp. 550–563.
M.F. Tappen, et al., “Recovering intrinsic images from a single image”, Pattern Analysis and Machine Intelligence, IEEE Transactions on PAMI, 2005, Vol. 27, N. 9, pp. 1459-1472.
R. Tan, K. Ikeuchi, “Separating reflection components of textured surfaces using a single image”, IEEE Transaction on PAMI, Vol. 27, N. 2, 2005, pp. 178–193.
E. Angelopoulou, “Specular highlight detection based on the Fresnel reflection coefficient”, ICCV proceedings, 2007, pp 1-8.
L. Fang, T. Jiandong, T. Yandong, W. Yan, “An Image Highlights Removal Method with Polarization Principle”, ICMMITA 2015 Proceedings, 2015, pp. 402-407.
S.A. Shafer, “Using color to separate reflection components”, Color Research and Applications, Vol. 10, N. 4, 1985, pp. 210–218.
K.J. Yoon, Y. Choi, I.S. Kweon, “Fast separation of reflection components using a specularity-invariant image representation”, ICIP Proceedings, 2006, pp. 973–976.
H-L. Shen, et al., “Chromaticity-based separation of reflection components in a single im-age”, Pattern Recogn. Vol. 41, N. 8, 2008, pp. 2461-2469.
Q. Yang, S. Wang, N. Ahuja, “Real-time specular highlight removal using bilateral filtering”, ECCV Proceedings, 2010, pp. 87-100.
H.L. Shen, Z.H. Zheng, “Real-time highlight removal using intensity ratio”, Applied Optics, Vol. 52, N.19, 2013, pp. 4483-4493.
L. Yu, M.S. Brown, “Single Image Layer Separation Using Relative Smoothness”, CVPR Proceedings, 2014., pp. 2752-2759.
A. Jurio, et al., “A Comparison Study of Different Color Spaces in Clustering Based Image Segmentation”, Information Processing and Management of Uncertainty in Knowledge-Based Systems. Applications, Vol. 81, pp. 532-541.
D.J. Bora, A.K. Gupta, F.A. Khan, “Comparing the Performance of L*A*B* and HSV Color Spaces with Respect to Color Image Segmentation”, International Journal of Emerging Technology and Advanced Engineering, Vol. 5, N. 2, 2015, pp. 192-203.
S. R. Vantaram, E. Saber, “Survey of contemporary trends in color image segmentation”, J. Electron. Imaging, Vol. 21, N. 4, 2012, pp. 040901-1-28.
T. Kanungo et al., “An efficient k-means clustering algorithm: analysis and implementation”, IEEE Transaction on PAMI, Vol. 24, N. 7, 2002, pp. 881-892.
F. Meyer, “Topographic distance and watershed lines”, Signal Processing, Vol. 38, 1994, pp. 113-125.
J. Matas, et al., “Robust wide baseline stereo from maximally stable extremal regions”, BMVC Proceedings, 2002, pp. 384-396.
K. Mikolajczyk, et al., “A comparison of affine region detectors”, IJCV, Vol. 65, N. 1/2, 2005, pp. 43-72.
Il-Seok Oh, Jinseon Lee, Aditi Majumder, Multi-scale Image Segmentation Using MSER, Computer Analysis of Images and Patterns Lecture Notes in Computer Science, Vol. 8048, pp. 201-208.
H.J. Yoo, et al., “Color correction of high dynamic range images at HDR-level”, ACM SIGGRAPH 2007 posters, Article 67.
T-R. Chou; S.-K. Chang, “Color Calibration of Recovering HDR Images”, Computer Science and Software Engineering, 2008 International Conference on, Vol. 6, 2008, pp. 286-289.
E. Reinhard, et al., “Calibrated Image Appearance Reproduction”, ACM Transactions on Graphics, Vol. 31, N. 6, 2012, art. 201, 11 pp.
D. Varghese, R. Wanat, R.K. Mantiuk, “Colorimetric calibration of high dynamic range images with a ColorChecker chart”, HDRi 2014, 2014, Article 4.
T. Mitsunaga, S. Nayar, “Radiometric self calibration”, CVPR Proceedings, 1999, pp. 374–380.
D. Pascale, “RGB coordinates of the Macbeth ColorChecker”, 2006.
M. Granados, et al., “Optimal HDR reconstruction with linear digital cameras”, CVPR Proceedings, 2013, pp. 215- 222.
C. Aguerrebere, et al., “Best Algorithms for HDR Image Generation. A Study of Performance Bounds”, SIAM Journal on Imaging Sciences, Vol. 7, N. 1, 2014, pp. 1-34.
R. Mantiuk, S. Daly, L. Kerofsky, “Display adaptive tone mapping”, ACM SIGGRAPH 2008 Proceedings, 2008, art. 68, 10 pp.
R. Szeliski, S. Avidan, P. Anandan, “Layer extraction from multiple images containing reflections and transparency”, IEEE Conference on CVPR Proceedings, 2000, pp. 246-253.
M. Irani, S. Peleg, “Image sequence enhancement using multiple motions analysis”, CVPR Proceedings, 1992, pp. 216–221.
Y. Tsin, S.B. Kang, R. Szeliski, “Stereo matching with reflections and translucency”, CVPR Proceedings, 2003, pp. 702–709.
R. Feris, et al., “Specular reflection reduction with multi-flash imaging”, SIBGRAPI04 Proceedings, 2004, Vol. 42, pp. 316–321.
A. Agrawal, et al., “Removing Photography Artifacts Using Gradient Projection and Flash-Exposure Sampling,” ACM Trans. on Graphics, Vol. 24, N. 3, 2005, pp. 828-835.
S. Lin, et al., “Diffuse-specular separation and depth recovery from image sequences”, ECCV Proceedings, 2003, pp. 210-224.
N. Snavely, S.M. Seitz, R. Szeliski, “Photo tourism: exploring photo collections in 3D”, ACM transactions on graphics (TOG), Vol. 25, N. 3, 2006, pp. 835-846.
G. Haro, A. Buades, J.M. Morel, “Photographing paintings by image fusion”, SIAM Journal on Imaging Sciences, Vol. 5, N. 3, 2012, pp. 1055-1087.
A. Buades, G. Haro, E. Meinhardt-Llopis, “Obtaining High Quality Photographs of Paintings by Image Fusion”, Image Processing On Line, Vol. 5, 2015, pp. 159–175.
X. Guo, X. Cao, Y. Ma, “Robust Separation of Reflection from Multiple Images”, CVPR proceedings, 2014, pp. 2195-2202.
A. Levin, A. Zomet, Y. Weiss, “Separating Reflections from a Single Image Using Local Features,” CVPR Proceedings, 2004, Vol. I, pp. 306-313.
A. Levin, Y. Weiss, “User assisted separation of reflections from a single image using a sparsity prior”, IEEE Transactions on PAMI, Vol. 29, N. 9, 2007, pp. 1647– 1654.
T. Brox, et al., “High accuracy optical flow estimation based on a theory for warping”, ECCV proceedings, 2004, pp. 25-36.
K. He, J. Sun, “Computing nearest-neighbor fields via propagation-assisted kd-trees”, CVPR proceedings, 2012, pp. 111-118.
P. Weinzaepfel, et al., “DeepFlow: Large displacement optical flow with deep matching”, ICCV proceedings, 2013, pp. 1385-1392.
H.Yang, W.-Y. Lin, J. Lu, “DAISY Filter Flow: A Generalized Discrete Approach to Dense Correspondences”, CVPR proceedings, 2014, pp. 3406-3413.
E. Tola, V. Lepetit, P. Fua, “DAISY: An efficient dense descriptor applied to wide-baseline stereo”, IEEE Transaction on PAMI, Vol. 32, N. 5, 2010, pp. 815-830.
J. Lu, et al., “Patchmatch filter: Efficient edge-aware filtering meets randomized search for fast correspondence field estimation”, CVPR proceedings, 2013, pp. 1854-1861.
M. Tau, T. Hassner, “Dense Correspondences across Scenes and Scales”, IEEE Transaction on PAMI, accepted.
J. Revaud, et al., “EpicFlow: Edge-preserving interpolation of correspondences for optical flow”, CVPR proceedings, 2015, pp.1164-1172.
Downloads
Published
Issue
Section
License
The "Cultura e Scienza del Colore - Color Culture and Science" journal is registered at the Court of Milan at n.233 of 24.06.2014.
The journal is an open access journal, free for readers and authors and has joined ROAD, the Directory of Open Access scholarly Resources, since 2014. Articles published in the “Cultura e Scienza del Colore - Color Culture and Science" journal are open access articles, distributed under the terms and conditions of the Creative Commons Attribution License (CC BY). The copyright is retained by the author(s).
