Multi-Image Restoration and fusion: from Applied Mathematics to the imaging Industry – MIRIAM

Our research program is divided in 9 tasks. Five of these tasks (3 to 7) concern image enhancement problems, revisited in the multiframe context. Each of these tasks is a well defined and specific restoration problem: HDR and denoising, superresolution and demosaicking, multi-frame decompression, video/multiview inpainting, and finally contrast and color enhancement. As explained before, using several images to solve these problems can lead to dramatic improvements in image quality. These five tasks hence constitute the core of our project. Tasks 1 and 2 concern prior research points that are common to these five enhancement tasks

Finally, task 8 describes the dissemination of our results to the scientific and industrial communities, while task 9 describes the validation of our result on real world images and applications.

The attribution of these tasks to the different members of the project relies on the fields of expertise of each participant:

- task 1a How to compare image patches ? : Almansa, Delon, Gousseau, Tupin, Nikolova, Salmon;
- task 1b Large scale learning of image-patch manifolds : Almansa, Salmon;
- task 2 Outliers and compressed sensing : Almansa, Durand, Ladjal, Nikolova;

- task 3 HDR and denoising : Delon, Gousseau, Ladjal, Desolneux, Morel;

- task 4 Superresolution and demosaicking : Gousseau, Ladjal;

- task 5 Multi-frame decompression : Almansa, Durand, Nikolova;

- task 6 Video/multiview inpainting : Gousseau, Masnou, Bretin, Galerne;

- task 7 Contrast and color enhancement : Delon, Gousseau, Nikolova, Galerne;
- task 8 Academic valorisation : Morel + all;
- task 9 Industrial validation and valorisation : Hauser, Facciolo, Tarchouna, Guichard + all

After 18 months the project has a good progress. Academic tasks 1-7 resulted in 7 publications in international scientific journals (more than half of which are multi-partner collaborations), and as many presentations at international conferences with proceedings and committee. This work has already led to two software publications with online demos (task 8), and a database containing degraded images with the corresponding reference images is being built by DxO (task 9). This database will serve as a benchmark for evaluating different recovery algorithms performed as part of this project, and their comparison to the state of academic and industrial art on the subject.

To distribute these partial results we organized 3 minisymposia at the largest international conference on mathematical imaging (SIAM IS16 in Albuquerque, NM) and a regular seminar in Paris (SMATI). Several members of the project were invited to Cambridge University (UK) and Tsinghua University (China) to present some of their results related to the project.

Besides the scientific production the project allows a close cooperation between the various parties, through co-supervision of PhD students and postdocs, joint seminars and exchange of results for validation.

This project enabled the recruitment of several PhD students and postdocs, and to animate close collaborations with specialists abroad. Other recruitments are planned for the next two years. With the team formed around this project we hope to significantly advance a common formalism for understanding and optimizing the kinds of multi-image restoration techniques that are used in computational photography and earth observation. We also hope to get some results that can be exploited by industry.

### Journal

[DG16] Delon, J., & Guillemot,T. (2016). Implementation of the Midway Image Equalization. IPOL. www.ipol.im/pub/art/2016/140/.

[DDG16] Delon, J., Desolneux, A.,& Guillemot, T. (2016). PARIGI: a Patch-based Approach to RemoveImpulse-Gaussian Noise from Images. IPOL www.ipol.im/pub/art/2016/161/

[MDG15] Mazin, B., Delon, J.,& Gousseau, Y. (2015). Estimation of Illuminants From Projections on the Planckian Locus. IEEE TIP <hal-00915853>

[PDGM16] Provenzi, E., Delon, J.,Gousseau, Y., & Mazin, B. (2016). On the second order spatiochromaticstructure of natural images. Vision Research. <hal-01054307>.

[RDM16] L. Raad, A. Desolneux andJ.-M. Morel (2016), A conditional multiscale locally Gaussian texture synthesis algorithm,to appear in JMIV.

[TLA15] Y. Traonmilin, S. Ladjal, A. Almansa. Robust Multi-image Processing With Optimal SparseRegularization. JMIV <hal-00940192>

*[TPG16] G. Tartavel, G. Peyré, Y.Gousseau, «Wasserstein Loss for Image Synthesis and Restoration«, to appear in SIIMS, <hal-01292843)>

### Conferences

* [FNSW15] Nikolova, M., Steidl, G.,& Weiss, P. (2015). Bilevel Image Denoising using Gaussianity tests. In SSVM

*[FSDH15b] Frigo, O., Sabater, N.,Delon, J., & Hellier, P. (2015). Motion driven tonal stabilization. In IEEE ICIP. <hal-01256874>

[FSDH16] O. Frigo, N. Sabater, J. Delon, P. Hellier (2016), Split and Match: Example-based AdaptivePatch Sampling for Unsupervised Style Transfer, CVPR’16. <hal-01280818>

[GBL16] Galerne, B., Leclaire, A.,& Moisan, L. (2016). Microtexture inpainting through Gaussian conditionalsimulation. ICASSP, <hal-01214695>

[RAGT16] P. Riot, A. Almansa, Y. Gousseau, F. Tupin. Penalizing local correlations in the residualimproves image denoising performance. (EUSIPCO 2016) <hal-01341968>

### Other

6 preprints. <hal-01107519> <hal-01280818> <hal-01280818> <hal-01334028> <hal-01341839> dev.ipol.im/~pariasm/video_nlbayes

This research program deals with the quality enhancement of digital images. Specifically, we consider the increasingly common situations where several acquisitions of the same scene are available, possibly through a video sequence. An issue of growing importance is to fuse these images in order to get a single enhanced image. Another crucial question concerns the joint restoration of several images of the same sequence.

Such an approach has two great advantages. First, going beyond the physical limitations of sensors becomes possible, in terms of dynamic range, resolution or signal- to-noise ratio. Second, the classical limitations of single shot imaging (blur, specular reflections, over or under-exposure, etc.) may be strongly attenuated. As a counterpart, multi-image restoration faces specific difficulties, the toughest of which are related to change detection, motion detection, outliers (aberrant pixels or regions) detection, inpainting, contrast and colour corrections.

The proposed work program gathers several classical problems from image processing together with complex issues in image analysis and comparisons. It builds on various mathematical tools, mainly statistical estimation, optimization, stochastic modeling, variational approaches and optimal transport. Within this program, our main goal will be to develop reliable and efficient numerical algorithms for each of the studied restoration modalities.