deliverable...o dng o fff o mef o nef o orf o pef o psd o raf o tiff o tif o raw o rw2 fotoinmotion...
TRANSCRIPT
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 1 Dissemination level: PU - Public
Project Acronym: FotoInMotion
Grant Agreement number: 780612 (H2020-ICT-20-2017-1, RIA)
Project Full Title: Repurposing and enriching images for immersive storytelling
through smart digital tools
Project Coordinator: INTRASOFT International
DELIVERABLE
D3.1 - 2D image processing requirements and mechanisms
Dissemination level PU- Public
Type of Document Report
Contractual date of delivery M9 – 30/09/2018
Deliverable Leader QdepQ Systems BV
Status - version, date V1.0, 28/09/2018
WP / Task responsible WP 3
Keywords: Image processing, post production, toolbox,
architecture, API
Ref. Ares(2018)5554358 - 30/10/2018
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 2 Dissemination level: PU - Public
Executive Summary This deliverable will contain the analysis of the requirements in terms of 2D image processing
and design specifications for the first demonstrator of the FotoInMotion application.
These technical requirements are derived from the functional requirements (as described in
D1.2). We also briefly describe the processes and methodology that we will use for our
services as guidelines for integration.
Only a preliminary version of our API is included in this document, as it will be a living
document that updates during implementation of functionality. To keep the integration
smooth and agile, the API will be designed in consultation with WP4.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 3 Dissemination level: PU - Public
Deliverable Leader: QDEPQ
Contributors: QDEPQ, ATC, NOOR
Reviewers: INESC, NOOR
Approved by: All Partners
Document History
Version Date Contributor(s) Description
0.1 14/09/2018 QDEPQ Initial Draft version
0.2 21/09/2018 NOOR, ATC Added input
0.3 25/09/2018 QDEPQ Feedback on first draft
1.0 28/09/2018 QDEPQ Final version
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 4 Dissemination level: PU - Public
Table of Contents EXECUTIVE SUMMARY ..................................................................................................................................... 2
DEFINITIONS, ACRONYMS AND ABBREVIATIONS ............................................................................................. 6
1 INTRODUCTION ...................................................................................................................................... 7
PURPOSE OF THIS DOCUMENT AND RELATIONS WITH OTHER WPS .......................................................................... 7 DOCUMENT STRUCTURE ................................................................................................................................. 7
2 2D IMAGE PROCESSING .......................................................................................................................... 8
INPUT/OUTPUT ............................................................................................................................................ 8 File Format ...................................................................................................................................... 8 Debayering ...................................................................................................................................... 9 White balancing .............................................................................................................................. 9
COLOR MANIPULATION .................................................................................................................................. 9 Color Balance .................................................................................................................................. 9 Brightness/Contrast ........................................................................................................................ 9
GEOMETRY .................................................................................................................................................. 9 Cropping .......................................................................................................................................... 9 Flipping ........................................................................................................................................... 9 Rotation .......................................................................................................................................... 9
IMPLEMENTATION ....................................................................................................................................... 10 Image manipulation workflow ...................................................................................................... 10 Data storage policy ....................................................................................................................... 11 Tools .............................................................................................................................................. 11
3 TIMELINE MANIPULATION .................................................................................................................... 12
TIMELINE SPECIFICATION ............................................................................................................................... 12 EVENTS ..................................................................................................................................................... 12
Image events ................................................................................................................................. 13 Viewport events ............................................................................................................................ 13 Sound events ................................................................................................................................. 13
TRANSITION EFFECTS .................................................................................................................................... 13 Fade in/out .................................................................................................................................... 13 Slide in/out .................................................................................................................................... 13 Curl effect ...................................................................................................................................... 13
2D NARRATIVE EFFECTS ............................................................................................................................... 14 Panning ......................................................................................................................................... 14 Zooming ........................................................................................................................................ 14 Rotation ........................................................................................................................................ 14 Special Effects ............................................................................................................................... 14
4 SOUND PROCESSING............................................................................................................................. 15
INPUT / OUTPUT ......................................................................................................................................... 15 TIMELINE ................................................................................................................................................... 15 EFFECTS ..................................................................................................................................................... 15
Volume .......................................................................................................................................... 15 Compression.................................................................................................................................. 15 Fade in/out .................................................................................................................................... 15
5 RENDERING .......................................................................................................................................... 16
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 5 Dissemination level: PU - Public
TRANSPORT ................................................................................................................................................ 16 PREVIEW ................................................................................................................................................... 17 RENDERING ................................................................................................................................................ 17 OUTPUT / PUBLISHING ................................................................................................................................. 17
6 CONCLUSION ........................................................................................................................................ 18
7 APPENDIX: PRELIMINARY API DOCUMENTATION ................................................................................. 19
INTRODUCTION ........................................................................................................................................... 19 Scope ............................................................................................................................................. 19
DATA STRUCTURES....................................................................................................................................... 19 RAW IMAGE IMPORTATION ............................................................................................................................ 19
Image reading ............................................................................................................................... 19 RAW IMAGE CONVERSION ............................................................................................................................. 20
User feedback ............................................................................................................................... 20 White balance ............................................................................................................................... 20 Brightness ..................................................................................................................................... 21 Contrast ........................................................................................................................................ 21 Loading ......................................................................................................................................... 21
STANDARD IMAGE IMPORTATION .................................................................................................................... 21 COLOR MANIPULATION ................................................................................................................................. 22 GEOMETRY MANIPULATION ........................................................................................................................... 22
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 6 Dissemination level: PU - Public
Definitions, Acronyms and Abbreviations Acronym Title
API Application Programming Interface
HTTP Hypertext Transfer Protocol
SSL Secure Sockets Layer
FIM FotoInMotion
Term Definition
Beneficiary EC term used to designate the legal entity which has signed the
Grant Agreement. This term is often substituted by the common
language term ‘partner’.
RESTful API A RESTful API is an application program interface (API) that
uses HTTP requests to GET, PUT, POST and DELETE data. A RESTful API -
- also referred to as a RESTful web service -- is based on representational
state transfer (REST) technology, an architectural style and approach to
communications often used in web services development.
Consortium Group of beneficiaries that have signed the Consortium Agreement
and the Grant Agreement (either directly as Coordinator or
by accession through the Form A).
Deliverable
Leader
Responsible for ensuring that the content of the deliverable meets the
required expectations, both from a contractual point of view and in
terms of usage within the project. Is also responsible for ensuring that
the deliverable follows the deliverable process and is delivered on time.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 7 Dissemination level: PU - Public
1 Introduction Purpose of this Document and Relations with other WPs
The purpose of this document is to specify the 2D functionalities of the FotoInMotion
application. This includes basic image manipulation: image decoding, color manipulation,
geometry manipulation. 2D narrative effects and the core tools needed to enable the creation
of a FIM video, including timeline and soundtrack edition, up two the rendering of the final
video.
This document summarizes the required functionalities and gives guidelines towards their
implementation and integration inside the FIM application.
The image manipulation part has no direct links to other work packages. The integration of
this system in the FIM application is done in cooperation with WP4.
Document Structure This document is structured as follows:
The second section is dedicated to image manipulation functionalities and file decoding.
Next in the third section the timeline manipulation functionalities are described as well as the
first set of 2D narrative effects.
Section 4 describes the sound manipulation tools that will be proposed and finally the last
section describes the different rendering and output options.
The functionalities described in the document will be implemented in the first version of the
application in WP3 (tools) and WP4 (integration).
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 8 Dissemination level: PU - Public
2 2D Image Processing
Input/Output The first step of the creation of a FIM video shall be the selection of an existing photo.
Oriented towards professionals, the application shall be able to read standard file formats as
well as raw camera file formats. For the later the application provides a basic set of tools for
loading and converting the file.
File Format
The application will be able to handle both raw and non-raw formats. A set of conversion tools
will be provided for the later. When a raw image is loaded, the raw image data will be used
for 2D to 3D conversion in the later phases of the project (task 3.2).
Here is a minimal list of file formats that must be handled by the application.
• Standard image file, those files are loaded directly. Only a minimal set of editing tools
is available:
o JPEG
o TIFF
o PNG
o BMP
• Raw file formats. Raw formats require some conversion prior to be loaded. For this,
some additional tools will be provided (see under):
o 3FR
o ARW
o CRW
o CR2
o CR3
o CAP
o IIQ
o DNG
o FFF
o MEF
o NEF
o ORF
o PEF
o PSD
o RAF
o TIFF
o TIF
o RAW
o RW2
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 9 Dissemination level: PU - Public
Debayering
The first step of raw image conversion is the reconstruction of the image colors from the red-
green-blue pixel matrix from the sensors. The application shall support commonly available
filter configurations.
White balancing
The application will provide a color correction tool that will allow the user to select either
camera white balance either from a set of standard white balance and finally an interactive
tool to manually set the white balance from an area in the image.
Color Manipulation These tools are available for both raw and standard image formats.
Color Balance
Manipulation of color in the RGB and HSV (HSL) color space. For raw files this affect directly
the transfer curve during image conversion.
Brightness/Contrast
For raw files this affect directly the transfer curve during image conversion.
Geometry These tools are available for both raw and standard image formats.
Cropping
Interactively reframe the image. For raw files this affect directly the transfer curve during
image conversion.
Flipping
Flip the image left/right or upside/down. For raw files this affect directly the transfer curve
during image conversion.
Rotation
Rotate the image per 90/180/270 degrees.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 10 Dissemination level: PU - Public
Implementation
Image manipulation workflow
All image processing will be executed on the QdepQ server cluster. The workflow for any of
the operations listed is quite easy:
1. The user uploads its image, the system returns a unique ID.
2. The user sends the preferred operation with parameters and the image ID, the
systems acknowledges.
3. The user calls a render operation with parameters of the quality of the render (e.g. for
preview purposes or as a final full resolution render), the system executes the
processing and returns a link to the result.
During step 2 of the process, there is a possibility of sending multiple image operations to the
system that will be stored in an instruction list on the server. When the preview or full render
is requested, this list will be executed all together.
Figure 1: Image manipulation workflow
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 11 Dissemination level: PU - Public
Data storage policy
All data on the QdepQ server cluster will be temporarily stored. To ensure privacy, every
datafile will be stored under a unique identifier, making an orphaned file untraceable to a
user.
The duration of a file being stored for processing will depend on a server-side setting.
Depending on storage space available, legal requirements and processing status of an image,
the storage time may be minutes or days. The intention is to keep the final rendering time of
an image under 1 minute, so unless a project work-in-progress, the storage of files should be
minimum.
During development the storage will be set to a longer time period for analysis of the system
and the results.
Tools
As a part of the Image Processing & Narration Toolbox (IPNT), 2D image processing
functionalities will be presented via a set of RESTful services. Internally, the raw file
manipulation tools will be provided by the libRaw library around which an interface will be
built to match the specifications of the RESTful services.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 12 Dissemination level: PU - Public
3 Timeline manipulation These are the basic tools to build a FIM video from the decoded photography. The timeline
describes the different events that occur during the video and how they are organized over
the duration of the video.
Timeline specification The first version of the application will focus on the process of creating a video from single
photography. Events are the main concept used for the specification of the timeline: A
photography being selected for the construction of the video, the user creates the
FotoInMotion by adding event to the timeline. Events will affect either the way the
photography is displayed, either the associated soundtrack. Events have an effect for a limited
duration of time, from zero to the entire length of the video, that is to be specified by the
user.
Once an event has ended, its effect remains, this means that events are mainly used to specify
transitions from one state to another (e.g. zooming in over a period of time then remaining
focused on the zoomed in area).
There is no limitation to how many events can occur simultaneously, however, some events
might conflict with each other. Specific rules then apply that will be specified on a per case
basis.
The timeline is the conjunction of an image and a list of events, to be triggered at specific
moments, that will affect the way the photography shall be displayed. As well as the
mechanisms to trigger the actions described by the events and strategies to solve
inconsistencies between effects.
Creating a FotoInMotion video is therefore the task of adding events to a timeline.
Events Events introduce an effect in the video over a certain period of time, itself resulting in a
change in the final rendering. Events affect either the image itself, either the camera either
the sound.
Therefor events have a target. A first rule to apply to avoid conflicts is to prevent events
having the same target to happen at the same time.
Events can also be linked together so as to create more complex effects (eg. link a fade out
with a fade in).
Events affecting the 3D map will be added later, however the general model remains.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 13 Dissemination level: PU - Public
Image events
Image events affect the pixel of the image itself (but not the geometry of the image). Image
events, as part of the conflict avoidance rules, are divided into further subcategories, such as
color and texture events.
Viewport events
These event describe the general motion of the camera for rendering. In this first version,
only 2D motions are considered. The effects triggered by viewport events are part of the
narrative effects toolbox described below.
There is no limitation into combining viewport events together as long as they have different
targets.
Sound events
These effects trigger effects that target the soundtrack of the video (see below). Only one
sound vent can be triggered at the same time but they have no effect on other events.
Transition effects Transition effects are duration effects that reveal the image in a gradual way. These effects
are initially made for a single image, but if there is a demand for multiple images, this
functionality will be added for a later demonstrator. Like other effects, transitions can be
combined
Fade in/out
A fade in/out effect on an image will be a linear gradual increase/decrease of the image
opacity in respect to a certain background color.
Properties of the fade will be the duration and the direction (in/out) as well as the color of
the background.
Slide in/out
The slide in/out effect makes use of a change in panning to reveal / hide the image.
The properties of the slide effect are a destination (slide-out) / source (slide-in) of the image,
as well as the partition of the image that will be in-/outside the viewport and the duration of
the event.
Curl effect
This transition is a special effect that curls over an image and folds it out of the viewport,
revealing the underlying background.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 14 Dissemination level: PU - Public
2D Narrative Effects The first version of the application will provide a basic set of narrative effects. They are mainly
camera motion effect that will allow to move a virtually camera over the photography. The
first demonstrator(D3.3) is limited to basic narration tools, though in the second
demonstrator(D3.4) the annotation data of WP2 will be used to suggest narrative effects.
Panning
Defines the translations of the camera in a plane parallel to the image.As a part of an event,
this effect is defined over a period by a vector defining the displacement of the camera during
this period. The speed of the camera remains constant over this period. This effect can be
used in conjunction with zooming or rotation.
Zooming
Zooms over the image in and out. It is specified over a duration of time and by a single number
as zoom factor. The speed of the transformation remains constant. Alone, it magnifies the
center of the field of view. It can be used in conjunction with Panning and Rotation.
Rotation
Rotates the image by a given angle. The effect is specified over a duration and the speed of
rotation remains constant over that period. When used alone, the center of rotation can be
set interactively. Used in conjunction with panning or zooming, the center of rotation is the
center of the field of view.
Special Effects
Some general purpose effects will be proposed in this first version. These are part of the
rendering pipeline and therefore affect the final image after all other effects have been
applied.
The first set of special effect includes:
Motion blur, 2D motion blur the images is affected by a 2D blur along a direction specified by
the motion of the camera.
Spotlight, the image is darkened except for a circular area defined by the user and,
automatically tracked when the camera moves.
Selective blur, simulate depth of field effects by selectively blurring areas of the image.
Intelligent tools will be provided to helps select areas to be blurred of not. The blur can be
applied progressively for a given time period.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 15 Dissemination level: PU - Public
4 Sound Processing This section describes the sound manipulation functionalities of the software. The application
provides a basic set of tools that allows to build the soundtrack of the FIM video.
Input / Output The tool shall be able to read commonly available sound files such as wav, mp3, ogg, etc….
The volume of each sound file is normalized to the same range at load time. Sound file
reading and decoding capabilities will be provided via ffmpeg and made available via a RESTful
interface.
Sound output will be provided by a master bus which outputs the mix of all video files audio
effects can be applied to either audio sample or the master output bus
Timeline In a way similar to video effects, sound effects are placed on the timeline and work in a similar
way. However, it is possible to load successively or simultaneously several sound files and to
play the simultaneously.
Effects Sound effects either act on a specific audio sample either on the master bus.
Volume
Set the volume of a sound sample. The effect is applied immediately. This effect cannot be
used while a fade in or out is playing.
Compression
Dynamics compression, low volume sounds are amplified while high volume ones are not.
This effect is effective over an entire sample (or the whole video when applied on the master
bus).
Fade in/out
Bring in/out a sound sample by progressively changing its volume until the desired setting, or
zero, is reached. The final (or start) volume is set using the volume effect while it is
progressively brought in/out over a user defined duration at beginning, resps. end, of the
sample.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 16 Dissemination level: PU - Public
5 Rendering This paragraph describes the functionalities of the rendering pipeline whose role is to process
the selected media (image and audio samples) to produce the final video. The rendering
pipeline is double: first oriented towards the user interface to allow the user to preview its
works and a final rendering pipeline to a video file or media platform.
Transport HTML5 video streaming enables the manipulation of multimedia content using the <video>
HTML tag. Using the <video> tag creates a native HTML5 video player, and allows additional
options to customize and display the content. The standard <video> tag provides the
functionality for displaying and hiding video controls including: play/pause, volume
adjustment, full screen toggle as well as seek slider. Either a complete video file or an ever-
updating video stream as a source can both be supported. Technically speaking, in case of a
video stream, the HTML5 specification is agnostic of which protocols, transports, or codecs
are used with HTML5 video. Though, the combinations supported by most browsers are
described below:
HLSi:
HLS is h264-compressed video with aacii- or mp3-compressed audio, transported with MPEG-
TS. The stream is divided into chunks described in m3u8iii playlists and is transmitted over
HTTP.
MPEG-DASHiv:
MPEG-DASH typically is h264/h265-compressed video with aac audio transported with mp4,
or vp8/vp9v transported with WebMvi. The standard however is not bound to any particular
codecs, protocols, or transports. As with HLS, the stream can be divided into chunks but it’s
optional in this case.
i https://en.wikipedia.org/wiki/HTTP_Live_Streaming ii https://en.wikipedia.org/wiki/Advanced_Audio_Coding iii https://en.wikipedia.org/wiki/M3U iv https://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP v https://en.wikipedia.org/wiki/VP9 vi https://en.wikipedia.org/wiki/WebM
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 17 Dissemination level: PU - Public
Preview The preview pipeline is intended to provide the user with a real time impression of a work in
progress. Therefore performance is crucial for the preview pipeline and only a downscaled
version of the final video is processed. Further, frame rate can also be reduced if necessary.
The rendered video and audio streams are then sent to an ffmpeg server that encodes and
streams it to the client. The output format and codec for the preview pipeline are yet to be
defined depending on the requirements of te video player component.
Rendering The final video is finally rendered at its full size and resolution, or a a resolution constrained
by the chosen output format or platform. Ffmpeg will be the tool of choice for this task and a
selection of the available output formats and options will be proposed. Calls to ffmpeg shall
be made via an API providing RESTful services.
Output / Publishing The final rendered video will temporary be stored on the QdepQ server clusters and the user
gets notified with a download link via a secured connection (SSL). Once the user downloads
the video and concludes the project, it is immediately deleted from the server (this setting
may be overruled during development to acquire data on performance).
Alternatively, the results can directly be pushed to an online platform. The application shall
provide at least access to the most important ones (Youtube, Dailymotion, Vimeo). Also, the
user shall be able to provide a custom server where the final video is to be sent. The work can
then be published automatically once the rendering done with no further user intervention,
even if the user is not connected.
In any case, the rendered video will not be kept on the server longer that the time necessary
for the user to download or publish it.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 18 Dissemination level: PU - Public
6 Conclusion This document summarizes the basic functionalities of the first version of the IPNT service,
with a focus on 2D image processing. 2D functionalities include basic image preprocessing
and raw file manipulation functions. Definition of a timeline upon which narrative effects can
be arranged and the definition of a basic set of narrative effects. And finally, the rendering
pipelines and publishing functionalities.
Together these features will form the backbone of the first version of the IPNT service to be
implemented in the first viable version of the application (D3.3)
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 19 Dissemination level: PU - Public
7 Appendix: Preliminary API documentation Introduction
The 2D API of the FotoInMotion application is intended to provide the necessary functionalities for
opening and pre-processing image files. In the case of raw files, this includes the conversion of the raw
image to standard 8 bits images for rendering and the manipulation of the conversion parameters. For
standard images, basic color and image geometry manipulation is also provided.
These functionalities are accessible via actions the user interface that trigger processing on the server
side. Each of them are therefore specified from the front end and the back end side.
In this appendix we focus on the image processing functionalities and leave aside the effects based
around the time line, as specification of the later is out of the scope of this deliverable.
Scope
This document a preliminary work that intends to clarify the basic needs in terms of functionalities as
well as the general mechanisms underlying the client/server communication and the feedback to the
user. It also enlightens the basic data structures that will be needed to perform these tasks. These
specifications are subject to constant evolution in accordance with the Agile methodology used in the
project (see D4.1).
Data structures In terms of image data, the application relies on three main image data classes:
• RawImage
a RawImage class instance is used to manipulate the raw image conversion tasks. It contains
the raw sensor data, EXIF data and a thumbnail 8 bits image. Instance of RawImage only
exist on the back-end side.
• FIMImage
Internal image data class, this class embeds the converted (or read) 8 bits image data at full
resolution. Floating point image data consisting of either the debayered raw image data or the
floating-point conversion of the 8 bits data if the former is not available. A low resolution 8
bits thumbnail image. Instances of a FIMImage only exist on the back en side.
• FIMThumbnail
This is the class for providing feedback to the user interface. It consists of a downscaled
version of the image in its 8 bits version. Manipulations to either RawImage conversion
parameters or to FIMImage data are applied correspondingly on the corresponding
FIMThumbnail instance. FIMThumbnail instances are mirrored automatically on the front-
end side when modified on the back end.
Raw image importation
Image reading
These functions allow the importation of raw images and their conversion to standard 8 bits images, as
well as the manipulation of the conversion parameters. The image is read on the front-end device and
sent in a binary format to the back end server.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 20 Dissemination level: PU - Public
Raw image manipulations are widely based on the libRaw libraryvii and functionalities described her
are essentially an abstraction of the libRaw API tailored to the needs of the FIM application.
• FIM_Front_ReadRawImage(String path)
Opens the image file in path, establish a connect to the back end server and sends a stream of
binary data from the file.
• FIM_Back_ReadRawImage(InputStream stream, out RawImage img)
Read the data sent by FIM_Front_ReadRawImage and build a RawImage instance that
gathers the EXIF metadata, Raw image data, conversion parameters as well as a the data for a
thumbnail image used to provide feedback to the user (see 7.4.1) .
Raw image conversion Once the raw data has been made available, it is possible to manipulate the conversion parameters.
Depending on the limitations of the libRaw, not all parameters might be available for all kind of
cameras.
User feedback
In order to provide feedback to the user, each function call leads to an update of the thumbnail image.
That in return will be sent back to the front end. The thumbnail image is a preview of the imported
image, converted to 8 bits with the current settings and downscaled so that to insure smooth interaction.
These functionalities are handled internally so that each call to a front-end function will trigger an
update of the preview image in the UI. For these purpose two functions are made available:
• FIM_Front_UpdatePreview()
Make a request to the back end to send the up-to-date thumbnail image.
• FIM_Back_SendPreview()
Update thumbnail image with the current conversion parameters and sends it back to the front
end.
White balance
• FIM_Front_RawImageSetWhiteBalancePreset(int preset)
Send a flag corresponding to a white balancing preset to the back end.
• FIM_Back_RawImageSetWhiteBalancePreset(int preset)
Set image white balance to a predefined value (Camera, Automatic, …). The exact set is
depending of the camera and defind by the libRaw interface (see:
https://www.libraw.org/docs/API-datastruct.html#libraw_output_params_t).
• FIM_Front_RawImageSetWhiteBalancePoint(point2D point)
Send the coordinates of a user selected point to set white balance parameters to the back end.
• FIM_Back_RawImageSetWhiteBalancePoint(point2D point)
Compute white balance parameters in a small window around the designated point.
vii https://www.libraw.org/
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 21 Dissemination level: PU - Public
• FIM_Front_RawImageSetWhiteBalanceTemperature(int temp)
Send a user defined white balance temperature value to the back end.
• FIM_Back_RawImageSetWhiteBalanceTemperature(int temp)
Compute white balance parameters according to a given temperature setting.
Brightness
• FIM_Front_RawImageSetBrightness(float brightness)
Send a user defined brightness value to the back end. Brightness is a floating-point value
within [0,1].
• FIM_Back_RawImageSetBrightness(float brightness)
Set the brightness setting to the given value.
Contrast
• FIM_Front_RawImageSetContrast(float contrast)
Send a user defined contrast value to the back end. Contrast is a floating point value within
[0,1].
• FIM_Back_RawImageSetcontrast(float contrast)
Set the contrast setting to the given value.
Loading
• FIM_Front_ConvertRawImage()
Trigger the conversion of the full resolution image on the back end.
• FIM_Back_ConvertRawImage(out FIMImage img)
Apply the conversion parameters to the whole RawImage and produce a FIMImage instance.
FIMImage is the internal image format used for further processing. It embeeds te converted 8
bits data as an openCV image, the raw data converted to floating point that will later be used
for 3D conversion as well as thumbnail images for user feedback.
Standard image importation The mechanisms for reading a standard 8 bits image are similar to the ones for raw images. The main
difference being that openCV will be the background library of choice for those files. Also there is no
need for conversion and related functions. The functions for reading standard images are:
• FIM_Front_ReadImage(String path)
Opens the image file in path, establish a connect to the back end server and sends a stream of
binary data from the file.
• FIM_Back_ReadImage(InputStream stream, out FIMImage img)
Read the data sent by FIM_Front_ReadImage and directly build a FIMImage instance
where the raw data is a floating point version of the image data.
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 22 Dissemination level: PU - Public
Color manipulation These operations apply to the 8 bits version of the image contained in a FIMImage and the thumbnail.
Preview mechanisms for user feedback are identical to the ones ones for setting the conversion
parmeters. The raw data is left untouched.
• FIM_Front_FIMImageSetColorBalance(float r,float g, float b)
Send color balance parameters to the back end. Color balance is defined by three
multiplication factors within [0,1].
• FIM_Back_FIMImageSetColorBalance(float r,float g, float b, in/out FIMImage img, )
Apply the color balance to the FIM image by multiplying each color channel by the
corresponding factor. Previously, these factors have been rescaled so to match the range of
values in the image from [0,1] to [0, min(img.r/img.g/img.b)/255].
• FIM_Front_FIMImageSetBrightness(float brightness)
Send a user defined Brightness value to the back end. Brightness is a floating-point value
within [0,1].
• FIM_Back_FIMImageSetBrightness(float brightness, in/out FIMImage img)
Apply the Brightness setting to the image
• FIM_Front_FIMImageSetContrast(float Contrast)
Send a user defined Contrast value to the back end. Contrast is a floating-point value within
[0,1].
• FIM_Back_SetContrast(float Contrast, in/out FIMImage img)
Apply the Contrast setting to the image.
Geometry manipulation The functions apply to the FIMImage instance and modify both the 8 bits and the raw image as well as
the thumbnail.
• FIM_Front_FIMImageCrop(Point2D topLeft, Point2D bottomRight)
Send the top left and bottom right coordinates of a cropping rectangle. The coordinates are
sent relative to the dimensions of the thumbnail image
• FIM_Back_FIMImageCrop(Point2D topLeft, Point2D bottomRight, in/out FIMImage img)
Compute the coordinates of the full size rectangle and crop the FIMImage instance
accordingly.
• FIM_Front_FIMImageRotateLeft()
• FIM_Front_FIMImageRotateRight()
Send to the back end a message for rotating the image by 90 degrees to the left or to the right.
• FIM_Back_FIMImageRotateLeft(in/out FIMImage img)
• FIM_Back_FIMImageRotateRight(in/out FIMImage img)
Rotate img 90 degrees to the left or to the right.
• FIM_Front_FIMImageFlipHorizontal()
FotoInMotion D3.1 - Final
H2020 Contract No. 780612 v1.0, 30/09/2018
Page | 23 Dissemination level: PU - Public
• FIM_Front_FIMImageFilpVertical()
Send to the back end a message for flipping the image left/right or upside/down.
• FIM_Back_FIMImageFlipHorizontal(in/out FIMImage img)
• FIM_Back_FIMImageFilpVertical(in/out FIMImage img)
Flip img left/right or upside/down.