spot & datacontainer daniel (me) henryk & thomas & siavash
TRANSCRIPT
Agenda• The way we work with data and metadata NOW and how we
want to improve it• Make codes shorter and easier to read
– Basic Clean-up using DataContainer– Two new operator constructor syntaxes
• Avoid/detect programming mistake -> more robust code• Some Things to Keep in Mind
– Children operator dimension– Operation History– Dynamic Operator
What is Metadata?
Image Source: USGS
--------------> Data
1. Unit on the axis2. How far apart are the sample3. The label on the axis4. <etc>
--------------> Metadata
How We Treat Metadata
We have to change the metadata ourselves.
Data and metadata are separated.
What if we load different data and the unit is different
Why keeping track of metadata ourselves is error prone?
How We Treat MetadataWhy keeping track of metadata ourselves is error prone?
How do we know what actually happened in the function?
How We Treat MetadataWhy keeping track of metadata ourselves is error prone?
Data is structured differently than we thought:What if the data we get actually has space-domain on 1st dimension and time domain on 2nd dimension?
How We Treat MetadataWhy keeping track of metadata ourselves is error prone?
Keep track of metadata is a total pain for higher dimension data. Remember there are other metadata too, not just unit.
What if We Want to Keep Track of Metadata Automatically
Operator(s) Data
MetaData
Operator (not us) should modify BOTH the data and metadata at the same time.
Originally
We
What if We Want to Keep Track of Metadata Automatically
Operator(s)
Data
MetaData
DataContainer
What we want
So what does a DataContainer look like?
Same Vector Operation as MATLAB Array
x = randn(3,5);
>> x1 = vec(x);>> x2 = x’;>> x3 = norm(x);>> x4 = 2*x + 3*x – 4*x>> % and more
MATLAB array
C = iCon(randn(3,5));
>> C1 = vec(C);>> C2 = C’;>> C3 = norm(C);>> C4 = 2*C + 3*C – 4*C>> % and more
DataContainer
DataContainer : Works like MATLAB array
What DataContainer Enables to DoOriginally,We don’t know if the data is structured the way we expected.We don’t know if the function is doing things properly.
We can put something to check the metadata.Here we check unit as an example.
Agenda• The way we work with data and metadata NOW and how we
want to improve it• Make codes shorter and easier to read
– Basic Clean-up using DataContainer– Two new operator constructor syntaxes
• Avoid/detect programming mistake -> more robust code• Some Things to Keep in Mind
– Children operator dimension– Operation History– Dynamic Operator
Example code
The data
metadata
Let’s Clean up the Code!
Clean-up Stage 0: Put data and metadata into DataContainer
New Operator Constructor Syntaxes
A = opDFT(model.n(1))
A = opDFT(size(C,1))Version 0
A = opDFT({C,1})Version 1
A = opDFTVersion 2
Clean-up Stage 1: Using new Syntax to Clean Up and Give more info to the spot operator
New Operator Constructor Syntaxes
A = opDFT(model.n(1))
A = opDFT(size(C,1))Version 0
A = opDFT({C,1})Version 1
A = opDFTVersion 2
Clean-up Stage 2: Using Dynamic Operator
Dynamic Operator
Simple Dynamic Operator Example : opDFT
opDFT( )
DataContainer
Dimension: 3 X 1opDFT( ) *3
Clean-up Stage 2: Using Dynamic Operator
Agenda• The way we work with data and metadata NOW and how we
want to improve it• Make codes shorter and easier to read
– Basic Clean-up using DataContainer– Two new operator constructor syntaxes
• Avoid/detect programming mistake -> more robust code• Some Things to Keep in Mind
– Children operator dimension– Operation History– Dynamic Operator
Agenda• The way we work with data and metadata NOW and how we
want to improve it• Make codes shorter and easier to read
– Basic Clean-up using DataContainer– Two new operator constructor syntaxes
• Avoid/detect programming mistake -> more robust code• Some Things to Keep in Mind
– Children operator dimension– Operation History– Dynamic Operator
Keep in Mind - Operator
opKron( , ) Size : dim1 X dim2 X dim3
(DataContainer)
For operating onDataContainer:
opKron(m x n) (n x 1)XFor operating on
MATLAB Array:
(MATLAB Array)
Keep in Mind - Operator
opTaper = opKron(opDirac(size(C,3)),opDirac(size(C,2)),opLinMute(size(C,1),0.1*size(C,1),0.2*size(C,1)));
opTaper = opKron(opDirac({C,2:3}),opLinMute({C,1},0.1,0.2)); ({<Container>, <selected dimensions>})
To Properly Define the Operator:
opTaper = opKron(opDirac,opDirac, opLinMute({C,1},0.1,0.2)); Dynamic operator
Keep in Mind - DataContainer
In our example, we must initialize the container as 3-dimensional THEN vectorize it.
Keep in Mind – DataContainerKeeps Track of Metadata History
res2 should have the same metadataas container C.
History Stack
A.ID Mode = 1 Delta, Origin, Unit,…, etc
A.ID Mode = 1 Delta, Origin, Unit,…, etc
res2 = A’ * A * C
How operating on DataContainer helps recover information that we usually can’t recover
A.ID Mode = 2
res1 = A * C
Keep in Mind - DataContainer
• When doing addition or subtraction between 2 DataContainers, we keep the history of the one on the left.
Container 1
C1_history
Container 2
C2_history+ =
Result Container
C1_history
Keep in Mind – Dynamic Operator
• A dynamic operator is activated in the scope it is used.
Advantage?Disadvantage?
Keep in Mind – Dynamic Operator
• Some dynamic operators cannot activate in adjoint mode.– They require that they had been used previously
in the forward mode.
Some dynamic operators require that they had been used previously in the forward mode.
A = opCurvelet(23,29)A = opCurvelet(?)
A’ * A * C Example
History Stack
A.ID Mode = 1 Delta, Origin, Unit, info for A,…, etc
A.ID Mode = 1 Delta, Origin, Unit, info for A,…, etc
A.ID Mode = 2
C = iCon(randn(23,29))
A * C
A’ * A * C
Keep in Mind – Dynamic Operator
• The operator will have different ID when created at different time.
A.ID Mode = 1 …
Conclusion
Why consider using DataContainer?• Works just like MATLAB array.
– Not too difficult to change– Added feature
• Organize your data and metadata• Can be saved to file – no more editing your script
– Apply changes to your metadata• Shorten your code => easier to read• Dynamic operator
– Automatically initialize operator for you• Use the same script for different size data => easily scale to larger data sets
• Catch error that MATLAB array can’t normally catch
• The new spot operators are backward compatible