home r fast documentation - · pdf filehome_r fast documentation olivier mestre, ... •...
TRANSCRIPT
![Page 1: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/1.jpg)
HOME_RFast
documentation
Olivier Mestre, Enric Aguilar
HOMER Training School
![Page 2: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/2.jpg)
Requirements• Software: homer.2.6.R under R 2.15
• Requires– GNU math library GSL
– R packages “cghseg”, “maps”, “mapproj”
![Page 3: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/3.jpg)
Main functions
• Basic checks (CLIMATOL)
• Fast QC
• Plotting series
• Detection (pairwise,joint detection,ACMANT)
• Correction
![Page 4: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/4.jpg)
Directories/files
Raw data
Quality controlled data
Homogenized data
Graphic files
Main programQC files, coeffs.
Network file
Detected breaks
Temporary files
• Main directory
![Page 5: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/5.jpg)
Network filennnnnnstations.txt nnnnnn network numbercreated by... you!filename, latitude (°' ''), longitude, altitude, na meSeparator : tabStation indices: 8 charactersStation names: no spaces allowed
![Page 6: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/6.jpg)
Data file• Year + 12 monthly values; separator: Tab
• Missing flag: -999.9
• Filename: hhppmxxxxxxxxd.txt
hh = prefix (ra for raw, qc for QCed, ho for corrected)
pp = parameter (ex: tn, tx, rr...)
• xxxxxxxx station id (8 characters)
![Page 7: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/7.jpg)
Detected breaks
• Corresponds to network file nnnnnndetected.txt• Created by HOMER
Station id
Nature of changeNature of change
Year of change
Month of change
Flag"v" = validatedby metadata
![Page 8: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/8.jpg)
Initialization menu (part 1)Network 000005
qctxmxxxxxxxxd.txt filesin qc directory
What you want, could havebeen "Maximum Temperatures"
Temperatures = additive parameter (return)
Interactive option allows control interactively detection outputs
Will produce a nice "°C" in outputs"
![Page 9: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/9.jpg)
Initialization menu (part 2)For small networks within thesame climate region, keep allseries; otherwise, correlationneighbourhood (1st differencedseries) is recommended
Ex : Will select all series withCorrelation > 0.9
If less than 6 series, will alsopick up series less correlated(up to 6 series)
Recommanded for temperatures
Adds linear trend/smoothed trend
Fills with color above/below mean
rb = red above, blue below
Also available : g (green), y (yellow)
![Page 10: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/10.jpg)
Input checksChecks if directories are present, and creates them if necessary
Checks consistency between station files and data files (wrong network or header)
![Page 11: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/11.jpg)
Actions : Main MenuAutomatic gender
detection not implemented yet!
ACMANT (a)Assess Month of change (m)
Those options appear onlyafter a first correction round
![Page 12: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/12.jpg)
CLiMATOL Checks (i)• General properties for nnnnnnstations.txt network
• Stored in meta/hhppnnnnnndiagnosis
![Page 13: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/13.jpg)
Fast Quality Control (f)Performs QC on input files or on
corrected files
Running…Here with correlationneighbourhood.
![Page 14: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/14.jpg)
meta/control_qctxxxxxxxxx.pdf
Outlier for Île d'Yeu*series, january 1979
![Page 15: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/15.jpg)
Create Outlier file (o)
Creates outlier file
Action (r) changes all entries to
missing in the corresponding
datafiles
![Page 16: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/16.jpg)
Visualisation (v)
Assesses significance of the linear trend
(non significant here: large p-value)
Creates window (interactive option turned on)
Files created in "fig" directory
Season option turned on
DJF = winter seriesJJA – summer serie
Etc… also created
![Page 17: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/17.jpg)
Pairwise detection (d)Running pairwise on computed neighbourhoods
Probable breaks on St Georges TX
![Page 18: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/18.jpg)
Pairwise detection• ''d'' : performs detection on specified files (input or corrected)
• Uses predefined neighbourhoods – changed using ''n'' optionFor clarity reasons, pop up windows only show 6 pairwisedetections with lowest sigma (interactive option turned on)
• Detection performed on annual+seasonal, annual or monthly (not recommended)
• Does not create "detected" file
![Page 19: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/19.jpg)
Pairwise detection output files
Pairwise series Position and amplitude Position and neighbour
of detected breaks of detected breaks
fig/detect_qctx49281001_a.pdf fig/detect_qctx49281001_b.pdf fig/detect_qctx49281001_c.pdf
Sigma values (noise intensity)
![Page 20: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/20.jpg)
Joint detection (j)
• Automatically creates a "detected" file
• Interactive window for modifications
![Page 21: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/21.jpg)
Joint detection (j)
Pairwise adds a break removes a break(annual, JJA, DJF) position on y axis is not important
+ = joint detection + interactive modifications
(automatic)
![Page 22: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/22.jpg)
Result of joint detection• St Georges
– 1970 2001 Automatic– 1955 1977 Added interactively– 1986 Removed interactively
![Page 23: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/23.jpg)
Correction (c)• Always performed on input files - not on already corrected data
• A pairwise detection phase on corrected series is launchedimmediately afterwards
![Page 24: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/24.jpg)
Common problemA common mistake is to have only missing values between two breaks.
Shall this situation occur, the program will stop and ask you to remove one of the two breaks. In the example below, Rennes series has no data between 1928 and 1930, hence estimation cannot be performed. One of the two breaks has to be removed.
![Page 25: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/25.jpg)
Correction (c)• Corrected data are stored in ho/hoppmxxxxxxxxd.txt
• Correction neighbourhoods and coefficients are provided in meta/metappxxxxxxxx.txt
![Page 26: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/26.jpg)
Correction
Each time a correction is performed, the current version of the detected file isstored in the "tmp" directory, so you cango back to earlier version at any time:
nnnnnndetected.txt.1
nnnnnndetected.txt.2
nnnnnndetected.txt.3
Etc…
![Page 27: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/27.jpg)
ACMANT detection (a)
• Option available only for additive parameters
• Runs with pre-homogenised reference seriesOption available only after a 1st correction round
• Strength: joint detection of changes in annualmeans and seasonal cycle
• Automatic
![Page 28: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/28.jpg)
ACMANT Detection (a)
Dotted lines = ACMANT detections
Grey line = dates already in nnnnnndetected.txt
Grey (+) symbols = previous joint detections
Maybe interactively modified (same as joint detection)
![Page 29: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/29.jpg)
Assess month (m)• Option available only for additive parameters
• Runs with pre-homogenised reference series
Option available only after a 1st correction round
performed on serial deseasonalized series
![Page 30: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/30.jpg)
Break file creation/modification (b)
• Create or add dates to nnnnnndetected.txt
Automatic Menu
Asks if more dates are to
be added for each series
![Page 31: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/31.jpg)
Break file edition (e)• Edits nnnnnndetected.txt content (R editor)
• Convenient to modify dates/validation flags
Line inserted using (b)
![Page 32: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/32.jpg)
CCreate data filesCreate station file
GRAB METADATA!
CLIMATOL Checks
Fast QC
Pairwise detection + joint detection
First guess of changes on input data (use alsometadata)
Correction
ACMANT detection (input data)
Correction
Pairwise detection + joint detection on corrected data
Improved guess of changes (use also metadata)
CorrectionACMANT Detection on input data
Assess month of change
Correction + pairwise detection on corrected data
![Page 33: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/33.jpg)
Warning
Avoid validate very close change-points
Assess month of change only in the end of the process
![Page 34: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/34.jpg)
Recommendations
• Use QCed data• Use metadata• Do not hesitate to get rid of poor series• Ensure all data periods are non missing in
several series
Control your nnnnnndetected.txtclose change-points are not recommended
![Page 35: HOME R Fast documentation - · PDF fileHOME_R Fast documentation Olivier Mestre, ... • xxxxxxxx station id (8 characters) ... pick up series less correlated](https://reader030.vdocument.in/reader030/viewer/2022021509/5a8551167f8b9a882e8c2680/html5/thumbnails/35.jpg)
Citing HOMER+Licence
homer.2.6.R is distributed under GNU General Public licence, version 3 or newer. Please check: http://www.gnu.org/licenses/
HOMER has no dedicated publication yet, and combines different published methods(PRODIGE, ACMANT, CLIMATOL) that gave good results during COST Action ES0601, alltogether with "cghseg" package , that you shall cite:
Caussinus H. and Mestre O. (2004) Detection and correction of artificial shifts in climate series. Applied Statistics, 53, part 3, 405-425.
Domonkos P., R. Poza, and D. Efthymiadis (2011) Newest development of ACMANT. dv. Sci. Res., 6, 7-11, 2011. doi:10.5194/asr-6-7-2011
Picard F., Lebarbier, E., Hoebeke, M., Rigaill, G., Thiam B. and Robin S. (2011). Joint segmentation, calling, and normalization of multiple CGH profiles. Biostatistics. doi:10.1093/biostatistics/kxq076
Guijarro J.A. (2011): User's guide to Climatol, 40 pp. http://www.meteobal.com/climatol/climatol-guide.pdf