best practices for publishing online services · 2019-08-28 · overwrite feature service – from...

Best Practices for Publishing Online ServicesMichael DangermondPaul DoddKeith VanGraafeiland

Agenda

• Feature Services• Image Services• Multidimensional Image Services

Section SubheadFeature Services

Keith VanGraafeiland

Hosted Feature Services Examples

Feature Services

• Overwrite Feature Service – from web content• Hosted Feature Views• Collaborative Group

• Examples- Coral Reef Stations- USA Weather Watches and Warnings- Arctic Sea Ice Extent

Overwrite Feature Services

• Keeping it updated• There’s a script to do that?!?

- https://www.arcgis.com/home/item.html?id=d45f80eb53c748e7aa3d938a46b48836

• Append functionality coming soon!• Link to webinar

Updating Hosted Feature Services

• Append- More Flexible / Efficient loader- Stream to Database – Bulk Load- Preferred over

- Batch: Add / Update / Apply Edits

• When is Overwriting practical?- High Row Replacement- No Unique Identifier (ex. weather data)

- Append / Delete too expensive

• Enterprise Equivalent to Overwrite- Disable Indexes- Truncate Data- Append Updates- Rebuild Indexes!

• Overwrite will impose Downtime!- Features are inaccessible during update- Enabling CDN cache helps!- Doesn’t have to be! (more later)

Leveraging Hosted Feature Views

• Create from Hosted Feature Service- Never Duplicate Data- Not effected by Service Overwrite!- Points to Host Service Layers

• Provides High Level of Control- Rename Layers- Alter Aliases- Control Caching- Limit access to sensitive data Fields- Limit Feature access by extent- Control Sharing

File Item

Hosted Feature Service & Item

Hosted Feature Views & Items

Internal View

External View

Public View

Feature Service ‘A’

Overwrite Feature Services – Layer Swizzling

• To Swizzle:1. To alter a reference by changing what it

points to.2. To present or change the appearance of

something based on something else.Ex: Lookup Table or Coded Value Domain

• Simply put:- Change what your Layers point to!

• Overwrite w/o Downtime!• Frequent / Extended Update Times

Feature Service ‘B’

Private Items

Private Items

Shared View

View Template

View Template

Active Service Active ServiceUpdate Target Update Target

Update, Swizzle, Repeat!

Enterprise Example – Table Swizzling

• National Water Model – Time Series- 2.7 Million Features (river reaches)

- x 80 - 3hr Forecasts (medium range, 4/day)

- x 18 - 1hr Forecasts (short range, 24/day)

- = 2+ Billion Rows Updated Daily!

Public Map Service

Feature to Table Join

(view)Line

Featureclass

DB Table ‘A’ DB Table ‘B’

Active Table Active TableUpdate Target Update Target

Update, Swizzle, Repeat!

Miscellaneous Tips

• Time Series Layers- REST admin console to refine Time details- Stick to Date and Numeric fields for Time!

• GeoDatabase Attribute Fields- Use Appropriate Type & Length for Data- Limit to a practical number, set visibility- Only add Indexes to Query-Likely Fields- Leverage Coded Value Domains!- Use Reasonable Names and Aliases- Add Units to Alias of Numeric Fields

- Ex: ‘Speed (km/h)’ or ‘Temperature (kelvin)’

• Geometries- Watch your Feature complexity!

- 10k vertices / feature ok- Too many Parts can slow load / draw times!

- Use ‘Dice’ GP tool to break up Features!

• GeoDatabase Feature Datasets- Don’t use as a content organizer!

- Locks applied to every object when accessing one!

Image Services

Michael Dangermond

Thematic Image Services

Mosaic Dataset+ overviews

Raster

Image Service

Clients

Thematic Image Served From a Mosaic

Easier to Maintain• Upgrading can be this easy:

Turn off the service.Replace the TIFF image in the mosaic with one with the same name.Turn the service back on.

• In other words, make any changes within the mosaic and refresh the service.• The ID, content item, popup, URL stays the same.

Mosaic Dataset

Rasters

Image Service

Clients

Mosaic Datasets+ overviews

BLM NPS USFS

AlaskaHawaiiCONUS

Puerto Rico

USA Federal Lands Service

Mosaic Dataset

Rasters

Image Service

Clients

Mosaic Datasets+ overviews

BLM NPS USFS

AlaskaHawaiiCONUS

Puerto Rico

ProcessingTemplatesServer Functions and

Definition Queries

USA Federal Lands Service

Attribute Table Function

• ClassName field appears in legend• Red, Green, Blue fields color the map• Other text fields display

in an online popup• Maximum 2000 Records

in attribute table

Time-EnabledMosaic Dataset

2018Rasters

Time-EnabledImage Service

Time-AwareClients

Mosaic Datasets+ overviews

20162017 2018

AlaskaHawaiiCONUS

Puerto Rico

USA NLCD Land Cover Service

Creating a Thematic Image Service

1. Organize or Create Rasters2. Create an Attribute Table3. Create a Mosaic4. Publish an Image Service

Download the Checklist

Step by step instructions help you through the details as you publish services.https://tinyurl.com/tichecklist

Organize rasters

• Common projection, nodata symbol, bands, bit depth, type, etc.• Strip off tables and colormaps from rasters.

Apply attribute table function on the mosaic using a processing template.

Organizing Your Rasters

• Recommended workflow: Use CopyRaster() to createTIFF images with LZW compression. Use the same NoData symbol for each raster.

• Use an equal area projection so client area counts are accurate.USA – Albers ProjectionsWorld – Mollweide Projection has the least distortion. We checked.

Prepare an Attribute Table

• Geodatabase table• Value and count fields are required• Add ClassName, Red, Green, Blue

fields for the legend and map colors,plus additional fields to drive popups,if desired.

Create mosaic

• Same projection, nodata symbol, bands, bit depth, etc.• Before creating the mosaic decide:

- Mosaic’s name- Service name- Projection

OverviewsBest practices

• Overviews are a lower resolution version of your rasters.

• Edit the mosaic footprint table,change all MaxPS and HighPS values to the maximum value in the table.

• Define Overviews• Build Overviews

Thematic Image Service ChecklistHere’s where to download the checklist:

https://tinyurl.com/tichecklist

Image Services

• Simple Example - Topobathy• More Complex Example – USA Landcover? Cropscape?• Raster Function Templates• Using Image Services in Pro

Keith VanGraafeiland

Multidimensional Image Services

Oceanographic Information for the World

• Five variables using on-the-fly calculations from HYCOM

AT YOUR FINGERTIPS

• Sea Surface Elevation

• Temperature, Salinity, Eastward Velocity, Northward Velocity

- 40 Standard Depth Levels

• 7 Day Forecast

• 30 Day Hindcast

Multidimensional Image Services

• What we see:- Global high resolution data- Temporal component- Continuous updates

• How it works- S3 used for storage- ALF Processor / ALF Deployer- Applying RFT’s for additional

functions

HYCOM CapabilitiesTemperature Salinity Eastward

VelocityNorthward

Velocity

2.9 GB 2.9 GB37 MB

Elevation• Sea Surface Elevation

• Temperature

• Salinity

• Water U

• Water V

Files are downloaded and converted to Meta Raster Format (MRF).

Placed in Amazon s3

Referenced to mosaic

Additional Raster Functions applied for unit conversion

HYCOM by the Numbers

• 1 x Sea Surface Elevation

• 40 x Temperature

• 40 x Salinity

• 40 x Water U

• 40 x Water V

• 40 + 40 + 40 + 40 +1 = 161 file sets

Every 3 hour Interval

644 files (4 files per MRF 161 x 4)2.43 GB

Each day

5,152 files (644 x 8 intervals)19.44 GB

Entire Volume

190,624 files (5,152 x 37 days)719.28 GB

Elevation Temperature Salinity Eastward Velocity

Northward Velocity

Future Enhancements

• Multi-Dimensional MRF or CRF- Consolidating down to 5 files per time slice (one for each variable)

- Limits the amount of requests to s3- Distributing files between multiple s3 buckets (sharding)- Reduce s3 as the “bottleneck”- Programmatic

Please Share Your Feedback in the App

Download the EsriEvents app and find

your event

Select the session you attended

Scroll down to “Survey”

Log in to access the survey

Complete the survey and select “Submit”