specification of the fims media soa framework repository service interfaces loic barbou & ashraf...

SPECIFICATION OF THEFIMS

MEDIA SOA FRAMEWORK

REPOSITORY

SERVICE INTERFACESLoic Barbou

&

Ashraf Tadros

Services• Repository Service

– Repository Service Operations– Repository Capabilities Registry (RCR) Operations

• Repository Notification Service– Asynchronous Operations Notifications– Events Subscriptions

Repository Service - Operations» Lock, Unlock, Clear Lock, Retrieve Open Locks» Generate Unique ID» Add Object» Add Content (Async.)» Update Object (Replace, Update Properties)» Update Status *» Delete and UnDelete Object/Content» Purge Object (Async.)» Purge Content (Async.)» Get BM Content(s)» Retrieve Essence (Async.)» Media Query

Repository Service - RCR» Repository State (online, offline, failover)» Storage Capacity (Names, available and total space)» Performance (Performance Indicator %)» Vendor Specific Report (Report HTML)» General Capability (supported functionalities)» Vendor Defined Schema Extensions» Updatable and Searchable Properties» Vendor Defined Query Result Set Definitions» Repository Definitions (name & Description)» Retrieve Administrative Super Lock Token» Supported Object/Content Status

Notification Service - Async. Operations» Add Content (Notify and Fault)

» Purge Content (Notify and Fault)

» Purge Object (Notify and Fault)

» Retrieve Essence (Notify and Fault)

Notification Service - Events Subscription

– System Events» Exceptions» Performance

– Asset Events» Status Changed» Object/Content Created» Object Deleted» Content Deleted/Purged» Object/Content Format/Essence Locator Update

Clarifications• Asynchronous Operations Notifications

» Request => ResourceType => AsyncEndpointType» Ack: timeStamp + operationId» Notify Result: result + operationId» Notify Fault: RepositoryFaultType + some related info +

operationId

• Lock Token » RepositoryLockTokenType» RepositorySuperLockTokenType

Clarifications• Status

» Object - BMContentStatusType

» Content - BMEssenceLocatorStatusType

» BMStatusType - new, online, offline, deleted, purged, invalid, processing.

» Vendor specific - BMCustomStatusExtensionType.

» RepositorySupportedStatusType

Essence Locator StatusBMContentType Status

Online Offline Deleted Purged Invalid New ProcessingOnline x x x x x x xOffline x x x x x xDeleted x x xPurged x xInvalid x x x x x x xNewProcessing x x x

Clarifications• Object

» BMContentType

• Content» BMEssenceLocatorType - SimpleFileLocatorType,

ListFileLocatorType or FolderLocatorType)

• Light Object» BMContentLightType

Clarifications• Delete vs. Purge

» Delete: flags for deletion.» Purge: removes completely from DB and repository storage.» Purge object purges all related contents.

• Update Object vs. Update Properties» Update object: fully replaces the object.» Update properties: updates the specific passed property only.

• RepositoryProfileType

Repository Service Operations

GenerateUniqueID• Description

» Generates a Unique ID within the scope of a given repository.» Must be obtained before Object or Content Add operations.» Doesn’t have to be generated by this operation and can be

created by an external service as long as ID uniqueness is respected.

• Request» RepositoryProfileType

• Response» ResourceIDType

AddObject• Description

» Uses the generated Unique ID.» Submits a new BMContentType to be added to the system.

• Request» AddObjectRequestType

• Response» BMContentType

AddContent• Description

» Associate a physical essence – BMEssenceLocatorType (SimpleFileLocatorType, ListFileLocatorType or FolderLocatorType) to an existing object – BMContentLightType.

» Uses the generated Unique ID.» BMContentFormatType is optional input.

• Request» AddContentRequestType

• Ack» AddContentOperationAckType

AddContent Notification• Notify Result

» NotifyAddContentResult Operation => AddContentOperationNotificationType

• Notify Fault» NotifyAddContentFault Operation =>

AddContentOperationFaultNotificationType

ReplaceObject• Description

» Updates “metadata” properties of a BMContent object.» Behaves as an object replace not as update properties.» Sets omitted properties in input object to null. » A new version will be created (If versioning supported).

• Request» ReplaceObjectRequestType

• Response» BMContentType

UpdateProperties• Description

» Updates “metadata” of BMContent object (BMContentType).» Applies only on addressable and updatable properties –

RepositoryUpdatablePropertyInfoType (defined by the RCR).» PropertyInfosType parameter includes properties Xpath, values

and RepositoryActionType (update, new, remove).

• Request» UpdatePropertiesRequestType

• Response» BMContentType

UpdateStatus ( Remove !)

• Description» Updates object status only (BMContentStatusType not

BMEssenceLocatorStatusType)» Should the service allow status update from outside the repository

system?» Status automatically updated through the object and essence life

cycle and other operations (DeleteObject, PurgeContent ..)

• Request» UpdateStatusRequestType

• Response» BMContentType

Lock• Description

» Create a lock on a given resource or add a resource to an existing lock token.

» Adds a reference to be locked to the RepositoryLockTokenType .» Lock expiration is per RepositoryLockTokenType and not per

resource.

• Request» LockRequestType

• Response» RepositoryLockTokenType

UnLock• Description

» Removes the lock from multiple references.» If no references passed in the request, will unlock all resources

associated with the RepositoryLockTokenType .

• Request» UnLockRequestType

• Response» RepositoryLockTokenType

ClearLock• Description

» A mechanism to override a lock.» A RepositorySuperLockTokenType must be obtained from the

RCR before calling the operation.

• Request» ClearLockRequestType

• Response» RepositoryLockTokenType

RetrieveOpenLocks• Description

» Gets all open locks within the system.» A RepositorySuperLockTokenType must be obtained from the

RCR before calling the operation.

• Request» RetrieveOpenLocksRequestType

• Response» RepositoryLockTokensType

DeleteContent• Description

» Sets the BMEssenceLocatorStatusType for an active BMEssenceLocatorType (SimpleFileLocatorType, ListFileLocatorType or FolderLocatorType) to ‘deleted’.

» No physical bytes are deleted

• Request» DeleteContentRequestType

• Response» BMContentType

UnDeleteContent• Description

» Sets the BMEssenceLocatorStatusType for the BMEssenceLocatorType (SimpleFileLocatorType, ListFileLocatorType or FolderLocatorType) to an active status (invalid, online, offline ..).

» Repository system is responsible about tracking and resetting the status in the undelete operation implementation.

• Request» UnDeleteContentRequestType

• Response» BMContentType

DeleteObject• Description

» Sets the BMContentStatusType for an active BMContentType to ‘deleted’

» Repository system is responsible about setting related essences - BMEssenceLocatorStatusType for the BMEssenceLocatorType (SimpleFileLocatorType, ListFileLocatorType or FolderLocatorType) to ‘deleted’.

• Request» DeleteObjectRequestType

• Response» BMContentType

UnDeleteObject• Description

» Resets the BMContentStatusType for a deleted BMContentType.» Repository system is responsible about tracking and resetting the

status in the undelete operation implementation.» Repository system is responsible about resetting related essences

- BMEssenceLocatorStatusType for the BMEssenceLocatorType.

• Request» UnDeleteObjectRequestType

• Response» BMContentType

PurgeContent• Description

» Can only be executed on an Essence in a “deleted” state.» Physically removes essence from repository database & storage.

Physical bytes are deleted.» If the essence is the last representation in a

BMContentFormatType, the format entry will be removed from the BMContentType.

• Request» PurgeContentRequestType

• Ack» PurgeObjectOperationAckType

PurgeContent Notification• Notify Result

» NotifyPurgeContentResult Operation => PurgeContentOperationNotificationType

• Notify Fault» NotifyPurgeContentFault Operation =>

PurgeContentOperationFaultNotificationType

PurgeObject• Description

» Can only be executed on BMContentType in a “deleted” state.» Physically removes asset information and all related essences

from repository database & storage.

• Request» PurgeObjectRequestType

• Ack» PurgeObjectOperationAckType

PurgeObject Notification• Notify Result

» NotifyPurgeObjectResult Operation => PurgeObjectOperationNotificationType

• Notify Fault» NotifyPurgeObjectFault Operation =>

PurgeObjectOperationFaultNotificationType

GetBMContent

GetBMContents• Description

» Retrieves multiple BMContentType(s).» RepositoryMetadataFilterOutType: physical, editorial

• Request» GetBMContentsRequestType

• Response» BMContentsType

RetrieveBMEssence• Description

» Copy an essence BMEssenceLocatorType (SimpleFileLocatorType, ListFileLocatorType or FolderLocatorType) to a known supported destination by the repository – RepositoryKnownLocationType

• Request» RetrieveBMEssenceRequestType

• Ack» RetrieveBMEssenceOperationAckType

RetrieveBMEssence Notification• Notify Result

» NotifyRetrieveBMEssenceResult Operation => RetrieveBMEssenceOperationNotificationType

• Notify Fault» NotifyRetrieveBMEssenceFault Operation =>

RetrieveBMEssenceOperationFaultNotificationType

Query• Description

» Consumer is an orchestration engine.» A search parameter must match a property defined in the asset

schema BMContentType or RepositorySchemaExtensionType and defined in the RCR - RepositorySearchablePropertyInfoType.

» The max number of supported search parameter across all query operations is defined in RCR.

• Operation Structure

Query Payload

Query DefinitionQuery Definition

Output DefinitionOutput Definition

ParametersParameters

OperandsOperands

GroupGroup

Amount of DataAmount of Data

Data Structure Data Structure

Query

Query• QueryExpressionType : Simple Query: Title = NY

QueryParameterType• name = Title• value = NY• valueType = string• operand = equal• negated = false <queryExpression >

<resourceID>00000000-0000-0000-0000-000000000001</resourceID> <parameter> <resourceID>00000000-0000-0000-0000-000000000002</resourceID> <name>Title</name> <value>NY</value> <valueType>string</valueType> <Operand>equal</Operand> <negated>false</negated> </parameter></queryExpression>

Query

• Title = NY• OR• Title = Boston• OR

• (DateModified > 2010• AND• DateModified < 2012• AND• Not (DateCreated=2011))

• QueryExpressionType: Compound Query:

Title = NY OR Title = Boston OR (DateModified > 2010 AND DateModified < 2012 AND Not (DateCreated=2011))

Repository Capabilities RegistryRCR

RCR ?• Definition

» Datastore containing a list of properties describing the implementation behavior of the repository interface.

» Provide the ability for a vendor to implement a FIMS repository interface on top of their product based on its capabilities

» Service interoperability is achieved by having a common interface with a preset list of behaviors and settings (Support Lock, Extension Registration, Queryable Properties,… )

» External service can query the RCR to retrieve settings and behaviors

RCR Clarifications

GetRepositoryState• Description

» Retrieve configuration parameters indicating the repository service status: online = active, offline = inactive or not available and failover = instance of the service is not in use, but another instance is active.

• Request» RepositoryProfileType

• Response» RepositoryStateType: online, offline, failover

GetStorageAvailableCapacity• Description

» Retrieves the repository storage components and their available and total space.

• Request» RepositoryProfileType

• Response» StorageComponentsType

GetPerformanceIndicator

GetRepositoryReport• Description

» Retrieves a vendor specific health or general condition report.

• Request» RepositoryProfileType

• Response» RepositoryReportType

GetRepositoryGeneralCapability• Description

» Retrieves general supported functionality.

» RepositoryEventSubType: assetStatusChange, assetObjectCreated, assetContentCreated, assetContentDeleted, assetContentPurged, assetObjectUpdated, assetContentFormatUpdated, assetContentEssenceLocatorUpdated, assetObjectDeleted, repositorySystemException, performanceEvent.

» FormatProfileMetadataProcessingType: mustBeProvided, autoGenerated, autoGeneratedWhenNotProvided

» RepositoryEventListenerEndpointTypes: soap, rest

• Request» RepositoryProfileType

• Response» RepositoryGeneralCapabilityType

GetSchemaExtensions• Description

» Retrieve vendor defined metadata schema extension.

• Request» RepositoryProfileType

• Response» RepositorySchemaExtensionsType

GetUpdatableProperties• Description

» Retrieves the list of properties the repository system allows to be updated.

» RepositoryActionType: update, new, remove.

• Request» RepositoryProfileType

• Response» RepositoryUpdatablePropertyInfosType

GetQueryResultsetDefinitions• Description

» Gets a set of vendor specific defined result sets retuned by the Query operation.

• Request» RepositoryProfileType

• Response» QueryResultsetDefinitionsType

GetSearchableProperties• Description

» Retrieves list of properties exposed by the system to be used in the QueryParameterType.

• Request» RepositoryProfileType

• Response» RepositorySearchablePropertyInfosType

GetRepositoryDefinitions• Description

» Retrieves the internal repository system volumes.

• Request» RepositoryProfileType

• Response» RepositoryDefinitionsType

GetSuperLockToken• Description

» Retrieves a super token to override existing RepositoryLockTokenType to unlock specific resources.

• Request» RepositoryProfileType

• Response» RepositorySuperLockTokenType

GetSupportedStatus• Description

» Retrieves the system supported statuses: BMContentStatusType, BMEssenceLocatorStatusType, and vendor specific BMCustomStatusExtensionType for the content and object.

• Request» RepositoryProfileType

• Response» RepositorySupportedStatusType

Repository Notification Service

RepositoryEvent• Description

» Single event per message.» SOAP and REST» Ordered delivery: events are sent in the same order they are generated» One time delivery: no duplicate events are sent» Push to a single end point with no throttling for V1.0. Pull is out of scope for v1.0.» Client event Listener Endpoint (SOAP or REST) and supported vendor event types defined in

the RCR – RepositoryGeneralCapabilityType.» Customer can disable in RCR supported vendor event types» RepositoryEventType: system, asset» RepositoryEventSubType: assetStatusChange, assetObjectCreated, assetContentCreated,

assetContentDeleted, assetContentPurged, assetObjectUpdated, assetContentFormatUpdated, assetContentEssenceLocatorUpdated, assetObjectDeleted, repositorySystemException, performanceEvent.

» RepositoryEventClassificationType: error, warning, information.» RepositoryOperationType: ingest, archive, delete

RepositoryEvent• Message

– RepositoryEventEnvelopeType• RepositoryEventPayloadType

– SystemRepositoryEventType» RepositorySystemExceptionEventType» RepositorySystemPerformanceEventType

– AssetRepositoryEventType» AssetStatusChangeEventType» AssetObjectCreatedEventType» AssetContentCreatedEventType» AssetContentDeletedEventType» AssetContentPurgedEventType» AssetObjectUpdatedEventType» AssetContentFormatUpdatedEventType» AssetContentEssenceLocatorUpdatedEventType» AssetObjectDeletedEventType

Questions?

Questions?• Do we need BMContentLightType?• BMContentType inheritance from BMContentLightType and move

BMContentLightType to baseMediaService.xsd.• Changing baseMediaService effect on previous fims versions

implementation and rules for fims XSD(s) versioning.• RepositoryProfileType: rename to something more generic and

descriptive, like CredentialsProfile or AuthorizationProfile. Its scope can exceed the Repository and be used in other services. Perhaps, it should even live in the baseMediaService.xsd.

• Do we need UpdateStatus operation?

Questions?• Will the Lock operation on an object - BMContentType locks the

associated contents – BMEssenceLocatorType?• ResourceIDType as a parameter – REST.• Many classes inheriting from bms:Resource – REST.• I understand that if my repository exposes a custom searchable and

updateable property it should be returned in GetSchemaExtensions, GetUpdatableProperties and GetSearchableProperties. Is this correct?

Questions?• What about "FIMS standard" supported properties (the ones defined

in description-V1_0_7.xsd)? Should we also return them in GetSchemaExtensions, GetUpdatableProperties and GetSearchableProperties?

• Could we also expose standard FIMS technical metadata to be used in the searches? (For example, "title = NY and aspect_ratio = 16/9")

Questions?• Locking is an optional feature, the token parameter should be

optional in ReplaceObjectRequest, UpdatePropertiesRequest and DeleteContentRequest - DONE.

• Should we add a lock token to DeleteObjectRequestType?• Naming can be confusing at first sight. The functions and structures

that deal with "BMContent" use "Object" and the ones that deal with "EssenceLocator" use "Content", which could be thought as BMContent. For coherence, the GetBMContent(s) functions should be GetObject(s) and RetrieveBMEssence() should be RetrieveContent()

Questions?• QueryParameteType.valueType and

RepositorySearchablePropertyInfoType.dataType, I am not sure if specifying datatypes by name using a string can create interoperatibility problems. Perhaps we should define and use a Resource "DataType" and provide a list of well-known fims data types, which could be extended by inheritance.

• Do we plan to have a way of extending QueryParameterOperand with custom operands? It would be desirable. enumeration vs. object (resource) in xsd.

Questions?• Purge Operations – Rename• Delete operations - Rename• AddObject – Rename• AddContent – Rename (RegisterContent)• Responses that contain BMContentType can we have an option in

the content to return BMContentLightType instead. The use of container object with “choice” would this be a good represntation.

• RCR to define max. lock expiration duration (seconds, minutes, hours?) allowed by the system, null means no restriction.

Questions?• RetrieveOpenLocks operation – Rename• RCR to define if dirty read allowed• When a parent is locked will this lock the children?• Do we need GetBMContentor GetBMContents should be enough?