data archiving: a key to performance and data governance
TRANSCRIPT
Data Archiving: A Key to Performance & Data Governance
Jonathan BruceDirector of Product [email protected]@jonbruce
Jessica HarmanSupv, R&IMPhillips 66@zz_jess
Jennifer McClainDirector Product ManagementCloudlock@jenniferDigital
The Need for Data Archive is Accelerating and ShiftingFive Key Industry Trends
1. Manage Retention and Compliance2. Protect & Optimize App Performance3. Retain Managed & Secure Access4. Position for future Data Analytics5. Need to reduce application costs
Compliance is a Primary Driver for Data Archiving
Hardening Standards mean ComplianceNeed for a single focal point to document and enforce data-retention policies across all customer data
E-Discovery ReadinessAgreed-upon process require data & metadata retention to facilitate Identification, Preservation, Collection, Processing, Review and Production
Explosion of Data Drives Storage Costs High, Increasing the Need for Cloud Archive
Reduce Costs Across App PortfolioArchiving production data and retiring legacy reduces storage costs across the application portfolio
Scheduled ArchivingArchiving infrequently accessed records to a highly accessible location has net effect of optimizing app performance
Increase R&D VelocityOptimized data footprint reduces risks, thrash and accelerate output for R&D
Consistent & Secure Access Across Data Life Cycle
Accessibility maintained across archive life-cycle boundaryConsistent security model for user access to archived data via familiar API and UI experiences.
Maintain Role and User based controlRole based security to manage who can and cannot report on archive data
Big Data Set to Become the Primary Driver for Data Archive
“By 2016, 75% of structured data archiving will incorporate support for Big Data analytics and by 2017, archiving support of Big Data Analytics will surpass archiving for compliance as the primary use case for structured data archiving” - Gartner
Source: “Magic Quadrant Structured Data Archive Application & Retirement” - Gartner
Key Challenges in Data Archiving
Legacy TechnologyHard to balance speed, design and functionality
Diverse Developer Skill SetCan’t find, train, or keep them
Growing Enterprise RequirementsGovernance, control and security
Current Data Archive Methodologies are PainfulComplex mix of integration & storage, governance, hidden costs
Introducing: Data Archive
Data Archive
Policy & Programmatic Data ArchiveTools, repositories and patterns to retain recordsEstablish Data Retention PoliciesRetain all data across your life-cycle
Access Retained Data at ScaleNormalized on big data back-end for performance
Comply with Industry RegulationsSecure data archive with the highest trust standards
Near-line storage for Salesforce
Pol Policy driven storage service for data retention and compliance
BigObjects let you store manage billions of records nativelyBigObject
Data Archive Storage and Services
Data Archive
Object query language Resilient async SOQL SOQL Async Query
API
BigObjects means 100s of billions of records on force.com
Data persistence optimized for high-volume data Geared for 1, 10 100s of billions of records Immutable data – archive, events, external data, historical data
Familiar, object-based development model Simple data types – string, number, date, JSON Exposed in SOAP, REST, Bulk, and Metadata APIs New contracts for synchronous and asynchronous query patterns
High throughput Ingress & EgressNew Bulk API Implementation geared for 1billion record/day ingest
High-volume storage for Saleforce.com - reliable, highly-available & secure
BigObject
Salesforce Data Archiving Using Different Methods
Programmatic Package Assisted Policy Criteria
High Effort / Flexible Low Effort / Targeted
1. Define source SObject records
2. Define target BigObject(s)
3. Define SObject to BigObject field mappings
4. Use AsyncQuery or Pipelines to copy records from SObject to BigObject storage.
5. Conceive and orchestrate the delete process driven by the parent IDs via APIs
Subtle Highlight Color
Programmatic Approach (Pilot)Follow 5 Steps
BigObject
SObject1
2
3 5
Programmatic Data Archive
4
Archiving Mapping Scenarios
Significant flexibility with mappings between SOBject -> BigObject● 1:1● Many:1● 1:Many
Important Best Practices● Always store parent child-Id - important for delete● Platform Encryption considerations● Custom VisualForce / Lightning for UI● Manage production-archive field relationship lifecycle
Programmatic Approach (Pilot)Key Considerations
BigObject
SObject1
2
3 4
Programmatic Data Archive
User Responsibilities
1. Define source SObject records
2. Define target BigObject(s)
3. Define mapping (if necessary)
4. Customize, enable and deploy the policy
Platform Responsibilities● Manage field definition production-archive life-cycle (limited)● Fully manage initial and on-going Delete phase● Platform Encryption enforcement
Declarative Approach (Next Year)4 Steps
BigObject
SObject1
2
3
Policy Data Archive
4
Potential Platform Policies
1. Improved Storage management- Last Modified Criteria- Least Recently Accessed- ....
2. Data life-cycle for Compliance - Age based archive- Field-value based archive
Custom Policies● Criteria & Rules based policies
Declarative Approach (Next Year)Potential Policies
BigObject
SObject1
2
3
Policy Data Archive
4
“Data Lifecycle is defined and dictated by the business ”
StateFarm Insurance
Kip Davis, State Farm
17,700 agents, 343 claims, StateFarm harness their customer interactions on force.comGenerates massive data volumes with on Salesforce force.com platform
Data Archive is pivotal for operational responsiveness and compliance
Data Archive
How to Engage
BigObject
AsyncQuery
Data Pipelines
Engage in All PilotsEach of these products have an active pilot, apply with your AE today to participate.
Pilot participation is free!
Make your Voice HeardEngage and discuss on the Dreamforce App, Communities or Twitter - find me at @jonbruce
Build Out Your Use CasesLeverage our Implementation Want your voice heard?
USERS &
APPS
DATA
INFRASTRUCTURE
● Behavioral Anomaly● 3rd Party Apps granted access to data
● Cloud Data Protection & Governance
● Regulatory Compliance
● Audit Logs● Security APIs
CloudLock Enables Customers to Securely Embrace the Cloud
IT Security
App Developer
Homegrown Apps
ISV Cloud Apps
Enterprise
SaaS
force.com
PaaS and IaaS
Content Classification
User Behavior Analytics
. . .Encryption
ManagementApps
Firewall
force.com
IDaaS
Configuration Security
CloudLock Security Fabric 2.0: Cybersecurity-as-a-Service
INFRASTRUCTURE
CloudLock and Salesforce Shield
CloudLock Overview
Top Use Cases• Account Compromise• Data Breach• Cloud Malware• Regulatory Compliance• Security Ops & Auditing
CloudLock: Healthcare and Financial Policy Packs
CloudLock: New Data Retention & Archival Policies
• Automated, Policy-Driven Response Actions to selectively archive records based on policy criteria, such as content or object type
Data Compliance – Value & Engagement
Jessica HarmanRecords & Information Management, Supervisor [email protected]@zz_jess
How do you assess the business?
What do I really do?
I’d rather….
Data ComplianceIs it a necessary component?
FIPS CFR DOT
PCI PIIHIPPAHIPAA
Retention Schedule
The smoking gun
Retention Policy Disposition Cycle
Poor actions/strategy
Lack of policy Disposition at will Business process differ for
system/content
Good process but lacks execution
Policy statement
Standards and Procedures
Disposition is set for the RM system only
How do you see your future? In good faith but reality shows:
Create a standard for usage guidelines
Designate classifications of data and ‘where’ it should be located
Build an “Information Map” that shows inventory, content record types, ownership model, permission structure and disposition time frames
Define the company policy for Records & Information Management
Publish a retention schedule
Create a standard of compliance for the company policy
Generate an Accountability Network
R&IM Program Basics
Design & Scope
Training
Assessments
Communications
Engaging the Business Action Plan
User Adoption
Decrease in User Errors
Reduction in Training Costs
Productivity
Engaging the Business Benefits
Is your data authentic, reliable, and usable? What would your search results look like?
Thank you