Optimize, Store and Protect SharePoint 2010 Server

.

…Best Practices James Baldwin Strategic Solutions Group EMC Corporation Session W21

• SharePoint architecture and performance factor

• SharePoint and FAST Search virtualization

• Storage best practices – sizing and performance

• Remote BLOB Storage

• Storage options and features

• SharePoint farm Protection

Agenda

Apologises in advance…..for the word… FAST FAST…as in Search… FAST…as in VP… FAST…as in “let’s get out of here!”

Proven Solutions approach

Capture & Define

Test and Validate Document Publish

Singapore Shanghai, China

Cork, Ireland

Durham, NC Santa Clara, CA

Vienna, Austria

1 2 3 4

Server

‒ CPU ‒ Memory ‒ HBA/CNA ‒ NIC

BLOB Storage

(Optional)

SharePoint performance - The user experience

Storage

− Content/Metadata − Search − System Network Client

Document Request Web Front End

SQL Server

BLOB Retrieval/Creation

Domain Controller

Technet: http://technet.microsoft.com/en-us/library/cc287790(office.12).aspx

Operations Acceptable user response time

• Browsing to the home page • Browsing to a document library <3 seconds

• Creating a subsite Creating a list • Uploading a document to a document library <5 seconds

• Backing up a site • Creating a site collection <7 seconds

Authentication

SharePoint Farm Topologies

Web

Application

Database

Small Medium Large

H/W Eval Prod

RAM 4GB 8GB

CPU 4 Cores

H/W Small Medium

RAM 8 GB 16 GB

CPU 4 Cores 8 Cores

Web/Query

All DBs

App

Web

Query/Crawl

Search DBs SharePoint DBs

App

Web Servers Groups

Query Crawl

Search DBs SharePoint DBs Content DBs

User requests Crawling/Admin

App Servers Groups

Central Admin /Office/Other

H/W Eval Prod

RAM 4GB 8GB

CPU 4 Cores

http://technet.microsoft.com/en-us/library/cc262485.aspx http://technet.microsoft.com/en-us/library/cc298801.aspx

Scale out approach

Key Benefits – Virtualizing SharePoint Consolidation

• Achieve 2-10x consolidation ratio, especially for larger deployments Performance

• Improved front end performance with more, smaller WFEs rather than few large WFEs Availability

• VM based protection for SharePoint provides homogeneous high availability (WSFC, VMware HA) Business Continuity

• Simplified DR management (Geo-Clustering, vCenter Site Recovery Manager) Maintenance

• Live migration of virtual machines (Hyper-V Live Migration, VMware vMotion) Load Balancing

• Maximized overall performance with maximum HW utilization (SCVMM PRO , VMware DRS)

Virtualizing Server Roles In SharePoint

Application (Excel, Doc Conv, etc)

Index/Crawl

SQL Server

Web Front End / Query

CPU – Application dependent Scaling out is more efficient

CPU – User concurrency, Search requests Scaling out is more efficient Network – segment virtual NICs and virtual Switches

Redundant (Non redundant in MOSS 2007) CPU – Crawling, indexing (depends on content type/size) Scale out (Up only with MOSS 2007) Memory intensive

CPU – Document updates, Search, Backup VMFS/RDM, VHD/Pass-Through Scale up/out (Hyper-V ≤ 4 vCPU, vSphere ≤32 vCPU) Failover Clustering, Mirroring, VMware HA

Server Roles/Priority What To Consider

4th

3rd

2nd

1st

Understanding your existing workload (WFE to SQL) and requirements is better than any general best practice!!!

A Day in the life of SharePoint… SQL Server CPU

The majority of load comes from systematic operations…

Sample anonymous customer data SharePoint Timer Job Reference - http://technet.microsoft.com/en-us/library/cc678870.aspx

Virtualized SharePoint - Reference Architectures Enterprise-Class SharePoint (Virtual) and FAST Search (Physical)

Role Qty VM specs

Hyper-V Servers 3 Nodes 4-socket quad-core (16 cores), 128 GB RAM

SQL Server 2008 2 4 vCPU, 32 GB RAM

Web front ends 4 4 vCPU, 6 GB RAM Application servers (Incl. Central Admin) 2 2 vCPU, 4 GB RAM

Query SSA 1 4 vCPU, 8 GB RAM

Content SSA 1 4 vCPU, 8 GB RAM

Fast Search Physical Servers 5 2-socket hex-core (12 cores), 48 GB RAM

• 9 TB User content – (4TB SharePoint | 5TB Fileshare)

• Virtualized SharePoint 2010 Farm

• Virtualized Query and Content SSAs

• 5 Physical FAST Search Servers (12-core ea.)

• 2 Document Processors/Content Distributors

• 2 Index Servers (1 Partition, 2 Search Roles)

• 1 FAST Admin inc. Document Processor

• <1 Sec search latency with 22,000 Heavy Users @ 10% Concurrency

Whitepaper due for release in October timeframe

Virtualized SharePoint - Reference Architectures Enterprise-Class SharePoint and FAST Search (All Virtual)

Role Qty VM specs Hyper-V Servers (SharePoint Farm) 3 Nodes 4-socket quad-core (16 cores), 128 GB RAM

Hyper-V Servers (FAST Search Farm) 2 Nodes 2-socket hex-core (12 cores), 48 GB RAM

SQL Server 2008 2 4 vCPU, 32 GB RAM

Web front ends 4 4 vCPU, 6 GB RAM Application servers (Incl. Central Admin) 2 2 vCPU, 4 GB RAM

Query SSA 1 4 vCPU, 8 GB RAM

Content SSA 1 4 vCPU, 8 GB RAM Whitepaper due for release in October timeframe

• 9 TB User content – (4TB SharePoint | 5TB Fileshare)

• Virtualized SharePoint 2010 Farm

• Virtualized FAST Search !

• 2 Document Processors/Content Distributors

• 2 Index Servers (1 Partition, 2 Search Roles)

• 1 FAST Admin inc. Document Processor

• <1 Second search latency with 22,000 Heavy Users @ 10% Concurrency

SQL Server

SharePoint 2010 Storage Elements

Configuration Databases

Central Admin

tempdb, master ,model , msdb

Usage & Health Data Collection - Logging

Search – Admin, Property, Crawl

Web Analytics – Staging, Reporting

User Profiles – Profile, Synchronization, Social Tagging

Managed Metadata- Term Store

State

Business Data Connectivity

Secure Store

Search Index

Index Partition/s Query Component/s

System Volumes Boot/OS/VMs Web Parts & Features SharePoint Binaries

Service Application Data

SA Volumes

Content Databases

BLOB Store

Content DBs 8%

BLOBs 34%

System DBs 6%

SA DBs 5%

Search Index 15%

SA Data 12%

System Volumes

20%

% Total Capacity - Sample

Shared Service Applications System and Configuration Content

• SharePoint relies mainly on SQL but there’s important data outside SQL

• Typical storage usage is not I/O heavy • Search services in SharePoint generate most of the I/O

load • Content – Plan for capacity • Search and System – Plan for performance

1TB content farm – How much storage do I need?

2150 2910 3310 3290 4470

010002000300040005000

2 WFEs/noSearch

2 WFEs/1xIndex&Query

2 WFEs/2xIndex&Query

2 WFEs/1xIndex/2x Query

(Striped)

2 WFEs/2xIndex/4xQuery (4

Partitions,Mirrored(1-A,D,

etc)

Gigabytes of Storage

A Day in the life of SharePoint… SQL Server Storage I/O

Sample anonymous customer data

Plan for user load peaks, not systematic peaks…

SharePoint 2010 Storage I/O Characteristics

I/O Type Content Databases tempdb

Search Databases

(property & crawl stores)

Index Component

Query component

Read (KB) 16 8 8 32 32

Write (KB) 16 32 16 64 64

* Based on average workloads in a collaboration farm (Browse/Search/Modify – 80/10/10)

SQL Server In-box Search Servers

R:W Ratio 95:5 50:50 60:40 90:10 70:30

SQLIO/SQLIOSim are I/O stress tools - should not be considered as “performance requirements” !

Sha

reP

oint

FA

ST

FAST Search Servers Server Read (KB) Write (KB) R:W Ratio

FAST Index Server (Primary) 294 611 3:1

FAST Index Server (Backup) 31 710 1:30

FAST Servers (Document Processors ,etc) 13 26 1:66

SharePoint 2010 Storage performance Requirements/estimates for Search

Based on a Microsoft case study – Mileage may vary!!!

Database Role Microsoft’s Estimate IOPS Typical averages observed

Crawl Database 3,500 – 7,000 (R70:W30) 2,000

Property Database 2,000 (R30:W70) 600

Query Component 2,000 per Active/Failover pair (Load/Write/Merge)

450

Crawl Component 300-400 IOPS 80-100 http://technet.microsoft.com/en-us/library/cc298801.aspx

Crawler Query SQL Server

Crawl Component Query Component Crawl DB

Property DB

Search Admin DB

FAST Search Server 2010 for SharePoint Storage Requirements/estimates

Based on a Microsoft case study – Mileage may vary!!!

FAST server role Microsoft’s Estimate LUN size Typical averages observed

Web Analyzer Database (2GB per 1M items) SharePoint or Intranet (6GB per 1M items) Public Web content (25GB per 1M items)

1.2 GB

Index server (Primary)

X2.5 the size of a single index file set +50 GB synchronization data

910 GB (2 TB recommended to hold temp indexing files)

Index server (Secondary)

X2.5 the size of a single index file set +50 GB synchronization data

899 GB (2 TB recommended to hold temp indexing files)

http://technet.microsoft.com/en-us/library/ff381239.aspx

FAST Admin

S:\FASTSearch LUN size: 80GB

S:\FASTSearch LUN size: 80GB

S:\FASTSearch LUN size: 80GB

S:\FASTSearch LUN size: 2000GB

Document Processor 01

Document Processor 02

Index 01 (Primary)

Index 02 (Backup)

S:\FASTSearch LUN size: 2000GB

SharePoint Reference Architecture Storage Layout

Cost driven configuration ‒ 13,000 Heavy SharePoint users

‒ 1 TB User content with RBS FILESTREAM Externalization

‒ SAN based replication (RecoverPoint)

Storage Configuration ‒ RAID5 –VM Boot Luns, Content Databases

‒ RAID10 – Search Databases, tempdb

‒ Array SSD Cache to compensate for disk latencies

http://www.emc.com/collateral/software/white-papers/h8139-protection-virtualized-sharepoint-wp.pdf

Storage Role % of Corpus Size Typical sizes GB Recommended RAID

Virtual Machine Boot Volumes - 80 R-5

Application Volumes - 50 – 300+ R-5

Content Databases

Data File Volume - 100 – 200 R-5

Log File Volume 10% of Data

File 10 – 20 R-5 / R-10

tempdb Data File Volume 10% 100 – 300 R-10

Log File Volume 2% 10 – 100 R-10

SharePoint Storage Sizing Volume Sizing

Storage and SQL Server capacity planning and configuration http://technet.microsoft.com/en-us/library/cc298801.aspx Hardware and software requirements http://technet.microsoft.com/en-us/library/cc262485.aspx

Storage Role % of Corpus Size

Sample size (1TB Content)

Recommend RAID

SQL Search Databases

Crawl Store DB 0.046 × (sum of content databases) 46GB R-10

Property Store DB 0.015 × (sum of content databases) 15GB R-10

Search Server Volumes

Index Component 10% 100GB R5 / R10

Query Component 10 – 35% * 2.85 150 – 1TB R-5 / R-10

SharePoint Configuration

Databases

- Data & Log Volume

SharePoint_Config - 5GB R-5

SP_AdminContent - 15GB R-5

Usage and Health Data Collection Database - (based on % monitoring) 50 - 500GB(?) R-5

SharePoint Storage Sizing Volume Sizing

SharePoint 2010 Database sizing characteristics http://technet.microsoft.com/en-us/library/cc298801.aspx http://technet.microsoft.com/en-us/library/cc678868(office.14).aspx

Storage Role Microsoft Data set Typical result observed (5.2 million items)

Recommend RAID

FAST Query SSA Databases

FASTQuerySSA_DB_guid N/A 460 MB R-10

FASTQuerySSA_PropertyStore_guid <0.4GB 158 MB R-10

FASTQuerySSA_CrawlStoreDB_guid 3.3 GB per million items 15 MB R-10

FAST Query SSA Databases

FASTContent_DB_guid N/A 99 MB R-10

FASTContent_PropertyStoreDB_guid <0.4GB 158 MB R-10

FASTContent_CrawlStoreDB_guid 3.3 GB per million items 38621 MB R-10

FAST Admin FASTAdminConfigDB <0.1GB 3.00 MB R-10

FAST Search server 2010 for SharePoint Storage Sizing Volume Sizing (estimates)

SharePoint 2010 Database sizing characteristics http://technet.microsoft.com/en-us/library/cc298801.aspx http://technet.microsoft.com/en-us/library/cc678868(office.14).aspx

SharePoint Storage Best Practices SQL Server – Storage Allocation

Use 64KB unit allocation size when formatting DB volumes

Physical volumes (RDM/Pass-through) for larger than 2TB when virtualized

Plan Database file sizes accordingly – Don’t rely on autogrowth

File growth can cause locking, set files size and autogrowth increments appropriately

– Using RBS would keep the SQL Database files small – Watch the SP Configuration database log file growth!

When using Thin/Virtual provisioning – Use the “Quick Format” option – Enable Instant file initialization*

Enhances the speed for data file creations, restores, data file growth Assign SQL service account to “Perform Volume Maintenance Tasks” permission

– Log files are fully allocated and zeroed upon creation or growths

http://msdn.microsoft.com/en-us/library/ms175935.aspx

* Hard to interpret due to virtualization of storage. Consider in combination with response times

SharePoint Storage Best Practices SQL Server - Performance

Data file Latency

Read/Write Operations

Log file Latency

Write Operations Recommendation

< 10 ms < 5 ms Very Good

< 20 ms 5 – 10 ms Acceptable

> 20 ms > 15 ms Investigate and Improve

Important Perfmon I/O counters Real Meaning! Average Disk/sec Read or Write Disk Latency Current Disk Queue Length* Outstanding I/Os Disk Reads/Writes per Second IOPS Average Disk Bytes/Read & Write I/O Size (KB)

Plan for optimal disk response times

• Thin Provisioned Pools ‒ Reduces initial storage requirements/costs for content databases ‒ Enables SharePoint administrators to grow volumes non-distruptively

• FAST VP (Fully Automated Storage Tiering) ‒ Helps to handle unanticipated performance requirements

– EMC Cluster Enabler (CE) – Automated multi-site Disaster Recovery (failover) for Hyper-V/Physical/Hybrid SharePoint farms – Can use any EMC replication technology (RecoverPoint/SRDF/MirrorView)

EMC Storage Technologies for SharePoint 2010

SharePoint Component Recommended Notes Search Index No Highly changing, throw-away data

Search Query Yes Highly-read data with small burst write changes

tempdb Yes Same blocks are re-used on disk and performance of TEMPDB directly affects SharePoint performance request

Content databases Yes/Maybe C-Databases which are defragmented regularly or complex Databases like NewsGator

BLOB Store No BLOB Store has low IOPs requirements

• LUN Compression/De-duplication/Thin Provisioning ‒ Reduces initial storage requirements/costs for content databases ‒ Enables SharePoint administrators to grow volumes non-disruptively

• FAST Cache

‒ Helps to handle unanticipated performance requirements

EMC Storage Technologies for SharePoint 2010

SharePoint Component

Performance Improvement

Notes

Search Index High Search Index re-uses the same blocks on disk as working space to process listitems during indexing, then that storage is quiet between crawls

Search Query Medium Query improves due to random large block reads/writes being serviced by FASTCache, but Query storage is large and costly in FASTCache

tempdb Medium TempDB can benefit from FASTCache as the way SQL Server uses TempDB (database page reuse) is ideally suited to FASTCache algorithms

Content databases Medium Low IOPs requirements does not require FastCache, DB Index operations see an improvement

BLOB Store Low Low IOPs requirements, large-block I/O

BLOB Externalization (RBS/EBS)

BLOB = Binary Large Object

BLOB externalization options ‒ EBS using 3rd party provider- Mainly for MOSS 2007 but still supported with SP2010

– RBS through SQL RBS FILESTREAM provider (Local/Remote)

– RBS using feature rich 3rd party provider

Pros – Lower TCO as BLOBs can reside on lower cost storage tier (SATA/NL-SAS) – More flexible and efficient storage configurations (CIFS/NFS/REST) – Performance improvement

– Large objects retrieval (>1Mb) – SharePoint Indexing operations – Various SQL operations (Index defragmentation, statistics operations, consistency checks)

• Cons – Backup and restore complexity – 2GB file size limit is not lifted by using RBS – Not supported with System Center DPM and/or Database Mirroring

http://technet.microsoft.com/en-us/library/ff628583.aspx http://technet.microsoft.com/en-us/library/cc262787.aspx

Remote BLOB Storage (RBS)

EMC has several RBS solutions – In partnership with several vendors for BLOB externalization (EMC Select-Metalogix StoragePoint) – EMC SourceOne also leverages RBS – Block/File/Object - VMAX, VNX, DataDomain, Isilon – Cloud Optimized Storage - Atmos/Atmos VE – Archival Storage - Centera

Feature SQL Server RBS FILESTREAM 3rd Party RBS provider

Externalization using SMB/CIFS/NFS protocols No (Local volumes only) Yes

User Interface Powershell Central Admin

BLOBs tiering based on Age/Size/Type/Hierarchy policies No Yes

Orphaned BLOB Garbage Collection Basic (RBS Maintainer) Policy-based

BLOB store volumes automatically included in VSS Backup Yes No

Considerations for Remote BLOB Storage

Replication, Backup and Recovery – Native/Item level backup (stsadm based) would include BLOBs (“deep copy”) – SQL based backup would only protect the content database – To maintain consistency:

Backup – First Content Databases then BLOB Store Restore – First BLOB Store then Content Databases

– For faster recovery, consider larger intervals of garbage collection jobs (Keeps previous BLOB versions) – For DR purposes always tie RBS volumes with SQL Server volumes

Block, File or Object storage?

– Performance: Block would be faster but RBS has typically low I/O – Storage Efficiency

– Block-level LUN Compression – increases storage efficiency, may affect backup performance – Filesystem-Deduplication – better performance and increased dedup rate

Farm Profile Total content ~1TB 4.36M documents Avg. File Size - 220K

Storage Profile EMC VNX5300 15K SAS (System and DBs) /7.2K NL-SAS (BLOB Store) CIFS Share for RBS

Results Max user capacity – 8,630 (10%) BLOBs consumed 92% of content databases Full crawl duration – 34 hours 8-10% user throughput (RPS) improvement using RBS 20% capacity saved with RBS FS de-duplication

Reference Architecture - BLOB Externalization File Storage (CIFS)

Farm Profile Total content 3.1TB 1.2M documents File Sizes – 200KB-5MB 16 Site Collections/40,000 Sites

Storage Profile EMC Symmetrix VMAX FAST VP controlled FC and SATA tiers

Results BLOBs consumed 94% of content databases 3.1 TB Externalized in 35 Hours

Reference Architecture - BLOB Externalization Block Storage

Storage Configuration Concurrency

Users (Heavy) (80%-Browse/ 10% -Search/ 10%-Modify)

Response Time

Baseline 10%

25,800 < 3 sec

Externalized 31,800

Protecting SharePoint to Enterprise Scales Replication Management for Microsoft SharePoint 2010

Leverage SAN based replication for Rapid full farm protection and item-level recovery….

– Hardware VSS-based coordinated SharePoint replication, Enabling farm-wide consistency

– Negligible impact to farm performance even during the first synchronization

– Utilizing EMC storage replication (Snaps/Clones/Bookmarks)

– Simple, intuitive SharePoint discovery and application set configuration (Configure protection in <8 minutes)

– Works with SQL RBS FILESTREAM

– Full farm restore include search index consistency on recovery

– Item level recovery enabled by Kroll Ontrack Powercontrols

SharePoint Replication Management Reference Architecture

Enabled by EMC Replication Manager, Kroll Ontrack PowerControls Hybrid farm (Physical/Virtual) 1.5 TB of content (6,818,250 files)

15 SharePoint content DBs

Both SnapView Snaps and Clones used

SharePoint action EMC SnapView STSADM

Full farm backup (2.5 TB) Clone: 3hr 11min Snapshot: 9min

39hr 30min (14.8 MB/s)

Daily incremental SharePoint backup (~1% daily change rate)

Clone: 11min Not tested in this environment

Content database recovery (online) 7min 3hr 12min (12 MB/s)

Item-level recovery 10min Not tested in this environment (requires recovery farm)

SAN

Continuous Replication

SQL

DB Mirroring Log Shipping

Point in Time

Backup PiT Replication

Business Continuity for SharePoint 2010: Options

Stretched Farm (Partial Replication)

Mirror Farm (Partial Replication)

Virtualized Farm (Complete Replication)

Consistency Group (RP/SRDF/MV) LUNs Grouping VMware

SRM Hyper-V

Cluster Enabler

Web Front Ends Boot + Query (optional) Protection Group Cluster Group

Query Servers Boot + Query Component Protection Group Cluster Group

Index Servers Boot + Index Component Protection Group Cluster Group

Application Servers Boot + Application Volumes Protection Group Cluster Group

SQL Server(s)

Boot + Pagefile (optional)

Protection Group Cluster Group

SQL System Databases

Configuration Databases

Search Databases

Content Databases

RBS BLOB Store

Automated DR

Designing DR consistency for SharePoint

Search consistency Content consistency

Production Site Outage

Protecting SharePoint Business Continuity – vCenter SRM with RecoverPoint CRR

Proven Solution Test Results 13,080 heavy users @ 10% concurrency -

Sustained Performance 30% reduced cost of BLOB Storage with EMC

LUN Compression <16 minutes to perform full end-to-end

failover

Test type Shutting down production VMs Preparing storage Restarting DR VMs Total time

Executing recovery plan for a fully operational farm (under load) 00:33 11:45 3:15 15:33

Failover Test Results

Key Takeaways • SharePoint is more than just SQL…

– Leverage EMC Proven solutions and Best Practices for SharePoint and FAST Search storage, networking and compute design – FAST, FAST Cache, VP improve efficiency & performance but require proper planning – Use RBS to improve scalability and TCO

• Full SharePoint and FAST Search farms virtualization has great advantages over physical/hybrid configurations – Horizontal scaling is more efficient – The best FULL farm protection when Integrated with EMC replication. – Simplifies, accelerates and automates SharePoint DR! (vCenter SRM, Hyper-V+EMC Cluster Enabler)

• EMC’s SharePoint VSS based replication can significantly accelerate replication and recovery of SharePoint – A must for large deployments (TBs) – Protects content and index – Fast and simple Item level recovery while integrating with EMC partners (e.g. Kroll)

• Block-level replication for Automated Disaster Recovery answers the difficult challenges in providing site protection

– Essential for critical SharePoint farms, where uptime and site resiliency is critical – Automates failover for the entire farm/s

Additional References EMC Solutions for SharePoint Portal

– http://www.emc.com/sharepoint

Technical Whitepapers

• SharePoint in a multi-tenancy virtualized environment (Vblock) – http://www.emc.com/collateral/software/white-papers/h8798-virtualizing-ms-apps-multi-tenancy-vblock1-wp.pdf

• Virtualized SharePoint 2010 (ESX 4.0, CX4-240)

– http://www.emc.com/collateral/software/white-papers/h8024-virtual-sharepoint-clariion-vsphere-wp.pdf

• Continuous protection for virtual SharePoint 2010 (ESX 4.1, RBS, RP, RM, CX4-120) – http://www.emc.com/collateral/software/white-papers/h8139-protection-virtualized-sharepoint-wp.pdf

• SharePoint 2010 BLOB externalization (Hyper-V, Metalogix StoragePoint, VNX-5300) – http://www.emc.com/collateral/hardware/technical-documentation/h8185-sharepoint-vnx-metalogix-psg.pdf

• SharePoint 2007 Business Continuity (Hyper-V, Cluster Enabler, CX4-120) – http://www.emc.com/collateral/solutions/reference-architecture/h7041-bc-sharepoint-clariion-recoverpoint-hyperv-ref-arc.pdf

SharePoint Blogs - EMC – http://sharepointintheprivatecloud.wordpress.com – http://sustainablesharepoint.com

Thank you! Q&A?

FAST Search Server… Physical to Virtual Test Results

Comparison between Physical and Virtual FAST Search environments…

Description Physical Virtual

# Physical Servers 5 2

Total CPU number 60 20

# Document Processors 52 24

Search Response Time <1sec <1sec

SharePoint Content full crawl rate (Avg. Doc Size 1.76MB)

1,513 Items/min

123.1 GB/Hr

770 Items/min

81.7 GB/Hr

File Share Content full crawl rate (Avg. Doc Size 1.56MB)

1,667 Items/min

152.5 GB/Hr

978 Items/min

82.98 GB/Hr