ibm storage: speichertechnologien gestern, heute und morgen
TRANSCRIPT
© 2010 IBM Corporation
IBM Deduplication
Tom Chandler – IBM ProtecTIER Solution Architect - EMEAStorage Technology 2010 - Juni 2010
© 2010 IBM Corporation
Storage Technology 2010
2
Deduplication is key to use disk more cost effectively!
© 2010 IBM Corporation
Storage Technology 2010
3
IBM Deduplication IBM Delivers Deduplication Solutions
LAN
SAN
LAN-free Backup Client
Backup Server
Desktop Clients
DiskTape Disk
Buffer
TSM 6.1 Dedup
SVCXIVDS8000DS 3/4/5*
ProtecTIERTS7650
ProductiveServer
Primary Storage Backup/Recovery StorageN Series
WANTSM 6.2 Client Dedup FastBack Dedup
Storage Manager 6
Storage Manager 6
VTL
DedupDedup DedupDedup
DedupDedup
DedupDedup
DedupDedup
© 2010 IBM Corporation
Storage Technology 2010
4
IBM Data De-Duplication Options
IBM N series– VMWare environments, user file spaces
IBM Tivoli Storage Manager– Data reduction with ‚Incremental Forever‘– Data de-duplication for disk pools– Future: client de-duplication!
IBM TS7600 family– Virtual tape library with ProtecTIER software– Hyperfactor industry-leading data de-duplication– solution for
• Open systems (TS7650) • System z (TS7680)
Storage Hierarchy
Disk buffer
Represented Capacity Physical
Capacity
© 2010 IBM Corporation
Storage Technology 2010
5
Backup with TSM Disk Buffer
Data reduction before / during backup– Incremental forever
TSM disk pools can be de-duplicated– More space for critical data– Smaller disk pool
Disk
LAN
SAN
LAN-free Client
BackupServer
Client
Disk Tape
Disk
Disk-bufferRepresented
CapacityPhysicalCapacity
De-duplicated disk buffer
© 2010 IBM Corporation
Storage Technology 2010
6
IBM N series System
Flexvol
VMDK VMDK VMDK
N series de-duplication provides the same benefits as VMware’s shared memory functionality
Storage consumption with N series data de-duplication
VI3 Server
Datastore A
N seriesde-duplication
removes redundant VMware
data
•Reduce OS & applications to a
single copy
•VMs only consume storage for their
unique data
•Reduce Storage Costs with
Virtualization
VMDK VMDK VMDK VMDK
VMDKDuplicate data removed
© 2010 IBM Corporation
Storage Technology 2010
7
Disk
LAN
SAN
LAN-free Client
BackupServer
LAN Client
DiskTape
Disk buffer
Disk-buffer
Backup Solution with IBM TS7600
Tape Virtualization
Represented Capacity
PhysicalCapacity
Possible reduction of required disk capacity– 1:5 … 1:25 , Strong dependency on
• Backup process used• Type of data
Bandwidth vs cost– Compare with using multiple physical drives
© 2010 IBM Corporation
Storage Technology 2010
8
Tivoli Storage Manager may be best when
TS7650G ProtecTIER VTL may be best whenGreater than 6 TB of data
backed up nightly
Data backed up nightly 6 TB or less
Spare resources are available to dedicate to
dedup processing
TSM manages majority of data on
tape
Heterogeneous tape management
Many backup servers, both TSM and other
Large disk caching requirements for
secondary storage
Larger or dedicated storage management staff
Moderately sized TSM server installation
Prefer an integrated software solution with no specific hardware dependencies
Larger or dedicated storage management staff
© 2010 IBM Corporation
Storage Technology 2010
9
1. Post Processing & Inline
2. Hash –Based Approach
3. IBM Hyperfactor
4. Case Studies
De-duplication Topologies
© 2010 IBM Corporation
Storage Technology 2010
10
Two Basic Implementations with Deduplication
#1 Inline
As data is received by the target device it is– deduplicated in real time– not temporarily stored on disk
Data written to the disk storage is de-duplicated
#2 Post Processing
As data is received by the target device it is
– temporarily stored on disk storage
Data is subsequently read back in to be processed by a de-duplication engine
© 2010 IBM Corporation
Storage Technology 2010
11
The Advantages of (Async Dedupe) Postprocess are:
No concerns about slowing down incoming backup speed
Allows for staggered implementation of dedupe
Allows you to copy last night's backups in its original format
The Disadvantages of (Async Dedupe) Postprocess are
It does have a lot more I/O work to do.
It requires the landing zone disk
It requires more configuration than an inline approach
It allows the vendor to advertise numbers that aren't quite real
It will delay replication to a remote site by minutes or hours, depending on whichproduct we're talking about. Source:http://www.backupcentral.com/content/view/134/47
© 2010 IBM Corporation
Storage Technology 2010
12
The Advantages of (Sync Dedupe) Inline are:
Less I/O work to perform
When you're done, you're done
Data can be replicated the second it shows up
Simpler configuration
No landing zone required
The Disadvantages of (Sync Dedupe) Inline are:
Possibly slow down the incoming backup speed
Source:http://www.backupcentral.com/content/view/134/47
© 2010 IBM Corporation
Storage Technology 2010
13
Hash-Based Approach
© 2010 IBM Corporation
Storage Technology 2010
14
1. Slice data into chunks (fixed or variable)
2. Generate Hash per chunk and save
3. Slice next data into chunks and compare hashes with table
4. Reference data previously stored
A B C D E
Ah ChBh Dh Eh
A B C D E
Hash Based Approach
© 2010 IBM Corporation
Storage Technology 2010
15
Assessment for Hash-Based Approach
Applicable for all chunking methods
Hash-Collisions must be handledMust be prevented through secondary comparison (additional metadata, second hash method, binary comparison)
Requires hash table to store hash of all chunks
– Hash table will grow with data volume
Hash Table must be quickly searchable and accessible
– Growing hash-table may become performance bottleneck (doesn’t fit into RAM)
– Scalability issues
© 2010 IBM Corporation
Storage Technology 2010
16
MD5 – 128 bits
Sha0 – 128 bits
Sha1 – 160 bit
.... sha384, sha512
Hashing
© 2010 IBM Corporation
Storage Technology 2010
17
Hash Collision
Hash Collison(n) – a term in computer programming for a situation that occurs when two distinct inputs into a hash function produce identical outputs.
The possibility of a hash collision (2 chunks of different data assigned the same hash) is not zero.
A 10 TB repository has 1.25 billion 8k blocks, even with a low probability, when you are managing that many blocks for a long time, the likelihood increases.
© 2010 IBM Corporation
Storage Technology 2010
18
© 2010 IBM Corporation
Storage Technology 2010
19
© 2010 IBM Corporation
Storage Technology 2010
20
© 2010 IBM Corporation
Storage Technology 2010
21
• Hash Collision
• Hash Index size
© 2010 IBM Corporation
Storage Technology 2010
22
Understanding the Knowledge Base
A key metric is the means used to map the user content– Balancing performance vs. capacity
With hash schemes the hash for a ‘chunk’ is remembered
an index
For example purposes imagine a chunk size of 8KB– 1 TByte repository has ~134,000,000 8 KB chunks– Each hash (signature) is 20 bytes long– Need pointers scheme to reference inside 1 TByte
The hashes require 2.9 GBytes of memory – no issue
With a 100 TByte repository
~300 GBytes of memory is required
The Index is Everything
© 2010 IBM Corporation
Storage Technology 2010
23
The Index is Everything
If the index gets too big it must be paged to disk – performance crashes to 40-70 MB/sec
Last but not least - performance
The performance of the hash based solution is good until the storage hits a size around 28-30TB. At that point the index needs to be stored on diskbecause it won’t fit in memory. When that happen, our performance tanks to 60 MBs or less. We end up ordering an extra VTL appliance to make up for the lossin performance. This means more administration overhead due to configuration ofthe extra repository and service costs.
A quote from our customer!
© 2010 IBM Corporation
Storage Technology 2010
24
The Index is Everything
If the index gets too big it must be paged to disk – performance crashes to 40-70 MB/sec
Last but not least - performance
Over a year ago, we (NEC) actually conducted tests with Hash-based vendor and verified that their maximum stated numbers were indeed *only* applicable for 100% duplicate data. For 0% duplicate data (i.e. all data actually written to disk), the max throughput numbers we measured with the same system were 45% lower.
A quote from vendor!
© 2010 IBM Corporation
Storage Technology 2010
25
Sizing Requirements for Hash Based Solution
© 2010 IBM Corporation
Storage Technology 2010
26
Big Financial Company’s Environment and Issues– NetBackup – Weekly 100TB fulls and 20TB incrementals nightly – 60-day retention– Data type: File system, Database and Exchange – Pain points:
• Backup window too short and shrinking• Recovery from tape too slow - Wanted to use disk for short term recoveries• Tried using straight disk but too expensive and managing file systems was painful• Tried NBU PureDisk but it slammed performance of backup servers
Review the following actual proposals and you’ll see why this customer choose IBM’s enterprise-class solution over hash-based approach attempt
– The following is an example of a real customer that needed a deduplication to improve their backup and recovery environment and how both Hash-based Vendor and IBM proposed to solve their problems
ProtecTIER versus Hash Based – Real World Example
© 2010 IBM Corporation
Storage Technology 2010
27
Backup Servers
Four separate Appliances
No global deduplication
No centralized management
4 head units and 24 shelves
No failover – no clustering
35TB
35TB
35TB
35TB
Clients
Hash-Based Vendor’s Proposed Solution:
© 2010 IBM Corporation
Storage Technology 2010
28
Dual node TS7650G Cluster with an IBM
Storage Array
Actual Results:
Full 100TB backup: 36 hours 809MB/s
Incr 20TB backup: 7 hours 832MB/s
Backup Servers
Clients
1000MB/s
Global deduplication
Centralized management
Semi-automated failover
Future scalability to 1PB
IBM’s Proposed ProtecTIER Solution:
© 2010 IBM Corporation
Storage Technology 2010
29
Dual node TS7650G Cluster with an IBM
Storage Array
Backup Servers
Clients
IBM’s PT Large Repositories: > 100 TBAT&T – 900TB repository Humana - 200TB repository JCP - 300TB repository MetLife - 480TBPepsi - 400TBThomson Reuters - 405TB UPMC - 200TBVanguard - 250TBLloyd's - 256TBMapfre - 128TBABSA – 250TBNAB (National Australia Bank) – 750TB
© 2010 IBM Corporation
Storage Technology 2010
30
VTL-Dedupe Hybrid (Differential) Approach(Inline)
TS7680 (z-OS)TS7650 (Open Systems & System i)
© 2010 IBM Corporation
Storage Technology 2010
31
1. Locate data in a backup stream similar to content stored in repository
2. After locating similar content, retrieve existing content from repository and run byte level check between existing and incoming data
3. Matches factored out – unique data added to repository
Element A Element B Element C
New Data Stream
HyperFactor Approach
© 2010 IBM Corporation
Storage Technology 2010
32
HyperFactor Approach
HyperFactor has two indexes–HyperFactor index used for backup
• Fixed size of 4 GB, stored in memory• Contains most similar data elements• Used to filter out similar elements from data
stream
–Restore Index used for restore• Dynamic index, growing• Includes reference of de-duped objects• Stored on disk system
© 2010 IBM Corporation
Storage Technology 2010
33
Assessment for HyperFactor
No Hash Table required
–No scalability issues• 4 GB index references 1 PB of physical data
No dependency of data format and application–Very flexible
HyperFactor index always fits in memory–Enables in-band de-duplication
Eliminates the phenomenon of missed factoring opportunities
–Looks for similarity between data not on exact chunk matches
© 2010 IBM Corporation
Storage Technology 2010
34
Repository
Backup Servers
FC Switch TS7650G
HyperFactor
Memory Resident Index
“Filtered” data
Existing Data
New Data Stream
Disk Arrays
Inside ProtecTIER TS7650G
© 2010 IBM Corporation
Storage Technology 2010
35
ProtecTIER Conceptual Flow Chart
Locatepattern
(RAM search)
Repositorydata
New backupdata
Read similarpattern fromrepository
Compute“delta”
Storenew backup
“delta”
LZH
Memoryindex
2:1
© 2010 IBM Corporation
Storage Technology 2010
36
• When data is written from the backup application to VTL, the VTL receives the write commandand the data, in this case LER001.
• The data is stored on the disk in segments and the VTL uses a “metadata file, or database, tokeep track of each segment of data.
• When the VTL recieves a read command from the backup application, it will use the ‘metadata‘or ‘restore index‘ to retrieve and present it to the backup application as sequential data.
ProtecTIER Conceptual Flow Chart
© 2010 IBM Corporation
Storage Technology 2010
37
HyperFactor Approach – IBM TS7650
Most de-duplicaton products are based on a simple standard cryptographic hash algorithm
Most use SHA-1 (160 bit) hashing algorithms
A hash is generated for each chunk of data
For each TB of repository, the index grows approximately 3 GBs
If the index gets too big it must be paged to disk – performance crashes to 40-60 MB/sec.
IBM Diligent first tried hash technology and discovered its limitiations
Hyperfactor developed by IBM Diligent engineers and Israeli mathematicians
A bit-for-bit comparison (differential) is used to guarantee unqiue data is not discarded
Only 4 GB is needed for a Petrabyte repository
Small Index remains in memory
Steady non-degrading performance of 600 MB/s or more, cluster solution equals 1000 MB/s
© 2010 IBM Corporation
Storage Technology 2010
38
• Active-Active 2 nodes cluster (architecture will allow for increasing node count over time)
• Full repository sharing among nodes– Writing data to the repository– Reading data from the repository (restore and read reference)– Access to all virtual devices
• No degradation on HyperFactor efficiency (regardless of the node through which the data is received)
• Minimum cluster down-time
ProtecTIER Clustering Overview
© 2010 IBM Corporation
Storage Technology 2010
3939 11-Jun-10
System Overview: Clustered ProtecTIER Nodes
Single Virtual Library Image
Network
MediaServer
Repository
CFSMetadata filesSTU data files
PT ManagerCluster: MyCluster
Node A
Node B
© 2010 IBM Corporation
Storage Technology 2010
4040 11-Jun-10
Hardware Deployment Diagram – TS7650G Two Node Cluster Configuration
ProtecTIER Gateway 2
ProtecTIER Gateway 1
Power Switch
Cluster Internal Network Switch 2
Cluster Internal Network Switch 1
Storage Fabric
Disk Arrays
© 2010 IBM Corporation
Storage Technology 2010
41
ProtecTIERServer
ProtecTIERServer
HOST
CPF
Unavailable Active
Virtual Drives
Unavailable Available
ALL PATHS TO NODE LOSTWITH CPF
Virtual Accessor
Control Path Failover in Dual-Node Configuration
© 2010 IBM Corporation
Storage Technology 2010
42
© 2010 IBM Corporation
Storage Technology 2010
43
Data Storage Repository
User DataUser Data
User Data
MetadataMetadata
User Data
MetadataMetadata Metadata
Metadata
User Data
User Data
RepositoryHyperFactor Index
Virtual Volume files
Library Configuration Data
Storage Management Data
User Datafrom Backup Application
RAID-10
RAID-5
© 2010 IBM Corporation
Storage Technology 2010
44
ESG Lab Valdiation Tests - 2009
© 2010 IBM Corporation
Storage Technology 2010
45
Scalable Capacity and Performance
Highest PerformanceLargest CapacityHigh Availability
Better Performance
Larger Capacity
Scalable
Better Performance
Larger Capacity
Scalable
Good Performance
Highly Scalable
Low cost
Highest Performance
Largest Capacity
Highest Performance
Largest Capacity
Up to 500 MB/sec
36 TB useable
Up to 100 MB/sec
7 TB useable
Up to 250 MB/sec
18 TB useable
Active-Active Cluster
Up to 500 MB/sec
36 TB useable
IBM TS7650 ProtecTIER® Deduplication Family
High PerformanceHigh Capacity
Flexible Storage
Highest Performancelargest CapacityHigh Availability
Single Node
Up to 500 MB/sec
1 PB TB useable
Active-Active Cluster
Up to 1000 MB/sec
1 PB TB useable
TS7650G GatewaysTS7650G Gateways
TS7650 ApplianceTS7650 Appliance
© 2010 IBM Corporation
Storage Technology 2010
46
IBM System Storage TS7650G ProtecTIER Deduplication Gateway
– Inline performance - sustainable at 1000 MB/sec – Up to 25:1 Data Reduction– Scalable to 1PB physical capacity 20:1 = 20PB Nominal – Single Node and cluster configuration– IBM HyperFactor®; industry leading inline deduplication– Enterprise Class Data Integrity – LTO drive emulation– Designed for performance scaling– IBM & Non-IBM disk support
• DS3XXX• DS4XXX• DS5XXX• DS8XXX• XIV• SVC• N-Series• EMC• HDS AMS1000 / USP• EMC CX• HP EVA
Only Inline High Availability Solution in the
Market Today
TS7650G Gateway
3Q08
HIGH Performance Data Deduplication
Clustered TS7650G Gateway
© 2010 IBM Corporation
Storage Technology 2010
47
End to End Data Deduplication for System z
Comprehensive solution builds on IBM z/OS, tape,
tape virtualization and ProtecTIER deduplication
Supports standard
Tape Applications Emulates
an IBM Tape
Library
Deduplicates with
ProtecTIER
Stores data on a variety
of disk storage
TS7680 Disk Cache
TS3500
VTL Deduplication
Tape Storage
(Less active data)
FICON Switch/Director
FICON Switch/Director
System z
© 2010 IBM Corporation
Storage Technology 2010
4848 11-Jun-10
Replication with ProtecTIER
Physical capacity
ProtecTIERGateway
Backup Server
Backup Server
Represented capacity
Primary Site
Represented capacity
Physical capacityProtecTIER
GatewayBackup Server
Secondary Site
Significant bandwidth reduction
PT-server based replication
© 2010 IBM Corporation
Storage Technology 2010
49
Up to 12 Branch Offices (spokes): Gateways and/or Appliances1 target (hub): Appliance, Gateway, single or two-node cluster
Physical capacityProtecTIER
Gateway
Backup Server
Central / DR Site
IP based NR links
Tape library
Virtual cartridges can be cloned to tape by the
Main-Site B/U server
Hub repository includes local backups and remote DR copies
ProtecTIER Many-to-one Replication Overview
Protect More. Store Less.™
© 2010 IBM Corporation
Storage Technology 2010
50
IBM Data Deduplication Case Studies
© 2010 IBM Corporation
Storage Technology 2010
51
Business challenge
Solution
10 TS7650G ProtecTIER™ Deduplication Gateways
Benefits
Executes backups to disk with a retention of 180 days providing faster backups and even quicker restores
Saved over 100+ square meters of floor space by eliminating tape libraries through this implementation
Off-site backups are no longer needed. Data is electronically copied and replicated safely and efficiently
Enables customer to re-use existing disk infrastructure
Lloyds Banking Group – formerly Halifax Bank of Scotland (HBOS)
Protect More. Store Less.™
IBM’s TS7650G ProtecTIER seamlessly integrated into an existing backup environment using TSM, removed the complexity of failed backup and restores and will help them contain the growth rate of their data sets
Lloyds created one of the largest SANs in Europe. Nightly backup numbers exceed 1,000TBs and they have more than 5PBs of centrally managed storage. Faced with shrinking backup windows, backup failures and data growth at 55% CAGR, contributed to them evaluating a new disk-based backup and recovery infrastructure. This client chose IBM’s TS7650G data deduplication solution over final contender Data Domain. This deal is now IBM’s largest ProtecTIER installation across Europe.
© 2010 IBM Corporation
Storage Technology 2010
52
© 2010 IBM Corporation
Storage Technology 2010
53
SEB BankBusiness challenge
Solution
IBM’s ProtecTIER TS7650G (6 in a clustered configuration)
IBM DS8000 disk arrays (2)Benefits
Provides industry-leading performance, scalability and availability with true global deduplication technology
IBM provided SEB a solution of 6 TS7650 Gateways vs. Hash-Based Vendor’s 34 appliances to handle the same amount of data
Enables SEB to manage their environment holistically and will enable SEB to meet their goal of going tapeless in the next 2-3 years
Protect More. Store Less.™
SEB wants to be tapeless with incremental data with total 4.2 PB in 2-3 years. SEB had already moved to a disk-based backup and recovery solution using hash-based VTL solution but was hampered by their inability to provide performance, scalability, and capacity to meet their backup and recovery requirements. As new datasets were added and their environment continued to grow, performance and capacity suffered. With the current VTL appliances, their only choice was to keep adding appliances to try and solve the problem. SEB decided not to invest any more time and money and opted for IBM’s TS7650G deduplication solution in a clustered configuration to have a more robust, dependable solution that could guarantee performance and scalability.
With industry- leading
performance, scalability and
capacity, ProtecTIER
continues to exceed expectations on
meeting customer requirements of all
sizes
© 2010 IBM Corporation
Storage Technology 2010
5454 11-Jun-10
TS7650G Dual Cluster Nodes / Standby – SEB Bank Stockholm, Sweden
Legato Networker 7.5–
DS8700
TS7650G
60 TB Repository
1500
Clients
TS7650G
60 TB Repository
1000
Clients
TS7650GStandby
(Grytet Node 1 )
Rissne Node 1
Rissne Node 2
Solaris 10
60TBPrime
60TBMirror
1TB P1TB P
1TB M
DS8700
60TBPrime
60TBMirror
1TB P1TB P
1TB M
RissneSAN
GrytetSAN
Dark Fibre
>20km
TS7650GStandby
(Rissne Node 1 )
Grytet Node 1
Grytet Node 2
Legato Networker 7.5–Standby
PT Test Server
MetroMirror(Bi-Directional)
© 2010 IBM Corporation
Storage Technology 2010
55
Some customer experiences with IBM TS7650
Helios– Dedupe rate 12:1– 8 TB usable – 96TB nominal– Databases, files, emails– Backup server: Backup Exec– Backup Restore/Requirements 300 MB/s
Hilti– Dedupe rate 16:1 – 30% databases (SQL, Oracle and SAP), 70% files– Retention time: 21 days = Files, 3 months = DB– Backup server: Netbackup– Backup Restore Requirements 400 MB/s
Ekom21– Dedupe rate 8:1 – OS, files (Incremental Forever) , emails, mySQL,Oracle, Informix– Daily full backups/Incrementals
Backup software: TSM 6.1Backup/Restore Requirements 300 MB/s
– Cartridge level IP replication
© 2010 IBM Corporation
Storage Technology 2010
56
Key Facts – IBM ProtecTIER VTL
Launched in Q4 2005- First VTL with Deduplication
Installed in all major industries
Over 1000 systems in production
In EMEA Current Install Base May 2010– Shipped 205 Clusters
• Average Cluster Repository – 58.3TB
– Shipped 120 Single Nodes • Average Single node Repository – 39TB
– Shipped 60 Appliances
Open Systems, AS/400, z/OS Support
Disk-Based and IP-Based Replication Support
56
© 2010 IBM Corporation
Storage Technology 2010
57
Questions?
Merci!
Danke!
© 2010 IBM Corporation
Storage Technology 2010
58
Notices and Disclaimers
Copyright © 2008 by International Business Machines Corporation. All rights reserved.
No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation.
Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. This document could include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the product(s) and/or program(s) described herein at any time without notice. Any statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Any reference to an IBM Program Product in this document is not intended to state or imply that only that program product may be used. Any functionally equivalent program, that does not infringe IBM's intellectually property rights, may be used instead.
THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. IBM shall have no responsibility to update this information. IBM products are warranted, if at all, according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. IBM makes no representations or warranties, expressed or implied, regarding non-IBM products and services.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to:
IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785U.S.A.