cluster components compute server disk storage image server
TRANSCRIPT
Compute Server:Stanford Research Computing Facility (RCF)
• Accessible only to Stanford faculty and authorized sub users for their research
• 128 nodes provided by the provost– 8 cores each (16 threads), 64GB RAM, Infiniband card– Ivy Bridge V2 with 1866GHz memory – 1024 cores, 2048 threads, 8TB RAM total in the initial
cluster– Cluster re-evaluated every 2-3 years for
upgrade/expansion
Compute Server:Stanford Research Computing Facility (RCF)• 1PB of scratch space (not backed up)
– Must push data to the local storage prior to running jobs– Home directories allocated to individual users (~10-15GB/user),
but may be combined among users within a group– Expansion units will be available for purchase later on
• Two additional nodes with 1.5TB RAM each– Jobs running on these machines must meet minimum RAM
requirements or else they will be terminated and sent to the general cluster
• Infiniband 56Gbits/sec Ethernet lines
Purchasing into the RCF: Hardware
• Purchasers will always have access to the main cluster• Base system purchase
– 8 cores (16 threads), 64GB RAM (4GB/thread), Infiniband card, $5000
• Custom systems can be purchased– i.e., 8 cores (16 threads), 128GB RAM (8GB/thread), ~$7000
• Flexible configurations can be made to purchased machines– i.e., 4 active threads (12 “wasted” threads, 128GB RAM with
32GB/thread)– Can be changed as long as DIMM configuration remains optimal
Purchasing into the RCF: Administration
• Can assign priority of jobs among authorized users that have access to a purchased machine.
• Server administration is included in purchase price. No annual fees.
• Purchase announcements made ~3 times per year.– May have to wait a couple months before purchased
machine is available.• Other users’ access to purchased machines that are
idle is TBD– Compliance/audit issues remain
Loading Software on the RCF cluster
• Can request to have software packages on the general cluster– More likely to be installed if widely used– Matlab is on the list to be added, but is currently a
source of frustration• Can install software on a purchased machine– Recommended for software that is not widely used
(i.e., offline.recon)– Need to evaluate current licenses for transfer to
purchased machines (i.e., COMSOL, SEMCAD, etc.)
RCF Timeline
• February– Designated “friendly” users to begin testing– One each from Rutt group and McNab group
• March– Cluster becomes available for general use
• April-June?– 1st round of purchased machines available
RCF Considerations• May be advantageous to purchase early due
to our high RAM requirements• Purchase costs may increase next year• Unclear if purchased machines are available to
other users• Job priority rules are still being decided
Compute Server Action Items
• Leaning towards RCF purchase of 2 x 128GB systems to be used as a mini cluster.– Jennifer’s group gets top priority on one machine and
Brian’s group gets top priority on other.– Users from both groups are authorized to use both
machines.
• ASAP: Designate friendly users for RCF testing. One each from Brian’s group and Jennifer’s group.– Jason/Simone + Grant Yang
Disk Storage
• Synology desktop NAS or rack-mounted• RAID6 configuration– Main Unit: 48TB capacity 40TB actual storage– Expansion Units: 48TB capacity 48TB actual storage
• Must decide maximum expansion capacity prior to purchasing main unit– Main + 2xExpansion = 144TB system 136TB RAID6– Main + 1xExpansion = 96TB system 88TB RAID6
Desktop vs. Rack Storage vs. Max Capacity
DESKTOP NAS SYSTEM 88TB RAID6 136TB RAID6 RACK MOUNTED SYSTEM 88TB RAID6 136TB RAID6 Main Unit DS2413+ DS3612xs Main Unit RS2212+ RS3614xs+44TB Capacity (36TB RAID 6) $1,900.00 $3,000.00 44TB Capacity (36TB RAID 6) $1,900.00 $5,000.00 Main Unit Drives WD Red 4TB WD Red 4TB Main Unit Drives WD Red 4TB WD Red 4TB12 Drives @ $189 each $2,268.00 $2,268.00 12 Drives @ $189 each $2,268.00 $2,268.00 MAIN UNIT INITIAL COST $4,168.00 $5,268.00 MAIN UNIT INITIAL COST $4,168.00 $7,268.00
Expansion Unit DX1211 DX1211 Expansion Unit RX1214 RX121444TB Capacity $1,300 $1,300 44TB Capacity $1,700 $1,700 Expansion Drives WD Red 4TB WD Red 4TB Expansion Drives WD Red 4TB WD Red 4TB12 Drives @ $189 each $2,268.00 $2,268.00 12 Drives @ $189 each $2,268.00 $2,268.00 EXPANSION UNIT COST $3,568.00 $3,568.00 EXPANSION UNIT COST $3,968.00 $3,968.00
Maximum Expansion Units 1 2 Maximum Expansion Units 1 2MAX EXPANSION TOTAL COST $7,736.00 $12,404.00
MAX EXPANSION TOTAL COST $8,136.00 $15,204.00
COST PER TB $87.91 $91.21 COST PER TB $92.45 $111.79
Image Server
• Want a multi-user DICOM server and database– Need a way for people to be able to view and pull
DICOM images remotely– Adding all these users as recognized DICOM nodes
is cumbersome– A web interface is an effective way to accomplish
this
OsiriX
• Has a fairly basic web layout– Rudimentary search– Organized by albums (which cannot be managed through the
website, I think)– Excellent image viewer– Can be customized to our heart’s content though
• Advanced user management with customized access rights for each individual
• Try it out: http://171.65.72.243:3333/– Just press login