Download - Newlug presentation- OpenSolaris
04/15/2010
2
A brief history● SunOS 1 - 4
Based on original BSD Sun 680x0 workstations Later added Sun SPARC
● AT&T & Sun collaboration in 1987 Merging the best parts to create SVR4
● New operating system named Solaris 2 ( SunOS 4 retroacively named Solaris 1 ) Solaris brand to identify whole system
OS, Graphics system, Desktop software
04/15/2010
3
A brief history● 1989 – AT&T forms Unix System Labs● 1991 – First Solaris 2 release● 1993 – AT&T sells USL to Novell● 1994 – Sun buys out license from Novell
No more license/royalties, Sun owns the code
● Early 2004 – OpenSolaris planning begins● Late 2004 – OpenSolaris pilot project starts● 2005 – Open to the public
Dtrace source, website & lists, more code mid-year
● 2007 – Ian Murdock joins Sun, project Indiana starts
04/15/2010
4
Solaris/OpenSolaris Today● Supports SPARC, x86 CPUs● Both 32 and 64 bit kernels
SPARC systems 64 bit kernel only Mixed 32 & 64 bit programs on 64 bit kernel
● Highly scaleable 32 CPUs on 32 bit x86, 256 on 64 bit, 512 on SPARC
04/15/2010
5
Licensing● CDDL - Common Development & Distribution License
Kernel, drivers, and most of userland programs
● OpenSolaris Binary License Not open source, but is freely redistributable http://hub.opensolaris.org/bin/view/Main/no_source
● Lots of other code with their own licences GPL, BSD, MIT, Apache, Perl Artistic, etc.
04/15/2010
6
About the CDDL● Comon Development & Distribution License● Based on Mozilla Public License v1.1● File Based● Allows combination with code that has other
compatible licenses● Requires modifications shared back● Explicit patent license, “patent peace” clause● http://hub.opensolaris.org/bin/view/Main/licensing_faq
04/15/2010
7
CDDL Misconceptions● Gives Sun ownership of your code
You retain ownership/copyright of your code Seperate contributor agreement for including your source
code in Sun's code base
● Designed to be GPL incompatible Compatible with lots of other licences File based, does not require everything to be under CDDL
● http://blogs.sun.com/cmh/entry/cddl_is_it_so_bad● http://blogs.sun.com/harpster/entry/cddl_me_elmo
04/15/2010
8
Building an operating system● Many seperate consolidations
Operating System/Networking ( aka ON ) Kernel, drivers, networking, libraries, services, & commands
X11 Desktop
Gnome, Firefox, Thunderbird
SFW ( Sun Freeware, various open source software ) Globalization ( includes internationalization & localization )
● Not all consolidations open sourced SPARC Graphics Drivers OpenWindows, CDE ( obsolete, removed )
04/15/2010
9
SMF: Service Management Facility● Framework for defining persistent system services● Each service identified by FMRI
( Fault Management Resource Identifier )
● A service can have multiple instances
04/15/2010
10
SMF: Service Management Facility● Configuration stored in repository
Persistant, transaction based, safe rollback
● XML manifests used to import/export service● Tracks dependencies, manages parallel startup
04/15/2010
11
SMF: Service Management Facility● Restarters responsible for managing services● New process attribute: contract
Restarter alerted if contract becomes empty Decides what action to take
● Restarters responsible for managing services● New process attribute: contract
Restarter alerted if contract becomes empty Decides what action to take
04/15/2010
12
SMF: Service Management Facility● Long running & transient ( run once ) services● Service manifest defines how service runs
User/Group Credentials Process Privileges Working Directory Environment variables Project ID / Resource Pool
● Delegated administration of services● Users assigned profiles for managing services
[ demo 1 ]
04/15/2010
13
Fault Management Daemon● Manages diagnosis engines, response agents, logging● Recieves telemetry from other parts of system
Hardware error reports ( CPU, memory, IO bus bit errors ) Driver faults ( NIC reporting corrupted data ) Disk errors ( bad blocks, disk not responding )
● Diagnoses problem, takes action when needed Offline/retire faulty CPUs & memory pages Reroute IO ( Multipath disks, aggregated NICs ) Activate disk hot spares Restart affected services via SMF Send alerts via syslog, SNMP
04/15/2010
14
ZFS● Pooled storage
Does away with dealing with partitions/slices Eliminates extra volume manager layer Simplifies administration, improves performance
Storage Pool
Volume Filesystem FS
04/15/2010
15
ZFS● Pool built on one or more vdevs
Stripe, Mirror, Raidz1, Raidz2, Raidz3 Able to add additional vdevs to a pool
● NFSv4/NTFS style ACLs● Instant read-only snapshots
Can be cloned to create new writeable filesystem
● Delegated administration Allow users & groups ability to manage own filesystems Fine grained control of allowed tasks
● Compression, deduplication● In place upgrades for new capabilities
04/15/2010
16
ZFS● Guaranteed data protection
Checksums used end to end, catches & repairs corruption Transactional writes, always consistent state on disk Copy on write, never alters live data, no “write hole” Checksum Trees, checksum stored in parent block pointer
Avoids corruption hitting both data & checksum together Validate entire pool by walking whole tree
Background scrubbing & resilvering
● Maximum performance Writes spread across all devices in pool Most disk writes sequential due to copy on write/transactions Smart prefetch, detects length & striding reads
04/15/2010
17
ZFS● Self healing ( as long as there's redundancy )● Redundant copies of all filesystem metadata● Copies of user data optional, setable on the fly
Mirror Mirror Mirror
AppAppApp
04/15/2010
18
ZFS & SSDs● Seperate Intent Logs ( aka “slogs” )
Intent log used for synchronous writes Normally uses space from disks in pool Move to SSD for fast, short lived writes ( NFS, iSCSI )
● Level 2 ARC ( aka “L2ARC” ) ARC == Adaptive Replacement Cache in memory L2ARC best for read optimized SSDs Faster than disk, slower & cheaper than RAM
04/15/2010
19
ZFS vs btrfs● 2001 start, 2005 release● CDDL License● Checksum: fletcher2,
fletcher4, sha256, off● Always copy-on-write
● No fsck needed● Rollback from snapshot
● 2007 development start● GPL License● Checksum: crc32, off
● Selective copy-on-write● Device removal, defrag● Online/offline fsck
04/15/2010
20
DTrace● Dynamic Tracing of kernel & userland code● Simplified C-like “D” language
Added functions and variables for tracing Thread-local variables
● Completely safe to run on production systems● Zero probe effect when dtrace not running
04/15/2010
21
DTrace● Provider: syscall, pid, proc, io, fbt ( functional block trace )
● Module: Process executable, library, driver
● Function● Name: entry, return, connection-start, connection-done
● Applications can also have their own providers
04/15/2010
22
IPS: Image Packaging System● Replaces old SVR4 packaging system● Network based software repository & delivery● GUI and CLI package tools● Multi-variant packages
04/15/2010
23
IPS: Image Packaging System
04/15/2010
24
IPS: Image Packaging System● Uses ZFS to create & manage boot environments
04/15/2010
25
Desktop● Gnome 2.28● Xorg 1.7● Compiz ( on live CD )● Firefox 3.5.8● Thunderbird 3.0.1● Device Driver Utility
04/15/2010
26
Desktop● Time Slider
Automated ZFS Snapshots Addon to Nautilus File Manager Enable/disable per filesystem
● [ demo ]
● Future improvements Backup to external media Better retention configuration
04/15/2010
27
Crossbow: Network Virtualization● Great for server & network consolidation● Add virtual NICs, switches, & vlans● Assign cpu resources, priorities, bandwidth limits
Local IP/network & remote IP/network Local & remote network port Transport type ( tcp, udp, sctp, icmp, icmpv6 ) Limit max bandwidth, set relative priority Extended accounting for flows for tracking usage
04/15/2010
28
Zones● OS level virtualization, “chroot on steroids”
One kernel, isolated userland instances Seperate naming services inside each zone ( DNS, NIS, LDAP ) Processes isolated within zones
Broad resource controls Processor pools, process, thread, memory, swap limits
Limited privileges & device access Must be granted by configuration Shared ip stack networking or exclusive access to NIC
04/15/2010
29
Branded Zones● Support for non-native environments
Interpretation layer to translate from brand to native System calls, signals, process & thread handling
Different brand types native: on Solaris 10 / Solaris Express releases ipkg: for OpenSolaris releases lx: Linux 2.4 kernel & 32 bit apps lx26: Experimental Linux 2.6 support solaris8 & solaris9: on SPARC ( commercial support ) s10brand: to support transition of Solaris 10 to future versions
● [ zones demo ]
04/15/2010
30
xVM● OpenSolaris as dom0 on top of Xen
Xen 3.4 hypervisor & tools Live migration support Full & Paravirtualized domU on top of hypervisor
Intel VT or AMD-V instruction set needed for Full HVM OpenSolaris as full PV guest
● Use Crossbow for managing networking● Can use ZFS volumes for storage
04/15/2010
31
COMSTAR● Common Multiprotocol SCSI Target● Seperates storage backend from access methods● LUN masking & mapping● Multipathing over different transports
COMSTAR
04/15/2010
32
COMSTAR● Logical Unit Providers
Disk, Tape, Virtual Tape Library logical units Actual storage via ZFS volume, raw disk, file, tape Supports thin provisioning
COMSTAR
04/15/2010
33
COMSTAR● Port Providers
Exports LUNs to other hosts Looks like normal block device to the host iSCSI, FibreChannel, FCoE, SAS, iSER ( iSCSI over RDMA )
COMSTAR
Server 1
Server 2
Server 3
04/15/2010
34
CIFS Server● Native, in kernel CIFS server● Can join Windows domain● LDAP/Windows AD user maps● Map Windows Security ID to
Solaris user/group● NT ACL stored natively in ZFS● ZFS snapshots work with
Windows Shadow Copy● Virus scan engine support
Integrated into ZFS, uses ICAP Protects NFS & local access too
04/15/2010
35
Trusted Extensions● Replaces previous seperate “Trusted Solaris” OS● System enforced mandatory access control policies● Explicit authorization needed to cross security labels● Device & network access also labeled
● [ demo ]
04/15/2010
36
Drawbacks● Uses more memory● No KDE ( Other distros include KDE )● ZFS block pointer rewrite a ways off yet
For online pool resizing, re-compress/dedup/defrag
● ZFS encryption still being developed● IPS going through heavy development● Power Management, Suspend/Resume● Limited Wi-Fi chipsets supported● Proprietary media codecs
Fluendo commercial gstreamer plugins
04/15/2010
37
Other Distributions● OpenSolaris: http://opensolaris.com/
Official Sun/Oracle sponsored distribution
● BeleniX: http://belenix.org/ First non-Sun Live CD & GUI distro KDE, GNOME, & XFCE packages
● Nexenta: http://nexenta.org/ OpenSolaris kernel, Ubuntu userland & packaging Easy NAS/SAN Seutp Commercially supported NexentaStor storage platform
04/15/2010
38
Other Distributions● EON: http://sites.google.com/site/eonstorage/
Embedded Operating System/Networking Small NAS focused distribution
● MilaX: http://www.milax.org/ Started as mini/rescue CD experiment Full CD/USB based distro, enough for older hardware
● Jaris: http://jaris.jp/ Japanese based distribution
04/15/2010
39
Other Distributions● Schillix: http://schillix.berlios.de/
First non-sun distribution Lead by Joerg Schilling
● MarTux First SPARC distribution
● Korona: http://techbase.kde.org/Projects/KDE_on_Solaris/OpenSolaris/Korona
KDE4 based test drive Live DVD Experimental, currenly on KDE 4.3.98
