smf deepdive tran
DESCRIPTION
soalris smf docTRANSCRIPT
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SOLARIS 10DEEP DIVE: SMF
Christine Tran
US Solaris Adoption Practice
Sun Microsystems
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Topics
• Part I: SMF Fundamentals> Motivation for SMF> Core concepts and terminology> Command overview> New boot process
• Part II: SMF in Action> Quick tour of service administration> Managing inetd services> Tips for troubleshooting> Recovering a repository
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Topics
• Part III: Advanced SMF> Inside the manifest> A little about XML> Writing a manifest from scratch> Exploring the repository with svccfg and svcprop
• Part IV: More Advanced SMF> Security aspects> Snaphots> Handling signals> Portable services
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Topics
• Part V: SMF Tools and SMF Community> Tools: Webmin> Tools: xmllint, jedit, Morphon> SMF Community on OpenSolaris> Resources> Review> Q & A> Closing
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Part I:SMF Fundamentals
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Motivation for SMF
• What is a service?• Is running a software program the same as delivering a
service? What exactly is a service?> Answer> Answer> Answer
• Does there currently exist an OS framework for service support and management?> Answer> Answer> Answer
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
How are services started today?
• init(1M) vi rc scripts in /etc/rc?.d/*> Long-time running or one-time initializations
• inetd(1M) as defined by inetd.conf> Short-lived to provide network functions
• init(1M) as defined by /etc/initab > Restartable or one-time functions
• How are these services grouped? Could they be related? How are the relationships described?
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Diagnostic ability
• An application fails to start> Missing configuration file?> Failed to mount a filesystem?> Database is late in starting up?> Missing another component?
• Lack of knowledge of service boundary and service interdependencies limits error handling ability of system.• Lacks common framework of error-handling• Lacks restart capability
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service Management Today
• Thousands of different text files, arbitrarily grouped and managed with multiple administration techniques.• Undeclared, often unknown dependencies; linear startup
is a by-product.• Lacks common interface, each service is started a
different way.• Does not address multiple instances of a service, for
example, web servers.• Does not address services that may span multiple hosts,
for example, a grid.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Motivation for SMF
A Service Oriented Architecture requires a definition of service
and a more robust framework to deliver and manage it.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SMF gives you
• A consistent service model in a common framework: command interface, service description, property specification, status view, etc.• A meaningful system view.• Ability to state dependencies (by-product is parallel boot).• Restart capabilities • All data stored in persistent, transaction-based repository.• Snapshots allow undo and rollback to a working
configuration.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Core concepts and terminology
• A service is:> An abstract description of a long-lived software object.> May describe object on a collection of machines (grid).> May have multiple instances (httpd).> Each instance of a service has a well-defined state and a well-
defined error boundary [process contract].> Each service defines methods: start, stop, refresh, etc.> Each service defines dependencies: what it depends on, and
may define dependents: what depends on it.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Core concepts and terminology
• A service has a unique identifier called FMRI – Fault Management Resource ID
svc://localhost/network/login:rlogin
Schemesvc – SMF managed servicelrc – legacy RC script
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Core concepts and terminology
• A service has a unique identifier called FMRI – Fault Management Resource ID
svc://localhost/network/login:rlogin
Locationlocalhost – currenthostname – future release
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Core concepts and terminology
• A service has a unique identifier called FMRI – Fault Management Resource ID
svc://localhost/network/login:rloginFunctional category
applicationsystemdevicenetworkmilestoneplatformsite
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Functional Categories
• Application – traditional daemons• Device – useful for dependencies• Milestone – similar to run levels• Network – converted services from inetd.conf• Platform – platform-specific services• System – platform-independent system services• Site – reserved for local site use
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Core concepts and terminology
• A service has a unique identifier called FMRI – Fault Management Resource ID
svc://localhost/network/login:rlogin
Service Description – related to method or RC script
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Core concepts and terminology
• A service has a unique identifier called FMRI – Fault Management Resource ID
svc://localhost/network/login:rlogin
Instance – “default” is the default instance
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Core concepts and terminology
• FMRI can be addressed by the shortest unique match, for example:> svc://localhost/network/login:rlogin> svc:/network/login:rlogin> network/login:rlogin> rlogin
• Some common names have changed to a different FMRI, for example: syslog is now system-log
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service states
• online – the service instance is enabled and has successfully started.• offline – the service instance is enabled, but the service is
not yet running or available to run, usually due to a dependency that has not been satisfied, or an error in the start method.• disabled – the service instance is not enabled and is not
running.• maintenance – the service instance has encountered an
error that must be resolved before it could be started again. There are many reasons why a service could be in this state.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service states
• legacy_run – the legacy service is not managed by SMF, but the service can be observed.> Faults not handled by SMF, no automated restart> Administrative error undetected> Software or hardware error results in process death
• degraded – the service instance is enabled, but is running at a limited capacity.• uninitialized – this state is the initial state for all services
before their configuration has been read.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service dependency
• A dependency is a formal description of other services that are required to start a service.• A service can be dependent on another service or files.
When a dependency is not met, the service stays offline. Whether a dependency is satisfied depends on its type:> require_all – all services are running or all files are present.> require_any – at least one is running or at least one file is
present.> optional_all – all are running, disabled, in maintenance, or not
present. For files, this type is the same as require_all.> exclude_all – all are disabled, in maintenance, or when files
are not present.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Dependency actions
• Once running (online or degraded states are considered running) if a service dependency is stopped or refreshed, SMF determines whether to restart the service depending on the restart_on attribute specified.
restart_on attributeReason for dependency stop None Error Restart RefreshError no yes yes yesNon-error stop no no yes yesRefresh no no no yes
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service restarter
• /lib/svc/bin/svc.startd is the master process starter and restarter. It> reads the Service Configuration Repository and start services
on boot;> restart services that have failed;> shutdown services whose dependencies are no longer
satisfied;> run legacy RC scripts at run-level transitions;> provides system view of service status.
• There is a delegated restarter, inetd.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Delegated restarter
• Assumes responsibility for executing a set of services.• Manages service faults; may gracefully handle conditions
such as signals or core dumps.• May define/require additional service configuration.• Often provides functionality common to a specific set of
services, for example: inetd
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SMF Manifest
• Description of the initial configuration of a service in XML.• Loaded into the SMF repository at boot time.• Sun-delivered services live in /var/svc/manifest. • For ISVs, manifests should be placed in the appropriate
subdirectory /var/svc/manifest.• Manifests for anything that is specific to the customer's
site only can go in /var/svc/manifest/site. This is reserved for local use.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SMF Manifest
• A manifest identifies, at the very least: > the service name> what the service is dependent on> ways to start and stop the service
• Can contain other attributes, such as the delegated restarter, config files, log files and others.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service Configuration Repository
• Located in /etc/svc.• Distributed between local memory (volatile) and local file
(repository.db).• repository.db holds the persistent service description read
from the manifests.• volatile is a tmpfs allocated from swap containing
transient data (lock files, init state, log files).• The underlying DB engine is sqlite 2.8• Zones have their own repository.• Changes to services should be made against repository.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Profile and Archive
• Profile – a set of service instances and their enabled or disabled state. Useful for copying service states between systems.• Generated by svccfg extract• Archive – a complete set of persistent data for all service
instances. Useful for copying service definitions between systems.• Generated by svccfg archive• Output is an XML file similar to manifests.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SMF Snapshots
• snapshot – a complete collection of properties for a service instance.• It provides a historical view of a service and simplifies
rollback of service configuration changes.• No snapshot-at-will capability yet.• Snapshot types:> initial – taken on the first import of the manifest> last_import – taken during last import> running – properties of the currently running service instance> previous – properties of the service instance just before running> start – taken at the last successful service start
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Key files and directories
• /var/svc – manifests, logs and profiles• /lib/svc – binaries, methods, seed, support files• /etc/svc/repository.db – global configuration database• /lib/svc/seed/global.db – minimal configuration database• /lib/libscf.so.1 – repository APIs
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
New filesystems
• /etc/svc/volatile> Directory where svc.startd stores log files in early stages of
boot, before /var is mounted read-write. Contains transient data.
• /system/contracts> Primary interface into contract subsystem, to allow service
instances to be restarted. Indirectly related to SMF.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Command overview
• General commands> svcs(1) service status listings> svcadm(1M) administrative actions> svccfg(1M) general property manipulation> svcprop(1) property reporting
• inetd management commands> inetadm(1M) administrative actions/property modification> inetconv(1M) conversion of legacy inetd.conf entries
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Command overview
• Daemons> svc.startd(1M) dependency engine, master restarter> svc.configd(1M) repository, authorization> inetd(1M) delegated restarter
• Contracts subsystem> ctrun(1M) execute with process contract> ctstat(1M) display active contracts> ctwatch(1M) monitor contract events> libcontract(3LIB) contract APIs
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
New boot process
• Instead of booting to run level, SMF introduces the concept of milestone.• milestone – a service which specifies a collection of
services which declare a specific state of system-readiness.• You can boot to specific milestone, services not part of
that milestone are temporarily disabled.• The default milestone is the milestone you will transition
to at every boot. It is usually milestone “all”.• init S, boot -s and friends still work.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Milestones and run levels
SVR4 Run Level SMF Milestone - none s, S single-user 2 multi-user 3 multi-user-server - all
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Legacy services
• Milestones are services, as such, they have manifests which have defined dependencies.• After all its dependencies have been satisfied, the start
method of the SMF major milestone executes each 'S' script within /etc/rc?.d/ with the argument 'start'. • /var/svc/manifest/milestone/single-user.xml specifies a
series of dependencies, and the start method is /sbin/rcS start.• SMF services first, legacy services last, run once only.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Part II:SMF in Action
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service administration
svcs – reports service status
• svcs -a show all services• svcs -x show services not running, impact• svcs -d show dependencies (what this depends on)• svcs -D show dependents (what depends on this)• svcs -p show member processes• svcs -l/-v show additional details
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcs
# svcs -a
STATE STIME FMRI
legacy_run Oct_12 lrc:/etc/rc2_d/S20sysetup
online Oct_12 svc:/network/ssh:default
disabled Oct_12 svc:/application/print/server:default
offline Oct_12svc:/application/print/rfc1179:default
...
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcs
# svcs -x
svc:/application/print/server:default (LP print server)
State: disabled since Wed Oct 12 16:18:38 2005
Reason: Disabled by an administrator.
See: http://sun.com/msg/SMF-8000-05
See: lpsched(1M)
Impact: 2 dependent services are not running. (Use -v for list.)
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcs
# svcs -d http
STATE STIME FMRI
online Oct_12 svc:/network/loopback:default
online Oct_12 svc:/network/physical:default
# svcs -D loopback
STATE STIME FMRI
disabled Oct_12 svc:/network/dns/client:default
disabled Oct_12 svc:/network/dns/server:default
disabled Oct_12 svc:/network/rarp:default
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcs
# svcs -p sendmail
STATE STIME FMRI
online 16:09:32 svc:/network/smtp:sendmail
16:10:32 23724 sendmail
16:10:32 23725 sendmail
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcs
# svcs -l rarp
fmri svc:/network/rarp:default
name Reverse Address Resolution Protocol (RARP) server
enabled false
state disabled
next_state none
state_time Wed Oct 12 16:18:41 2005
restarter svc:/system/svc/restarter:default
dependency require_all/error file://localhost/etc/nsswitch.conf (online)
dependency require_any/error
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service Administration
svcadm – administer services-t temporary-r recursive-s synchronously waits for completion
• svcadm enable/disable enable/disable services• svcadm enable/disable -t temporarily, until rebooted• svcadm restart restart specified service• svcadm refresh re-read service configuration• svcadm clear/mark clear/mark maintenance state• svcadm milestone transition to a milestone• svcadm -d milestone transition and set as default
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcadm
# svcs sendmail
STATE STIME FMRI
online 16:49:59 svc:/network/smtp:sendmail
# svcadm disable -t sendmail
# svcs sendmail
STATE STIME FMRI
disabled 16:51:27 svc:/network/smtp:sendmail
• Does not show service in temporary state. The presence of this property tells us that this setting is temporary.
# svcprop sendmail | grep general_ovr/enabled
general_ovr/enabled boolean false
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcadm keyserv
# svcs keyserv
STATE STIME FMRI
disabled Oct_12 svc:/network/rpc/keyserv:default
# svcadm enable keyserv
# svcs keyserv
STATE STIME FMRI
maintenance 16:58:57 svc:/network/rpc/keyserv:default
• Keyserv is in maintenance mode, why?
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcadm keyserv
# svcs -x
svc:/network/rpc/keyserv:default (RPC encryption key storage)
State: maintenance since Wed Oct 19 16:58:57 2005
Reason: Start method exited with $SMF_EXIT_ERR_CONFIG.
See: http://sun.com/msg/SMF-8000-KS
See: keyserv(1M)
See: /var/svc/log/network-rpc-keyserv:default.log
Impact: This service is not running.
• Actually, it is because the domain name is not set on this host.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcadm keyserv
# domainname test.com
# domainname
test.com
• There is a service called domain, which takes care of system identifying info; it has to be restarted.
# svcs -o fmri,desc domain
FMRI DESC
svc:/system/identity:domain system identity(domainname)
# svcs domain
STATE STIME FMRI
online Oct_12 svc:/system/identity:domain
# svcadm restart domain
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcadm keyserv
# svcs keyserv
STATE STIME FMRI
maintenance 16:58:57 svc:/network/rpc/keyserv:default
• Keyserv is still in maintenance, why?# svcadm clear keyserv
# svcs keyserv
STATE STIME FMRI
online 17:02:48 svc:/network/rpc/keyserv:default
• Will this survive a reboot?
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcadm milestone
• What milestone am I in?# svcprop -p options_ovr/milestone system/svc/restarter:default
svcprop: Couldn't find property `options_ovr/milestone' for instance `svc:/system/svc/restarter:default'.
• If you see this, you are in milestone “all”.# svcadm milestone single-user
# Requesting System Maintenance Mode
Console login service(s) cannot run
Oct 19 17:19:00 zone1 syslogd:going down on signal 15
svc.startd: The system is coming down for administration. Please wait.
svc.startd: Killing user processes: done.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcadm milestone
svc.startd: Killing user processes: done.
Root password for system maintenance (control-d to bypass):
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode
# svcprop -p options_ovr/milestone system/svc/restarter:default
svc:/milestone/single-user:default
# reboot
# svcprop -p options_ovr/milestone system/svc/restarter:default
svcprop: Couldn't find property `options_ovr/milestone' for instance `svc:/system/svc/restarter:default'.
• I'm back in milestone “all”.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svcadm milestone
# svcadm milestone -d single-user
# Requesting System Maintenance Mode
# reboot
# svcprop -p options_ovr/milestone system/svc/restarter:default
svc:/milestone/single-user:default
• Every subsequent reboot will get me back here. To get back to full service, set default milestone to be “all”.
# svcadm milestone -d all
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Managing inetd services
• Definition for inetd-started services were in inetd.conf but inetd.conf is now deprecated. Look how short it is!• inetd.conf entries are converted to SMF manifests and
imported into the repository on initial boot.• The old way of adding a service by modifying inetd.conf
and HUPing inetd no longer works.• If you have an application that depends on what's in
inetd.conf, there is a tool called inetconv that will convert the inetd.conf entry to an SMF manifest.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
inetd
• inetd is now a delegated restarter for inet services.• It manages inet-specific properties such as> tcp_wrappers> max connection rate
• It manages state for inet services.• It listens for connections, etc.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
inetconv
# grep tftp inetd.conf
# TFTPD - tftp server (primarily used for booting)
# tftp dgram udp6 wait root /usr/sbin/in.tftpdin.tftpd -s /tftpboot
• It's commented out. Uncomment the entry. You can HUP inetd but take my word that that won't do anything.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
inetconv
# inetconv
inetconv: Notice: Service manifest for 100235/1 already generated as /var/svc/manifest/network/rpc/100235_1-rpc_ticotsord.xml, skipped
tftp -> /var/svc/manifest/network/tftp-udp6.xml
Importing tftp-udp6.xml ...Done
inetconv: Notice: Service manifest for 100083/1 already generated as /var/svc/manifest/network/rpc/100083_1-rpc_tcp.xml, skipped
inetconv: Notice: Service manifest for 100068/2-5 already generated as /var/svc/manifest/network/rpc/100068_2-5-rpc_udp.xml, skipped
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
inetconv
# svcs tftp
svcs: Pattern 'tftp' doesn't match any instances
# svcs -l udp6
fmri svc:/network/tftp/udp6:default
name tftp
enabled true
state online
next_state none
state_time Wed Oct 19 17:40:16 2005
restarter svc:/network/inetd:default
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service Administration
inetadm – administer inetd-started services
• inetadm list all services managed by inetd• inetadm -p show default inetd service property values• inetadm -l show all properties for a service• inetadm -e/-d enable/disable a service• inetadm -m modify property values• inetadm -M modify inetd default property values
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
inetadm
# inetadm -l udp6
SCOPE NAME=VALUE
name="tftp"
endpoint_type="dgram"
proto="udp6"
isrpc=FALSE
wait=TRUE
exec="/usr/sbin/in.tftpd -s /tftpboot"
user="root"
...
• Let's say you want to run with some other option, -X instead of -s.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
inetadm
# inetadm -m udp6 exec="/usr/sbin/in.tftpd -X /tftpboot"
# inetadm -l udp6
SCOPE NAME=VALUE
name="tftp"
endpoint_type="dgram"
proto="udp6"
isrpc=FALSE
wait=TRUE
exec="/usr/sbin/in.tftpd -X /tftpboot"
user="root"
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
inetadm
• Can also use inetadm to manage inetd-services as well as svcadm.
# svcs udp6
STATE STIME FMRI
online 17:49:44 svc:/network/tftp/udp6:default
# inetadm -d udp6
# svcs udp6
STATE STIME FMRI
disabled 18:01:53 svc:/network/tftp/udp6:default
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
inetadm
• Usually if you don't use a service, it's enough to disable it using svcadm. But you want it really gone then delete it from your repository using svccfg delete FMRI.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service Administration
svcprop – get service configuration properties
• -p property FMRI fetch a property• -s snapshot FMRI fetch property from the snapshot
# svcprop -p inetd_start/exec tftp/udp6
/usr/sbin/in.tftpd\ -s\ /tftpboot
# svcprop -p restarter/state tftp/udp6
online
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Service Administration
svccfg – modify service configurations
• select FMRI select a service• listprop show properties for a service• setprop change a property value for a service• delprop delete a property value for a service• delete delete a service• export|import export|import a service description in XML• extract|apply extract|apply a profile• archive dump a full XML service description for
all services
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svccfg
# svccfg -s print/server listprop 'lpsched/fd_limit'
lpsched/fd_limit count 0
# svccfg
svc:> select print/server
svc:/application/print/server> listprop lpsched/fd_limit
lpsched/fd_limit count 0
svc:/application/print/server> setprop lpsched/fd_limit = 8192
svc:/application/print/server> exit
# svcadm refresh print/server
# svccfg -s print/server listprop 'lpsched/fd_limit'
lpsched/fd_limit count 8192
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svccfg
• Can be used in batch or interactive mode.• Understands globbing to form regex• We'll work through a more pertinent example later.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Tips for troubleshooting
• If a single service is broken, -x is your friend. svcs -x will display the service logfile, if it exists.• See service start message:> boot -m verbose.
• For a system that hangs during boot:> boot -m verbose,milestone=none> log in at prompt> svcadm milestone all
• Watch system progress with svcs.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Tips for troubleshooting
• truss the service starting up by trussing the restarter• Or modify the start method, wrapping it in truss# svcprop -p start/exec apache2
/lib/svc/method/http-apache2 start
# svccfg -s apache2
svc:/network/http:apache2> setprop start/exec = “/usr/bin/truss -ealfo /tmp/apache2.truss /lib/svc/method/http-apache2 start”
• Don't forget svcadm clear once fault is rectified.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Last resorts
• Have changes been made to repository?> How many services are affected?> What snapshots exist?> Can you revert back to a previous snapshot?
• Read /lib/svc/share/README.• Restore repository from backup.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Revert a snapshot
# svccfg -s svc:/network/dns/server:default
svc:/network/dns/server:default> listsnap
initial
last-import
previous
running
start
svc:/network/dns/server:default> revert start
svc:/network/dns/server:default> exit
# svcadm refresh dns/server
# svcadm restart dns/server
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Recovering a repository
• The SMF repository can be recovered in case of corruption.• You may want to save the current repository in
/etc/svc/repository.db for post mortem.• Restore script is /lib/svc/bin/restore_repository.• If you restore from seed, next boot would be the initial
boot, you will see SMF reading and importing manifests.# zlogin -C zone1
[Connected to zone 'zone1' console]
40/104
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
restore_repository
# /lib/svc/bin/restore_repository
The following backups of /etc/svc/repository.db exist, from oldest to newest:
boot-20051012_125317
manifest_import-20051012_130044
boot-20051012_161835
boot-20051019_172301
boot-20051019_180815
# Please enter one of:
1) boot, for the most recent post-boot backup
2) manifest_import, for the most recent manifest_import backup.
3) a specific backup repository from the above list
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
restore_repository
4) -seed-, the initial starting repository. (All customizations will be lost.)
5) -quit-, to cancel.
Enter response [boot]: -seed-
After confirmation, the following steps will be taken:
svc.startd(1M) and svc.configd(1M) will be quiesced, if running.
/etc/svc/repository.db -- renamed --> /etc/svc/repository.db_old_20051019_182048
/lib/svc/seed/nonglobal.db -- copied --> /etc/svc/repository.db
and the system will be rebooted with reboot(1M).
Proceed [yes/no]? yes
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Part III:Advanced SMF
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Manifest revisited
• Manifest is the description of the initial configuration of a service, delivered in XML.• Automatically imported into repository on boot, install,
upgrade, and pkgadd by svc:/system/manifest-import.• Ignored once imported unless modified.• Do not modify Sun-delivered manifests! If you need to
change service properties, do it in the repository. We'll talk more about this.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Order is important
• Service bundle• Service name• Create default instance• Single instance• Dependencies• Exec methods• Property groups• Stability level• Template
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
A little about XML
• XML markups contain data about data, metadata, on the information they bracket.> <time>1200</time>
• Tells you at 1200 is a time unit, instead of weight or distance. Whatever processes XML data can recognize that data for what it is.• Can contain any kind of data. How do we differentiate?
An XML file of train schedule will have different formatting than an XML file of TV schedule, not just formatting, but different processing and different usage. Yet, <time>1200</time> can appear in both XML files.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Introducing DOCTYPE
• Look at the top of each manifest, they all have this line:> <!DOCTYPE service_bundle SYSTEM
'/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
• DTD – Document Type Definition, contrast with DOCTYPE for HTML.• Tells you whether you (especially if you are an XML-
processing program) are reading a train schedule or a TV Guide listing.• DTD is one example of a type of markup called
declaration. Specifically, the DTD is used to constrain the grammar of an XML file.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Grammar?
• Grammar governs the “correctness” of an XML document. A train schedule must have: source, terminus, track number, departure and arrival time. It won't have channel, show name, show time, duration, starring, rating.• Looking at just data, they look very similar. The DTD
defines and enforces how they must be marked up in XML, one way and one way only.• DTD is a collection of rules, or declarations, describing
markup objects.• An element declaration defines what it can contain and in
which order, also known as a content model.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Define your own language
• Strictly speaking, XML is not a markup language. It has no tag with absolute meaning. <time> can mean absolute time or elapsed time. Contrast this with <b></b>, this always means bold.• XML lays down framework for you to construct your own
markup language.• Adding elements in the DTD adds objects to your
language. An element not declared in the DTD is illegal. An element containing anything not declared in the DTD is illegal.• We'll see why this is important later on.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
What's in the DTD?
• A bunch of things! None resembling what you might find in a manifest.> <!-- comments --> are comments> <!ELEMENT> or <!ATTLIST> are declarations> <blah /> is same as <blah></blah>
• Ordering in DTD is not same as the required ordering in manifest. For example, declaration for property group is before declaration of service FMRI, and service bundle is declared last.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Here's an snippet
<!ELEMENT dependency
( service_fmri*, stability?, ( propval | property )* ) >
<!ATTLIST dependency
name CDATA #REQUIRED
grouping ( require_all | require_any | exclude_all |
optional_all ) #REQUIRED
restart_on ( error | restart | refresh | none ) #REQUIRED
type CDATA #REQUIRED
delete ( true | false ) "false" >
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Why XML?
• Think about what's in /etc/rc?.d today. Anyone can write anything in any format and drop it in /etc/rc?.d. In fact, what's in /etc/rc?.d is closer to methods than manifests.• A unified service management tool depends on having
uniform data to process. The repository is a database which relies on having correct data in the correct “cell”.• With XML you get a parser and a validator for free.
Anything that XML can't validate won't get imported.• The parser is in a library called libxml2. It comes with
xmllint, which is a generic XML validator.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
A quick tour inside the manifest
• We said a manifest must have at least> the service name> what the service is dependent on> ways to start and stop the service> in theory you need a restarter too but there is the default
svc.startd.
• House-keeping robot analogy> Lights> Depends on Power> On/Off switch on the wall, flip up or down> Robot is the default restarter
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
tftp-udp6.xml
service
name='network/tftp/udp6'
type='service'
<exec_method
type='method'
name='inetd_start'
exec='/usr/sbin/in.tftpd -s /tftpboot'
<exec_method
type='method'
name='inetd_offline'
exec=':kill_process'
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
tftp-udp6.xml
<restarter>
<service_fmri value='svc:/network/inetd:default' />
</restarter>
• Note that the method isn't “start”, but “inetd_start”.• Wait! No dependencies? Well, having inetd as your
delegated restarter implicitly marks you as dependent on inetd. Who starts inetd, then? It's started in /var/svc/profile/generic_open.xml• If you look at /var/svc/manifest/network/nfs/server.xml,
you will see that there is no explicitly defined restarter.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Writing a manifest from scratch
• Not all Sun products are delivered in a manifest, the reason seems to be time and effort. The README file for MySQL still says to link a startup script in /etc/rc3.d to the mysql.server script.• If no manifest is delivered, the service can still run as a
legacy service.• We're going to convert MySQL into an SMF service.• You may want to do this in a zone. Remember that each
zone has their own manifests and SMF repository.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Setting up MySQL database
• First, we need to set up the database. Instructions are in /etc/sfw/mysql/README.solaris.mysql. The startup script is /etc/sfw/mysql/mysql.server.
root@mysql~> /usr/sfw/bin/mysql_install_db
Preparing db table
Preparing host table
Preparing user table
Preparing func table
Preparing tables_priv table
Preparing columns_priv table
Installing all prepared tables
13:38:46 /usr/sfw/sbin/mysqld: Shutdown Complete
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
MySQL
• Add users and groupsroot@mysql~> groupadd mysql
root@mysql~> useradd -g mysql mysql
root@mysql~> chgrp -R mysql /var/mysql
root@mysql~> chmod -R 770 /var/mysql
• Add configuration filesroot@mysql~> installf SUNWmysqlr /var/mysql d 770 root mysql
root@mysql~> cp /usr/sfw/share/mysql/my-medium.cnf
/var/mysql/my.cnf
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
MySQL
• Start MySQL manually to see that everything worksroot@mysql~> /usr/sfw/sbin/mysqld_safe --user=mysql &
Starting mysqld daemon with databases from /var/mysql
• Set the passwordsroot@mysql~> /usr/sfw/bin/mysqladmin -u root password 'mysql'
root@mysql~> /usr/sfw/bin/mysqladmin -u root -h mysql password 'mysql'
• Check the status of MySQLroot@mysql~> /usr/sfw/bin/mysqladmin -u root -pmysql status
Uptime: 37 Threads: 1 Questions: 1 Slow queries: 0 Opens: 6 Flush tables: 1 Open tables: 0 Queries per second avg: 0.02
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Copy and modify
• Great! Everything works. Now you are ready to write your manifest!• Find a manifest similar to your application and tweak it.
Since MySQL is an application, I'll look in /var/svc/manifest/application. The print/server.xml seems closest to what I want, so I'll start from that. Remember that site-specific manifests go into “site”.• Most everything is already filled out, I just change a few
relevant properties.root@mysql~> cp /var/svc/manifest/application/print/server.xml/var/svc/manifest/site/mysql.xml
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Name
• Change the name<service_bundle type='manifest' name='SUNWsqlr:mysql'>
<service
name='application/mysql'
type='service'
version='1'>
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Dependency
• I leave the existing dependencies alone. I may want to add a dependency on the config file, so MySQL doesn't start if it's missing.
<dependency
name='database'
grouping='require_all'
restart_on='none'
type='path'>
<service_fmri value='file://localhost/var/mysql/my.cnf'/>
</dependency>
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
This just in
• File dependencies going may be going away.• CR 6240573> Synopsis: File dependencies are currently overused by SMF
consumers. The problem is that a file dependency means "wait for this file to become available before starting". That's not the usual case. Most users of file dependencies are trying to express the concept that "if this file is missing, there is a problem."
• No file event to signal SMF if a file has changed.• Actions in a file dependency doesn't do anything, yet.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Methods
• Change the start/stop methods<exec_method
type='method'
name='start'
exec='/etc/sfw/mysql/mysql.server start'
timeout_seconds='60' />
<exec_method
type='method'
name='stop'
exec='/etc/sfw/mysql/mysql.server stop'
timeout_seconds='60' />
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Description and documentation
• For now, delete the property groups as they don't apply to mysql• Change the description and documentation location<loctext xml:lang='C'>
MySQL server
</loctext>
<documentation>
<manpage title='mysql' section='1M'
manpath='/usr/sfw/man' />
</documentation>
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Importing your manifest
• You're done! Now import it.root@mysql~> svccfg import mysql.xml
root@mysql~> svcs -l mysql
fmri svc:/application/mysql:default
name MySQL server
enabled false
state disabled
next_state none
state_time Tue Jun 07 19:20:51 2005
restarter svc:/system/svc/restarter:default
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Importing your manifest
dependency require_all/none svc:/system/filesystem/local (online)
dependency require_all/none svc:/system/filesystem/usr (online)
dependency require_all/refresh svc:/system/identity:domain (online)
dependency optional_all/none svc:/system/system-log (online)
dependency require_all/none file://localhost/var/mysql/my.cnf (online
root@mysql~> svcadm enable mysql
root@mysql~> svcs mysql
STATE STIME FMRI
online 19:21:19 svc:/application/mysql:default
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Done!
• Congratulations! You have successfully written a service manifest. Now MySQL is an SMF-managed service.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SMF on OpenSolaris
• Collection of manifests and methods are at http://www.opensolaris.org/os/community/smf/manifests/• There is a different manifest for MySQL submitted there.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Exploring the repository
• We said the repository is a database containing all descriptions of all services SMF knows about, and their states.• Make changes to service against its content in the
repository, not in the manifest. The days of looking in a text file to find what's running is over. In fact, what's in a manifest can differ vastly with the running configuration.• We'll fix Bug ID 6270245 as an example.• Bug ID: 6270245> Synopsis: apache2 needs svc:/system/filesystem/local
dependency when Sun Volume Manager metadevices are in use
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Two ways to fix
• Two ways to fix:• Delete the service from the repository, fix the manifest
and re-import. This is not the recommended way.• Fix the currently running service configuration. This is the
recommended way, and we'll talk about why after we've finished the exercise.• We're going to do this both ways, in order to better
understand the repository and the manifest.• In this case the service can be addressed as http or
apache2, http is the service, apache2 is the instance.• Disable the service, then delete it from the repository.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Fix 1: deleting http from repository
# svccfg
svc:> list network/http
network/http
svc:> delete network/http
svc:> exit
• Or
# svccfg delete apache2
# svcs apache2
svcs: Pattern 'apache2' doesn't match any instances
STATE STIME FMRI
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Modifying manifest
• Once again, look around for something to copy from, I find my filesystem/local dependency in inetd-upgrade.xml and ssh.xml, respectively:inetd-upgrade.xml: <dependency name='filesystem'ssh.xml: <dependency name='fs-local'
• Note the name, filesystem and fs-local. These are just common names, they are not FMRI. In fact, these can be anything descriptive.• Add this to my
/var/svc/manifest/network/http-apache2.xml• Import your fixed manifest and check.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
filesystem/local dependency
dependency name='myfs-local'
grouping='require_all'
restart_on='none'
type='service'>
<service_fmri value='svc:/system/filesystem/local'/>
</dependency>
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Compare with DTD
<!ELEMENT dependency
( service_fmri*, stability?, ( propval | property )* ) >
<!ATTLIST dependency
name CDATA #REQUIRED
grouping( require_all | require_any | exclude_all |
optional_all ) #REQUIRED
restart_on ( error | restart | refresh | none ) #REQUIRED
type CDATA #REQUIRED
delete ( true | false ) "false" >
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
In depth
• name is myfs-local, a descriptive name.• grouping is require_all, self-explanatory, but when is
grouping not require_all? When the dependency is a milestone, for example.• restart_on is none. If you depend on a service, and its
state changes, you may want to re-examine your own service.• Refer back to slide “reason for dependency stop”• type is service; can be service or file.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
restart_on
• restart_on none: do not restart, you require this dependency to be satisfied once, on startup, that's all.• restart_on error: restart when this service faults. Why?
Opposite of previous case, you depend on this service all the time. If it faults (restart is implicit) and does not come back up, you need to restart and check your dependency. What state would you be in?• restart_on restart: restart when this service has an error
stop, or a non-error stop.• restart_on refresh: restart on all the above conditions,
and when this service re-read its configuration file.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
svccfg import
# svccfg import http-apache2.xml
# svcs -l apache2
fmri svc:/network/http:apache2
name Apache 2 HTTP server
enabled false
state disabled
next_state none
state_time Mon Jun 06 18:26:41 2005
restarter svc:/system/svc/restarter:default
dependency require_all/error svc:/network/loopback:default (online)
dependency optional_all/error svc:/network/physical:default (online)
dependency require_all/none svc:/system/filesystem/local (online)
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Fix 2: modify properties in repository
• Fix by modifying current service properties in the repository.• Now that I've written the manifest, I know better what to
do. Basically, take that block of dependency and some how get it into the running service configuration of apache2.• Disable and delete apache2 from the repository.• Re-import the original, unrepaired manifest; it should only
have two dependencies.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Put back original manifest
# svccfg -s apache2 delete
# svccfg import http-apache2.xml
# svccfg -s apache2 listpg
loopback dependency
physical dependency
start method
stop method
refresh method
httpd application
startd framework
general framework
restarter framework NONPERSISTENT
restarter_actions framework NONPERSISTENT
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
What I'm interested in
# svccfg
svc:> help
General commands: help set repository end
Manifest commands: inventory validate import export archive
Profile commands: apply extract
Entity commands: list select unselect add delete
Snapshot commands: listsnap selectsnap revert
Property group commands: listpg addpg delpg
Property commands: listprop setprop delprop editprop
Property value commands: addpropvalue delpropvalue setenv unsetenv
svc:>
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Adding properties
• Add a new dependency to my property groupsvc:/network/http:apache2> addpg myfs-local dependency
• In your manifest, this conforms to <dependency name='myfs-local'
• Add other propertiessvc:/network/http:apache2> setprop myfs-local/grouping = astring: ”require_all”
svc:/network/http:apache2> setprop myfs-local/restart_on = astring: ”none”
svc:/network/http:apache2> setprop myfs-local/type = astring: ”servicr”
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Oops! A typo.
svc:/network/http:apache2> listprop
myfs-local dependency
myfs-local/group astring require_all
myfs-local/restart_on astring none
myfs-local/type astring servicr
svc:/network/http:apache2> editprop (invokes vi)
# Property group "myfs-local"
# delprop myfs-local
# addpg myfs-local dependency
# setprop myfs-local/group = astring: (require_all)
# setprop myfs-local/restart_on = astring: (none)
# setprop myfs-local/type = astring: (servicr)
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Add the FMRI
• Change this line tosetprop myfs-local/type = astring: “service”
• Upon quitting vi, the temp file is executed. This corresponds to
grouping='require_all'
restart_on='none'
type='service'>
• Now add the FMRI of the dependencysvc:/network/http:apache2> setprop myfs-local/entities = fmri: “svc:/system/filesystem/local”
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Check your configuration
svc:/network/http:apache2> listprop
...
filesystem-local dependency
filesystem-local/group astring require_all
filesystem-local/restart_on astring none
filesystem-local/type astring service
filesystem-local/entities fmri svc:/system/filesystem/local
...
svc:/network/http:apache2> exit
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Refresh the service
# svcadm refresh apache2
# svcs -l apache2
fmri svc:/network/http:apache2
name Apache 2 HTTP server
...
restarter svc:/system/svc/restarter:default
dependency require_all/error svc:/network/loopback:default (online)
dependency optional_all/error svc:/network/physical:default (online)
dependency require_all/none svc:/system/filesystem/local (online)
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
New property added
# svccfg -s apache2 listpg
loopback dependency
physical dependency
start method
stop method
refresh method
httpd application
startd framework
general framework
restarter framework NONPERSISTENT
restarter_actions framework NONPERSISTENT
myfs-local dependency
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
The reason why ...
• You've modified the service properties for apache2 in the repository. Don't be confused the next time you look at http-apache2.xml and wonder what happened. Manifests are read and imported only once (unless they are modified.)• Don't modify manifests in place. Upon boot
svc:/system/manifest-import searches /var/svc/manifest for new XML files and ones which have changed and import them.• This allows us to upgrade service properties (via a patch,
for example) by dropping new manifest in, instead of manipulating the repository using svccfg.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
The reason why ...
• When svccfg import is run on a new manifest for services which already exist, it first compares the current values to those in the last-import snapshot.• If values don't differ, it assumes that there's been no
customization, and upgrades the properties to what's in the manifest.• Otherwise it issues warnings to the user, but leaves the
customizations in place.• So any patch which alters a manifest will cause manifest-
import to try to upgrade the corresponding service, no post-install action necessary.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
The reason why ...
• If you had modified that manifest in place and re-imported it, or even if you copied it somewhere else, modified it, and re-imported it, svccfg will think that the modified version is the original, official version (via the last-import snapshot comparison), will will override all your customization after the patch.• The moral of the story: leave those manifests alone.
Make changes against the repository with svccfg.• man smf_bootstrap(5) for the gory details.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Part IV:More Advanced SMF
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SMF security
• SMF uses RBAC to delegate service management responsibility. We'll talk about managing specific services first.• SMF provides these propval to be put into individual
manifest to authorize specific services:> action_authorization – can execute method> value_authorization – can change value of existing properties> modify_authorization – change, add, delete properties
• These are fixed keywords.• Let's take a look smtp-sendmail.xml and ssh.xml as
examples.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
action_authorization
smtp-sendmail.xml:
<property_group name='general' type='framework'>
<propval name='action_authorization' type='astring' value='solaris.smf.manage.sendmail' />
</property_group>
ssh.xml:
<property_group name='general' type='framework'>
<!-- to start stop sshd -->
<propval name='action_authorization' type='astring' value='solaris.smf.manage.ssh' />
</property_group>
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
prof_auth
• Where will you find this? /etc/security/prof_authMail Management:::Manage sendmail &
queues:auths=solaris.smf.manage.sendmail;help=RtMailMngmnt.html
Network Security:::Manage network and host security:auths=solaris.smf.manage.ssh,solaris.network.*;help=RtNetSecure.html• These RBAC roles have this privilege, which privilege is
this? The one defined by the keyword action_authorization, which allows you to execute methods.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Delegating to a user
• But you haven't set up roles properly, you want to delegate service administration to a user.
# echo christine::::auths=solaris.smf.manage.ssh >> /etc/user_attr
• You have to re-login for authorization to take effect.# su – christine
$ id
uid=85228(christine) gid=10(staff)
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Delegating to a user
$ svcs ssh
STATE STIME FMRI
online Oct_19 svc:/network/ssh:default
$ date
Fri Oct 21 15:47:40 EDT 2005
$ svcadm restart ssh
$ tail /var/svc/log/network-ssh:default.log
[ Oct 21 15:47:42 Stopping because service restarting. ]
[ Oct 21 15:47:42 Executing stop method (:kill) ]
[ Oct 21 15:47:42 Executing start method ("/lib/svc/method/sshd start") ]
[ Oct 21 15:47:42 Method "start" exited with status 0 ]
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
A little problem
$ svcadm disable ssh
svcadm: svc:/network/ssh:default: Permission denied.
$ svcadm disable -t ssh
$ svcs ssh
STATE STIME FMRI
disabled 15:55:43 svc:/network/ssh:default
$ svcprop ssh |grep general_ovr
general_ovr/enabled boolean false
• Service can only be temporarily disabled, why?See BugID 6303496.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Needs value_authorization
• Bug ID 6303496> Synopsis cron cannot be smf managed with
solaris.smf.manage.cron
• It turns out that you also need value_authorization, the keyword to change existing property values.• Why? Because you're actually changing the value of
some property. Which property?• And exactly where is that general_ovr property listed?• general and general_ovr appear to be property groups of
type framework. Check the DTD.• We'll talk more about instances later.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Looking for general_ovr
# svccfg -s ssh
svc:/network/ssh> listprop general*
general framework
general/action_authorization astring solaris.smf.manage.ssh
general/entity_stability astring Unstable
general/single_instance boolean true
# svccfg -s ssh:default
svc:/network/ssh:default> listprop general*
general framework
general/enabled boolean true
general_ovr frameworkNONPERSISTENT
general_ovr/enabled boolean false
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
value_authorization
# svccfg -s ssh
svc:/network/ssh> listprop
general/action_authorization astring solaris.smf.manage.ssh
• Do you need to addpg?svc:/network/ssh> setprop general/value_authorization = astring: "fooblat"
svc:/network/ssh> exit
• You still need to make one change for this to work.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
value_authorization
# tail /etc/user_attr
christine::::auths=solaris.smf.manage.ssh,fooblat
# svcadm refresh ssh
# su - christine
$ svcadm disable ssh
$ svcs ssh
STATE STIME FMRI
disabled 16:42:47 svc:/network/ssh:default
$ svcprop ssh|grep general_ovr
$
• Yes, it's a permanent change.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Value can be anything
• The value of the auth used has no specific meaning except that it must exist in /etc/user_attr. • In the manifest<propval name='value_authorization' type='astring' value='fooblat' />
• ... is equal to<propval name='value_authorization' type='astring'
value='solaris.smf.manage.ssh' />
• ... but the later is more descriptive. In fact:# auths christine
solaris.smf.manage.ssh,fooblat,solaris.device.cdrw,s
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SMF Security
• If you look at /etc/security/prof_attr, you will seeService Management:::Manage
services:auths=solaris.smf.manage, solaris.smf.modify• These are two main authorizations in SMF:> solaris.smf.modify – add, delete, modify services> solaris.smf.manage – restart, refresh, or other state
modification
• They don't appear in any manifest because they apply all services.• There are others: method, dependency, application, and
framework. man smf_security(5) for the gory details.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Blueprint
Restricting Service Administration in the Solaris 10 Operating System by Glenn Brunette (June 2005)This Sun BluePrints Cookbook describes how to use the Solaris 10 Service Management Facility (SMF) to require specific authorizations for certain types of operations. Using this capability, it is possible to delegate access to core service management functions based on the concept of least privilege--if a user or service does not strictly need to have some degree of privilege, then that privilege should not be granted. SMF allows organizations to have much finer grained access control policies than was possible before the Solaris 10 Operating System.
http://www.sun.com/blueprints/0605819-2887.pdf
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Snapshots
# svccfg -s ssh
svc:/network/ssh> listsnap
Instance not selected.
svc:/network/ssh> select default
svc:/network/ssh:default> listsnap
last-import
initial
running
• Where are snapshot start and previous?# svcs ssh
STATE STIME FMRI
disabled Oct_21 svc:/network/ssh:default
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
What's the difference?
# svccfg -s ssh listprop '*general*'
general framework
general/entity_stability astring Unstable
general/single_instance boolean true
general/action_authorization astring solaris.smf.manage.ssh
general/value_authorization astring yadayada
# svccfg -s ssh:default listprop '*general*'
general framework
general/enabled boolean false
general_ovr framework NONPERSISTENT
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
instance
• Instances are units of administrations.• Services are mechanism for sharing properties.• Instance can have properties, and its parent service can
have properties.• Instances have snapshots, services do not.• If unnamed, that instance is called “default”.• Snapshots are composite view of the instance and
service properties at one point in time.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
revert
svc:/network/ssh:default> listsnap
last-import initial
running start
svc:/network/ssh:default> selectsnap initial
[initial]svc:/network/ssh:default> revert
svc:/network/ssh:default> unselect
svc:/network/ssh> listprop general*
general framework
general/action_authorization astring solaris.smf.manage.ssh
general/entity_stability astring Unstable
general/single_instance boolean true
• value_authorization has disappeared.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Snapshots
• Snapshots are not good for rolling back a single change because > No control over when snapshots are taken> Alters the service properties as well
• To undo a property change, it's simpler to use svccfg setprop commands• Can use svcprop or svccfg export to take textual
snapshots. • However, when you use svccfg export, you use the
service name, even though the output file contains the description of the instance(s).
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
ignore_error property
• From http-apache2.xml:<property_group name='startd' type='framework'>
<!-- sub-process core dumps shouldn'trestart session -->
<propval name='ignore_error' type='astring'value='core,signal' />
</property_group>
• SMF is instructed to ignore signals sent to a service or core dumps by a service; otherwise SMF will try to restart the service.• man svc.startd(1M) for the gory details; description at the
startd/ignore_error paragraph.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Signaling httpd
# ps -ef|grep http
webservd 29855 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k start
root 29853 27175 0 11:46:45 ? 0:00 /usr/apache2/bin/httpd -k start
webservd 29857 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k start
webservd 29856 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k start
webservd 29858 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k start
webservd 29854 29853 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k start
# kill -HUP 29853 <-- started by root
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Logs
# tail -f /var/apache2/logs/error_log
(nothing ...)
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 11:46:44 Executing start method ("/lib/svc/method/http-apache2 start") ]
[ Oct 26 11:46:45 Method "start" exited with status 0 ]
(nothing ... )
• What would happen, by the way, if you kill just httpd processes started by webservd?
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Keep on killing httpd
# ps -ef|grep http
webservd 29854 27175 0 11:46:46 ? 0:00 /usr/apache2/bin/httpd -k start
# kill -9 29854 <-- last one
# tail -f /var/apache2/logs/error_log
[Wed Oct 26 12:46:53 2005] [notice] Digest: generating secret for digest authentication ...
[Wed Oct 26 12:46:53 2005] [notice] Digest: done
[Wed Oct 26 12:46:54 2005] [notice] Apache/2.0.52 (Unix) DAV/2 configured -- resuming normal operations
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
httpd restarted by SMF
# tail -f /var/svc/log/network-http:apache2.log
Stopping because all processes in service exited. ]
[ Oct 26 11:55:05 Executing stop method ("/lib/svc/method/http-apache2 stop") ]
httpd (pid 29853?) not running
[ Oct 26 11:55:05 Method "stop" exited with status 0 ]
[ Oct 26 11:55:05 Executing start method ("/lib/svc/method/http-apache2 start") ]
[ Oct 26 11:55:06 Method "start" exited with status 0 ]
# svcs apache2
STATE STIME FMRI
online 12:46:53 svc:/network/http:apache2
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Removing ignore_error property
# svccfg -s apache2
svc:/network/http:apache2> listprop startd*
startd framework
startd/ignore_error astring core,signal
svc:/network/http:apache2> delprop startd/ignore_error
svc:/network/http:apache2> listprop startd*
startd framework
svc:/network/http:apache2> exit
# svcadm refresh apache2
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Log of httpd refreshing
# tail -f /var/apache2/logs/error_log
[Wed Oct 26 12:55:30 2005] [notice] Graceful restart requested, doing restart
[Wed Oct 26 12:55:30 2005] [warn] (128)Network is unreachable: connect to listener
[Wed Oct 26 12:55:30 2005] [notice] Digest: generating secret for digest authentication ...
[Wed Oct 26 12:55:30 2005] [notice] Digest: done
[Wed Oct 26 12:55:31 2005] [notice] Apache/2.0.52 (Unix) DAV/2 configured -- resuming normal operations
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 12:55:30 Rereading configuration. ]
[ Oct 26 12:55:30 Executing refresh method ("/lib/svc/method/http-apache2 refresh") ]
[ Oct 26 12:55:30 Method "refresh" exited with status 0 ]
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
httpd without ignore_error# kill -9 29999 <-- started by root
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 13:13:02 Stopping because process received fatal signal from outside the service. ]
[ Oct 26 13:13:02 Executing stop method ("/lib/svc/method/http-apache2 stop") ]
httpd (pid 29999?) not running
[ Oct 26 13:13:02 Method "stop" exited with status 0 ]
[ Oct 26 13:14:02 Method or service exit timed out. Killing contract 720 ]
# svcs apache2
STATE STIME FMRI
maintenance 13:14:02 svc:/network/http:apache2
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Clearing and restarting
# svcadm clear apache2
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 13:16:04 Leaving maintenance because clear requested. ]
[ Oct 26 13:16:04 Enabled. ]
[ Oct 26 13:16:04 Executing start method ("/lib/svc/method/http-apache2 start") ]
[ Oct 26 13:16:05 Method "start" exited with status 0 ]
• What would happen if you kill any httpd processes started by webservd, now that you no longer have the ignore_error property?
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Logs
# tail -f /var/apache2/logs/error_log
[Wed Oct 26 13:24:23 2005] [notice] caught SIGTERM, shutting down
[Wed Oct 26 13:24:25 2005] [notice] Digest: generating secret for digest authentication ...
[Wed Oct 26 13:24:25 2005] [notice] Digest: done
[Wed Oct 26 13:24:26 2005] [notice] Apache/2.0.52 (Unix) DAV/2 configured -- resuming normal operations
• SMF stops and restarts httpd:apache because svc.startd will interpret any externally-generated signal as a failure of the service, and will execute the stop and start method to try and recover.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Logs
# tail -f /var/svc/log/network-http:apache2.log
[ Oct 26 13:24:23 Stopping because process received fatal signal from outside the service. ]
[ Oct 26 13:24:23 Executing stop method ("/lib/svc/method/http-apache2 stop") ]
[ Oct 26 13:24:23 Method "stop" exited with status 0 ]
[ Oct 26 13:24:24 Executing start method ("/lib/svc/method/http-apache2 start") ]
[ Oct 26 13:24:25 Method "start" exited with status 0 ]
• What's different though? Why isn't it in maintenance mode?
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Portable SMF
• Perhaps you have a large number of servers, or even zones, on which you'd like to imprint an identical set of services.• Well, this is not easy. This is “provisioning”, and SMF is a
service management tool.• You can't just “pack up” the database, or the XML dump
generated by svccfg archive, or all the manifests ... Why not? • What about the methods, the config files, the binaries,
everything else that makes up the apps?
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Duplicating service properties
• If, however, you have an application already installed, and you would like to transfer service properties (such as dependencies, the exec method, anything that might be defined by the manifest), you can do that by exporting and then importing a manifest.
# svccfg export svc:/network/http > apache2.export
• Move apache2.export to another system.# svccfg import apache2.export
• Note: the argument to svccfg export is a service, not an instance, although the XML file will contain definition of the instance(s).
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Duplicating service states
• You can get a profile, or an XML file that describes the state of all services by using svccfg extract.• This XML file can be used to duplicate the on|off state of
services from system to system. Use svccfg apply.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Profiles
• A profile is an XML description of all service instance states on a system.• Profiles are in /var/svc/profile.• Profiles are applied in order: generic, platform, site• Never modify existing profiles, use site.xml for local
customization.• An example of something that may create a use a profile
is the Solaris Security Toolkit 4.2 (aka JASS).
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Part V:SMF Tools and Community
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Webmin
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
About Webmin
• A web-based interface for UNIX system administration• It comes with Solaris 10, or get it at
http://www.webmin.com• Version 1.190 an
later supports SMF,even has a feature tocreate a new manifest.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SMF in Webmin
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Create new service
• This button will walk you through creating a new service manifest. • You don't need to know XML, but you do need to
understand what goes into a manifest.• Will ask you a series of questions such as:> Name of service> What are the dependencies,> Which is the delegated restarter> Service properties> Man pages
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
GUI Tools
• With Webmin you can monitor and manage SMF services through a browser.• A GUI interface to SMF is being developed.• Watch for a SunMC SMF module due near the end of this
year.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
More tools
• Use xmllint to check for errors in the manifest file without importing the manifest.• svccfg validate does something similar, without changing
the repository.• Jedit is a free text editor with XML with syntax checker
and validator. Comes with Solaris 10 (/usr/demo/jds/bin/jedit) or get it from http://www.jedit.org.• Morphon Technologies has a good and free XML editor,
but sadly no longer in development, get it from http://www.morphon.com.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SFM community on OpenSolaris
• http://www.opensolaris.org/os/community/smf/• Submit manifests or methods you've written here:
http://www.opensolaris.org/os/community/smf/manifests/• [email protected] migrating to
[email protected]• Subscribe here:
http://www.opensolaris.org/os/community/smf/discussions• Read it here:
http://www.opensolaris.org/jive/forum.jspa?forumID=24• File bugs here:• http://bugs.opensolaris.org/
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Resources• Internal> Greenline project page: http://greenline.eng> greenline-interest alias.
• External> BigAdmin: http://www.sun.com/bigadmin> Especially the Service Developer Intro athttp://www.sun.com/bigadmin/content/selfheal/sdev_intro.html> SMF community: http://opensolaris.org/os/community/smf/> SMF mailing list: [email protected]
• Blogs> Liane Praza's blog: http://blogs.sun.com/roller/page/lianep> Stephen Hahn's blog: http://blogs.sun.com/sch
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Review
•
•
•
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Questions?
•
•
•
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
And lastly ...
Thank you!
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
References
• PresentationsL. Praza. smf(5): Service Management Facility.R. Manus, G. Hiregoudar. Solaris Management Framework – Workshop.
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
Changes
• Last modified: 03/10/2005
Sun Proprietary/Use by Permission Only
SOLARIS 10 DEEP DIVE: SMF
SOLARIS 10DEEP DIVE: SMF
US Solaris Adoption Practice
Sun Microsystems