smiii-103 configuration management (marimba) deployment
DESCRIPTION
testTRANSCRIPT
9/3/2006
BMC Configuration Management (Marimba) Best Practices and Troubleshooting
Andy SantosaSenior Technical Support Analyst
9/3/2006 ©2006 BMC Software2
Agenda
› CM Infrastructure› CM Inventory› CM Subscription› CM Software Distribution (App. Packager and Content Replicator)› CM Deployment Manager
9/3/2006 ©2006 BMC Software3
CM Infrastructure
› What’s New in Tuner 7.0?› What’s New in Transmitter 7.0?
9/3/2006 ©2006 BMC Software4
What’s New in Tuner 7.0?
› Status Report– Returns general information (for example: tuner version, release date, VM version,
heap-size, arguments, and operating system information) and checks if the tuner is running
• http://hostname:7717/workspace?status
› Debug Report– Returns specific information and checks for problems with the tuner (for example:
tuner properties, thread dump information, license information, RPC configuration and connections)
• http://hostname:7717/workspace?debug&<option>=t includes the option&<option>=f excludes the option
• http://hostname:7717/?debug&config=t• http://hostname:7717/?debug&threaddump=t
› Log Report– Returns the tuner and channel history logs
• http://hostname:7717/workspace?log• http://hostname:7717/workspace/http://hostname:5282/Marimba/Current/SubscriptionService?log
9/3/2006 ©2006 BMC Software5
Status Report
9/3/2006 ©2006 BMC Software6
What’s New in Tuner 7.0? (Cont’d)
› Thread Dump– You can get thread dump information remotely from both client endpoints and
servers.• runchannel http://hostname:5282/Version7/TunerAdministrator -getthreaddump -tuner hostname:7717
-username admin -password "" -output C:\Temp\threaddump.txt
– If you want to generate thread dumps remotely, the Java VM must be launched with the –Xdebug Option. • On servers (transmitters, proxies, etc.), profiles do not include the -Xdebug
option for performance reasons.
› Minimal Mode– Minimal mode is now a separate process known as minituner.exe and is no longer
part of the tuner. This makes minimal mode more robust and the tuner more reliable.
9/3/2006 ©2006 BMC Software7
What’s New in Tuner 7.0? (Cont’d)
› Improved Robustness and Stability– IPC service switched from DDE to named pipe.
• Comparing to DDE, named pipe provides a more efficient and generic bidirectional communication mechanism. It does not require applications to be GUI-based, and data exchanged can be in arbitrary format.
› Receipt Service– BMC® Remedy® Change Management could initiate a change request/task on a
Marimba Tuner endpoint through Deployment Manager and Policy Manager.Tuner is to provide a receipt service so as to leave a meaningful “audit trail” for the work performed on an endpoint.
– Receipts are stored in the tuner workspace.
9/3/2006 ©2006 BMC Software8
What’s New in Transmitter 7.0?
› More robust Transmitter storage› Detection of ungraceful Transmitter shutdowns› Tx Verify improvements› New debug flags:
– Tuner• TUNER/RECEIPT
– Transmitter• TX/GC • TX/DIFF• TX/DSL• TX/HTTP
9/3/2006 ©2006 BMC Software9
Infrastructure - Recommendations
› Separate CMS, Transmitters and Windows Patch Source
› Java Launch Arguments:– marimba.launch.javaArgs=-Xms128m -Xmx512m -XX:PermSize=32m -
XX:MaxPermSize=128m
› Using Profile Property– If the tuner profile property doesn’t exist -> ONLY Kernel Upgrade– If the tuner profile property exists, but Transmitter doesn’t have the profile segment -
> No Kernel Upgrade– If the tuner profile property exists, but Transmitter has the profile segment -> Kernel
upgrade + Profile applied
› Health Check Tools– Replication Monitor– Tx Verify– Transmitter Guardian
9/3/2006 ©2006 BMC Software10
Troubleshooting - Scheduler
› Update schedule vs. start schedule– Subscription/Policy Service -> update.schedule– Patch Service -> update.schedule– Inventory/Scanner Service -> start.schedule– Infrastructure Service -> start.schedule
› Missed update/start schedule while the machine was off the network/turned offExample:– Subscription update schedule at 2 AM– Inventory start schedule at 3 AM– Marimba schedule start delay=10000
› Schedule dumper to help debugging scheduler issue
9/3/2006 ©2006 BMC Software11
CM Inventory
› What’s New in Inventory 7.0? › Performance› Debugging
9/3/2006 ©2006 BMC Software12
What’s New in Inventory 7.0?
› In 6.x, compliance will ALWAYS re-send all data (even when no data changes between scans). This results in an expensive delete of all compliance data followed by a re-insert of all data. This was fixed in 7.0.
› Using AMT Data As mac_id– Intel AMT data is also now part of Intel inventory scans. Identify if machine is AMT-enabled
and fetch persistent Tuner ID from AMT chip.
› New Diffing Logic– Maintain a full report cache on the endpoint. – Diff against the current inventory data in the DB if there is a checksum mismatch.– Purge full report cache based on size and/or time interval.
› Double-Byte Storage
› Mirror Forwarding– Mirrors on WAN can forward Inventory reports to LAN Mirrors/Master – Reduce the chances of database blocking or deadlocks
› Task Receipts (CCM)– Scan and report task receipts
9/3/2006 ©2006 BMC Software13
What’s New in Inventory 7.0? (Cont’d)
› Old 6.X Diff Over-Write That Resulted in Full Scan Report› [15/May/2006:22:23:33 -0400] - warning - 6672 Over-writing older report for machine: wuscxdw221(win32:3888154505819947868)
Diff 1
Inventory Service Plugin
DBDiff 1
Diff 2
Inventory Service Plugin
DBDiff 1
Disk Queue:Diff 2
Diff 3
Inventory Service Plugin
DBDiff 1
Disk Queue:Diff 2Diff 3`
Endpoint
1
2
3
Diff Checksum Mismatch: Diff Scan Old Checksum is
for Diff 2, Plugin Local Checksum Cache has Diff 1.
Request Full Scan report from Endpoint
4
Endpoint sends Full Scan Report Full
Scan Report
Inventory Service Plugin
DBDelete AllInsert All
Disk Queue:
5
9/3/2006 ©2006 BMC Software14
What’s New in Inventory 7.0? (Cont’d)
› 7.0 Special Diff Scan Report (Instead of 6.x Full Report)– [15/May/2006:22:23:33 -0400] - warning - 6672 Over-writing older report for machine: wuscxdw221(win32:3888154505819947868)
Diff 1
Inventory Service Plugin
DBDiff 1
Diff 2
Inventory Service Plugin
DBDiff 1
Disk Queue:Diff 2
Diff 3
Inventory Service Plugin
DBDiff 1
Disk Queue:Diff 2Diff 3`
Endpoint
1
2
3
Diff Checksum Mismatch: Diff Scan Old Checksum is
for Diff 2, Plugin Local Checksum Cache has Diff 1. Request Special Diff report
from Endpoint
4
Endpoint generates new special diff scan report
between the full report at Diff 1 state and the full
report at the Diff 3 state and resends back to the plugin
Diff 3'
Inventory Service Plugin
DBDiff 3'
Disk Queue:
5
9/3/2006 ©2006 BMC Software15
Performance
› SQL Server – Report Center queries are basically unusable while inventory insertions are running in the background.
› Highly recommend enabling “dirty reads” if inventory insertions need to be run while Report Center queries take place. Reason:
• Inserts/Updates/Deletes issue an exclusive lock• Select statements issue a shared lock• A shared lock cannot be issued when an exclusive lock is in place and an
exclusive lock cannot be issued when a shared lock is in place• You can change the behavior of locking via a lock hint:
- Nolock - Enables dirty reads where a Select does NOT issue a shared lockand Select does NOT honor exclusive locks
- 6.x – Only Report Center interactive queries can use dirty reads (enabled through property)
- 7.0 – Report Center interactive queries, email reports, collections, and processing queries that another application requests can use dirty reads (enabled through UI)
9/3/2006 ©2006 BMC Software16
Performance (Cont’d)
› Inventory Plugin Tuning– Set the scan schedule as far apart as possible (24 hours, if possible).– Use the scheduling “vary” option to reduce the load all insert at the same time load
(due to time sync’d machines).– 3 to 5 inventory plugin Oracle database connections appears to be optimal. More
connections just result in more resources and waiting instead of increasing throughput.
– Disk I/O is low.– CPU utilization is low. – Memory is low.– Java GC tuning is lower priority since currently GC’s / heap size growth does not
appear to be causing issues.
9/3/2006 ©2006 BMC Software17
Debugging Customer Performance Issues
› Guide to Debugging Customer Performance Issues– Get inventory plugin logs from all masters/mirrors/repeaters.– Gather a few sample Endpoint Inventory Service channel workspaces.– Oracle redo logs and/or archive logs (if archive log mode is enabled).
• If gathering archive logs, also request character sets, dictionary file, version/platform Oracle is running on.
– Tuner/History logs where inventory plugins running.– System Architecture – Scan schedule, # endpoints, # repeaters, # mirrors, load
balancer, which plugins insert directly into database, etc. – Gather any queued-up scan reports in the inventory plugin disk queue.– Find out if the scan schedule is too aggressive and check if it is possible to reduce
the scan schedules.– In more complex environments, see if client IP Load Balancer persistence might
help reducing checksum mismatches.
9/3/2006 ©2006 BMC Software18
CM Subscription
› What’s New in Subscription 7.0?
9/3/2006 ©2006 BMC Software19
What’s New in Subscription 7.0?
› Install Priority • You can now set install priority values by typing numeric values per package on the
Edit Policy page.• In M6, you can only scroll arrow up and down to change the install priority.
› User-Based/Machine-Based Policies• Policy will be assigned based on either user-based, machine-based, or both. In M6, the
only option was both.• Property: ‘marimba.subscriptionplugin.resolvetype’ Value: user or machine, otherwise
both.
› Directory Service Hierarchy Structure
• Policy Manager now uses an updated LDAP container structure.
• Enables you to more effectively extend the schema of the Active Directory
infrastructure, and store and secure policies.
9/3/2006 ©2006 BMC Software20
What’s New in Subscription 7.0? (Cont’d)
• Supports install order • Supports staging new url before doing an autoupdatefrom• Will not updatefrom if marimba.subscription.nodelete=true
• Will revert url if channel fails to install
9/3/2006 ©2006 BMC Software21
Compliance – Architectural Overview
Inventory Plug-In• Implements differencing for compliance. It only inserts compliance data that
has been modified from the previous inventory scan on the endpoint.
Subscription Service• Invoked as a custom scanner by Inventory Service to calculate compliance
on the endpoint.
Query-Based ComplianceTwo types of queries in compliance engine:
1. Inventory-only query; query based solely on inventory scan data2. Latest-data query; query based on inventory, LDAP-synced policy and
Transmitter
9/3/2006 ©2006 BMC Software22
Compliance (Cont’d)
› Scheduling LDAP Sync• The LDAP synchronization process is new in this release.• LDAP synchronization optimizes the accuracy of compliance queries and reports by
updating the configuration database with the latest available policy data.
› To optimize policy management workflow, schedule the following processes in sequence:• Update policy service• Perform inventory scans• Schedule LDAP Sync• Typically, you schedule the processes on a daily basis • You can schedule LDAP Sync using the LDAP-to-Database Synchronization Service link
on the Data Source tab in CMS System Settings
› Compliance Reporting• In addition to viewing policy compliance information based on targets, you can now view
compliance based on a specified package.
9/3/2006 ©2006 BMC Software23
New Features: Compliance (Cont’d)
9/3/2006 ©2006 BMC Software24
CM Application Packager/Content Replicator
› What’s New in Application Packager 7.0?› What’s New in Content Replicator 7.0?
9/3/2006 ©2006 BMC Software25
What’s New in Application Packager 7.0?
› Support for MSI 3.0– No need for ‘User Elevation’ when installing MSI packages
› Delaying the download of MSI package until pre-scripts are run› Repair of MSI now shows progress bar› Return code mapping› App-friendly name› DSL support
9/3/2006 ©2006 BMC Software26
What’s New in Content Replicator 7.0?
› Packaging and Installation of Content From– Existing
• 1 source to 1 target folder
– N source to N target folders› Installing Unix Packages
– Using XML Command file to specify packaging commands• {5.1 screen shot}
› Self-Installing Channels– New “package” command that will facilitate installation of self-installing channels
• {5.2 Screen shot with syntax}› Channel Signing (Credit Suisse)
9/3/2006 ©2006 BMC Software27
What’s New in Content Replicator 7.0? (Cont’d)
› No More Auxiliary Channels; Now Segments» (Segments also, only for n source - n target use case)
9/3/2006 ©2006 BMC Software28
CM Deployment Manager
› What’s New in DM 7.0?
9/3/2006 ©2006 BMC Software29
What’s New in DM 7.0?
› Main Features (Architecture Changes)– Database-backed workspace– Improved DS->DM communication– Improved handling of log data– Report Center integration
9/3/2006 ©2006 BMC Software30
Previous Workspace Architecture
› File System-Based– Object-tree is stored on disk in File System hierarchy– Each object consists of a directory with several files– Entire object-tree is loaded up and kept in memory at startup
› Problems– Keeping all nodes in memory doesn’t scale
• Start-up time grows linear to workspace size• Workspace size is restricted by available memory
– Hard to: • Query data• Share data with other applications• Maintain referential integrity• Guarantee safety of the data
9/3/2006 ©2006 BMC Software31
New Workspace Architecture
› Database-Backed– Object-tree is stored in a database under DBTree
• Simple– Similar to what DM uses today (PTree). Easy to learn and replace.
• Yet powerful– Lazy loading: In-memory caching and batching of object-updates, etc.
– An object is a DBTree dir-node• All common files are merged into a set of node-properties.• Other data stored as child-nodes.• Exception: Log data is stored in the logging schema.
9/3/2006 ©2006 BMC Software32
DB-Backed Workspace
› How Does This Solve The Problems?– Workspace-size scalability
• Tree no longer needs to be traversed at start up; start up time will be minimal• Number of objects no longer limited by amount of memory
– Querying the data• Logging data is stored in logging schema; querying will be easy• Object data is stored in DBTree-schema; querying is possible but somewhat
complicated– Sharing data
• All data is in the database– Referential integrity
• Yes, we can guarantee it. – Safety of data
• Increases; databases are built to be safe datastores.
9/3/2006 ©2006 BMC Software33
DS->DM Communication
› Problems– DS communicates too frequently– Protocol is inefficient and simplistic– DS queue logs in memory
› Solutions– Tighter control of DS communication
• Intelligent flushing of queue– New, efficient protocol
• Binary • Support for compression• Support for new client commands (re-try later, etc.)
– Better use of HTTP• Be efficient about keep-alives (partly done in 603S releases)
9/3/2006 ©2006 BMC Software34
Handling of Log Data
› Problems– DM does “on-the-fly” data processing (CPU-intensive)– Incoming data is immediately processed– UI becomes unusable when many clients upload
› Solutions– Incoming data is placed in disk-based queue– Limited set of processor threads read off of queue– We get in control of CPU resources used for log-processing vs. UI– May introduce latency in respect to reporting under high load
• This should be acceptable since the alternative is an unusable UI
9/3/2006 ©2006 BMC Software35
What’s New in Deployment Reports?
› Leverage The Fact That Data Is In The DB – Use Report Center for building reports
• Separate schema for reporting• Near real-time reports (for deployment status)• After-the-fact reports
– Reports to be available in DM UI• RC query-tree is accessed remotely• Report can be selected from DM UI
9/3/2006 ©2006 BMC Software36
Q & A