bravepoint progress system tables dan foreman bravepoint, inc. email: [email protected]
TRANSCRIPT
BravePoint
Progress System Progress System TablesTables
Dan ForemanDan Foreman
BravePoint, Inc.BravePoint, Inc.
Email: [email protected]: [email protected]
BravePoint
Introduction - Dan ForemanIntroduction - Dan Foreman• Progress User since 1984 (V2.1)Progress User since 1984 (V2.1)• Presenter at lots of Progress Presenter at lots of Progress
conferences dating back to 1990conferences dating back to 1990
BravePoint
Introduction - Dan ForemanIntroduction - Dan ForemanAuthor of:Author of:• Progress Performance Tuning GuideProgress Performance Tuning Guide• Progress Database Admin GuideProgress Database Admin Guide• Progress Virtual System TablesProgress Virtual System Tables• Pro D&LPro D&L - Dump/Load with short down time no - Dump/Load with short down time no
matter how large the Database ismatter how large the Database is• ProMonitorProMonitor – DB monitoring tool (uses VSTs) – DB monitoring tool (uses VSTs)• ProCheckProCheck – Monitor AppServers & WebSpeed – Monitor AppServers & WebSpeed
BravePoint
What are Virtual System TablesWhat are Virtual System Tables
Virtual System tables (VSTs) provide Virtual System tables (VSTs) provide access to Database Shared Memory data access to Database Shared Memory data that was previously only accessible that was previously only accessible through through promonpromonIt’s possible to customize the user It’s possible to customize the user interfaceinterfaceAccessible Locally or Remotely via the Accessible Locally or Remotely via the 4GL or SQL4GL or SQL
BravePoint
What are Virtual System TablesWhat are Virtual System Tables
VSTs are Schema TablesVSTs are Schema TablesThey do not interfere with Application tables and IndexesVSTs are not part of the empty Database in V8Enable with proutil <db> -C enablevstAre part of the V9/V10/V11 empty DatabaseAre visible in the Dictionary (must view ‘Hidden’ tables)
What are Virtual System TablesWhat are Virtual System Tables
Some new tables and fields are not in Some new tables and fields are not in the “native” empty DB unless:the “native” empty DB unless:
proutil –C updatevst is run
Example: New VST fields added in V10.2B SP6
The feature must be enabled:Auditing: proutil enableauditing
TDE
Multi-Tenancy
BravePoint
BravePoint
What are Virtual System TablesWhat are Virtual System Tables
VSTs in the Standard empty DBVSTs in the Standard empty DBV8.2V8.2 30 tables30 tablesV8.3V8.3 35 tables35 tablesV9.0-V9.1CV9.0-V9.1C 39 tables39 tablesV9.1DV9.1D 42 tables42 tablesV10V10 45 tables45 tablesV10.1BV10.1B 47 tables47 tables
What are Virtual System TablesWhat are Virtual System Tablespromonpromon can lag behind VSTs; can lag behind VSTs; Examples:Examples:
_TableStat & _IndexStat (V8.3)_AreaStatus (V9 and later)
Conversely VSTs can lag behind also:Conversely VSTs can lag behind also:_Startup
BravePoint
BravePoint
Resource Impact of VSTsResource Impact of VSTsThere is no performance penalty for accessing There is no performance penalty for accessing VSTs with 2 exceptions that will be covered VSTs with 2 exceptions that will be covered laterlaterVSTs take up no additional disk space except VSTs take up no additional disk space except for the VST schemafor the VST schemaVSTs take up a small amount of memory to VSTs take up a small amount of memory to cache the additional schemacache the additional schema
Except –tablerangesize & -indexrangesize can increase DB Shared Memory by a surprising amount
BravePoint
VST ImplementationVST Implementation
Whenever a record is requested, the Whenever a record is requested, the table is checked to see if it’s a VST table is checked to see if it’s a VST table.table.
VST tables have table numbers between -16385 to -32768.The check is simple arithmetic which takes place regardless of whether VSTs have been enabled or not
BravePoint
VST ImplementationVST Implementation
If a requested record is a VST record, If a requested record is a VST record, the fields are populated from shared the fields are populated from shared memory structures instead of from a memory structures instead of from a database recorddatabase record
BravePoint
VST BasicsVST BasicsEach VST has one, unique index on the Each VST has one, unique index on the ‘Id’ field which is a ‘virtual’ RECID ‘Id’ field which is a ‘virtual’ RECID Each VST record is numbered starting Each VST record is numbered starting at 1 and continues sequentially (some at 1 and continues sequentially (some exceptions)exceptions)It is not possible to add new indexes or It is not possible to add new indexes or triggers to VSTstriggers to VSTs
BravePoint
VST BasicsVST BasicsSome VST Record Counts are controlled by startup Some VST Record Counts are controlled by startup parametersparameters_connect_connect -n & -Mn-n & -Mn_LockReq_LockReq -n & -Mn-n & -Mn_UserIO_UserIO -n & -Mn-n & -Mn_UserLock_UserLock -n & -Mn-n & -Mn_UserStatus_UserStatus -n & -Mn-n & -Mn_Trans_Trans -n-n_Servers_Servers -Mn-Mn_Lock_Lock -L-L
BravePoint
VST BasicsVST BasicsThe tables on the last slide are pre-populated The tables on the last slide are pre-populated at Database Broker startup timeat Database Broker startup timeThe empty slots have “?” valuesThe empty slots have “?” valuesTo avoid displaying the empty slots:To avoid displaying the empty slots:FOR EACH _connect FOR EACH _connect
WHERE _connect-usr <> ?WHERE _connect-usr <> ?The <> operator is not a performance The <> operator is not a performance problem since VSTs are not really ‘indexed’problem since VSTs are not really ‘indexed’
BravePoint
Which DatabaseWhich Database
If a Client is connected to multiple If a Client is connected to multiple Databases, the VST Statistics will be Databases, the VST Statistics will be based upon the Current Working based upon the Current Working DatabaseDatabase
BravePoint
Updatable VSTsUpdatable VSTs
_Startup-spin_Startup-spin -spin-spin
_Startup-APWQTime_Startup-APWQTime -pwqdelay-pwqdelay
_Startup-APWSTime_Startup-APWSTime -pwsdelay-pwsdelay
_Startup-APWBuffs_Startup-APWBuffs -pwscan-pwscan
_Startup-APWMaxWrites_Startup-APWMaxWrites -pwwmax-pwwmax
_Startup-_Startup-
Updateable in V10.2B SP6Updateable in V10.2B SP6
_Startup._Startup-LRU-Skips_Startup._Startup-LRU-Skips
_Startup._Startup-LRU2-Skips_Startup._Startup-LRU2-Skips
_Startup._Startup-NmsgWait_Startup._Startup-NmsgWait
_Startup._Startup-Pefetch-Delay_Startup._Startup-Pefetch-Delay
_Startup._Startup-Prefetch-Factor_Startup._Startup-Prefetch-Factor
_Startup._Startup-Prefetch-Num-Recs_Startup._Startup-Prefetch-Num-Recs
_Startup._Startup-Prefetch-Priority_Startup._Startup-Prefetch-Priority
BravePoint
BravePoint
Updatable VSTsUpdatable VSTs
_MyConnect-NumSeqBuffers_MyConnect-NumSeqBuffers
Added in V9Added in V9
Same as –Bp parameterSame as –Bp parameter
Number of Sequential Read BuffersNumber of Sequential Read Buffers
Makes it possible to run a large report Makes it possible to run a large report with a less severe impact on the -B with a less severe impact on the -B Buffer CacheBuffer Cache
Example on the next slideExample on the next slide
BravePoint
Updatable VSTsUpdatable VSTs
DO FOR _MyConnect TRANSACTION :DO FOR _MyConnect TRANSACTION :
FIND FIRST _MyConnect.FIND FIRST _MyConnect.
_myconnect-numseqbuffers = 32._myconnect-numseqbuffers = 32.
END.END.
RUN killer_report.p.RUN killer_report.p.
DO FOR _MyConnect TRANSACTION : DO FOR _MyConnect TRANSACTION :
FIND FIRST _MyConnect.FIND FIRST _MyConnect.
_myconnect-numseqbuffers = 0._myconnect-numseqbuffers = 0.
END.END.
BravePoint
Updatable VSTsUpdatable VSTs
_StatBase._TableBase_StatBase._TableBase_StatBase._IndexBase_StatBase._IndexBaseV9 and laterV9 and laterUsed to set the window size of Tables and Used to set the window size of Tables and Indexes monitored using the _TableStat Indexes monitored using the _TableStat and _IndexStat VSTs or to reset (zero out) and _IndexStat VSTs or to reset (zero out) the statistics by moving the “window” up the statistics by moving the “window” up and then downand then down
BravePoint
Table & Index I/O VSTsTable & Index I/O VSTs
Note that _TableStat and _IndexStat Note that _TableStat and _IndexStat are available in V8.3 and very helpful are available in V8.3 and very helpful for Storage Area Planningfor Storage Area PlanningUse _IndexStat to identify unused Use _IndexStat to identify unused indexes; COMPILE/XREF is only good indexes; COMPILE/XREF is only good for finding indexes that are defined but for finding indexes that are defined but not used in the code; COMPILE/XREF not used in the code; COMPILE/XREF is useless for Dynamic Queriesis useless for Dynamic Queries
BravePoint
VST ProblemsVST Problems
Potential Performance KillersPotential Performance Killers_Block ( V8, useless in V9/V10/V11 )
_Lock ( FOR EACH _lock WHILE … )
_UserLock ( DO loops of 512 iterations )
Too Much Data for One ScreenToo Much Data for One Screen_Block-block x(1024)
_UserLock-* Several arrays of 512
Missing DataMissing Data_DBStatus
BravePoint
VST ProblemsVST ProblemsStorageStorage
VSTs are defined as INTEGERs which limits the maximum value to about 2 billion – prior to V10.1B
Poorly documented (see my Poorly documented (see my System Table System Table GuideGuide for better documentation) for better documentation)Stale DataStale Data
_Lock (total # of records) compared to _DbStatus-NumLocks
System Tables - MetaSchemaSystem Tables - MetaSchema
_File_File
_Field_Field
_Index_Index
_Index-Field_Index-Field
_File-Trig_File-Trig
_Field-Trig_Field-Trig
_Db_Db
_Sequence_Sequence
BravePoint
System Tables - OtherSystem Tables - Other
SQLSQLAuthentication (GRANTs & REVOKEs)
Views
Constraints
Query Optimizer statistics
AuditingAuditing
EncryptionEncryption
Multi-TenancyMulti-Tenancy
BravePoint
BravePoint
Usefulness Usefulness
VSTs are useful for two types of VSTs are useful for two types of Progress usersProgress users
Application Developers
Database Administrators
Who else is there?
BravePoint
For DevelopersFor Developers
Can view the lock status for an individual Can view the lock status for an individual recordrecord_Lock VST shows all active locks _Lock VST shows all active locks andand who who has the lockhas the lockWe can easily find the Table name of the We can easily find the Table name of the record being locked by doing a join to the record being locked by doing a join to the _File table_File table_Trans VST shows all active transactions_Trans VST shows all active transactions
BravePoint
For DevelopersFor DevelopersDeadly Embrace (Deadlock) DetectionDeadly Embrace (Deadlock) Detection
_Lock_Connect
““Delinquent” Active Transactions that Delinquent” Active Transactions that grow the BI File to an abnormally large grow the BI File to an abnormally large sizesize
_Connect_Trans
BravePoint
For DevelopersFor Developers
Improve code efficiency by measuringImprove code efficiency by measuring::User DB Accesses, Reads, etc.
Reads, Writes, Creates, Deletes by Table_TableStat
Unused indexes; Logical Scatter Factor_IndexStat
For DevelopersFor Developers
Name & attributes of a FieldName & attributes of a Field
find _file “customer” no-lock.find _file “customer” no-lock.
for each _field of _file no-lock :for each _field of _file no-lock :
displaydisplay
_field-name_field-name
_data-type_data-type
_format_format
_extent._extent.
end.end.
BravePoint
BravePoint
DBAs - Tables of interestDBAs - Tables of interest
Monitoring the BI High Water MarkMonitoring the BI High Water Mark_DbStatus._dbstatus-bisize
Available in V8.3 and later
V8: BI Size in Bytes
V9 & later: BI Size divided by BI Block Size
BravePoint
DBAs - Tables of interestDBAs - Tables of interestMonitoring Database SizeMonitoring Database Size
promon doesn’t have the correct data
Each Storage Area has a separate High Water Each Storage Area has a separate High Water MarkMark
_AreaStatus_AreaStatus-AreaName_AreaStatus-HiWater_AreaStatus-TotBlocks_AreaStatus-LastExtent_AreaStatus-Extents
BravePoint
DBAs - Tables of InterestDBAs - Tables of InterestRecord FragmentationRecord Fragmentation
_Record._Record-RecRead
_Record._Record-FragRead
dbanalys/tabanalysdbanalys/tabanalys only shows only shows how many Fragments exist, not how many Fragments exist, not how often the Fragments are how often the Fragments are accessedaccessed
DBAs - Tables of InterestDBAs - Tables of InterestWhat Client is running what ProgramWhat Client is running what Program
_Connect-Cache*
Also Line# for 4GL Code
Or SQL Statement
Since V10.1C
What Tables is a Client HittingWhat Tables is a Client Hitting_UserTableStat
Since V10.1B
BravePoint
BravePoint
For More InformationFor More Information
Progress System Tables GuideProgress System Tables Guide200+ pages
Covers V8 through V11
Covers VSTs and System Tables
Has lots of undocumented information
BravePoint
ProMonitorProMonitorWritten using (mostly) VSTsWritten using (mostly) VSTsSource Code provided for most ProgramsSource Code provided for most ProgramsThreshold Alerts warn of problemsThreshold Alerts warn of problemsMonitor an unlimited number of DBsMonitor an unlimited number of DBsMonitors more than just VST StatisticsMonitors more than just VST Statistics
Session Temp File I/OClient Memory UtilizationDB Log File MonitoringDB Fragmentation/Scatter MonitoringAppServers, WebSpeed, AdminServer, etc