Bringing Internals to Bringing Internals to the Surface the Surface
Internals Concepts for Highly Effective
Engines
Mark ScrantonPrincipal Consultant/Trainer
Informix Denverwww.markscranton.com
2
useruser .conference.conferenceInformixInformix
Who Am I?Who Am I?
• Principal Consultant/Trainer, Informix Denver
• I do engines - XPS and IDS
• 5+ yrs with Informix
• website - www.markscranton.com• tips/tricks• sql/scripts• documents, presentations
• Principal Consultant/Trainer, Informix Denver
• I do engines - XPS and IDS
• 5+ yrs with Informix
• website - www.markscranton.com• tips/tricks• sql/scripts• documents, presentations
3
useruser .conference.conferenceInformixInformix
Presentation OverviewPresentation Overview
• This presentation will identify “internals” concepts that appear to be trivia
• But they could be useful daily for maintenance and support of IDS
• A strong working knowledge of IDS is assumed
• More info available in the “IDS Internal Architecture” class
• This presentation will identify “internals” concepts that appear to be trivia
• But they could be useful daily for maintenance and support of IDS
• A strong working knowledge of IDS is assumed
• More info available in the “IDS Internal Architecture” class
4
useruser .conference.conferenceInformixInformix
TopicsTopics
• Physical pages vs. logical pages? • Discussion about the value of knowing
the difference
• The dog has swallowed it’s tail • ...or the tale of physical log overflow
• PARTNUMs - and the story of getting to data• We’ve all seen them…what does it really mean?
• Physical pages vs. logical pages? • Discussion about the value of knowing
the difference
• The dog has swallowed it’s tail • ...or the tale of physical log overflow
• PARTNUMs - and the story of getting to data• We’ve all seen them…what does it really mean?
5
useruser .conference.conferenceInformixInformix
TopicsTopics
• The maximum number of extents for a table? • Is there REALLY a max?
• Where did my row go?• The long tale of the forward pointer
• The last topic
• The maximum number of extents for a table? • Is there REALLY a max?
• Where did my row go?• The long tale of the forward pointer
• The last topic
6
useruser .conference.conferenceInformixInformix
Topic 1: Logical Pages vs. Physical PagesTopic 1: Logical Pages vs. Physical Pages
• Understanding the difference can help in:• Interpretation of msg log assertion failures• Interpretation of the extent list of a partition page• Lock level identification with respect to rowid(s)• Oncheck usage• A bunch of other stuff
• Understanding the difference can help in:• Interpretation of msg log assertion failures• Interpretation of the extent list of a partition page• Lock level identification with respect to rowid(s)• Oncheck usage• A bunch of other stuff
7
useruser .conference.conferenceInformixInformix
Physical PagesPhysical Pages
• Always with respect to a chunk
• Numbering starts with 0
• Format:
0xCCCPPPPP
ccc - chunk numberppppp - page offset into the chunk
• Always with respect to a chunk
• Numbering starts with 0
• Format:
0xCCCPPPPP
ccc - chunk numberppppp - page offset into the chunk
8
useruser .conference.conferenceInformixInformix
Physical PagesPhysical Pages
30023c 30023d
300243300242300241300240
30023f30023e
300247300246300245300244
Chunk 3
0xCCCPPPPPc - chunk #p - page offset
9
useruser .conference.conferenceInformixInformix
Logical PagesLogical Pages
• Always with respect to a tablespace
• Numbering starts with 0
• Format simply 0 through x
• Always with respect to a tablespace
• Numbering starts with 0
• Format simply 0 through x
Logical PagesLogical Pages
30023c 30023d
300243300242300241300240
30023f30023e
300247300246300245300244
extent 1 for sparky
0
54
21
3
logicalpage num
create tablesparky...
physicalpage num
11
useruser .conference.conferenceInformixInformix
Usage Examples - Extent ListUsage Examples - Extent List
extent list on a partition page (oncheck -pt skippy:sparky)
chunk 0x4 page 0x33
12
useruser .conference.conferenceInformixInformix
Usage Examples - Bad PageUsage Examples - Bad Page
13
useruser .conference.conferenceInformixInformix
Usage Examples - Lock LevelUsage Examples - Lock Level
onstat -k
rowid format: 0xLLLLLLSSL - logical page in the tableS - slot/row # on the page
oncheck -pp 0x100016 2
14
useruser .conference.conferenceInformixInformix
Topic 2: Physical Log OverflowTopic 2: Physical Log Overflow
Physical Log File
1st before images of modified cache pages
75% full: checkpoint request is raised
what happens now?
back to the beginning, and keep writing!
15
useruser .conference.conferenceInformixInformix
Physical Log OverflowPhysical Log Overflow
Physical Log File
75% full: checkpoint request is raised
• Two potential scenarios:• the chkpt completes, and we give the physical log a
new logical beginning. In that case, we don’t care about the overwritten pages
• if the engine falls over, you’ll most likely get stuck in fast recovery when trying to come back up
• Two potential scenarios:• the chkpt completes, and we give the physical log a
new logical beginning. In that case, we don’t care about the overwritten pages
• if the engine falls over, you’ll most likely get stuck in fast recovery when trying to come back up
16
useruser .conference.conferenceInformixInformix
Topic 3: PartnumsTopic 3: Partnums
• Three areas focused on here:• What is a partnum? • How do we use it to get to stuff• What is a tablespace tablespace?
• Three areas focused on here:• What is a partnum? • How do we use it to get to stuff• What is a tablespace tablespace?
17
useruser .conference.conferenceInformixInformix
DefinitionsDefinitions
• Tablespace tablespace• Keeps track of tablespaces, or partitions,
in a dbspace• Each dbspace has one• Typical allocation is 50 pages for each extent
• Partition page• A page in a tablespace tablespace• Typically one page per tablespace• It “describes” the tablespace
• Tablespace tablespace• Keeps track of tablespaces, or partitions,
in a dbspace• Each dbspace has one• Typical allocation is 50 pages for each extent
• Partition page• A page in a tablespace tablespace• Typically one page per tablespace• It “describes” the tablespace
18
useruser .conference.conferenceInformixInformix
partnumpartnum
D D D L L L L L
dbspace number(starting at 1)
Logical page numwithin the tablespace tablespace (starting at 0)
1 1/2 bytes(3 nibbles, or 3 hex digits)
2 1/2 bytes(5 nibbles, or5 hex digits)
4 bytes
0x
19
useruser .conference.conferenceInformixInformix
dbspace4
rootdbs
systablesname partnumdatabase
sparky
0020000311-11-1999informixsparky
database tablespace
partnumflagscreatedownername
0020003cmudtable mud
extentsfor mud
tablespace tablespace page 3c
CREATE DATABASE sparky IN dbspace4;CREATE TABLE mud;
DATABASE sparky;SELECT * FROM mud;
20
useruser .conference.conferenceInformixInformix
Topic 4: Max Number of ExtentsTopic 4: Max Number of Extents
• What’s the maximum number of extents for a tablespace?
• Consider the previous topic, and the use of the partition page for a tablespace
• What’s the maximum number of extents for a tablespace?
• Consider the previous topic, and the use of the partition page for a tablespace
21
useruser .conference.conferenceInformixInformix
The Partition PageThe Partition Page
22
useruser .conference.conferenceInformixInformix
The Partition PageThe Partition Page
oncheck -pt skippy:sparky
23
useruser .conference.conferenceInformixInformix
Oh by the Way...Oh by the Way...
• Do you really want to know the max?
• Fragmentation changes the max for a logical table
• 4K vs. 2K pages
• Do you really want to know the max?
• Fragmentation changes the max for a logical table
• 4K vs. 2K pages
24
useruser .conference.conferenceInformixInformix
Topic 5: Forward PointersTopic 5: Forward Pointers
• Forward pointer - a 4-byte number that “points” to the remainder pieces of a row that spans pages
• Used with rows longer than a page
• Format is same as a rowid, although it’s NOT the rowid for the row
• Used with varchars or really long rows (rows longer than a page typically)
• We’ll focus on varchars ...
• Forward pointer - a 4-byte number that “points” to the remainder pieces of a row that spans pages
• Used with rows longer than a page
• Format is same as a rowid, although it’s NOT the rowid for the row
• Used with varchars or really long rows (rows longer than a page typically)
• We’ll focus on varchars ...
25
useruser .conference.conferenceInformixInformix
Forward Pointers w/ varcharsForward Pointers w/ varchars
• Problem - you can easily end up with a “chain” of forward pointer/remainder portions of rows
• This can happen when you’ve updated a varchar column and increased it’s size
• The new row size won’t fit into it’s old slot, and so we:• Move the row - all or some of it - to another page• Leave back a 4-byte forward pointer
on the home page
• Problem - you can easily end up with a “chain” of forward pointer/remainder portions of rows
• This can happen when you’ve updated a varchar column and increased it’s size
• The new row size won’t fit into it’s old slot, and so we:• Move the row - all or some of it - to another page• Leave back a 4-byte forward pointer
on the home page
26
useruser .conference.conferenceInformixInformix
Forward Pointers w/ varchars Forward Pointers w/ varchars
• This can happen as many times as it needs
• Every move adds an additional 8 bytes to the row length• 4 bytes for the forward pointer in the home page• 4 bytes for the slot table entry on the
new/remainder page
• This can happen as many times as it needs
• Every move adds an additional 8 bytes to the row length• 4 bytes for the forward pointer in the home page• 4 bytes for the slot table entry on the
new/remainder page
27
useruser .conference.conferenceInformixInformix
Forward PointersForward Pointers
28
useruser .conference.conferenceInformixInformix
What Can I Do? What Can I Do?
• To rid yourself of the “chaining” effect:• Unload/reload the table
• If the row will fit onto a single page, then there is no need for the forward pointer
• If it’s grown to be larger than a page (actual row length), then you’ll have at least one. This can’t be avoided - the row is now longer than a page
• Alter index to cluster• Same caveats as above
• To rid yourself of the “chaining” effect:• Unload/reload the table
• If the row will fit onto a single page, then there is no need for the forward pointer
• If it’s grown to be larger than a page (actual row length), then you’ll have at least one. This can’t be avoided - the row is now longer than a page
• Alter index to cluster• Same caveats as above
29
useruser .conference.conferenceInformixInformix
Topic 7: How Do You Find Out More? Topic 7: How Do You Find Out More?
• Attend the “IDS Internal Architecture” class• 5 days - very intense use of oncheck, and topics
covered in this presentation• Taught in Denver, Chicago, Menlo Park most often
• Watch TechNotes for a full white paper on this topic
• Send me email - [email protected]
• Checkout the website - www.markscranton.com
• Attend the “IDS Internal Architecture” class• 5 days - very intense use of oncheck, and topics
covered in this presentation• Taught in Denver, Chicago, Menlo Park most often
• Watch TechNotes for a full white paper on this topic
• Send me email - [email protected]
• Checkout the website - www.markscranton.com
30
useruser .conference.conferenceInformixInformix
Thanks...Thanks...
• For listening. Hope you learned something
• See you in class
• For listening. Hope you learned something
• See you in class