MySQL8Administrator'sGuide

Effectiveguidetoadministeringhigh-performanceMySQL8solutions

ChintanMehtaAnkitBhavsarHetalOzaSubhashShah

BIRMINGHAM-MUMBAI

MySQL8Administrator'sGuideCopyright©2018PacktPublishingAllrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,or transmitted inanyformorbyanymeans,without thepriorwrittenpermissionof thepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the informationpresented. However, the information contained in this book is sold without warranty, either express orimplied.Neithertheauthors,norPacktPublishingoritsdealersanddistributors,willbeheldliableforanydamagescausedorallegedtohavebeencauseddirectlyorindirectlybythisbook.

PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.

CommissioningEditor:AmeyVarangaonkarAcquisitionEditor:AmanSinghContentDevelopmentEditor:AaryamansinghTechnicalEditor:DharmendraYadavCopyEditors:SafisEditingProjectCoordinator:ManthanPatelProofreader:SafisEditingIndexer:TejalDaruwaleSoniGraphics:TaniaDuttaProductionCoordinator:AparnaBhagatFirstpublished:February2018

Productionreference:1140218

PublishedbyPacktPublishingLtd.LiveryPlace35LiveryStreetBirminghamB32PB,UK.

ISBN978-1-78839-519-9

www.packtpub.com

mapt.io

Maptisanonlinedigital librarythatgivesyoufullaccess toover5,000booksand videos, as well as industry leading tools to help you plan your personaldevelopment and advance your career. For more information, please visit ourwebsite.

Whysubscribe?Spendless timelearningandmoretimecodingwithpracticaleBooksandVideosfromover4,000industryprofessionals

ImproveyourlearningwithSkillPlansbuiltespeciallyforyou

GetafreeeBookorvideoeverymonth

Maptisfullysearchable

Copyandpaste,print,andbookmarkcontent

PacktPub.comDidyouknow thatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusatservice@packtpub.comformoredetails.

Atwww.PacktPub.com,youcanalsoreadacollectionoffree technicalarticles,signupforarangeoffreenewsletters,andreceiveexclusivediscountsandoffersonPacktbooksandeBooks.

Contributors

AbouttheauthorsChintan Mehta is a cofounder of KNOWARTH Technologies(www.knowarth.com)andheadscloud/RIMS/DevOps.Hehas rich,progressiveexperienceinserveradministrationofLinux,AWScloud,DevOps,RIMS,andopensourcetechnologies.HeisanAWSCertifiedSolutionsArchitect.

He has authoredMySQL 8 for Big Data andHadoop Backup and RecoverySolutions, and has reviewed Liferay Portal Performance Best Practices andBuildingServerlessWebApplications.

I would like to thank my coauthors. I would especially like to thank mywonderfulwife,Mittal,andmysweetson,Devam, forputtingupwith the longdays,nights,andweekendswhenIwascampinginfrontofmylaptop.Lastbutnot least, I want to thank my mom, dad, friends, family, and colleagues forsupportingmethroughout.

AnkitBhavsarisaseniorconsultantleadingateamworkingonERPsolutionsat KNOWARTH Technologies. He received an MCA from North Gujaratuniversity. He has had dynamic roles in the development andmaintenance ofERP solutions and astrologyportalsContentManagement that includingOOP,technicalarchitectureanalysis,design,developmentaswellasdatabasedesign,development and enhancement process, data and objectmodeling, in order toprovidetechnicalandbusinesssolutionstoclients.

First, I would like to thank the coauthors, reviewers, the wonderful team atPacktPub, and Aaryaman for this effort. I would especially like to thank mywonderfulwife,Avani,forputtingupwiththelongdays,nights,andweekends.Last,butnotleast,Iwanttothankmymom,dad,friends,family,andcolleaguesforsupportingmethroughoutthewritingofthisbook.

HetalOza anMCA from a reputable institute of India, is working as a leadconsultantatKNOWARTHTechnologies.ShehasrichexperienceinJava-basedsystemswithvariousdatabases.Her10yearsofexperiencecoversallstagesof

softwaredevelopment.Shehasworkedondevelopmentofweb-basedsoftwaresolutions on various platforms. She has good exposure to integration projectswith web-service-based and thread-based architecture. Her knowledge is notboundtoanysinglefieldbecauseshehasworkedonwiderangeoftechnologiesandtools.

Itgavemeimmensepleasuretobeanauthorofthisbook.First,Iwouldliketothankmyhusband,Suhag,andmysweetson,Om,forputtingupwithmeduringthelongdays,nights,andweekendswhenIwascampinginfrontofmylaptop.Second, I would like to thank Chintan Mehta, who showed trust in me andprovidedthisopportunity,andKrupalKhatriforhissupport.IwouldalsoliketothanktheteamatPacktPubfortheirgreathelp.

SubhashShahworks as aprincipal consultant atKNOWARTHTechnologies.HeholdsadegreeininformationtechnologyfromaHNGU.Heisexperiencedin developing web-based solutions using various software platforms. He is astrongadvocateofopensourcesoftwaredevelopmentanditsusebybusinessesto reduce risks and costs.His interests include designing sustainable softwaresolutions.Histechnicalskillsincluderequirementanalysis,architecturedesign,projectdelivery,applicationsetup,andexecutionprocesses.Heisanadmirerofqualitycodeandtest-drivendevelopment.

Iwouldliketothankmyfamilyforsupportingmethroughoutthecourseofthisbook. Itwould have been difficultwithout them being a source of inspiration.ThankstoPacktPublishing,especiallyAaryaman,fortheirsmoothcoordinationand support.Thanks to fellowauthors for being around all the time, for theirdedicationandcommitment.Lastbutnot least, thanks tomycolleagues forallthesupporttheyhaveprovided.

AboutthereviewersSahaj Pathak has been involved with backend technologies such as Java,Spring,Hibernate,anddatabases(MySQL,PostgreSQL,Oracle,andothers).Hisexperience also spans frontend technologies (HTML4/5, jQuery, AngularJS,Node.Js,JavaScript,andCSS2/3).Hehasspeedyversatilitywithanytechnologyandasharpdesireforconsistentchange.

HeworksatKNOWARTHTechnologiesasasoftwareconsultantwherehedealswithbigenterprise-product-basedprojects.

Ravi Shah is a highly versatile IT professional with more than 5 years ofexperience of handling high-end IT projects, with competencies inconceptualizingandsupportingcriticalITframeworksandapplications.Heisateam player, a software engineer with a can-do attitude, and possessesphenomenal timemanagement skills, and stronguser focus.Hehas developedseveralwebapplicationsandmainlyspecializesinhealthcareandinsurance.

He is skilled in all phases of software development, an expert in translatingbusinessrequirementsintotechnicalsolutions,anddevotedtoquality,usability,security and scalability. His expertise mainly includes Liferay, Java, Spring,Struts,Hibernate,MySQL,Lucene,Angular,andAgile.

He is a good trainer delivering training on J2EE and theLiferay portal in hisorganization.

IwouldliketotakethisopportunitytoexpressheartfeltthankstoKNOWARTHTechnologies and Packt Publishing for givingme this opportunity. Also, I amverythankfultomyparentsforalwayssupportingmeinallpossibleways.

Packt is searching for authorslikeyouIfyou'reinterestedinbecominganauthorforPackt,pleasevisitauthors.packtpub.com and apply today. We have worked with thousands of developers and techprofessionals,justlikeyou,tohelpthemsharetheirinsightwiththeglobaltechcommunity.Youcanmakeageneralapplication,apply foraspecifichot topicthatwearerecruitinganauthorfor,orsubmityourownidea.

TableofContents

Preface

Whothisbookisfor

Whatthisbookcovers

Togetthemostoutofthisbook

Downloadtheexamplecodefiles

Conventionsused

Getintouch

Reviews

1. AnIntroductiontoMYSQL8

OverviewofMySQL

MySQLasarelationaldatabasemanagementsystem

LicenserequirementsofMySQL8

Reliabilityandscalability

Platformcompatibility

Releases

CorefeaturesinMySQL

Structureddatabase

Databasestorageenginesandtypes

OverviewofInnoDB

OverviewofMyISAM

Overviewofmemory

Overviewofarchive

OverviewofBLACKHOLEasastorageengine

OverviewofCSV

Overviewofmerge

Overviewoffederated

OverviewoftheNDBcluster

ImprovedfeaturesinMySQL8

Transactionaldatadictionary

Roles

InnoDBautoincrement

Invisibleindexes

Improvingdescendingindexes

TheSETPERSISTvariant

ExpandedGISsupport

Defaultcharacterset

Extendedbit-wiseoperations

InnoDBMemcached

NOWAITandSKIPLOCKED

JSON

Cloud

Resourcemanagement

BenefitsofusingMySQL8

Security

Scalability

Anopensourcerelationaldatabasemanagementsystem

Highperformance

Highavailability

Cross-platformcapabilities

LimitationsofMySQL8

Numberoftablesordatabases

Tablesize

Joins

Windowsplatform

Tablecolumncount

Rowsize

InnoDBstorageengine

LimitationsofInnoDBstorageengine

Restrictions

Datadictionary

LimitationsofgroupreplicationinMySQL8

Limitationsofpartitioning

Constructsprohibition

Operators

Tables

UsecasesofMySQL

Socialmedia

Government

Mediaandentertainment

Frauddetection

Businessmapping

E-commerce

Summary

2. InstallingandUpgradingMySQL8

TheMySQL8installationprocess

Generalinstallationguide

DownloadingMySQL8

Verifyingthepackageintegrity

UsingMD5checksums

Usingcryptographicsignatures

InstallingMySQL8onMicrosoftWindows

Windows-specificconsiderations

MySQL8installationlayout

Choosingtherightinstallationpackage

TheMySQL8installer

Initialsetupinformation

Installationworkflow

InnoDBclustersandboxtestsetup

Serverconfiguration

MySQLinstallerproductcataloganddashboard

MySQLinstallerconsole

MySQL8installationusingaZIPfile

InstallingMySQL8onLinux

InstallationusingtheYumrepository

InstallationusingtheRPMpackage

InstallationusingtheDebianpackage

Post-installationsetupforMySQL8

Datadirectoryinitialization

SecuringtheinitialMySQLaccount

StartingandtroubleshootingMySQL8services

Executingcommandstotesttheserver

UpgradingMySQL8

Upgradingmethods

In-placeupgradeofMySQL

LogicalupgradeforMySQL8

UpgradingprerequisitesforMySQL5.7

MySQL8downgrading

Downgradingmethods

Logicaldowngrade

Manualchangesrequiredbeforedowngrading

Summary

3. MySQL8–UsingProgramsandUtilities

OverviewofMySQL8programs

MySQLprogramsinbrief

Startupprograms

Installation/upgradationprograms

Clientprograms

Administrativeandutilitiesprograms

Environmentvariables

MySQLGUItool

MySQL8command-lineprograms

Executingprogramsfromthecommandline

ExecutingMySQLprograms

ConnectingtotheMySQLserver

Specifyingoptionsforprograms

Optionsonthecommandline

Modifyingprogramoptions

Modifyingoptionswithfiles

group

opt_name

opt_name=value

Includedirectives

Command-lineoptionsaffectingoptionfilehandling

Settingprogramvariableswithoptions

Settingenvironmentvariables

Serverandserver-startupprograms

mysqld-theMySQLserverprogram

Options

mysqld_safe-MySQLserverstartupscript

mysql.server-MySQLServerstartupscript

mysqld_multi-managingmultipleMySQLservers

Installationprograms

comp_err-compilingtheMySQLerrormsgfile

mysql_secure_installation-improvingMySQLinstallationsecurity

mysql_ssl_rsa_setup-creatingSSL/RSAfiles

mysql_tzinfo_to_sql-loadingthetimezonetables

mysql_upgrade-checkingandupgradingMySQLtables

MySQL8clientprograms

mysql-thecommand-linetool

mysqloptions

mysqlcommands

help[arg],\h[arg],\?[arg],?[arg]

charsetcharset_name,\Ccharset_name

clear,\c

connect[db_namehost_name],\r[db_namehost_name]

edit,\e

exit,\q

prompt[str],\R[str]

quit,\q

status,\s

usedb_name,\udb_name

mysqllogging

mysqlserver-sidehelp

Executingsqlfromtextfiles

mysqladmin-clientforadministeringaMySQLserver

mysqlcheck-atablemaintenanceprogram

mysqldump-adatabasebackupprogram

Performanceandscalability

mysqlimport-adataimportprogram

mysqlpump-adatabasebackupprogram

mysqlsh-theMySQLShell

mysqlshow-showingdatabase,table,andcolumninformation

mysqlslap-loademulationclient

MySQL8administrativeprograms

ibdsdi-InnoDBtablespaceSDIextractionutility

innochecksum-offlineInnoDBfilechecksumutility

myisam_ftdump-displayingfull-textindexutility

myisamchk-MyISAMtable-maintenanceutility

myisamlog-displayingMyISAMlogfilecontent

myisampack-generatingcompressed,read-onlyMyISAMtables

mysql_config_editor-MySQLconfigurationutility

mysqlbinlog-utilityforprocessingbinarylogfiles

mysqldumpslow-summarizingslowquerylogfiles.

MySQL8environmentvariables

MySQLGUItools

MySQLWorkbench

MySQLNotifier

MySQLNotifierusage

Summary

4. MySQL8DataTypes

OverviewofMySQL8datatypes

Numericdatatypes

Integertypes

Fixedpointtypes

Floatingpointtypes

Problemswithfloatingpointvalues

Bitvaluetype

Bitvalueliterals

PracticalusesofBIT

Typeattributes

Overflowhandling

Dateandtimedatatypes

DATE,DATETIME,andTIMESTAMPtypes

MySQLDATETIMEfunctions

TIMEtype

Timefunctions

YEARtype

MigratingYEAR(2)toYEAR(4)

Stringdatatypes

CHARandVARCHARdatatypes

BINARYandVARBINARYdatatypes

BLOBandTEXTdatatypes

ENUMdatatype

SETdatatype

JSONdatatype

PartialupdatesofJSONvalues

Storagerequirementsfordatatypes

Choosingtherightdatatypeforcolumn

Summary

5. MySQL8DatabaseManagement

MySQL8serveradministration

Serveroptionsanddifferenttypesofvariables

ServerSQLmodes

SettingtheSQLmode

TheavailableSQLmodes

CombinationSQLmodes

StrictSQLmode

TheIGNOREkeyword

IPv6support

Serversidehelp

Theservershutdownprocess

Datadirectory

Thesystemdatabase

Datadictionarytables

Grantsystemtables

Objectinformationsystemtables

Logsystemtables

Theserver-sidehelpsystemtables

Timezonesystemtables

Replicationsystemtables

Optimizersystemtables

Othermiscellaneoussystemtables

Runningmultipleinstancesonasinglemachine

Settingupmultipledatadirectories

RunningmultipleMySQLinstancesonWindows

Componentsandpluginmanagement

MySQL8serverplugins

Installingtheplugins

Activateplugin

Uninstallplugin

Gettinginformationabouttheinstalledplugins

Rolesandpermissions

Cachingtechniques

Globalization

Charactersets

Charactersetsupport

Addingthecharacterset

Configuringthecharactersets

Languageselection

TimezonesettingsforMySQL8

Localesupport

MySQL8serverlogs

Theerrorlog

Componentconfiguration

Defaulterrorlogdestinationconfiguration

DefaulterrorlogdestinationonWindows

DefaulterrorlogdestinationonUnixandUnix-Likesystems

Thegeneralquerylog

Thebinarylog

Theslowquerylog

TheDDLlog

Serverlogmaintenance

Summary

6. MySQL8StorageEngines

Overviewofstorageengines

MySQLstorageenginearchitecture

Severaltypesofstorageengine

OverviewoftheInnoDBstorageengine

Customstorageengine

Severaltypesofstorageengines

Pluggablestorageenginearchitecture

Thecommondatabaseserverlayer

Settingthestorageengine

TheMyISAMstorageengine

TheMEMORYstorageengine

TheCSVstorageengine

TheARCHIVEstorageengine

TheBLACKHOLEstorageengine

TheMERGEstorageengine

TheFEDERATEDstorageengine

TheEXAMPLEstorageengine

TheInnoDBstorageengine

ACIDmodel

Multiversioning

Architecture

Lockingandtransactionmodel

Configuration

Tablespaces

Tablesandindexes

INFORMATION_SCHEMAtables

Memcachedplugin

Creatingacustomstorageengine

Creatingstorageenginesourcefiles

Addingengine-specificvariablesandparameters

Creatingthehandlerton

Handlinghandlerinstallation

Definingfilenameextensions

Creatingtables

Openingatable

Implementingbasictablescanning

Closingatable

Referenceforadvancedcustomstorageengine

Summary

7. IndexinginMySQL8

Anoverviewonindexing

UsesofindexesinMySQL8

SQLcommandsrelatedtoindexes

CreatinganINDEXcommand

Spatialindexcharacteristics

Non-spatialindexcharacteristics

Dropindexcommand

SPATIALindexcreationandoptimization

InnoDBandMyISAMindexstatisticscollection

Column-levelindexing

Columnindexes

Indexprefixes

FULLTEXTindexes

SpatialIndexes

IndexesintheMEMORYstorageengine

Multiple-columnindexes

B-Treeindex

Hashindex

Indexextension

Usinganoptimizerforindexes

Invisibleanddescendingindexes

Invisibleindex

Descendingindex

Summary

8. ReplicationinMySQL8

Overviewofreplication

WhatisMySQLreplication?

AdvantagesofMySQLreplication

Configuringreplication

Binarylogfilebasedreplication

Replicationmasterconfiguration

REPLICATIONSLAVEconfiguration

Addingslavestoreplication

Globaltransactionidentifiersbasedreplication

MySQLmulti-sourcereplication

Replicationadministrationtasks

Implementingreplication

Replicationformats

Statement-basedversusrow-basedreplication

Replicationimplementationdetails

Replicationchannels

Replicationrelayandstatuslogs

Evaluatingreplicationfilteringrules

Groupreplication

Primary-secondaryreplicationversusgroupreplication

Groupreplicationconfiguration

Groupreplicationusecases

Replicationsolutions

Summary

9. PartitioninginMySQL8

Overviewofpartitioning

Typesofpartitioning

Partitioningmanagement

Partitionselectionandpruning

Restrictionsandlimitationsinpartitioning

Typesofpartitioning

RANGEpartitioning

LISTpartitioning

COLUMNSpartitioning

RANGECOLUMNpartitioning

LISTCOLUMNpartitioning

HASHpartitioning

LINEARHASHpartitioning

KEYpartitioning

Subpartitioning

HandlingNULLinpartitioning

Partitionmanagement

RANGEandLISTpartitionmanagement

HASHandKEYpartitionmanagement

Partitionmaintenance

Obtainpartitioninformation

Partitionselectionandpruning

Partitionpruning

Partitionselection

Restrictionsandlimitationsinpartitioning

Partitioningkeys,primarykeys,anduniquekeys

Partitioninglimitationsrelatingtostorageengines

Partitioninglimitationsrelatingtofunctions

Summary

10. MySQL8–ScalabilityandHighAvailability

OverviewofscalabilityandhighavailabilityinMySQL8

MySQLreplication

MySQLcluster

OracleMySQLcloudservice

MySQLwiththeSolariscluster

ScalingMySQL8

Scalingusingcluster

Clientnode

Applicationnode

Managementnode

Datanode

Datastorageandmanagementofdisk-basedandin-memorydata

Automaticanduser-definedpartitioningoftablesorsharding

oftables

Synchronousdatareplicationbetweendatanodes

Dataretrievalandtransactions

Automaticfailover

Automaticre-synchronizationforself-healingafterfailure

ScalingusingmemcachedinMySQL8

NoSQLAPIs

Scalingusingreplication

Singleserverdependancy

Performance

Backupandrecovery

Loaddistribution

Asynchronousdatareplication

Geographicaldatadistribution

GTIDreplication

ZFSreplication

ChallengesinscalingMySQL8

Businesstypeandflexibility

Understandserverworkload

Read-writeoperationlimit

Maintenance

Masterserverfailure

Synchronization

Databasesecurity

Crossnodetransaction

Growingteamfordevelopment

Managechangerequest

Scale-upandscale-out

Achievinghighavailability

Purposeofhighavailability

Dataavailability

Securityofdata

Synchronizationofdata

Backupofthedata

Competitivemarket

Performance

Updatesinthesystem

Choosingthesolution

Advantagesofhighavailability

Summary

11. MySQL8–Security

OverviewofsecurityforMySQL8

Commonsecurityissues

Generalguidelines

Guidelinesforasecurepassword

Guidelinesforendusers

Guidelinesforadministrators

Passwordandlogging

SecureMYSQL8againstattackers

SecurityoptionsandvariablesprovidedbyMySQL8

Securityguidelinesforclientprogramming

AccesscontrolinMySQL8

PrivilegesprovidedbyMySQL8

Granttables

Verificationofaccesscontrolstages

Stage1-Connectionverification

Stage2-Requestverification

AccountmanagementinMySQL8

Addandremoveuseraccounts

Securityusingroles

SETROLE

CREATEROLE

DROPROLE

GRANT

REVOKE

SETDEFAULTROLE

SHOWGRANTS

Passwordmanagement

EncryptioninMySQL8

ConfiguringMySQL8touseencryptedconnections

Server-sideconfigurationforencryptedconnections

Client-sideconfigurationforencryptedconnections

Commandoptionsforencryptedconnections

ConnectwithMySQL8remotelyfromWindowswithSSH

Securityplugins

Authenticationplugins

SHA-2pluggableauthentication

Client-sidecleartextpluggableauthentication

No-loginpluggableauthentication

Socketpeer-credentialpluggableauthentication

Testpluggableauthentication

Theconnection-controlplugins

CONNECTION_CONTROL

Plugininstallation

VariablesrelatedtoCONNECTION-CONTROL

Thepasswordvalidationplugin

Installpasswordvalidationplugin

Variablesandoptionsrelatedtothepasswordvalidationplugin

MySQL8keyring

Installkeyringplugin

Systemvariablesrelatedtokeyringplugin

Summary

12. OptimizingMySQL8

OverviewofMySQL8optimization

Optimizingthedatabase

Optimizingthehardware

OptimizingMySQL8serversandclients

OptimizingdiskI/O

UsingNFSwithMySQL

Optimizingtheuseofmemory

Optimizinguseofthenetwork

Optimizinglockingoperations

Performancebenchmarking

Examiningthreadinformation

Optimizingdatabasestructure

Optimizingdatasize

Tablecolumns

Rowformat

Indexes

Joins

Normalization

OptimizingMySQLdatatypes

Optimizingformanytables

UseofaninternaltemporarytableinMySQL

Optimizingqueries

OptimizingSQLstatements

Optimizingindexes

Queryexecutionplan

Optimizingtables

OptimizationforInnoDBtables

OptimizationforMyISAMtables

OptimizationforMEMORYtables

Leveragingbufferingandcaching

InnoDBbufferpooloptimization

MyISAMkeycache

Summary

13. ExtendingMySQL8

AnoverviewofextendingMySQL8

MySQL8internals

MySQL8pluginAPI

MySQL8servicesforcomponentsandplugins

AddingnewfunctionstoMySQL8

DebuggingandportingMySQL8

Extendingpluginsandusingservicestocallthem

Writingplugins

Componentandpluginservices

Thelockingservice

Thekeyringservice

Addingnewfunctions

Featuresofauser-definedfunctioninterface

Addinganewuser-definedfunction

Addinganewnativefunction

Debuggingandporting

DebuggingMySQLserver

DebuggingMySQLclient

TheDBUGpackage

Summary

14. MySQL8BestPracticesandBenchmarking

MySQLbenchmarkingandtools

Resourceutilization

Stretchingyourbenchmarkingtimelines

Replicatingproductionsettings

Consistencyofthroughputandlatency

Sysbenchcandomore

Virtualizationworld

Concurrency

Hiddenworkloads

Nervesofyourquery

Benchmarks

Bestpracticesformemcached

Resourceallocation

Operatingsystemarchitecture

Defaultconfigurations

Maxobjectsize

Backlogqueuelimit

Largepagessupport

Sensitivedata

Restrictingexposure

Failover

Namespaces

Cachingmechanism

Memcachedgeneralstatistics

Bestpracticesforreplication

Throughputingroupreplication

Infrastructuresizing

Constantthroughput

Contradictoryworkloads

Writescalability

Bestpracticesfordatapartitioning

Horizontalpartitioning

Verticalpartitioning

PruningpartitionsinMySQL

Bestpracticesforqueriesandindexing

Datatypes

Notnull

Indexing

Searchfieldsindex

Datatypesandjoins

Compoundindex

Shorteningupprimarykeys

Indexingeverything

Fetchingalldata

Lettingtheapplicationdothejob

Existenceofdata

Limitingyourself

Analyzingslowqueries

Querycost

Summary

15. TroubleshootingMySQL8

MySQL8commonproblems

MostcommonMySQLerrors

Accessdenied

Can'tconnectto[local]MySQLserver

LostconnectiontoMySQLserver

Passwordfailswhenenteredincorrectly

Hosthost_nameisblocked

Toomanyconnections

Outofmemory

Packettoolarge

Thetableisfull

Can'tcreate/writetofile

Commandsoutofsync

Ignoringuser

Tabletbl_namedoesn'texist

MySQL8servererrors

Issueswithfilepermissions

Resettingtherootpassword

MySQLcrashesprevention

HandlingMySQLfulldisk

MySQLtemporaryfilesstorage

MySQLUnixsocketfile

Timezoneproblems

MySQL8clienterrors

Casesensitivityinstringsearches

ProblemswithDATEcolumns

ProblemswithNULLvalues

MySQL8troubleshootingapproach

Analyzingqueries

Real-worldscenario

Summary

OtherBooksYouMayEnjoy

Leaveareview-letotherreadersknowwhatyouthink

Preface

For any system, it ismust tomanagedata in anorganizedmanner. In a large-scale system, it is necessary to handle various configurations for securitypurposes.MySQL

is one of the popular solutions used to handle enterprise-level applications. Inthisbook,wewillexplainhowtoconfigureusers,theirroles,multipleinstances,andmuchmore.

ManyorganizationsuseMySQLfortheirwebsitesorcommercialproducts,andit's very challenging for them to manage data storage and analyze data inaccordance with the business requirements. This book will show you how toimplementindexingandqueryoptimizationforbetterperformance.Alongwiththis,we'llcoverhowscalabilityandhighavailabilityoftheMySQLservercanhelptomanagefailurescenarios.Inadditiontothat,replicationandpartitioningconceptsareexplainedindetailwithexamples.

ThebookdescribesvariousfeaturesofMySQL8bytargetingdifferentlevelsofusers, from beginners to database administrators. This book starts from theinstallationwithabasicunderstandingofMySQL8'sconcepts.Theweproceedtoadministrative-levelfeatureswithconfiguration.Attheendofthebook,you

will have learned about very interesting functionalities, such as optimization,extension,andtroubleshooting.

WhothisbookisforThisbook is intended forMySQLadministratorswhoare looking for ahandyguide covering all theMySQL administration-related tasks. If you are aDBAlookingtogetstartedwithMySQLadministration,thisbookwillalsohelpyou.Knowledgeofbasicdatabaseconceptsisrequiredtogetstartedwiththisbook.

WhatthisbookcoversChapter 1, An Introduction to MySQL 8, serves as an introductory guide toMySQL 8. It briefly defines the core features available inMySQL and newlyintroducedorenhancedfeaturesofMySQL8.Inthelaterpartofthechapter,wehighlightthebenefitsofMySQL8alongwithreal-worldapplications.

Chapter 2, Installing and Upgrading MySQL 8, describes detailed steps forinstallingMySQL8ondifferentplatforms.ItalsoexplainshowtoupgradetoordowngradefromMySQL8.

Chapter3,MySQL 8 –Using Programs andUtilities, introduces command-lineprogramsfortheMySQL8serverandclient.ItalsoprovidesinformationontheavailableGUItoolswithitsconfiguration.

Chapter4,MySQL8DataTypes, focusesonadetailedexplanationofMySQL8data types. It also explains data type categorization based on the types ofcontent.We cover data types alongwith their properties in each category.Wealsocoverstoragerequirementsfordatatypes.

Chapter5,MySQL8DatabaseManagement,mainlyexplores theadministrationpart of MySQL 8. This chapter covers components and plugin management,along with user and role management. In addition, it explains globalizationconfiguration, caching techniques, and different types of logs available inMySQL8.

Chapter6,MySQL8StorageEngines, explains several types of storage enginesanddetailsoftheInnoDBstorageengine.Thischapterprovidesinformationoncustom storage engine creation, along with steps to make it pluggable ininstalledMySQL8.

Chapter7,IndexinginMySQL8,explainsindexing,alongwiththepossiblewaysofimplementingit.Itcomparestypesofindexing.

Chapter8,ReplicationinMySQL8,explainsreplicationandthedifferenttypesofreplication available in MySQL 8. It also describes the configuration and

implementationofreplicationalongwithdifferentapproaches.

Chapter 9, Partitioning in MySQL 8, explains the setting of several types ofpartitioning,selection,andpruningofpartitioning.Italsoexplainshowtocopeupwithrestrictionsandlimitationswhilepartitioning.

Chapter 10,MySQL 8 – Scalability and High Availability, explains how to doscaling and how to handle different challenges during implementation. Thereader gets an understanding of diverse ways to achieve high availability inMySQL8.

Chapter 11,MySQL 8 – Security, focuses onMySQL 8 database security. Thischaptercoversgeneralfactorsthataffectsecurity,thesecurityofcoreMySQL8files, accesscontrol, and securing thedatabase system itself.This chapter alsoincludesdetailsofsecurityplugins.

Chapter12,OptimizingMySQL8,explainshowtoconfigureMySQL8forbetterperformance. This chapter also describes use cases with a few performanceresultstovalidate.ThiswillhelpyouknowvarioustouchpointstolookoutforwhendealingwithoptimizingMySQL8.

Chapter13,ExtendingMySQL 8, shows how to extendMySQL 8 and add newfunctions,alongwithdebuggingandportingtoMySQL8.

Chapter 14, MySQL 8 Best Practices and Benchmarking, explains the bestpractices of using MySQL. It also explains various benchmarkings done forMySQL8.

Chapter 15,TroubleshootingMySQL 8, explains many common and real-worldscenariosoftroubleshootingforMySQL8.

TogetthemostoutofthisbookWerecommendthatyougetsomebasicknowledgeofMySQL(anyversion)andSQLcommandsbeforeyoustartreadingthisbook.

Thisbookalsocoverspracticalscenariosandcommandexecution,soifpossible,installatoolforeasyexecutionofMySQLcommands.

Downloadtheexamplecodefiles

Youcandownloadtheexamplecodefilesforthisbookfromyouraccountatwww.packtpub.com.Ifyoupurchasedthisbookelsewhere,youcanvisitwww.packtpub.com/supportandregistertohavethefilesemaileddirectlytoyou.

Youcandownloadthecodefilesbyfollowingthesesteps:

1. Loginorregisteratwww.packtpub.com.

2. SelecttheSUPPORTtab.

3. ClickonCodeDownloads&Errata.

4. Enter the name of the book in the Search box and follow the onscreeninstructions.

Once the file is downloaded, please make sure that you unzip or extract thefolderusingthelatestversionof:

WinRAR/7-ZipforWindows

Zipeg/iZip/UnRarXforMac

7-Zip/PeaZipforLinux

ThecodebundleforthebookisalsohostedonGitHubathttps://github.com/PacktPublishing/MySQL-8-Administrators-Guide.Wealsohaveothercodebundlesfromourrichcatalogofbooksandvideosavailableathttps://github.com/PacktPublishing/.Checkthemout!

ConventionsusedThereareanumberoftextconventionsusedthroughoutthisbook.

CodeInText: Indicates code words in text, database table names, folder names,filenames, file extensions, pathnames, dummy URLs, user input, and Twitterhandles.Hereisanexample:"ItwilldownloadwinMD5Sum.exeontoyourcomputer."

Anycommand-lineinputoroutputiswrittenasfollows:

CREATETABLEworking_days(

yearINT,

weekINT,

daysBIT(7),

PRIMARYKEY(year,week));

Bold:Indicatesanewterm,animportantword,orwordsthatyouseeonscreen.Forexample,wordsinmenusordialogboxesappearinthetextlikethis.Hereisanexample:"ClickontheDownloadWinMD5Sumoptiononthepage."

Warningsorimportantnotesappearlikethis.

Tipsandtricksappearlikethis.

Getintouch

Feedbackfromourreadersisalwayswelcome.

General feedback:Emailfeedback@packtpub.comandmention thebook title in thesubjectofyourmessage.Ifyouhavequestions

aboutanyaspectofthisbook,pleaseemailusatquestions@packtpub.com.

Errata: Although we have taken every care to ensure the accuracy of ourcontent,mistakes

dohappen.Ifyouhavefoundamistakeinthisbook,wewouldbegratefulifyouwould

report this to us. Please visit www.packtpub.com/submit-errata, selecting your book,clickingontheErrataSubmissionFormlink,andenteringthe

details.

Piracy: IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,

we would be grateful if you would provide us with the location address orwebsite

name.Pleasecontactusatcopyright@packtpub.comwithalinktothematerial.

Ifyouareinterestedinbecominganauthor:Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineither

writingorcontributingtoabook,pleasevisitauthors.packtpub.com.

ReviewsPleaseleaveareview.Onceyouhavereadandusedthisbook,whynotleaveareviewonthesitethatyoupurchaseditfrom?Potentialreaderscanthenseeanduse your unbiased opinion to make purchase decisions, we at Packt canunderstand what you think about our products, and our authors can see yourfeedbackontheirbook.Thankyou!

FormoreinformationaboutPackt,pleasevisitpacktpub.com.

AnIntroductiontoMYSQL8

MySQL is a well-known open source structured database because of itsperformance,easinesstouse,andreliability.Thisisthemostcommonchoiceofweb applications for a relational database. In the currentmarket, thousands ofweb-based applications rely on MySQL including giant industries such asFacebook,Twitter,andWikipedia.

Ithasalsoproven tobe thedatabasechoiceforSoftwareasaService (SaaS)based applications such as Twitter, YouTube, SugarCRM, Supply Dynamics,Workday,RightNow,Omniture,Zimbra,andmanymore.Wewilldiscussthisindetail in the use cases of MySQL section later in the chapter. MySQL wasdeveloped byMySQLAB, a Swedish company, and now it is distributed andsupportedbyOracleCorporation.MySQLcarriesavaluablehistorywithit.

MySQL has continued to improve in order to become an enterprise-leveldatabasemanagement system.MySQL 8 is expected to be a game-changer astodayweareintheageofdigitization.

MySQL8isall tunedtoservemanynewusecases that inpriorversionsweredifficult to achieve. Some of the use cases an enormous amount of data isproduced are social networking, e-commerce, bank/credit card transactions,

emails, data stored on the cloud, and so on. Analysis of all such structured,unstructured, or semi-structured ubiquitous data helps to discover hiddenpatterns,markettrends,correlations,personalpreferences.

"Thereissomuchforeachofus"

- JamesTruslowAdamsLet's takean in-depth lookatMySQL8new features,benefits, use cases alongwitha few limitationsofMySQL8afterwehaveanoverviewofMySQL.Thisisgoingtobeexciting,let'sgetprepared.

OverviewofMySQL

Structured Query Language (SQL) is used to manipulate, retrieve, insert,update,anddeletedatainrelationaldatabasemanagementsystem(RDBMS).Tomakeitsimpler,SQLtellsthedatabasewhattodoandexactlywhatitneeds.SQL is a standard language that all RDBMS systems such as MySQL, MSAccess,MSSQL,Oracle,Postgres,andothersuse.

RDBMSisthebasisforSQLandforallmoderndatabasesystemssuchasMSSQLServer,IBMDB2,Oracle,MySQL,andMicrosoftAccess.

SQL allows users to access data fromMySQL and define andmanipulate thedata. To embed within other languages, you can leverage SQL modules,libraries,andprecompilers,whichcanhelpyoucreate/dropdatabasesandtables,allowuserstocreatetheview,andstoredprocedures,functions,andsoon,inadatabase. It can do various other operations such as allowing users to setpermissionsontables,procedures,andviews.

MySQL as a relational databasemanagementsystem

Datainarelationaldatabaseisstoredinanorganizedformatsothatinformationcanberetrievedeasily.Datawillbestoredindifferent tablesmadeupofrowsand columns. However, the relationship can also be built between differenttables thatefficientlystorehugedataandeffectively retrieve theselecteddata.Thisprovidesdatabaseoperationswithtremendousspeedandflexibility.

Asarelationaldatabase,MySQLhascapabilitiestoestablishrelationshipswithdifferenttablessuchasonetomany,manytoone,andonetoonebyprovidingprimarykeys,foreignkeys,andindexes.Itcanalsoperformjoinsbetweentablestoretrieveexactinformationsuchasinnerjoinsandouterjoins.

SQLisusedasaninterfacetointeractwiththerelationaldatainMySQL.SQLisanAmericanNationalStandardInstitute(ANSI)standardlanguagewhichwecanoperatewithdatasuchascreation,deletion,updating,andretrieval.

LicenserequirementsofMySQL8

Many industries prefer open source technology because of the technology'sflexibility and cost-saving features, whileMySQL has put its footprint in themarketbybecomingthemostpopularrelationaldatabaseforwebapplications.Open sourcemeans that you canview the sourceofMySQLand customize itbasedonyourneedswithoutanycost.Youcandownload thesourceorbinaryfilesfromitssiteandusethemaccordingly.

TheMySQLserveriscoveredundertheGeneralPublicLicense(GNU),whichmeansthatwecanfreelyuseitforwebapplications,studyitssourcecode,andmodify it to suit our needs. It also has the Enterprise Edition as well withadvancedfeaturesincluded.ManyenterprisesstillpurchasethesupportcontractfromMySQL

togetassistanceonvariousissues.

ReliabilityandscalabilityMySQL has great reliability to perform well without requiring extensivetroubleshooting due to bottlenecks or other slowdowns. It also incorporates anumber of performance enhanced mechanisms such as index support, loadutilities,andmemorycaches.MySQL

uses InnoDB as a storage engine, which provides highly efficient ACIDcomplianttransactionalcapabilitiesthatassurehighperformanceandscalability.To handle the rapidly growing database,MySQLReplication and cluster helpscaleoutthedatabase.

PlatformcompatibilityMySQL has great cross-platform availability thatmakes itmore popular. It isflexible to run on major platforms such as RedHat, Fedora, Ubuntu, Debian,Solaris,MicrosoftWindows, and ApplemacOS. It also providesApplicationProgramming Interface (APIs) to interconnect with various programminglanguagessuchasC,C++,C#,PHP,Java,Ruby,Python,andPerl.

Releases

HereisalistofmajorreleasesofMySQLsofar:

Version5.0GAwasreleasedon19thOctober,2005

Version5.1GAwasreleasedon14thNovember,2008

Version5.5GAwasreleasedon3rdDecember,2010

Version5.6GAwasreleasedon5thFebruary,2013

Version5.7GAwasreleasedon21stOctober,2015

Nowit'stimeforthemajorversionrelease--MySQL8--whichwasannouncedon12thSeptember,2016andisstillinthedevelopmentmilestonemode.

CorefeaturesinMySQLLet'slookbackandquicklyglancethroughsomeofthecorefeaturesinMySQL.We will be discussing various features throughout the book in detail as weprogress.

Structureddatabase

Structured databases are traditional databases that have been used by manyenterprisesformorethan40years.However,inthemodernworld,datavolumeis becoming bigger and bigger and a common need has taken its place--dataanalytics. Analytics is becoming difficult with structured databases as thevolumeandvelocityofdigitaldatagrowfasterbytheday;weneedtofindawayto achieve such needs in an effective and efficient way. The most commondatabase that is used as a structured database in the open source world isMySQL.

Manyorganizationsuseastructureddatabasetostoretheirdatainanorganizedwaywiththeformattedrepository.Basically,datainastructureddatabasehasafixedfield,apredefineddatalength,anddefineswhatkindofdataistobestoredsuch as numbers, dates, time, addresses, currencies, and so on. In short, thestructureisalreadydefinedbeforedatagetsinserted,whichgivesaclearerideaofwhatdatacanresidethere.Thekeyadvantageofusingastructureddatabaseisthatdatabeingeasilystored,queried,andanalyzed.

Anunstructureddatabase is theopposite of this; it has no identifiable internalstructure. It can have a massive unorganized agglomerate or various objects.Mainly, the source of structured data is machine-generated, which means

information is generated from the machine and without human intervention,whereasunstructureddataishuman-generateddata.Organizationsusestructureddatabases for data such as ATM transactions, airline reservations, inventorysystems,andsoon.Inthesameway,someorganizationsuseunstructureddatasuch as emails, multimedia content, word processing documents, web pages,businessdocuments,andsoon.

<strong>mysql>SHOWTABLESTATUSLIKE'admin_user'\G;</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Name:admin_user</strong><br/><strong>Engine:InnoDB</strong><br/><strong>Version:10</strong><br/><strong>Row_format:Dynamic</strong><br/><strong>Rows:2</strong><br/><strong>Avg_row_length:8192</strong><br/><strong>Data_length:16384</strong><br/><strong>Max_data_length:0</strong><br/><strong>Index_length:16384</strong><br/><strong>Data_free:0</strong><br/><strong>Auto_increment:3</strong><br/><strong>Create_time:2017-06-1914:46:49</strong><br/><strong>Update_time:2017-06-1915:15:08</strong><br/><strong>Check_time:NULL</strong><br/><strong>Collation:utf8_general_ci</strong><br/><strong>Checksum:NULL</strong><br/><strong>Create_options:</strong><br/><strong>Comment:AdminUserTable</strong><br/><strong>1rowinset(0.00sec)</strong>

<strong>mysql>SHOWENGINES\G;</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Engine:InnoDB</strong><br/><strong>Support:YES</strong><br/><strong>Comment:Supportstransactions,row-levellocking,andforeignkeys</strong><br/><strong>Transactions:YES</strong><br/><strong>XA:YES</strong><br/><strong>Savepoints:YES</strong><br/><strong>***************************2.row***************************</strong><br/><strong>Engine:MRG_MYISAM</strong><br/><strong>Support:YES</strong><br/><strong>Comment:CollectionofidenticalMyISAMtables</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************3.row

***************************</strong><br/><strong>Engine:MEMORY</strong><br/><strong>Support:YES</strong><br/><strong>Comment:Hashbased,storedinmemory,usefulfortemporarytables</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************4.row***************************</strong><br/><strong>Engine:BLACKHOLE</strong><br/><strong>Support:YES</strong><br/><strong>Comment:/dev/nullstorageengine(anythingyouwritetoitdisappears)</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************5.row***************************</strong><br/><strong>Engine:MyISAM</strong><br/><strong>Support:DEFAULT</strong><br/><strong>Comment:MyISAMstorageengine</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************6.row***************************</strong><br/><strong>Engine:CSV</strong><br/><strong>Support:YES</strong><br/><strong>Comment:CSVstorageengine</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************7.row***************************</strong><br/><strong>Engine:ARCHIVE</strong><br/><strong>Support:YES</strong><br/><strong>Comment:Archivestorageengine</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************8.row***************************</strong><br/><strong>Engine:PERFORMANCE_SCHEMA</strong><br/><strong>Support:YES</strong><br/><strong>Comment:Performance

Schema</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************9.row***************************</strong><br/><strong>Engine:FEDERATED</strong><br/><strong>Support:NO</strong><br/><strong>Comment:FederatedMySQLstorageengine</strong><br/><strong>Transactions:NULL</strong><br/><strong>XA:NULL</strong><br/><strong>Savepoints:NULL</strong><br/><strong>9rowsinset(0.00sec)</strong>

OverviewofInnoDB

InnoDBisthedefaultstorageenginebroadlyusedoutofallotheravailablestorageengines.ItwasreleasedwithMySQL5.1asapluginin2008.MySQL5.5andlater has InnoDB as a default storage engine. It has been taken over by OracleCorporationinOctober

2005,fromtheInnobaseOy,whichisaFinland-basedcompany.

InnoDBtablessupportACID-compliant commits, rollback, andcrash recoverycapabilitiestoprotect

user data. It also supports row-level locking, which helps with betterconcurrency

andperformance.ItstoresdatainclusteredindexestoreduceI/Ooperationsfor

all SQL select queries based on the primary key. It also supports FOREIGN KEY

constraintsthatallowbetterdataintegrityforthedatabase.ThemaximumsizeofanInnoDBtablecanscaleupto64TB,whichshouldbegoodenough

toservemanyreal-worldusecases.

OverviewofMyISAMMyISAMwas the default storage engine forMySQLprior to 5.5 1. MyISAM storageenginetablesdonotsupportACID-compliantasopposedtoInnoDB.MyISAM tablessupport table-level locking only, so MyISAM tables are not transaction-safe;however, they are optimized for compression and speed. It is generally usedwhenyouneedtohaveprimarilyreadoperationswithminimaltransactiondata.ThemaximumsizeofaMyISAMtablecangrowupto256TB,whichhelpsinusecasessuchasdataanalytics.MyISAMsupportsfull-textindexing,whichcanhelpincomplexsearchoperations.Usingfull-textindexes,wecanindexdatastoredinBLOBandTEXTdatatypes.

Overviewofmemory

Amemorystorageengineisgenerallyknownasaheapstorageengine.Itisusedtoaccessdataextremelyquickly.ThisstorageenginestoresdataintheRAMsoitwouldn'tneedI/Ooperation.AsitstoresdataintheRAM,alldataislostuponserverrestart.

Thistableisbasicallyusedfortemporarytablesorthelookuptable.Thisenginesupportstable-levellocking,whichlimitshighwriteconcurrency.

Importantnotesaboutmemorytablesareasfollows:

Becausememory table stores data in theRAM,whichhas a very limitedstoragecapacity;ifyoutrytowritetoomuchdataintothememorytable,itwillstartswappingdataintothediskandthenyoulosethebenefitsofthememorystorageengine

These tables don't support TEXT and BLOB data types, and it is not evenrequiredasithaslimitedstoragecapacity

This storage engine can be used to cache the results; lookup tables, for

example,orpostalcodesandthenamesofstates

MemorytablessupportB-treeindexesandHashindexes

OverviewofarchiveThisstorageengineisusedtostorelargeamountsofhistoricaldatawithoutany

indexes.Archivetablesdonothaveanystoragelimitations.Thearchivestorageengine

is optimized for high insert operations and also supports row-level locking.Thesetablesstoredata

inacompressedandsmallformat.ThearchiveenginedoesnotsupportDELETEorUPDATEoperations;itonlyallowsINSERT,REPLACE,andSELECToperations.

Overview of BLACKHOLE as astorageengine

Thisstorageengineacceptsdatabutdoesnotstoreit.ItdiscardsdataaftereveryINSERTinsteadofstoringit.

Now,what is theuseof this storageengine;whywouldanybodyuse it?WhywouldwerunanINSERTquerythatdoesn'tinsertanythingintothetable?

This engine is useful for replicationwith large number of servers. A BLACKHOLEstorageengineactsasafilterserverbetweenthemasterandslaveserver,whichdonotstoreanydata,butonlyapplyreplicate-do-*andreplicate-ignore-*rulesandwriteabinlogs.Thesebinlogsareusedtoperformreplicationinslaveservers.WewilldiscussthisindetailinChapter8,ReplicationinMySQL8.

OverviewofCSVThe comma separated values (CSV) engine stores data in the .csv file typeusing the comma-separated values format. This engine extracts data from thedatabaseandcopiesitto.csvoutofthedatabase.IfyoucreateaCSVfilefromthespreadsheetandcopyitinto

the MYSQL data folder server, it can read the data using the select query.Similarly,

ifyouwritedatainthetable,anexternalprogramcanreaditfromtheCSVfile.

This storage engine is used for the exchange of data between software orapplications.

ACSVtabledoesnotsupportindexingandpartitioning.AllcolumnsintheCSVstorage

engineneedtobedefinedwiththeNOTNULLattributetoavoiderrorsduringtablecreation.

Overviewofmerge

This storageengine isalsoknownasanMRG_MyISAM storageengine.This storageenginemergesaMyISAMtableandcreatesittobereferredtoasingleview.Foramergetable,allcolumns

are listed in the same order. These tables are good for data warehousingenvironments.

The table is used to manage log-related tables, generally. You can createdifferent

monthsoflogsinseparateMyISAM tablesandmergethesetablesusingthemergestorageengine.

MyISAMtableshavestoragelimitfortheoperatingsystem,butacollectionofMyISAM(merge) tablesdonothavestorage limits.Sousingamerge tablewouldallowyou

to split data intomultiple MyISAM tables,which can help in overcoming storagelimits.

Merge tables do not support partitioning. Also, you cannot partition a merge

table

oranyofamergetable'sunderlyingMyISAMtablesinadifferentpartition.

Overviewoffederated

Thisstorageengineallowsyoutocreateasingledatabaseonamultiplephysicalserver.Itopensaclientconnectiontoanotherserverandexecutesqueriesagainstatablethere,retrievingandsendingrowsasneeded.Itwasoriginallymarketedas a competitive feature that supported many enterprise-grade proprietarydatabaseservers,suchasMicrosoftSQLServerandOracle,butthatwasalwaysastretch, tosay the least.Although it seemed toenablea lotof flexibilityandneat tricks, it has proven to be a source ofmanyproblems and is disabled bydefault.Thisstorageengineisdisabledbydefault inMySQL;toenableit,youneedtostarttheMySQLserverbinaryusingthefederatedoption.

OverviewoftheNDBclusterNDBcluster(alsoknownasNDB)isanin-memorystorageengineofferinghighavailabilityanddatapersistencefeatures.

TheNDBclusterstorageenginecanbeconfiguredwitharangeoffailoverandloadbalancing options, but it is easiest to startwith the storage engine at theclusterlevel.NDBclusterusestheNDBstorageenginethatcontainsacompleteset of data, which is dependent only on other datasets available within thecluster.

The cluster portion of the NDB cluster is configured independently of theMySQLservers.InanNDBcluster,eachpartoftheclusterisconsideredtobeanode.

Eachstorageenginehasitsownadvantageandusability,asfollows:

SearchEngine:NDBClusterTransactionsdata:InnoDBSessiondata:MyISAMorNDBClusterLocalizedcalculations:MemoryDictionary:MyISAM

Thefollowingdiagramwillhelpyouunderstandwhichstoreengineyouneedtouseforyourrequirement:

Nowyouhaveabetter ideaaboutvariousstorageenginesalongwithdifferentusecases,whichwillhelpyoutomakeadecisionbasedonyourneeds.

It's time to move on to our next topic where we will look at delightful newfeaturesavailableinMySQL8.

ImprovedfeaturesinMySQL8

The MySQL database development team has recently announced its majorreleaseasMySQL

8DevelopmentMilestoneRelease(DMR). Itcontainssignificantupdatesandfixesforproblemsthatweremuchneeded.

Youmight bewonderingwhy it's 8 after 5.7!Were the intermediate versions,thatis,6and7,missout?Ofcoursenot!Actually,6.0waspreservedaspartofthe changeover to a more frequent and timely release, while 7.0 for theclusteringversionofMySQL.

Let'sseesomeexcitingfeaturesthathavebeenintroducedinthislatestversion,as depicted in the following diagram:

It's time to look atMySQL 8 features in detail, which makes us excited andconvincedaboutthereasonsforamajorversionupgradeofMySQL.

Transactionaldatadictionary

Upuntilthepreviousversion,theMySQLdatadictionarywasstoredindifferentmetadatafilesandnon-transactionaltables,butfromthisversiononwards,itwillhaveatransactionaldatadictionarytostoretheinformationaboutthedatabase.Nomore.frm,.trg,or.par files.All informationwillbestored in thedatabase,which removes the cost of performing heavy file operations. There werenumerous issueswith filesystemmetadata storage such as the vulnerability ofthe filesystem, exorbitant file operations, difficult to handle crash recoveryfailures,orreplication;itwasalsodifficulttoaddnewfeature-relatedmetadata.Now this upgrade has made it simple by storing information in a centralizedmanner,andwillhaveimprovedperformanceasthisdatadictionaryobjectcanbecachedinmemory,similartootherdatabaseobjects.

ThisdatadictionarywillhavedatathatisneededforSQLqueryexecutionsuchas catalog information, character sets, collations, column types, indexes,database information, tables, stored procedures, functions and triggers, and soon.

Roles

InMySQL 8, the privileges module has been improved by introducing roles,which means a collection of permissions. Now we can create roles with anumberofprivilegesandassignthemtomultipleusers.

The problem with the previous version was that we were not able to definegenericpermissionsforagroupofusersandeachuserhasindividualprivileges.Supposeif thereare1,000usersalreadyexistingthathavecommonprivileges,andyouwanttoremovethewritepermissionsforthese1,000users,whatwouldyouhavedone in thepreviousversion?Youwouldhavehad to take the time-consuming approach of updating each user, right? Arrgh! That's a long, longtask.

NowwithMySQL8, it is easy to update any change in privileges.Roleswilldefine all the required privileges and this rolewill be assigned to those 1,000users.Wejustneedtomakeanyprivilegechangesintheroleandalluserswillautomaticallyinherittherespectiveprivileges.

Rolescanbecreated,deleted,grantorrevokepermission,grantorrevokefromtheuseraccount,andcanspecifythedefaultrolewithinthecurrentsession.

InnoDBautoincrement

MySQL 8 has changed the auto-increment counter value store mechanism.Previously, it was stored in thememory,whichwas quite difficult tomanageduring server restarts or server crashes. However, now the auto-incrementcountervalueiswrittenintotheredologwheneverthevaluegetschangedand,on each checkpoint, it will be saved in the system table, which makes itpersistentacrosstheserverrestart.

Withthepreviousversion,updateoftheauto-incrementvaluemayhavecausedduplicateentryerrors.Supposeifyouupdatedthevalueofauto-incrementinthemiddleofthesequencewithalargerthanthecurrentmaximumvalue,butthensubsequent insertoperationscouldnot identify theunusedvalues,whichcouldcause a duplicate entry issue. This has been prevented by persisting the auto-incrementvalue,hencesubsequent insertoperationscanget thenewvalueandallocateitproperly.

If server restarthappened, theauto-incrementvaluewas lostwith thepreviousversionasitwasstoredinmemoryandInnoDBneededtoexecuteaquerytofindoutthemaximumusedvalue.Thishasbeenchanged,asthenewerversionhasthecapabilitytopersistitsvalueacrosstheserverrestart.

Duringtheserverrestart,InnoDBinitializesthecountervalueinmemoryusingthemaximum value stored in the data dictionary table. In case of server crashes,InnoDB initializes the auto-increment counter value that is bigger than the datadictionarytableandtheredolog.

<strong>ALTERTABLEtable1ALTERINDEXix_table1_col1INVISIBLE;</strong><br/><strong>ALTERTABLEtable1ALTERINDEXix_table1_col1VISIBLE;</strong>

ImprovingdescendingindexesDescendingindexesexistedinversion5.7too,buttheywerescannedinreverseorder,whichcausedperformancebarriers.To improveperformance,MySQL8hasoptimizedthisandscanneddescendingindexesinforwardorder,whichhasdrasticallyimprovedperformance.

Italsobringsmultiplecolumnindexesfortheoptimizerwhenthemostefficientscan order has ascending order for some columns, and descending order forothercolumns.

<strong>SETGLOBALmax_connections=1000;</strong>

<strong>SETPERSISTmax_connections=1000;</strong>

ExpandedGISsupportUntil thepreviousversion, it supportedonlyone coordinate system, a unitless2D

placethatwasnotreferencedtoapositiononearth.NowMySQL8hasaddedsupport for aSpatialReferenceSystem (SRS)with geo-referenced ellipsoidsand2Dprojections.SRShelpsassigncoordinates toa locationandestablishesrelationshipsbetweensetsofsuchcoordinates.ThisspatialdatacanbemanagedindatadictionarystorageastheST_SPATIAL_REFERENCE_SYSTEMStable.

DefaultcharactersetThe default character set has been changed from latin1 to UTF8. UTF8 is thedominatingcharacterset,thoughithadn'tbeenadefaultoneinpreviousversionsofMySQL. Along with the character set default, collation has been changedfromlatin1_swedish_ci to utf8mb4_800_ci_ai.With these changes globally accepted,charactersetsandcollationsarenowbasedonUTF8;oneofthecommonreasonsis because there are around 21 different languages supported by UTF8, whichmakessystemsprovidemultilingualsupport.

Extendedbit-wiseoperations

InMySQL5.7,bit-wiseoperationsandfunctionswereworkingforBIGINT(64-bitinteger)data typesonly.WeneededtopassBIGINTasanargumentanditwouldreturn the result as BIGINT. In short, it had maximum range up to 64 bits toperformoperations.AuserneedstodoconversiontotheBIGINTdatatypeincasetheywanttoperformitonotherdatatypes.Thistypecastingwasnotfeasiblefordatatypeslargerthan64bitsasitwouldtruncatetheactualvalue,whichresultedininaccuracy.

MySQL8hasimprovedbit-wiseoperationsbyenablingsupportforotherbinarydata types such as Binary,VarBinary, and BLOB.Thismakes it possible to performbit-wiseoperationsonlargerthan64-bitdata.

Nomoretypecastingneeded!Thisallowsthetakingofargumentsandreturningresultslargerthan64bits.

InnoDBMemcachedMultiple get operations are now possible with the InnoDB memcached plugin,whichwill really help in improving the read performance.Now,multiple keyvalue pairs can be fetched in a single memcached query. Frequentcommunicationtraffichasalsobeenminimizedaswecangetmultipledatainasingleshot.

RangequeriesarealsosupportedbytheInnoDBMemcachedplugin.Itsimplifiesrangesearchesbyspecifyingaparticularrangeandretrievesvalueswithin thisrange.

<strong>SELECT*FROMtable1WHEREid=5FORUPDATENOWAIT;</strong><br/><strong>SELECT*FROMtable1FORUPDATESKIPLOCKED;</strong>

JSON

JSONsupporthadbeenimplementedinMySQL5.7;itwaswell-acknowledgedfeature.

InMySQL8 ithasaddedvarious functions thatwouldallowus togetdatasetresultsinJSONdataformat,virtualcolumns,andtentatively15SQLfunctionsthatallowyoutosearchanduseJSONdataonserverside.InMySQL8thereareadditionalaggregationfunctionsaddedthatcanbeusedinJSONobjects/arraystorepresent loadeddata ina furtheroptimizedway.Thefollowingare the twoJSONaggregationfunctionsthatwereintroducedinMySQL8:

JSON_OBJECTAGG()

JSON_ARRAYAGG()

Cloud

InMySQL8anewoption is introducedinnodb_dedicated_server,whichwouldbehelpfulforverticalscalingoftheservers.Itactuallyautomatically

detectsthememoryallocatedtothevirtualserverandappropriatelysetMySQL8without

any need to change configuration files. These would be very handy featuresconsidering

theadoptionofvirtualizationandcloudisthere.Infactwiththisconfiguration,

youmightnotevenneed toget shellaccessof server toedit theconfigurationfiles.

You can do this with the new SET PERSIST feature that can set relevantconfigurationfromtheMySQLcommandlineitself,which

can enhance security further as you almost wouldn't need shell access of theserver.

Resourcemanagement

MySQL8hascomeupwithawonderfulresourcemanagementfeaturethatwillallowyou toallocate resource to threads runningona server,whichwouldbeexecutedbasedontheresourcesconfiguredforthegroup.Currently,CPUtimeisaresourcethatcanbeconfiguredforagroup.Withthis,youcantweakyourworkloadswithvirtualresourcemanagementwithinMySQLitself.MySQLwillidentifyonstartupnumbersofvirtualCPUsavailableandafter thatuserswithappropriateprivilegescanmapthevirtualCPUswithresourcegroupandalignthreadmanagementtothesegroups.

Weexpect to seemore featuresby the timeMySQL8 is available forgeneraluse.LetusnowlookatbenefitsofusingMySQL8.

BenefitsofusingMySQL8Whetheryouareadeveloperoranenterprise,youwouldobviouslychooseonethatprovidesgoodbenefitsandresultswhencomparedtootherrelatedproducts.MySQL

providesnumerousadvantagesas thefirstchoice in thiscompetitivemarket. Ithas various powerful features available that make it a more comprehensivedatabase.Let'snowgothroughsomebenefitsofusingMySQL.

Security

Thefirst thing thatcomes tomindissecuringdatabecausenowadaysdatahasbecomepreciousandcanimpactbusinesscontinuityiflegalobligationsarenotmet; in fact, it canbe sobad that it canclosedownyourbusiness inno time.MySQL is themost secure and reliable databasemanagement systemused bymanywell-knownenterprisessuchasFacebook,Twitter,andWikipedia.Itreallyprovidesagoodsecuritylayerthatprotectssensitiveinformationfromintruders.MySQLgivesaccesscontrolmanagementsothatgrantingandrevokingrequiredaccessfromtheuseriseasy.Rolescanalsobedefinedwithalistofpermissionsthatcanbegrantedorrevokedfortheuser.

All user passwords are stored in an encrypted format using plugin-specificalgorithms.

Scalability

Day by day, the mountain of data is growing because of extensive use oftechnologyinnumerousways.Becauseofthis,loadaverageisgoingthroughtheroof.Insomecases,itisunpredictablethatdatacannotexceeduptosomelimitor number of userswill not go out of bounds. Scalable databaseswould be apreferable solution so that, at any point,we canmeet unexpected demands toscale.MySQLisarewardingdatabasesystemforitsscalability,whichcanscalehorizontally and vertically; in terms of data, spreading database and load ofapplicationqueriesacrossmultipleMySQL

servers is quite feasible. It is pretty easy to add horsepower to the MySQLclustertohandletheload.

An open source relationaldatabasemanagementsystemMySQLisanopensourcedatabasemanagementsystemthatmakesdebugging,upgrading, and enhancing the functionality fast and easy. You can view thesourceandmakethechangesaccordinglyanduseitinyourownway.Youcanalso distribute an extended version of MySQL, but you will need to have alicenseforthis.

HighperformanceMySQL gives high-speed transaction processing with optimal speed. It cancache the results, which boosts read performance. Replication and clusteringmake the system scalable for more concurrency and manages the heavyworkload. Database indexes also accelerate the performance of SELECT querystatementsforsubstantialamountofdata.Toenhanceperformance,MySQL8

hasincludedindexesinperformanceschematospeedupdataretrieval.

Highavailability

Today, in theworldofcompetitivemarketing,anorganization'skeypoint is tohave their system up and running. Any failure or downtime directly impactsbusiness and revenue; hence, high availability is a factor that cannot beoverlooked.MySQLisquitereliableandhasconstantavailabilityusingclusterandreplicationconfigurations.

Cluster servers instantly handle failures andmanage the failover part to keepyoursystemavailablealmostallthetime.Ifoneservergetsdown,itwillredirecttheuser'srequesttoanothernodeandperformtherequestedoperation.

Cross-platformcapabilities

MySQL provides cross-platform flexibility that can run on various platformssuchasWindows,Linux,Solaris,OS2,andsoon.IthasgreatAPIsupportforthe allmajor languages,whichmakes it very easy to integratewith languagessuch as PHP,C++, Perl, Python, Java, and so on. It is also part of theLinuxApache MySQL PHP (LAMP) server that is used worldwide for webapplications.

It's now time to get our handsdirty and look atMySQL8; let's startwith theinstallationofMySQL8onaLinuxplatforminourcase.WepreferMySQL8onaLinuxoperatingsystemas thathasbeenacommonusecaseacrossmanyorganizations.WewillbediscussingmoreinstallationinChapter2,InstallingandUpgradingMySQL8.Youcanuse itonotherplatforms thatMySQLsupports,suchasWindows,Solaris,HP-UNIX,andsoon.LinuxprovidesvariouswaystoinstalltheMySQLserver,asfollows:

RPMpackage

YUMrepository

APTrepository

SLESrepository

Debianpackage

TARpackage

Compilingandinstallingfromthesourcecode

LimitationsofMySQL8Acoin has two sides; similarly, benefits of also usingMySQL8would comealongwithafewlimitations.LetuswalkthroughafewareasofMySQL8now.

NumberoftablesordatabasesThenumberofdatabasesor tablesarenota limitationforMySQL8;however,theoperatingsystemfilelimitcanbealimitationforMySQL8.StorageEngineInnoDBisallowedtoscaleuptofourbilliontablesasitspeaknumber.

TablesizeYoumayhitmaximumtablesizelimit,whichisnotrestrictedfromMySQL8;however,itmaybebecauseofoperatingsystemfilesystemlimits.

JoinsInasinglejoin,onecanuse61tables,whichcanbereferred.Itisalsoapplicableto the tables that are referenced in view definition. Joins that are part ofsubqueriesandviewsarealsoconsideredtobepartofthelimitation.

Windowsplatform

There are few limitations when you have MySQL 8 used on the Windowsplatform:

Memory:32-bitarchitecturehaslimitationtouseonly2GBofRAMforaprocess.

Ports: In case you have a high number of concurrency youmight comeacross Windows platform limitation of having 4000 ports available forclientconnectionsintotal.

Case-insensitivity: The Windows platform doesn't have case sensitivity,which is why tables and databases need to be deliberately managed forcase-insensitivity.

Pipes:|, generally referred as pipe signs, they are not fully supported inWindows. You might come across them in a few scenarios while doingdatabaseadministrationactivities.

Pathname separator: MySQL 8 escape character is \, which is the

pathnameseparatorforWindows.Hencewhileusingpathseparatoryoucandoubleslashas"\\"asanalternativeforapathnameseparator.

TablecolumncountThe table column for each table inMySQL8 has a limit of 4096 columns. Itmightvarybasedonafewotherfactorsforcolumnscountlimit,asstatedinthefollowingsection.

RowsizeMySQLtableshavealimitof65,535bytesforarow,althoughstorageenginessuchasInnoDBarecapableofsupportinglargerchunks.

InnoDBstorageengineLimitations on InnoDB storage engine are what we will talk about a bit morespecificallyasInnoDBnowwithMySQL8willplayaprominentrole.

Limitations of InnoDB storageengine

WewillhaveaquickglanceatafewofthelimitationsofInnoDBstorageengine:

Thenumberofindexessupportedcanbemaximum64foratable

For tables thatusecompressedordynamic row format;3072 is the indexkeyprefixlengthlimit

Fortablesthatusecompactorredundantrowformat;767istheindexkeyprefixlengthlimit

Total columns in a table, which includes virtual generated columns, arelimitedtoamaximumof1,017

16columnsisthemaximumpermittedformulti-columnindexes

ThecombinedInnoDBlogfilesizecannotexceed512GB

MaximumtablesizesupportedbyInnoDBis256TB

AdminAPIisnotsupportedwhileusingunixsocketconnections

Multi-byte characters might give you unreliable aligned columns whileformattingofresultsinInnoDBclusters

Restrictions

WewillnowhaveaquickglanceatafewoftherestrictionsoftheInnoDBstorageengine:

Deletefromtablename: Itdoesn'tactuallydelete thecomplete table, insteaditdeleteseachrowofthetableoneafteranother.

Show table status: It wouldn't provide you accurate data all the time; itprovidesestimates.

When counting rows, the number of rows provided by count(*) is notaccuratebecauseofconcurrency; itwouldcountonly thosecountsvisibletotransactionscurrentlyavailable.

Ifthereismultipleanalyzetablequeriesexecuted,lateronewillbeblockeduntilthefirstonegetscompleted.

InnoDBkeepsanexclusive lockon the indexat theendassociatedwith theauto_incrementcolumn.

Inacasetheauto_incrementintegerrunsoutofthevalue;thefollowinginsertoperationswouldshowusduplicate-keyerrors.

Foreignkeysthatarecascadedcannotactivatetriggers.

There are a few column names reserved by MySQL that InnoDB uses forinternalpurposes.Thefollowingareafewsuchcolumnnames:

DB_ROW_ID

DB_TRX_ID

DB_ROLL_PTR

DB_MIX_ID

Wemightcomeacrossoutputshowninthefollowingexampleincaseofsuchreservedcolumnnamesused:

mysql>CREATETABLEchintan(c1INT,db_row_idINT)

ENGINE=INNODB;ERROR1166(42000):Incorrectcolumnname'db_row_id'

InnoDB locks are released immediately after the transaction is aborted orcommitted,whichisheldbyatransaction.

Theadditionoftablelocksarenotsupported,aslocksareimplicittocommitandunlocktables

Datadictionary

Letushavealookatafewknownlimitationsofdatadictionary:

IndividualMyISAMtablesforbackupandrestorearenotsupportedbymerelycopyingthefiles.

ManuallycreateddirectoriesfordatabasesarenotsupportedbyMySQL8.For instance, using mkdir would have no impact on MySQL server datadictionary.

DDLoperationswouldtakemoretimethanexpectedbecausesuchoperationsarewritten tostorage,undologsandredoinsteadof.frm filesaswhatwewouldhaveseeninpriorversionsofMySQL.

LimitationsofgroupreplicationinMySQL8

It'snowtimetodiscussafewlimitationsofgroupreplicationinMySQL8:

Largetransactions: Transactions that result toGTID contents cannot bereplicatedbetweentherestofthemembersofthegroupifthey'retoolarge.It is suggested to use smaller chunks of data that cannot be replicated inaroundfivesecondstogroupmemberstoavoidfailures.

Clusterfromagroup:Ifyoutrytocreateclustersfromanexistinggroupreplicationsetupitwillresult inanerrorastheinstancewouldalreadybepart of a replication group. This is noticed currently only in MySQL'swizardmodeonly;analternativesolutionfortheissueistodisablewizardmode.

Serializable isolation level: Serializable isolation level is not supportedwhenmulti-primarygroupsareused,whichisthedefaultconfiguration.

DDL and DML operations: If there is concurrent DDL and DML

operationsexecutedagainstthesamedataobjectbutondifferentserversisnotsupportedwhenmulti-primarygroupmodeisused.

Replication checksum: Currently MySQL design limitations createrestrictionsofhavingreplicationeventchecksums.

LimitationsofpartitioningWewillbediscussinglimitationsofpartitioninginthissection.

ConstructsprohibitionThefollowingaretheconstructsthatarenotallowedinexpressionsofpartitions:

DeclaredvariablesUservariablesStoredproceduresStoredfunctionsUDFsPlugins

OperatorsThere are a few operators that are not permitted in partition expressions suchas<<,>>,|,&,~and^.Resultsforarithmeticoperatorssuchas+,-,and*musthaveanintegervalueorNULL.

Tables

Thefollowingareafewspecificareasthatshowuslimitationsofpartitioningontables:

Themaximumnumberofpartitions supportedbyMySQL8 fora table is8192.Thislimitalsoconsiderssub-partitions.

Fulltextindexandsearchisnotsupportedonpartitionedtables.

Tablesthataretemporarycannotbepartitioned.

Logtablescan'tbepartitioned.

ForeignkeysarenotsupportedonpartitionedInnoDBstorageengine.

Thedata typeofpartitionkeys shouldbean integercolumnorcanbeanexpression to an integer. Expression or column values may be NULL;however,expressionsthatincludeENUMarenotsupported.

Upgradingpartitionedtables thathavebeenpartitionedbyKEYwouldhave

tobereloaded,whichstandstrueotherthantheInnoDBstorageengine.

Wehavesofardiscussedoverview, features,benefits,anda fewlimitationsofMySQL.

LetusnowwalkthroughthewonderfulusecasesofMySQL.

UsecasesofMySQL

MySQL has many advantages because it has its foot in many industries andvarioususecases across theglobe.The importance ofMySQLdoesn't dependonlyonhowmuchdatayouhave,it'sratherwhatyouaregoingtodowiththedata.Datacanbesourcedandanalyzedfromunpredictablesourcesandcanbeusedtoaddressmanythings.

Let's now look at use cases with real-life importance made on renownedscenarioswiththehelpofMySQL:

The preceding figure helps us understand where MySQL is serving variousindustries.

Thoughit'snotanextensivelistofindustrieswhereMySQLhasbeenplayingaprominentroleinbusinessdecisions,let'snowdiscussafewoftheindustries.

Socialmedia

Social media content is information, and so are engagements such as views,likes,demographics,shares,follows,uniquevisitors,comments,anddownloads.At the end of the day, what matters is how your social media-related effortscontributetothebusiness.

OnenotableexampleisFacebook,whereMySQLhadbeenusedextensively.Ontop of MySQL where petabytes of data was used to serve likes, shares, andcomments. Facebook has developed the RocksDB storage engine on top of theMySQL InnoDB storage engine, which leverages many advantages of InnoDBstorageengineasFacebookwantedtoprimarilyfocusonstorageoptimization.ThoughcurrentlyMySQLisstillusedlargelyforothercommonapplications.

Government

The era of MySQL has been playing a significant role in government too;government bodies have been using MySQL extensively because of splendidreturnoninvestmentsandpromotingopensource.Infact,thegovernmentsectoriscarryingoutahugenumberofimplementationsofMySQLworldwide.

Thismaycomeasasurprisetoyou;USNavyusesMySQLforitscriticalflightplanningactivities.Therearevariousactivitiessuchasweatherconditions,flightplans, fuel efficiency, maintenance of flights, and many more that are beingtrackedwiththehelpofMySQLasthedatabase.It'sano-brainerthatitneedstorun 24x7 with full redundancy;MySQL was able to achieve this serving USNavyaircraftacrosstheglobe.

MediaandentertainmentYouTube isalsooneof theprominentusersofMySQL.AnytimeyouwatchavideoonYouTube itgetsdata froma relationaldatabaseorablobstoreusingMySQL.YouTubealsousesVitess;aproject thatwas releasedbyYouTube tofrontendMySQL.VitesshelpstodolotsofoptimizationandactsasaproxytoserveeachdatabaserequestusingMySQL.MySQLreplicasareheavilyusedinYouTube's implementation; leveraging MySQL caching was one of the otherprominentfactorsforYouTube.

Frauddetection

Whenitcomestosecurity,frauddetection,orcompliance,andpreciselyifyoursolutionhelpsyouinidentifyingandpreventingissuesbeforetheystrike,thenitbecomesasweetspotforbusiness.Mostofthetime,frauddetectiontakesplacealongtimeafterthefraudhasoccurred,whenyoumighthavealreadysufferedloss. The next stepswould be obviously tominimize the impact of fraud andimproveareasthatcouldhelpyoupreventthisfrombeingrepeated.

Manycompanieswhoareintoanytypeoftransactionprocessingorclaimsusefraud detection techniques extensively. MySQL helps to analyze transactions,claims, and so on in real time, along with trends or anomalous behavior topreventfraudulentactivities.

PayPal is one of such use cases that has built fraud detection system usingMySQL.

PayPal has more than 100 million active users, which is distributed to US,Japanese, andEuropean data centers.High-availability for such use cases is akeycriteriaalongwithperformance,whichMySQLhasbeenabletodeliverasexpected.

BusinessmappingNetflix has millions of subscribers; it uses MySQL for running its billingsystems.

ThecorebillingsystemofNetflixonMySQLisaprominentbackboneforanybusiness.

Netflixhasbillionsofrowsofdataconcurrentlyupdatedandofconsistingdatasince

itsinceptiontwodecadesago.Compliancewasoneofthekeyfactorsalongwithmigration

fromOraclewithminimaldowntime;bothofthesewereachievedwithMySQLandhas

beenexpandingtremendouslyeveryotherday.

E-commerce

Uber is one of the other well-known customers of MySQL. Uber had beengrowingenormouslyworldwide,andscalability,high-availability,andreturnoninvestmentswereafewoftheimportantcriteriatobeworkedupon.UberusesMySQLasitsprimarydatabaseforitsknownprivatecartransportationservice.UberheavilyusesschemalessdatabasearchitectureasitsbackendasalayeronMySQL.

There are many real-world MySQL use cases that have changed humanity,technology, predictions, health, science and research, law and order, sports, e-commerce, power and energy, financial trading, robotics, and many more.MySQLisanintegralpartofourdailyroutine,whichisnotevidentallthetime,butyes,itplaysasignificantroleinwhatwedoinmanyways.

Summary

In this chapter, we started with an overview of MySQL along with majorfeatures of the MySQL database and explored the newly added features inMySQL8.Afterthis,wetookadeepdiveintoexcitingnewfeaturesofMySQL8alongwith

benefitsofusingMySQLforyourbusinessapplications.WeunderstoodMySQL8'scurrent

limitations and restrictions, which is important for us when performing theimplementations.

Finally,weglancedthroughafewimpressiveusecasesfromtherealworldthatplay

prominentrolesinourdailyroutine,andtheyalluseMySQLastheirdatabase.

In the next chapter, we will learn detailed steps for installing MySQL 8 ondifferent

platforms. The chapter also covers methods to upgrade or downgrade fromMySQL8,and

theywillallbediscussedindetail.

Installing and Upgrading MySQL8

Inthepreviouschapter,weprovidedanoverviewofMySQLalongwithMySQL8'snewfeatures,usecases,andlimitations.MySQLisveryflexibleintermsofplatforms, such as RedHat, Fedora, Ubuntu, Debian, Solaris, MicrosoftWindows, and so on. It has the support of an API to connect with differentlanguages, such as C, C++, C#, PHP, Java, Ruby, and many more. For anyprogrammingplatform,themostimportantandmonotonoustaskistosetuptheenvironment with the necessary software tools. That won't be the case forMySQL

8,asthischapterisfocusedonsettinguptheenvironmentwithMySQL8.

ThischapterexplainsMySQL8's installationsteps indetailwith thenecessaryprerequisites.

SeparateinstallationstepsareprovidedtosetupMySQL8onvariousplatforms.ThechapteralsocoversmethodstoupgradetoordowngradefromMySQL8.

Wewillcoverthefollowingtopicsinthischapter:

TheMySQL8installationprocess

Post-installationsetupforMySQL8

MySQL8upgrading

MySQL8downgrading

TheMySQL8installationprocessThis section will guide readers in MySQL 8 version selection, where to getMySQL 8 from, and how to install MySQL 8. It also explains the post-installationstepsrequiredforsetup.ThischapterprovidesinformationonhowtoupgradeordowngradefromMySQL8.

Generalinstallationguide

MySQL8 isavailableonmanyoperatingsystemswithdifferentversions.TheMySQL

8releaseismanagedintwoways:

Developmentrelease:Thishasthenewestfeaturebutisnotrecommendedforuseinproduction

Generalrelease:Thisisastablereleaseanduserscanuseitforreleaseinproductionalso

NamingconventionsarefollowedineachreleaseofMySQL8,whichindicatesitsstatus.

Eachreleasenameconsistsofthreedigitsandanoptionalsuffix.Forexample,mysql.8.1.2-rc.Thenumbersareinterpretedasfollow:

Thefirstnumber(8)indicatesamajorversionoftherelease.

The second number (1) indicates a minor version of the release. Acombinationofmajorandminornumbersdescribestheseriesoftherelease.

The third number (2) indicates the versionwithin the release series. It isincrementedoneachbugfixrelease.

Themostrecentversionofthereleaseisthemostpreferableforuse.ThesuffixgivenintheexampleindicatesthestabilityoftheMySQL8release.TheMySQL8

releasefollowsthreesuffixes:

DevelopmentMilestoneRelease (dmr):MySQL8 follows themilestonemodel,whereeachmilestoneindicatesthoroughlytestedfeatures.

ReleaseCandidate (rc):Anewfeaturemightget released in thisversionbuttheaimistofixbugswithinthepreviouslyreleasedfeatures.

Absence of a suffix: This indicates General Availability (GA) orproductionrelease.Thisreleaseisstableandpassedthroughearlierstages.

Itisreliableandsuitableforuseinproduction.

As described, preceding each release is the DMR, followed by the RC, andfinallytheGAreleasestatus.Now,afterdecidingtheMySQL8versionfortheinstallation,it'stimetoselectthedistributionformat.

Thebinarydistributionisrecommendedforgeneral-purposeuse.It isavailableinnativeformatsformanyplatforms.Forexample,theRPMpackageforLinuxandDMG

packageforOSX.

DownloadingMySQL8

TogetMySQL8fromtheofficialsite,refertothefollowingURL:http://dev.mysql.com/downloads/.MySQLalsoprovidesamirrorsite:http://dev.mysql.com/downloads/mirrors.html.Whenyoureachthedownloadpage,youcanseetheversionselectiontabatthebottomsideofthepage,wheretwotabsaredisplayed:

GenerallyAvailable(GA)release

Developmentrelease

Basedontheprevioussection,selectthesuitableversionfromthelistandclickontheDownloadbutton.

VerifyingthepackageintegrityThis is a stage where the downloaded package is available and ready for theinstallation.

It'sanoptionalstep,butwerecommendittoavoiderrorsduringtheinstallationprocess.Therearethreedifferentwaysavailabletocheckintegrity:

UsingMD5checksums

Usingcryptographicsignatures

UsingtheRPMintegrityverificationmechanism

<strong>E:\Softwares\md5>md5.exe<br/>E:\Softwares\mysql-installer-community-5.7.19.0.msi</strong><br/><strong>2578BFC3C30273CEE42D77583B8596B5<br/>E:\Softwares\mysql-installer-community-5.7.19.0.msi</strong>

Performthefollowingstepsforthegraphicaltoolexecution:

1. Openthelink:http://www.nullriver.com/index/products/winmd5sum.

2. ClickontheDownloadWinMD5Sumoptiononthepage.ItwilldownloadwinMD5Sum.exeontoyourcomputer.

3. RunthedownloadedInstall-winMD5Sum.exeandinstallitonyourlocalmachine.

4. Aftersuccessfulinstallation,openthewinMD5Sumtool.ThisopensonedialogboxwhereyouhavetoselectthedownloadedMySQL.msifile.

5. Clickonthecalculatebutton.ThiswillcalculatetheMD5checksumofthedownloadedfile.

6. EntertheMD5checksumavailableontheMySQLdownloadpageinthecomparetextboxandpressthecomparebutton.

<strong>cmd>gpg--verifypackage_name.asc</strong>

<strong>cmd>rpm--checksigpackage_name.rpm</strong>

ThistechniqueofverificationismorereliablethantheMD5checksumbutitisverycomplexandrequiresmoreeffortforintegritychecks.

Installing MySQL 8 on MicrosoftWindows

MySQLisavailableforboth32-bitand64-bitversions.Therearedifferentwaysavailable to install MySQL 8 on Microsoft Windows. The most commonapproachistouseaninstaller,whichinstallsandconfiguresMySQL8onyourlocalsystem.

BeforeinstallingMySQLCommunity8.0Server,makesurethattheMicrosoft Visual C++ 2015 redistributable package has beeninstalledonthesystem.

MySQL8 either runs as a standard application or runs as aWindows service.Useit,astheserviceenablesuserstocontrolandmeasureoperationsusingtheWindows service management tool. Three major distribution formats areavailableforeachplatform:

Installerdistribution:ThisincludestheMySQL8serveralongwithotherproductssuchasMySQLWorkbench,MySQL

for Excel, andMySQLNotifier.An installer is also useful for upgradingproductsintootherversions.

Sourcedistribution:Asthenameimplies,thiscontainsallthesourcecodealongwithallthesupportedfiles.TheVisualStudiocompilerisrequiredtomakeitexecutable.

Binary distribution: This distribution is available in ZIP file format. Itcontains all the required files except the installer.Theuserhas tounpackthefileintoaselecteddirectory.

Windows-specificconsiderations

BeforeinstallingMySQL8onMicrosoftWindowsconsiderfollowingpoints:

Antivirussoftware:Asweknow,antivirussoftwareusesthefingerprintingtechnique,whichwillconsiderrapidlychangedfilesasapotentialsecurityrisk.InMySQL8,therearesomedirectoriesthatcontainMySQL8relateddataandtemporarytablesinformationandareupdatedfrequently.So,thereisapossibilitythatantivirussoftwarewillconsiderthosefilesasspam.Thiswillalsoimpactperformance.

Antivirus software provides configurations to exclude some of thedirectories,soitisrecommendedtoexcludetheMySQL8datadirectoryandtempdirectory.MySQL

8, by default, stores temporary data into a Microsoft Windows temporarydirectory.

To change this default configuration inMySQL 8, refer to my.ini file's tempdirparameter.

Large table support:UseMySQL8 onNTFS or any new filesystem tosupportlargetableswhosesizeismorethan4GB.Fortheselargertables,theuserhastodefinetheMAX_ROWSandAVG_ROW_LENGTHpropertiesatthetimeoftablecreation.

MySQL8installationlayout

MicrosoftWindows, by default, considers the C:\Program Files directory for theMySQL8installation.However,wehaveachoiceforthedirectoryselectionatthe time of installation. Whatever the location of the installation, thesubdirectorystructureafterinstallationremainssame.FortheMicrosoftWindowlayout,refertothefollowingtable:

Directory ContentsofDirectory Notes

binmysqldserver,clientandutilityprograms

%PROGRAMDATA%\MySQL\MySQL

Server8.0\ Logfiles,databases

TheWindowssystemvariable%PROGRAMDATA%

defaultstoC:\ProgramData

examples Exampleprogramsandscriptsinclude Include(header)files

lib Libraries

share

Miscellaneoussupportfiles,includingerrormessages,charactersetfiles,sampleconfigurationfiles,SQLfordatabaseinstallation

Youcanlearnmoreaboutthistopicathttps://dev.mysql.com/doc/refman/8.0/en/windows-installation-layout.html.

Choosing the right installationpackage

There are multiple options available for package formats while installingMySQL 8 onWindows. MySQL provides a facility to debug the installationprocessusingprogramdatabase(pdb)files.ThesefilesareavailableinaZIPdistribution:

Theinstallerpackage:This isawizard-basedprocessandiseasy touse.Theinstallerpackageisavailable

for32-bitsonlybutcaninstallMySQL8onthe64-bitconfigurationalso.Itdoes

notcontainthedebuggingcomponentofMYSQL;wehavetodownloaditseparatelyin

the formof aZIP file.The installer package is available in twodifferentformats:

Web Community: As the name implies, this is available for webinstallation.ItmeanstheInternet

is required for the installation using the web community. Its size isapprox19MB.

ItsnameisdefinedasMySQL-installer-communityby theappendingversion.

Community:Thispackage format is used for offline installation. Itssizeisapprox301MB.

Its name is defined as MySQL-installer-web-community by theappendingversion.

An installer is themost commonway for MySQL product installationandupgrade.

TheNoinstallArchives:Thisisamanualinstallationprocessthatcontainsfilesfortheincompleteinstallation

package.As it is amanual process, noGUI is available.The user has tomanually

install andconfigureMySQL8andotherproducts if required.Unlike theinstaller,

itprovides twodifferent files for32-bit and64-bit configuration inaZIPformat.

TheMySQL8installer

The MySQL 8 installer is mainly used to reduce the complexity of theinstallationprocessalongwiththemanagementofMySQLproductsrunningontheWindowsplatform.

Intheproductlist,wecanconsider:

MySQLservers

MySQLapplications

MySQLconnectors

Documentationandsamples

TheMySQL8installerhastwoeditions:

CommunityEdition:Thiscanbedownloadedathttp://dev.mysql.com/download

s/installer/. As described in previous section, bothWeb Community andCommunitypackageformatsareavailablefortheinstaller.

CommercialEdition:Refer tohttps://edelivery.oracle.com/ todownload theCommercialEdition.TheCommercialEditioncontainsalltheproductsthatareavailableintheCommunityEditionalongwiththefollowingproducts:

WorkbenchSE/EE

MySQLEnterprisebackup

MySQLEnterprisefirewall

Initialsetupinformation

Asmentionedpreviously,theinstallerwillguideauserthroughthewizard.OncewestarttheinstallerinourhostmachineitwilldetectalreadyinstalledMySQL

productsandconsider themina listofproducts tobemanaged.Thefollowingarethestepsthatarerequiredintheinitialsetupoftheinstaller:

1. MySQLinstaller licensingandsupportauthentication:This is the stepwhere the user must accept the license agreement before starting theMySQL8installation.Afteracceptingtheterms,theuserisallowedtoadd,update,orremoveMySQLproducts.IntheCommercialEdition,credentialsare required to unbundle products andmustmatchwith the user'sOracleaccountinthesupportsite.

2. Choosingasetuptype:ThisisthestepwheretheusermustselectMySQLproducts for installation. The installer also provides the option of apredefined setup,whichcontains a setofMySQLproducts.So,youhavethe flexibility of selecting one setup type as per your requirements. Thefollowingaresomesetupsavailableintheinstaller.

3. Developerdefault:This installs theversionofMySQL8 server thatwasselectedatthetimeofdownload:

MySQLserver

MySQLshell

MySQLrouter

MySQLWorkbench

MySQLforVisualStudio

MySQLforExcel

MySQLnotifier

MySQLconnectors

MySQLutilities

MySQLdocumentation

MySQLsamplesandexamples

4. Serveronly:ThisinstallsonlytheMySQLserver.

5. Clientonly:Thisisthesameasthedeveloperdefaultsetuptype,exceptitdoesnotcontaintheMySQL8serveroranyclient-specificpackageaddedtoit.

6. Full:ThisinstallsalltheavailableproductsofMySQL,suchasmysql-server,mysql-client,mysqladmin,andafewmore.

7. Custom: This option installs only those products that are selected by theuser from the catalog.Here, the user has the freedom to choose only therequiredproducts,ratherthaninstallingthecompletebundleofproducts.

8. Path conflicts: When the hosting system already contains a MySQLproductandtheuseris tryingtoinstalladifferentversionofthatMySQL

productonthesamepath,thentheinstallerwillshowapathconflicterrorin the wizard. The installer enables the user to take action on the pathconflictinthefollowingways:

ChooseadifferentlocationusingtheBrowsebuttonfromthewizard

Chooseadifferentsetuptypeorversionbycustomselection

Overwritetheexistingfolderbymovingontothenextstep

Cancelthewizardsteps,deleteexistingproducts,andstarttheinstalleragain

9. Check requirements: Each MySQL product has a package-rules.xml fileattachedtoit,whichcontainsalltheprerequisitesoftwarelists.Duringtheinitialsetup,theinstallerwillchecktheavailabilityoftherequiredsoftwareandprompttheusertoupdatethehostincaseofmissingrequirements.

10. MySQLinstallerconfigurationfiles:The installerconfigurationfilesarelocatedatC:\ProgramFiles.Thefollowingaretheconfigurationfiledetails:

FileorFolder

Description

FolderHierarchy

MySQLinstallerforWindows

This folder contains all of the filesneededtorunMySQLinstallerandMySQLinstallerConsole.exe, a command-lineprogramwithsimilarfunctionality.

C:\ProgramFiles(x86)

Templates

TheTemplatesfolderhasonefileforeachversionofMySQLserver.Templatesfilescontainkeysandformulastocalculatesomevalues

C:\ProgramData\MySQL\MySQL

installerfor

Windows\Manifest

dynamically.

package-

rules.xml

This file contains the prerequisites foreveryproducttobeinstalled.

C:\ProgramData\MySQL\MySQL

installerfor

Windows\Manifest

produts.xml

The products file (or product catalog)containsa listofallproductsavailablefordownload.

C:\ProgramData\MySQL\MySQL

installerfor

Windows\Manifest

Product

Cache

The Product Cache folder contains allstandalone MSI files bundled with thefullpackageordownloadedafterward.

C:\ProgramData\MySQL\MySQL

installerforWindows

Reference:https://dev.mysql.com/doc/refman/8.0/en/mysql-installer-setup.html

Installationworkflow

TheMySQLinstallerfollowsaworkflowforeachproduct:

1. Product download: The installer will download all the required productMSIfilesintotheProductCachefolder.

2. Product installation:The installermanages thestatusofeachproductbyReady|Install|Installing|Complete.

3. Product configuration: This phase uses a step-by-step configurationprocess for products. The installer will change the status from Ready |Configure.

4. Installationcomplete:Thisfinalizestheinstallationandtheusercanstartusingtheapplicationaftertheinstallation.

InnoDB cluster sandbox testsetup

Therearetwooptionsavailableforhigh-availabilityimplementationinMySQL8,usingtheinstaller:

StandaloneMySQLserver/ClassicMySQLreplication (default):Thisoption configuresmultiple servers manually or uses the latest version ofMySQLShelltoconfiguretheInnoDBcluster.

InnoDBclustersandboxtestsetup(fortestingonly):Thisisalsoknownasthesandboxcluster.ThisoptionallowsyoutocreateanInnoDBclusteronthe local system for testing only. The MySQL installer toolbar providesconfigurationsforanumberofinstancesinInnoDBclustering.

Clusternodesrunondifferentports.Afterconfiguration,clickontheSummarytabtogettheportdetailsofeachcluster.

Serverconfiguration

The MySQL installer performs some basic configurations for the MySQL 8server,including:

Theinstallerwillcreateamy.iniconfigurationfilefortheMySQL8server.Filecontentswillbedecidedaspertheselectedoptionsoftheinstallationprocess.

By default, the installer will add theWindows service for theMySQL 8server.

TherequireddefaultinstallationanddatapathsoftheMySQL8serverwillbeprovidedbytheinstaller.

TheinstallerwillcreatesomeuseraccountswithrolesandpermissionsforMySQL

8server.ItcancreateaWindowsuserwithlimitedprivilegestotheMySQL8server.

Using Show Advanced Options, MySQL installer allows for definingcustom paths for logging options. For example, you can configure theseparatepathforanerrorlog,showaquerylog,andmuchmore.

ThefollowingserverconfigurationisrequiredforMySQL8:

Serverconfigurationtype:Basedontheserverconfigurationtype,systemresourceswillbeassignedtotheMySQL8server.

Development: By considering the host as a personal workstation, itconfiguresMySQL8tousetheminimumamountofmemory.

Server: As servers, some other applications are also running on themachine,soitwillconfigureamediumamountofmemory.

Dedicated: In case of a dedicatedmachine for theMySQL8 server, thisoptionconfiguresthemaximumuseofavailablememoryfortheMySQL8server.

Connectivity:ThisoptionindicatestheconnectionfortheMySQL8server.Thefollowingoptionsareavailable:

TCP/IP: This option enables TCP/IP connection with MySQL 8.Users are allowed to define the port number alongwith the firewallsettingfortheportonthenetworkaccess.

Namedpipe:Thisoptionallowsyou todefine thepipelinenamefortheconnection.

Sharedmemory:ThisallowsyoutodefinethememorynamefortheMySQL8server.

Advanced configuration: This configuration enables additional loggingfeatureswhichwillmanage logs in individual files.Users are allowed toconfigurepathsforindividualfiles.Forexample,configuringacustompath

forabinarylog.

MySQL Enterprise Firewall: This option is used for the CommercialEditiononly.Check theEnableEnterpriseFirewalloption toenable thefirewall.

Accountsandroles:Accountsandrolesareusedtomanageaccessrightsfor theusers.During the installationprocess, theMySQL installer allowsyoutosetrootaccountpasswordsanduseraccounts.

Rootaccountpassword: It is requiredtoenterrootpasswordduringtheinstallationprocess.Theinstallerwillcheckthepasswordstrengthandgiveawarningifthereisaviolationofapredefinedpolicy.

MySQLuseraccounts:ThisisanoptionalstepwhereanewMySQLuser account defines with existing user roles. Predefined roles havetheirownprivileges.

Windows service: The MySQL 8 service can be configured in thefollowingtwoways:

Configureas aWindow service:This is thedefault option selectedduringtheinstallationprocess.Itfurtherprovidestwooptions:

Start service on system startup: This option is selected bydefault and will start the MySQL 8 service automatically atsystemstartup.

RunWindow service as: This option allows attaching the useraccount with the MySQL 8 service. By default, the systemaccount is selected where the service is considered as networkservice.

Withacustomuser,itfirstsetsprivilegesfortheuserbyusingthe“localsecuritypolicy”inMicrosoftWindows.

Configure as an executable program: This deselects theWindowsServiceoptionduringtheinstallationprocess.

Pluginsandextensions:Thisstepisavailableforanewinstallation.IftheuserwantstoupgradefromanolderMySQLversion,thentheuserneedstochoosetheReconfigureoptionintheMySQLinstaller.

Advance options: To enable this option, select the Show advanceconfiguration check box in theType andNetworking step. This optionenablestheusertodefineaspecificpathforlogfiles,suchasanerrorlog,agenerallog,aslowquerylog,andbinlog.

Applyserverconfiguration:Oncealltheconfigurationhasbeendonebythe user in theMySQL installer, click on the Execute button to make itavailable.WhentheinstallationhasbeencompletedbypressingtheFinishbutton, the MySQL installer and all the MySQL installed products areavailableintheWindowsStartmenu.

MySQL installer product cataloganddashboard

This section contains details on how theMySQL installer handles productcatalogsandmanagesdashboards.

Theproduct catalog is a componentwhere a list of all the releasedMySQLproductsisavailable,whichsupportMicrosoftWindows.TheMySQLinstallerupdates the catalog on a daily basis and the option is also available for themanual update of the catalog. The product catalog performs the followingactionstomanagethelist:

Populatetheavailableproductslistonaregularbasis

Checkfortheproduct'supdateasinstalledinthehost

Theproductcataloglistsalltheproductsthatareavailableinthedevelopment,general,oranyminorrelease.

TheMySQL installer dashboard provides the facility to manage MySQL

productsinstallationinthehostworkstation.

Thefollowingarethewaystomanageproductsusingthedashboard:

The MySQL installer provides a configuration to update the catalog atspecifictimeintervals.Theusercanenableordisableautomaticupdatesbytheconfiguration.

The dashboard shows a special icon at the product level when its newversionisavailable.

Theusercanmanageproductswiththefollowingactions:

Add:Usetodownloadandinstalloneormoreproducts.

Modify:Usetoaddorremovefeaturesininstalledproducts.

Upgrade: Use to upgrade products. Make sure the checkbox isselected at the product level for upgrading in the upgradeableproductspane.

Remove:Usetouninstallproductsfromthepopulatedlist.

The dashboard provides the reconfiguration feature, where the user canchangealreadyconfiguredoptionsandvalues.Afterapplyingchanges,theMySQLinstallerwillstoptheMySQL8serverandrestartitagaintomakethemavailable.

The dashboard provides the facility to download the products catalogwithoutupgradingit.TheDonotupdateatthistimecheckboxisavailabletocheckcurrentchangesrelatedtoproductswithoutdownloading.

Toperformthisfunctionality,selectthecheckboxandclickonthecataloglink.

MySQLinstallerconsoleTheMySQLinstallerincludestheMySQLinstallerConsole.exefile,whichprovidesthefunctionalitytoexecutecommandsusingCommandPrompt.

This functionality is installed by default during the initial installation of theMySQL installer. There are some commands available to manage MySQLproducts.Toseethedetailsofthesecommands,executethehelpcommand.

<strong>[mysqld]</strong><br/><strong>#setbasedirtoyourinstallationpath</strong><br/><strong>basedir=E:\\mysql</strong><br/><strong>#setdatadirtothelocationofyourdatadirectory</strong><br/><strong>datadir=E:\\mydata\\data</strong>

<strong>E:\>“E:\MySQL\MySQLServer8\bin\mysqld”</strong>

<strong>mysqld:readyforconnections</strong><br/><strong>Version:'8.0.4'socket:''port:3306</strong>

<strong>E:\>“E:\MySQL\MySQLServer8\bin\mysqld”--install</strong>

<strong>E:\>“E:\MySQL\MySQLServer8\bin\mysqld”--installMySQL--defaults-file=E:\my-opts.cnf</strong>

<strong>E:\>“E:\MySQL\MySQLServer8\bin\mysqld”--install-manual</strong><br/><strong>E:\>"E:\MySQL\MySQLServer8\bin\mysqld"--remove</strong>

InstallingMySQL8onLinux

ForMySQL8installationonLinux,varioussolutionsareavailable.Theusercanchoose any of the distributions for his requirement. Th following are thedifferentdistributionsandamongthem,threeofwhicharedescribedindetail:

InstallationusingtheYumrepository

InstallationusingtheAPTrepository

InstallationusingtheSLESrepository

InstallationusingtheRPMpackage

InstallationusingtheDebianpackage

InstallationusingDocker

InstallationusingtheNativeSoftwarerepository

InstallationusingJuju

<strong>shell>sudoyumlocalinstallpackage_name.rpm</strong><br/><strong>shell>yumrepolistenabled|grep"mysql.*-community.*"</strong>

<strong>shell>yumrepolistall|grepmysql</strong>

<strong>shell>sudoyum-config-manager--disablemysql57-community</strong><br/><strong>shell>sudoyum-config-manager--enablemysql80-community</strong>

<strong>mysql80-community]</strong><br/><strong>name=MySQL8.0CommunityServer</strong><br/><strong>baseurl=http://repo.mysql.com/yum/mysql-8.0-community/el/6/$basearch/</strong><br/><strong>enabled=1</strong>

<strong>shell>yumrepolistenabled|grepmysql</strong>

<strong>shell>sudoyuminstallmysql-community-server</strong>

<strong>shell>sudoservicemysqldstart</strong><br/><strong>shell>sudoservicemysqldstatus</strong>

Duringtheinitialstartup,thefollowingtasksareperformed:

TheserverisinitializedTheSSLcertificateandkeyfilesaregeneratedinthedatadirectoryThepluginsnamevalidate_password_pluginisinstalledandenabledAsuperaccountiscreated

<strong>shell>sudoservicemysqldstart</strong>

Aswithotherinstallations,theRPMpackageinstallationalsocreatesfilesanddirectoriesinthesystemonfollowingpath:

FilesorResources LocationClientprogramsandscripts /usr/bin

mysqldserver /usr/sbin

Configurationfile /etc/my.cnf

Datadirectory /var/lib/mysql

ErrorlogfileForRHEL,OracleLinux,CentOS,orFedoraplatforms:/var/log/mysqld.logForSLES:/var/log/mysql/mysqld.log

Valueofsecure_file_priv /var/lib/mysql-files

SystemVinitscriptForRHEL,OracleLinux,CentOS,orFedoraplatforms:/etc/init.d/mysqldForSLES:/etc/init.d/mysql

Systemdservice ForRHEL,OracleLinux,CentOS,orFedoraplatforms:mysqldForSLES:mysql

Pidfile /var/run/mysql/mysqld.pid

Socket /var/lib/mysql/mysql.sock

Keyringdirectory /var/lib/mysql-keyring

Unixmanualpages /usr/share/man

Include(header)files /usr/include/mysql

Libraries /usr/lib/mysql

Miscellaneoussupportfiles(forexample,errormessages,andcharactersetfiles)

/usr/share/mysql

<strong>shell>tar-xvfmysql-server_MVER-DVER_CPU.deb-bundle.tar</strong>

<strong>shell>sudoapt-getinstalllibaio1</strong>

<strong>shell>sudodpkg-preconfiguremysql-community-server_*.deb</strong>

<strong>shell>sudoapt-get-finstall</strong>

TheMySQL8configurationfilesareavailableunderthefollowingpathintheDebianpackage:

Configurationfilesarestoredunder/etc/mysqlThedatadirectoryisstoredunder/var/lib/mysqlBinaries,libraries,andheadersarestoredunder/user/binandunderuser/sbin

Post-installationsetupforMySQL8Post-installationisaprocessthatdescribesthebasicstepsorconfigurationthattheuserhastoperformafterMySQL8installation.

<strong>E:\>bin\mysqld–-initialize</strong><br/><strong>E:\>bin\mysqld--initialize-insecure</strong>

<strong>E:\>bin\mysqld--initialize--basedirE:\mysql--datadir:\mydata\data</strong>

Theusercanalsospecifythesedirectoriesinaseparatefile,knownastheOptionfile,underthemysqldparameter.ThisconfigurationisdescribedindetailundertheOptionfilesectioninthischapter.Whentheuserexecuteseitherofthecommands,mysqldperformsthefollowingstepsintheexecution:

1. Itcheckstheexistenceofthedatadirectory

2. TheMySQL8servercreatesasystemdatabaseanditstable,granttables,helptables,andtimezonetables

3. ItinitializesasystemtablespacewithdatastructureforInnoDBtables

4. Theroot@localhostsuperuseraccountandotherreservedaccountswillbecreated

<strong>shell>mysql-uroot-p</strong><br/><strong>Enterpassword:(entertherandomrootpasswordhere)<br/></strong>

<strong>mysql>ALTERUSER'root'@'localhost'IDENTIFIEDBY'newPassword';</strong>

<strong>mysql-uroot</strong>

<strong>mysql>ALTERUSER'root'@'localhost'IDENTIFIEDBY'newPassword';</strong>

Afterassigningthenewpassword,youhavetousethenewpasswordwheneveryouwanttoconnectwithMySQL8.

<strong>shell>sudoservicemysqldstart</strong>

<strong>shell>sudoservicemysqldstatus</strong>

<strong>shell>mysql-uroot-p</strong>

Theprecedingcommandpromptsforthepassword,soenterthepasswordandpresstheEnterkey.TheMySQLpromptwillbedisplayedwhereyoucanentermysqlcommandsforexecution.Duringtheexecutionoftheprecedingcommands,somecommonproblemsmayarrive,soherewewillpresentthefollowingtroubleshootingsuggestions:

1. Checkintothelogfilestofindtheexacterrorthatoccurredduringtheservicestartup.Asmentionedintheprevioussection,theerrorfileandlogfilesarelocatedunderthedatadirectory.Itsnamingconventionsarehost_name.errandhost_name.log.Byreadingthelastfewlinesofthefile,youcanidentifytheproblemthatoccurredduringthelastcommandexecuted.

2. Checkthattherequiredport/socketisavailable.Thefollowingerrorswillindicatethattherequiredportsandsocketsarenotavailableforuse,meaningthattheyareinusewithotherprograms.Toidentifythis,trackalltheproblemsbydisablingtheservice.Anotherreasonisyourfirewallsettingsareblockingtheaccessoftherequiredport,somodifythefirewallsettingandgivepermissiontotherequiredports:

Can'tstartserver:BindontheTCP/IPport:theaddressisalreadyinuseCan'tstartserver:BindontheUnixsocket

3. DefiningthespecificparametersintotheOptionfileis

recommended.IftheparametersarenotdefineintothefilethenMySQL8willconsiderthedefaultparameters,sorefertoalltheavailableparametersprovidedbyMySQL8beforeusingit.

4. Verifythatthedatadirectorypathandpermissionareproperlydefinedornot.ThisdirectoryisusedasthecurrentdirectoryforMySQL8.Tofindthecurrentlysetpathofthedatadirectory,executethemysqldcommandwiththe--verboseand--helpcommand.IfthedatadirectoryislocatedatadifferentplaceotherthantheMySQLinstallationdirectory,thenusethe--datadiroptionwiththemysqldcommand.Forpermission,youwillgetanErrorcode13,whichindicatesthepermissiondeniederror.Toovercomethisissue,changethepermissionoftherequiredfilesandfolder.Anotherwayistologinwiththerootuser,butthisisnotpossibleinallthescenarios,sothefirstapproachisrecommendedtoovercomethepermissionissue.

<strong>shell>bin/mysqladminversion</strong>

<strong>mysql>mysqlshow</strong><br/><strong>mysql>mysqlshowmysql</strong>

Thefirstcommandshowsthelistofdatabasesavailableintheserver.Listsmayvaryasperthesystem,butmysqlandinformation_schemamustbeavailableinthelist.Thesecondcommandlistsallthetablescreatedunderthemysqldatabase.

UpgradingMySQL8In previousversionsofMySQL, thedata dictionary is stored in the file-basedsystemwhileinMySQL8itisstoredinthedatadictionarystructure.So,theup-gradation process will move the file-based structure into the data dictionarystructure. Up-gradation into MySQL 8 is possible from the MYSQL 5.7 GAversion, which means from 5.7.9 or higher. For non-GA versions of 5.7, up-gradationisnotpossible.Beforestartingtheup-gradationprocess,thefollowingpointsneedtobeunderstood.

UpgradingmethodsTwo methods are in use for up-gradation that are differentiated by theirimplementationmethod.Letusdiscussthesemethodsindetail.

<strong>ALTERINSTANCEROTATEINNODBMASTERKEY;</strong>

<strong>SETGLOBALinnodb_fast_shutdown=1;--fastshutdown</strong><br/><strong>SETGLOBALinnodb_fast_shutdown=0;--slowshutdown</strong>

<strong>mysqladmin-uroot-pshutdown</strong>

<strong>mysql_upgrade-uroot-p</strong>

7. Afterup-gradation,shutdownandrestarttheservertocheckwhetherallthechangeshavebeenappliedornot.

<strong>mysqldump-uroot-p--add-drop-table--routines--events--all-databases--<br/>force>data-for-upgrade.sql</strong>

<strong>mysqld--initialize--datadir=/path/to/8.0-datadir</strong>

<strong>mysqld_safe--user=mysql--datadir=/path/to/8.0-datadir</strong>

<strong>mysql-uroot-p--force<data-for-upgrade.sql</strong>

<strong>mysql_upgrade-uroot-p</strong>

<strong>mysqlcheck-uroot-p--all-databases--check-upgrade</strong>

<strong>SELECTTABLE_SCHEMA,TABLE_NAME</strong><br/><strong>FROMINFORMATION_SCHEMA.TABLES</strong><br/><strong>WHEREENGINENOTIN('innodb','ndbcluster')</strong><br/><strong>ANDCREATE_OPTIONSLIKE'%partitioned%';</strong>

<strong>ALTERTABLEtable_nameENGINE=INNODB;</strong><br/><strong>ALTERTABLEtable_nameREMOVEPARTITIONING;</strong>

<strong>SELECTCONSTRAINT_SCHEMA,TABLE_NAME,CONSTRAINT_NAME</strong><br/><strong>FROMINFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS</strong><br/><strong>WHERELENGTH(CONSTRAINT_NAME)>64;</strong>

5. MakesureMySQL5.7doesn'tcontainfeaturesthatarenotavailableinMySQL8,forexample:

IfatableusedastorageenginethatisnotsupportedbyMySQL8,sowasalteredwiththesupportedstorageengineAconfigurationchangewhereyouuseanoptionorvariablethatisnotavailableinMySQL8

MySQL8downgrading

Downgradingisthereverseprocessofup-gradation,wherewewillmovefromahigher version of MySQL to a lower version ofMySQL. In this section, wecoverhowtodowngradefromMySQL8toMySQL5.7.AdowngradethatdoesnotsupportaversionskipmeansthatdowngradingfromMySQL8toMySQL5.6 isnot supported.Within the sameserieswhereaversion skip is supportedmeansyoucandowngradefromMySQL8.ztoMySQL

8.xbyskippingtheMySQL8.yversion.First,wewillexplainsomebasicpointsthatneedtobeunderstoodbeforestartingdowngrading.

<strong>rmib_logfile*</strong>

<strong>mysqld_safe--user=mysql--datadir=/path/to/existing-datadir</strong>

<strong>mysql_upgrade-uroot-p</strong>

6. ShutdownandrestarttheMySQLserveragaintocheckifallthechangeshavebeenappliedornot.

ForMySQLinstallationbasedonAPT,SLES,andtheYumrepositoryinstallations,in-placedowngradesarenotsupported

<strong>mysqldump-uroot-p</strong><span><strong>--add-drop-table--routines--events</strong><br/><strong>--all-databases--force>data-for-downgrade.sql</strong><br/></span>

<strong>mysqladmin-uroot-pshutdown</strong>

<strong>mysqld--initialize--user=mysql</strong>

<strong>mysqld_safe--user=mysql--datadir=/path/to/new-datadir</strong>

<strong>mysql-uroot-p--force<data-for-upgrade.sql</strong>

<strong>mysql_upgrade-uroot-p</strong>

<strong>mysqladmin-uroot-pshutdown</strong><br/><strong>mysqld_safe--user=mysql--datadir=/path/to/new-datadir</strong>

<strong><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>columns_priv<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span><spanclass="tokenkeyword">component</span><spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>db<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>default_roles<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>engine_cost<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>func<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><span

class="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>general_log<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>global_grants<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>gtid_executed<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>help_category<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>help_keyword<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>help_relation<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>help_topic<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=

</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>innodb_index_stats<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>innodb_table_stats<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span><spanclass="tokenkeyword">plugin</span><spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>procs_priv<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>proxies_priv<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>role_edges<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><span

class="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>server_cost<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>servers<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>slave_master_info<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>slave_relay_log_info<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>slave_worker_info<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>slow_log<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>tables_priv<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=

</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone_leap_second<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone_name<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone_transition<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone_transition_type<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><spanclass="tokenkeyword"><br/>ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span><spanclass="tokenkeyword">user</span><spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span></strong>

<strong>ALTERTABLEmysql.columns_privENGINE='MyISAM'<br/>STATS_PERSISTENT=DEFAULT</strong>

<strong>ALTERTABLEmysql.userdropCreate_role_priv;</strong><br/><strong>ALTERTABLEmysql.userdropDrop_role_priv;</strong>

InnoDBchanges:Beforestartingin-placedowngrading,shutdownMySQLusingtheinnodb_fast_shutdownoption.Shutdowntheserverwithinnodb_fast_shutdown=0.Removingtheredologsisrecommendedforin-placedowngrading.

Summary

Tochoosethepropersoftwarewithitsversionfordevelopmentisanimportantphase,right?Inthischapter,weunderstoodhowtoselecttheproperversionofMySQL8

by understanding its version pattern. We also learned the execution steps ofMySQL

8 installationusing the installerandcommand line inMicrosoftWindows.Forthe Linux platform, we installed MySQL 8 using the Yum repository, RPMpackage,andDebianpackage.Post-installationdescribesthebasicconfigurationtostartwithMySQL8.

Finally, we explained how to upgrade and downgrade from MySQL 8 withexecutionsteps.

Inthenextchapter,wewilllearnaboutvariousprogramsandutilitiesavailableforMySQL8. Itmainly focuses on how to use these programs forMySQL8alongwithcommand-lineexecutions.

MySQL 8 – Using Programs andUtilities

Inthepreviouschapter,weinstalledMySQL8andgottoknowalternativewaystoinstallMySQL8.WealsolearnedhowtomigrateandupgradetoMySQL8.Thefollowingarethesummarytopicsexplainedinthepreviouschapter:

MySQL8Installation

PostInstallationSetup

MySQL8Upgrading

MySQL8Downgrading

In this chapter, the reader will learn about various programs and utilitiesavailable inMySQL8.The readerwill alsoget toknowhow touseprogramsand utilities in MySQL 8. The reader will learn about using command-lineprogramsusedinMySQL8.

The readerwill learn the syntax for theprogramandhow theyarebeingusedwithspecificoptionstoperformspecificoperations.Thefollowingisasummaryofthetopicscoveredinthischapter.

OverviewofMySQL8programs

MySQL8command-lineprograms

MySQL8clientprograms

MySQL8administrativeprograms

MySQL8environmentvariables

MySQLGUItools

OverviewofMySQL8programs

There are various different programs in the MySQL installation. A briefoverviewof theseprograms iscovered in this section.Upcomingsectionswillcover a detailed description for eachof themand the descriptionwill have itsowninvocationsyntaxandoptionstoperformtheoperation.

MostoftheMySQLdistributionwillhavealltheseprograms,apartfromthosethat are platform-specific; for example, server startup script not used inWindows.RPM

(Red-hatpackagemanager)distributionsareveryspecializedandarepartoftheexceptions toallprogramsavailable indistributions.What isspecializedaboutRPMdistributions?

Well, they have different programs for different operations; for example, oneprogramwillbeexecutedfortheserver,asecondprogramwillbeexecutedfortheclient, and soon. If it looks likeoneormoreprograms ismissing inyourinstallation,thendon'tworry.SeeChapter2,InstallingandUpgradingMySQL8,for information on the types of distributions available andwhat is included inthem.Itmightbethecasethatthedistributionwhichyouhavedoesnotincludealltheprogramsandyouneedtoinstallanadditionalpackage.

EachoftheMySQL8programswillhavetheirownoptions,butmostofthemwillhavea--helpoption thatcanbeused to retrievedescriptionsaboutall theoptionsthattheprogramhas.Forexample,trymysql--helponthecommandline(thatisyourshellorCommandPrompt).

The description on the first few lines will have specific version informationabout MySQL that is installed along with operating system and licenseinformation.ThenextlinewillstartwithUsage:mysql[OPTIONS][database],thatis,a syntax of the program command usage, and later lines describe the optionsavailabletobeusedalongwiththemaspertheusagedescription.Thiswasjustaglimpseofwhatwewillbelookingat:programdetailswithoptions,theirusagesanddefaultoptions,overridingdefaultoptionvaluesonvarious command-lineprograms,clientprograms,administrativeprograms,andsoon.

For detailed information about executing programs and specifying programoptionsinthecommandlineseetheMySQL8commandlineprograms section,whichwillbe followedbya listof installations,clientandserverstartup,andotherutilityprograms.

MySQLprogramsinbrief

LetusstartwiththeMySQLserverprogramsfirst!!

mysqld is thefirstprogram,alsoconsidered tobe themainprogramforMySQLinstallation.

Itworkswithseveral scripts tohelpwithstartingandstopping theserver.Thefollowing are the programs divided into categories based on their operationalscope:

Startupprograms

Installation/upgradationprograms

Clientprograms

Administrativeandutilitiesprograms

Startupprograms

ThestartupprogramsaretheprogramsthatareusedduringMySQLstartupandinitiatetherequiredbackgroundservicesbasedontheconfiguration.

mysqld: This is theMySQL server daemon. All the other client programsinteractwiththedatabaseusingthisserverprogram.Itmustbestartedandberunningatalltimesexceptformaintenance.

mysqld_safe:Thisisoneoftheserverstartupprogramscriptsandattemptstostartthemysqldprogram.

mysql.server:Another server startupprogram script,which is used in thosesystems, uses V-style run directories containing scripts. It starts systemservicesatparticularrunlevels.

Itcallsmysqld_safetostarttheMySQLserver.

mysqld_multi:Asthenamesuggests,thisisastartupprogramscripttostartorstopmultipleMySQLserversonthesystem.

Installation/upgradationprograms

Theprogramsregardingtheoperationsofinstallationandupgradationarelistedherewiththeirrespectiveusage:

comp_err: This program is used to compile error message files from errorsourcefilesanditisusedduringtheMySQLbuildorinstalloperation.

mysql_secure_installation: This program is used to update the securityconfigurationinordertoenablesecurityduringinstallationofMySQL.

mysql_ssl_rsa_setup:Asthenamesuggests,thisprogramisforgeneratingSSLcertificatesandkeyfilesandRSAkey-pairfiles,ifthosefilesaremissingandarerequiredtosupportsecureconnections.

mysql_tzinfo_to_sql: This program gets the content of the host system zoneinfodatabase(filesdescribingtimezones)andloadstheinformationinthetimezonetablesofMySQL.

mysql_upgrade: As the name suggests, it is used for upgrade operations. Itchecksforany incompatibilityandmakesrepairs if it isnecessary. Italso

updatesthegranttableswithanychangesinnewversionsofMySQL.

Clientprograms

TheclientprogramsareamongtheprogramsthatarecommonlyusedtoconnecttotheMySQLdatabaseandperformdifferentqueryoperations:

mysql: This is the most commonly used program. It is an interactivecommand-linetoolforexecutingSQLstatementsdirectlyorusingafileinbatchmode.

Detailed information is followed in the next MySQL 8 command lineprogramssection.

mysqladmin: This is the program responsible for performing variousadministrativeoperations,suchascreatingordroppingdatabases,flushingtables, reloading grant tables, reopening log files, and much more. Theprogram is also used to retrieve information from the server, such asversion,process,andstatus.

mysqlcheck: This is the client program used for maintenance of tables,performinganalysis,checks,repairs,andoptimizingtables.

mysqldump:This is theclientprogram thatdumps theMySQLdatabase to afile in text, SQL, orXML formats. It is commonly known as a databasebackupprogram.

mysqlimport:ThisistheclientprogramthatimportstextfilesintorespectivetablesusingLOAD_DATA_INFILE. It isalsocommonlyknownasthedataimportprogram.

mysqlpump:TheclientprogramwhichdumpsMySQLdatabaseintoSQLfile.

mysqlshow:The client that shows informationof databases, tables, columns,andindexes.

mysqlslap: This is the client program that is used to check client loadcapability for the MySQL server. The program mimics multiple clientsaccessingtheserver.

Administrative and utilitiesprograms

The following are the programs that perform various administrative activities.Theyaredepictedalongwithsomeoftheutilitieswhichhelpinadministrativeoperations:

innochecksum:TheprogramfortheInnoDBofflinefilechecksum.

myisam_ftdump: The utility program gives information of full-text indexes inMyISAMtables.

myisamchk: The programused to check, describe, repair andoptimize MyISAMtables.

myisamlog:TheutilityprogramforprocessingaMyISAMlogfilecontents.

myisampack:Theutilityprogramthatproducessmallerread-onlyMyISAMtablesthroughcompression.

mysql_config_editor: The utility program that enables authenticationcredentials storage in an encrypted and secure login path file namedmylogin.cnf.

mysqlbinlog:Theutilityprogramthatcanreadbinary logfilestatements. Intheeventofaservercrash,abinarylogfileexecutedstatementscanbebighelp.

mysqldumpslow:Theutilityprogramthatcanreadandsummarizethecontentsofaslowquerylog.

Environmentvariables

TheMySQL client programs that communicatewith theMySQL server usinglibrariesusethefollowingenvironmentvariables:

MYSQL_UNIX_PORT:Thisvariable is responsible for thedefaultUnixsocket filewhichwillbeusedforconnectingtoalocalhost

MYSQL_TCP_PORT: This variable is responsible for providing the default portnumberandisusedinTCP/IPconnections

MYSQL_PWD:Thisvariableisresponsibleforprovidingthedefaultpassword.

MYSQL_DEBUG: This variable is responsible for providing debug trace optionsduringdebuggingoperations

TMPDIR: This variable is responsible for providing the directory where thetemporaryfilesandtableswillbecreated

Foradetailedlistandusesofenvironmentvariablesinprograms,

seeMySQL8environmentvariablessection.TheuseofMYSQL_PWD isinsecure.

MySQLGUItoolTheMySQLWorkbenchGUI tool, providedbyOracle corporation, is used intheadministrationofMySQLserversanddatabases,forcreating,executing,andevaluating queries. It is also used for migrating schema and data from otherrelational database management systems to be used with MySQL. There areotherGUI tools, includingMySQLNotifier,MySQLforExcel,phpMyAdmin,andmanymore.

MySQL8command-lineprogramsIntheprevioussection,wewentthroughvarioustypesofprogramsprovidedbyMySQL

8andoutlinedtheirusageinbrief.

In this section, we will look at command-line programs and learn aboutexecuting programs from command lines.Wewill take a detailed look at theprovisionforoptionsandhowtheycanbeutilizedfortheadministration.

Executing programs from thecommandlineExecutingprogramsfromthecommandline(shellorCommandPrompt)isoneof themostused formsof administration inMySQL.Plentyofprogramshavebeenaddedalongwithoptionsforadministration.

<strong>shell>mysql--verbose--help</strong><br/><strong>shell>mysql--user=root--password=********mysampledb</strong><br/><strong>shell>mysqldump-urootpersonnel</strong><br/><strong>shell>mysqlshow--help</strong>

Therearenonoptionarguments,argumentswithoutanyleadingdash,givingsupplementaryinformationtotheprogram.Asanexample,ifyouseethesecondlineoftheprecedingexample,ithasathirdnonoptionargumentwithadatabasenamemysampledb,sothecommandmysql--user=root--password=********mysampledbtellsthemysqlprogramthatyouwantedtousemysampledbasthedatabasename.

Argumentsbeginningwithsingleordoubledash(-,--)areusedforspecifyingtheprogramoptions.Specifyingtheprogramoptionsindicatesthetypeofconnectiontheprogramwillconnecttotheserverorwillaffectthemodeofoperation.Syntaxfortheoptionsareexplainedwithdetails,seetheSpecifyingoptionsforprogramssection.

<strong>shell>mysql</strong>

<strong>shell>mysql--host=localhost--user=root--password=mypwdmysampledb</strong>

Asyoucanseeintheprecedingexamplewithspecificoptionvalues,thehostistobeconsideredasthelocalhostandtheuservalueisprovidedasmyname.Thepasswordisalsospecifiedand,finally,anon-optionargumentisspecifiedthattellstheprogramtousemysampledbasthedefaultdatabasename.

Specifyingoptionsforprograms

In theearliersection,wehaveseenhowprogramoptionschangetheoperationmodebasedonargumentvaluesspecified for theoption in theclientprogram.Here we will look at several ways to specify options for MySQL programs.Theseinclude:

Providingtheoptionsonthecommandlinefollowedbytheprogramname.This is the common way of providing options but it will be appliedspecificallytotheexecutionoftheprogramatthattimeonly.

Providingtheoptionsintheoptionsfile that isbeingreadbytheprogrambeforeitstartsexecution.Thisisthecommonwayforprovidingtheoptionsthatyouwanttheprogramtouseeachtimeitexecutes.

Providingtheoptions in theenvironmentvariables.Byusing thismethod,youcanalsospecifyoptionsthatyouwanttoapplyeverytimetheprogramisbeingexecuted.

Ingeneralpractice,usingoptionfilesiscommonlyusedforthispurposebutspecifyingoptionvaluesintheenvironmentvariablesisalsoveryusefulin

some cases; for example, when runningmultipleMySQL instances on aUnixsystem.

TheMySQLprogramcheckswhichoptionsare tobegivenfirstbyexaminingthe related environment variables, then it processes option files, and then itconsiders option arguments in the command line. Thus, the command-lineoptions have the highest precedence and the environment variables have thelowest.However,thereisoneexceptionthatappliesandthatisthemysqld-auto.cnfoptionfile inthedatadirectoryprocessedlast,soit takeshigherprecedencetothecommand-lineoptions.

<strong>shell>mysqladmin--count=1k--sleep=10ping</strong>

Forfilenameoptionsvalues,avoidusingthe~metacharacterbecauseitwillnotbeinterpretedasperexpectation.Optionvaluescontainingspacesmustbeenclosedbyquotationmarkswhenavalueisspecifiedonthecommandline.

<strong>--disable-column-names</strong><br/><strong>--skip-column-names</strong><br/><strong>--column-names=0</strong>

Asyoucanseeintheprecedingexample,the=0suffixandthe--skipand--disableprefixeshavethesameeffect.Itisalsoapplicablewhenturningtheoptiononwiththe=1suffixandthe--enableprefix.

Iftheoptionisspecifiedwiththe--looseprefix,andiftheoptionspecifieddoesnotexist,theprogramwillissueawarninginsteadofexiting.

Forsomeoftheprograms,the--maximumprefixisavailabletobeusedwiththeoptionnameforspecifyingthelimit.Itcanalsobeusedwithenvironmentvariables.

Modifyingoptionswithfiles

MostoftheMySQLprogramscanreadstartupoptionsfromtheoptionfiles,alsosometimescalledconfigurationfiles. It isaveryconvenientwayofprovidingthe options that are commonly used and once specified you need not specifythemeach timeyouexecute theprogram.Tocheckwhether theprogramreadsoption files, use the --help option. For example, consider the mysqld program,whichshoulduse--verboseand--help, if it readsoptionfiles.Thehelpmessagewillindicatewhichoptionfileitlooksforandforwhichoptiongroup:

TheMySQLprogramwiththe--no-defualtsoptiondoesnotreadanyoption files apart from .mylogin.cnf. If the server program startedwith theoption persisted_globals_load systemvariable disabled thentheprogramdoesnotreadthemysqld-auto.cnffile.

Themajorityoftheoptionfilesareplaintextfilesthatcanbecreatedandeditedby any text editors. Exceptions among those files are .mylogin.cnf , which hasloginpathoptions,encryptedbythemysql_config_editorutilityprogram.

MySQLchecksoptionfilesforWindowsandUnixsystemsinspecificorderandfollows the precedence that starts from reading global options, such as in theWindowssystem:

%PROGRAMDATA%\MySQL\MySQL Server 8.0\my.ini and %PROGRAMDATA%\MySQL\MySQL Server

8.0\my.cnf

%WINDIR%\my.iniand%WINDIR%\my.cnf

C:\my.iniandC:\my.cnf

BASEDIR\my.iniandBASEDIR\my.cnf

Thefilespecifiedwith--defaults-extra-file,ifany

Loginpathoptionsin%APPDATA%\MySQL\.mylogin.cnf(clientprogramonly)

For system variables persisted with SET_PERSIST or PERSIST_ONLY (if it is theserverprogram)inDATADIR\mysql-auto.cnf

Similarly, in Unix systems it follows the following order of precedence forreadingoptionfiles:

/etc/my.cnf

/etc/mysql/my.cnf

SYSCONFDIR/my.cnf

$MYSQL_HOME/my.cnf(serverprogramonly)

Thefilespecifiedwith--defaults-extra-file,ifany

~/.my.cnfforuser-specificoptions

~/.mylogin.cnfforuser-specificloginpathoptions(clientprogramonly)

For system variables persisted with SET_PERSIST or PERSIST_ONLY (if it is theserverprogram)inDATADIR\mysql-auto.cnf

Intheprecedingoptions,~referstothecurrentuser'shomedirectory.

Emptylinesinoptionfilesareignored,alongwithcomments.Commentscanbespecifiedusing#or;charactersand#canstartinthemiddleofanylineaswell.

groupgroupisthenameoftheprogramorgroupforwhichoptionsaretobeset.Theyare not case sensitive. Once a group line is added to the option file, all thefollowinglinesapplytothenamedgroupuntilanothergrouplineisspecifiedorattheendoftheoptionfile.

opt_nameThis is similar to the --opt_name in the command line turning on the namedoptimization.

opt_name=valueThisissimilartothe--opt_nameinthecommandlinebutinplaceofthevalue,youcanspecifythevaluewithspaces,whichyoucannotinthecommandline.

IncludedirectivesItispossibleusing!includedirectivesintheoptionfilestoincludeanotheroptionfileand!includedirtosearchforspecificdirectoriestocheckforoptionfiles.Forexample,!include/home/dev/myopt.cnfand!includedir/home/dev, fordirectories.Theonly thing that MySQL does not consider is any order during the directorysearch.

Anyoptionfilestobeusedinthe!includedirdirectiveonaWindowssystemmustendwiththe.inior.cnfextensionandinUnixsystemstheymustendwith.cnf.

Command-line options affectingoptionfilehandlingMost of theMySQL programs support option files. As they affect option filehandling,theymustbegiveninthecommandlineandnotaspartofanoptionfile.

Inordertomakethemworkproperly,theymustbegivenbeforeotheroptions.Some of the exceptions are as follows : --print-defaults might be usedimmediatelyafter--login-path,--defaults-file,ordefaults-extra-file.

--no-defaultsand--print-defaultsarealsousedtomodifyoptionfilehandling.

Setting program variables withoptionsManyMySQLprogramshaveinternalvariablesthatwecansetduringruntimeoperations

using the SET statement and also using the same syntax by which we specifyoptionvalues.This

willworkwhentheprogramisstarted.Forexample,ifweusetheoptionvaluesyntax

thenwehavetospecifylikethis:shell>mysql--max_allowed_packet=16M .Tospecifythe runtime option using the SET, we can specify like this: mysql> SET GLOBAL

max_allowed_packet=16*1024*1024.

To findout if theoptionvariable syntax iscorrect, youcango tomysqlandusethefollowing:mysql>showvariableslike'max%'.

<strong>SETUSER=your_user_name</strong>

<strong>MYSQL_TCP_PORT=3306</strong><br/><strong>exportMYSQL_TCP_PORT</strong>

<strong>setenvMYSQL_TCP_PORT3306</strong>

Thecommandstosetenvironmentvariablesthatareexecutedwillimmediatelyaffecttheprograminexecutionbutifyouwantedtogettheenvironmentvariabletopersist,youneedtospecifyitontheinterfaceprovidedbythesystemoryoumaysetitupinthestartupfilethatthecommandprocessorusesatstartup.OnWindows,thiscanbesetupfromthecontrolpanelsoptiontosetenvironmentvariablesandinUnix,thiscanbesetupbasedonthecommand-lineprocessoryouuse.Forbash,youneedtoputthevaluein.bashrcor.bash_profileandfortcshuse.tcshrc.

Server and server-startupprogramsTherearespecificprogramsprovidedbyMySQLwhichyouneedtoexecutefirstinordertomakeMySQLworkcorrectly.Inthefollowingsections,wewilllookattheserverprogramsandrelatedstartupprogramsthatcanbeusedwithseveraloptionsasperyourrequirement.

mysqld - the MySQL serverprogramTheMySQLserver isadaemonprogram.Allotherprogramsconnectwiththedatabase

through this server, so it shouldbe running at all times.Thedaemonprogramusually

getsstartedfromascriptcalledmysqld_safe.Theprogramscriptisrequired,asitsetstheappropriateenvironmentvariables

andexecutesthemysqldprogramwiththerequiredarguments-optionvalues.

Options

Thefollowingaretheoptionsbriefedindetailforvariousoptionsavailablefromcommandline:

-?,-I,--help:Displaystheusageinformationoftheprogram.

-#debuglevel,--debug=debuglevel:Setsthedebugginglevelasspecified.

-b directory, --basedir=directory: Specifies the base directory used todetermineallotherrelateddirectories.

--big-tables: Used to allow large result sets. They are saved as temporaryresultsinafile.

--bind-address=ip-number:SpecifiestheIPaddresstheserverwillbinditto.

-h directory, --datadir=directory: Specifies the directory where the databasedatafilesarestored.

-l [logfile], --log[=logfile]: Add various log information, which includes

connections and error information. If an argument is not provided, thenhostname.logisusedasthelogfile,andherehostnameisthenameoftheservermachine.

--log-isam[=logfile]: Adds changes to the data (ISAM) files in logs. If anargument is notprovided, then isam.log is used as the log file and the loggenerated by this option can only be read and manipulated with thetheisamlogutility.

--log-update[=number]: Logging the database updates info. The log file getsnamedashostname.num,wherethehostname is thenameoftheservermachineandthenumistheargumenttotheoptionorgeneratesauniquenumberiftheargumentisnotspecified.

-L=language, --language=language: To specify the language (English, French,German,andsoon)fortheserver.

-n,--new:Toenablenewroutines(andpossiblyunsaferoutines).

-S,--skip-new:Todisable/enablenewroutines(andpossiblyunsaferoutines).

-Ovariable=value,--set-variablevariable=value:Tospecifyandsetvalueforthevariables

--pid-file=file:TogetthenameofthefilehavingtheprocessID (PID)oftherunningserver.Thedefaultvaluefor thefile ishostname.pid ,wherethehostnameistheservermachine'sname.

-Pport,--port=port:Tospecifythenetworkportnumber.

--secure: To enable network security checks. But this reduces databaseperformance.

--skip-name-resolve: To specify using only IP numbers (not names) for theconnections.Thisincreasesthenetworkperformance.

--skip-networking: To disable network connections, with only local accessbeingallowed.

--skip-thread-priority:Forgivingallthethreadsthesamepriority.

-Sg:Todisableaccesschecking.Thisallowsall theusersfullaccess toallthedatabases.

-Sl:Tospecifynottoperformthreadlocking.

--use-locking:Toenablethreadlocking.

--socket=file:TospecifythefilenamefortheUnixsocket.

-T,--exit-info: Used to display debugging information during the shuttingdownoftheserver.

-v,-V,and--version:Toshowtheversioninformationoftheserver.

mysqld_safe - MySQL serverstartupscriptThisisthemostrecommendedwaytostarttheMySQLserverinaUnix-basedsystemasitaddsafewsafetyfeatures,suchaslogginginformationtoerrorlogifanyerroroccursatruntimeandrestartingtheserverifthereisanerror.

Insomeof theUnixplatforms,MySQLinstallations fromRPMorDebian packages include the systemd support to manage MySQLstartupandshutdownoperationsandsomysqld_safe isnotinstalledonthosesystems.

mysqld_safe attempts to execute mysqld and to override the default behavior tospecifythenameoftheserverthatyouwantedtoexecute.Theoptiontospecifythe directory using --ledir is also available so that mysqld_safewill look for theserver in the directory.Most of the options in mysqld_safe are also available inmysqldandifthespecifiedoptionisunknowntomysqld_safethenitgetspassedontomysqld.mysqld_safe,whichreadsoutalltheoptionsfromthemysqld,server,andmysqld_safe sections in option files. For backward compatibility, mysqld_safe alsoreadssafe_mysqldsectionsbutyoushouldrenamesuchsectiontobeonthecurrentonethatismysqld_safe.

Asstatedpreviously,thereareverycommonoptionsspecifiedinbothmysqldandmysqld_safe,sosomeoftheoptionsareexcludedinthefollowinglistofoptions:

--core-file-size=size

Tospecifythesizeofthecorefilewhichmysqldshouldcreate.

--ledir=dir_name

Ifmysqld is not able to find the server, then use this option to specify the pathnameof the directory inwhich the server is located. This option can be usedonlyonthecommandline,andnotinoptionfiles.Onplatformsthatusesystemd,thevalueshouldbegiveninthevalueofMYSQLD_OPTS.

--mysqld-safe-log-timestamps

Thisoptionistospecifytheformatfortimestampsinthelogoutputproducedbymysqld_safe.

--mysqld=prog_name

To specify the server program name contained in the ledir directory that youwanttostart.Ifmysqld_safecannotfindtheserver,usethe--lediroptiontospecifythepathnametothedirectorywheretheserverwiththespecifiednameislocated.Thisoptionisonlyacceptedonthecommandline,andnotfromtheoptionfiles.

--open-files-limit=count

Thenumberoffileswhichmysqldcanopen.

--plugin-dir=dir_name

Tospecifythepathandnameoftheplugindirectory.

--timezone=timezone

Thisoptionistosetthetimezoneenvironmentvariabletothegivenoptionvalue,dependingonoperatingsystemtimezonespecificationformats.

--user={username|user_id}

Runthemysqldserverasifyouhavethenameoftheuser.Specifytheuser_nameorspecifythenumericuserIDasuser_id.

<strong>basedir=dir_name</strong>

<strong>datadir=data_dir</strong>

<strong>pid-file=file_name</strong>

<strong>service-startup-timeout=seconds</strong>

Tospecifyinsecondshowlongtowaitforconfirmationoftheserverstartup.Iftheserverdoesnotstartwithinthistime,mysql.serverexitswithanerrorindication.Thedefaultvaluefortheoptionis900secondsandavalueof0meansnottowaitatallforstartupandprovidingnegativevaluesmeanstowaitforever(thereshouldnotbeatimeout).

<strong>shell>mysqld_multi[options]{start|stop|reload|report}[GNR[,GNR]...]</strong>

Intheprecedingsyntax,start,stop,reload(stopandrestart)andreportreferstotheoperationtobeperformed.BasedontheGNRlistvaluesspecified,youcanperformtargetedoperationsonsingleormultipleservers.

PleasemakesurethatthedatadirectoryforallserversisfullyaccessibletotheUnixaccountbywhichthespecificmysqldprocessisstarted.Donotusearootaccountunlessyouknowexactlywhatyouaregoingtodowithit.

InstallationprogramsTheprogramsdiscussedinthissectionareusedduringtheinstallationprocessorwhen upgrading theMySQL, somake sure you understand it correctly beforedoinganymodificationsontheprogram.

comp_err - compiling theMySQLerrormsgfileThis creates the errmsg.sys file which is used by mysqld to identify the errormessages and display individual error codes. comp_err is normally runautomaticallywhenMySQLisbuilt.Theerrmsg.sysfileiscompiledfromthetextfile located inMySQLdistributionsat sql/share/errmsg-utf8.txt. It alsogeneratessql_state.h,mysqld_ername.h,andmysqld_error.hheaderfiles.

comp_err has several options and can be retrieved using the --help option in thepreviouscommand.

<strong>shell>mysql_secure_installation[options]</strong>

Youcanusethe--helpoptionhereandretrievealistofotheroptionswheneverrequired.

<strong>shell>mysql_ssl_rsa_setup[options]</strong>

Usethe--helpoptionhereandretrievealistofotheroptionsifrequired.

Usingsslmysql_ssl_rsa_setuplowersthebarriertosslandmakesiteasiertogeneratetherequiredfilesbutthefilesthataregeneratedareself-signed,whichisnotverysecure.YoucanconsiderobtainingaCAcertificatefromtherespectiveauthority.

<strong>shell>mysql_tzinfo_to_sqltz_dir</strong><br/><strong>shell>mysql_tzinfo_to_sqltz_filetz_name</strong><br/><strong>shell>mysql_tzinfo_to_sql--leaptz_file</strong>

Afterrunningthemysql_tzinfo_to_sqlprogram,itishighlyrecommendedtorestarttheserversothatitwillnotuseanypreviouslycachedtimezonedata.

mysql_upgrade - checking andupgradingMySQLtablesAs the name suggests, this is used for upgrade operations. It checks for anyincompatibilityandmakes repairs if it is necessary and also updates the granttableswithanychangesinnewversionsofMySQL.Italsoupdatesthesystemtables so you can take advantage of any new privilege or compatibility thatmighthavebeenaddedinthenewerversion.

BeforeperforminganupgradealwaysbackupyourcurrentMySQLinstallation.

Ifmysql_upgradefindsthatatablehaspossibleincompatibility,itperformsatablecheckandattemptsrepairingthetable,andifitcannotrepairit,itwillaskforamanualtablerepair.

mysql_upgradeshouldbeexecutedeachtimeMySQLisupgraded.Itcommunicatesdirectly with the MySQL server and sends the required SQL statements toperformanupgrade.

Oncewerunmysql_upgrade,weshouldrestart theserver. Ifanychangesmade tosystemtablesaretakenintoeffect,beforerunningit,youshouldmakesuretheserverisrunning.

Executemysql_upgradewiththefollowingsyntax:

shell>mysql_upgrade[options]

Youcanusethe--helpoptionhereandretrievealistofotheroptionswheneverrequired.

MySQL8clientprogramsThe MySQL 8 client programs are the programs that are commonly used toconnecttotheMySQLdatabaseandperformdifferentqueryoperations.

Theprogramsinformationdetailedinthefollowingsubsectionincludesmysql—commandlinetoolswithmanycommandsandrelatedoptionsandconfigurationforlogging,mysqlcheck,mysqldump,mysqlimport,mysqlsh,mysqladminandsoon.

mysql-thecommand-linetoolThis is themost commonly used program.The command-line tool is used forexecutingSQLstatementsdirectlyorusingafileinbatchmode.Ithassupportforboth interactiveandnon-interactivemodes. In this section,wewill lookatthemysqlcommand lineand thevariousoptions,commands, logging,andotherrelatedprograms.

mysqloptions

mysqlisacommandlinetoolthathasbeenprovidedforalongtimeandsoithasplentyofoptions togetyourworkdone.The following is the tableofoptionswithformatsanddescriptions:

Format Description--auto-rehash Enablesautomaticrehashing--auto-vertical-

output Enablesautomaticverticalresultsetdisplay--batch Donotusethehistoryfile--binary-as-hex Displaysbinaryvaluesinhexadecimalnotation

--binary-modeDisables\r\n-to-\ntranslationandtreatmentof\0asend-of-query

--bind-addressUsesspecifiednetworkinterfacetoconnecttoMySQLserver

--character-sets-dir Directorywherecharactersetsareinstalled--column-names Writescolumnnamesinresults--column-type-info Displaysresultsetmetadata

--commentsAscertainswhethertoretainorstripcommentsinstatementssenttotheserver

--compress Compressesallinformationsentbetweenclientandserver

--connect-expired-

password

Indicatestoserverthatclientcanhandleexpiredpasswordsandboxmode

--connect_timeout Numberofsecondsbeforeconnectiontimeout--database Thedatabasetouse

--debugWritesdebugginglog;supportedonlyifMySQLwasbuiltwithdebuggingsupport

--debug-check Printsdebugginginformationwhenprogramexits

--debug-infoPrintsdebugginginformation,memory,andCPUstatisticswhenprogramexits

--default-auth Authenticationplugintouse--default-character-

set Specifiesdefaultcharacterset--defaults-extra-

file Readsnamedoptionfileinadditiontousualoptionfiles--defaults-file Readsonlynamedoptionfile--defaults-group-

suffix Optiongroupsuffixvalue--delimiter Setsthestatementdelimiter--enable-cleartext-

plugin Enablescleartextauthenticationplugin--execute Executesthestatementandquit--force ContinuesevenifanSQLerroroccurs--get-server-public-

key PathnametofilecontainingRSApublickey--help Displayshelpmessageandexit--histignore Patternsspecifyingwhichstatementstoignoreforlogging--host ConnectstoMySQLserverongivenhost--html ProducesHTMLoutput

--ignore-spaces Ignoresspacesafterfunctionnames

--init-command SQLstatementtoexecuteafterconnecting--line-numbers Writeslinenumbersforerrors--local-infile EnablesordisableforLOCALcapabilityforLOADDATAINFILE--login-path Readsloginpathoptionsfrom.mylogin.cnf--max_allowed_packet Maximumpacketlengthtosendtoorreceivefromserver

--max_join_size Theautomaticlimitforrowsinajoinwhenusing--safe-updates

--named-commandsEnablesnamedmysqlcommands

--net_buffer_length BuffersizeforTCP/IPandsocketcommunication--no-auto-rehash Disablesautomaticrehashing--no-beep Donotbeepwhenerrorsoccur--no-defaults Readsnooptionfiles

--one-databaseIgnoresstatementsexceptthoseforthedefaultdatabasenamedonthecommandline

--pager Usesthegivencommandforpagingqueryoutput--password Passwordtousewhenconnectingtoserver--pipe OnWindows,connecttoserverusingnamedpipe--plugin-dir Directorywherepluginsareinstalled--port TCP/IPportnumbertouseforconnection--print-defaults Printdefaultoptions--prompt Settheprompttothespecifiedformat--protocol Connectionprotocoltouse--quick Donotcacheeachqueryresult--raw Writescolumnvalueswithoutescapeconversion

--reconnectIftheconnectiontotheserverislost,automaticallytriestoreconnect

--i-am-a-dummy,--

safe-updates

AllowsonlyUPDATEandDELETEstatementsthatspecifykey

values--secure-auth Donotsendpasswordstoserverinold(pre-4.1)format

--select_limit TheautomaticlimitforSELECTstatementswhenusing--safe-updates

--server-public-key-

path PathnametofilecontainingRSApublickey

--shared-memory-

base-name

Thenameofsharedmemorytouseforshared-memoryconnections

--show-warnings Showswarningsaftereachstatementifthereareany

--sigint-ignoreIgnoresSIGINTsignals(typicallytheresultoftypingControl+C)

--silent Silentmode--skip-auto-rehash Disablesautomaticrehashing--skip-column-names Donotwritecolumnnamesinresults--skip-line-numbers Skipslinenumbersforerrors--skip-named-

commands Disablesnamedmysqlcommands--skip-pager Disablespaging--skip-reconnect Disablesreconnecting

--socketForconnectionstolocalhost,theUnixsocketfileorWindowsnamedpipetouse

--ssl-ca PathoffilethatcontainslistoftrustedSSLCAs

--ssl-capathPathofdirectorythatcontainstrustedSSLCAcertificatesinPEMformat

--ssl-cert PathoffilethatcontainsX509certificateinPEMformat--ssl-cipher Listofpermittedcipherstouseforconnectionencryption--ssl-crl Pathoffilethatcontainscertificaterevocationlists

--ssl-crlpathPathofdirectorythatcontainscertificaterevocationlistfiles

--ssl-key PathoffilethatcontainsX509keyinPEMformat

--ssl-mode Securitystateofconnectiontoserver

--syslog Logsinteractivestatementstosyslog--table Displaysoutputintabularformat--tee Appendsacopyofoutputtonamedfile--tls-version Protocolspermittedforencryptedconnections--unbuffered Flushesthebufferaftereachquery--user MySQLusernametousewhenconnectingtoserver--verbose Verbosemode--version Displaysversioninformationandexit

--verticalPrintsqueryoutputrowsvertically(onelinepercolumnvalue)

--waitIftheconnectioncannotbeestablished,waitandretryinsteadofaborting

--xml ProducesXMLoutput

Reference:https://dev.mysql.com/doc/refman/8.0/en/mysql-command-options.html

Togetmoreinformationonindividualoptions,usethatoptionalongwiththe--helpoption.

mysqlcommandsEachof theSQLstatements thatyouissuearesent to theserverforexecution.Thereisalsoalistofcommandsthatmysqlitselfinterprets.Togetthelistofallthosecommands,type\hor\helpatthemysql>prompt.

Eachofthecommandshavebothlongandshortformthatcanbeused;exceptshort form cannot be used inmulti-line comments. The long form is not casesensitivebuttheshortformcommandiscasesensitive.

help[arg],\h[arg],\?[arg],?[arg]Thehelparg[]commandisusedtodisplayhelpmessages,alongwithlistingalltheavailablecommandsinmysql.

charset charset_name, \Ccharset_nameTochangedefaultcharsets,issuingtheSET_NAMESstatement.

clear,\cToclearthecurrentoutputorpreviousqueryresultsfromthecommandline.

connect[db_namehost_name], \r[db_namehost_name]Toreconnecttheserverbyprovidingdatabaseandhost_namearguments.

edit,\eToedittheinputstatementcurrentlyprovided.

exit,\qToexitthemysqlcommandline.

prompt[str],\R[str]Tospecifyastringandreconfigureitwiththemysqlprompt.

quit,\qToexitthemysqlcommandline.

status,\sThis is used to check for the status of the server connection that is currentlybeingused.

usedb_name,\udb_nameTospecifyusingtheprovideddb_nameasthedefaultdatabase.

mysqlloggingThemysqlprogramcandologgingasperthefollowingtypes.

On Unix systems, it writes the logs to the history file with the default name.mysql_historyinthehomedirectory.

Onallplatforms,ifthe--syslogoptionisprovided,itwritesthestatementstothesystemloggingimplementation.

OnUnix,itissyslog,onWindows,itiseventlogs,onLinuxdistributions,itoftengoestothe/var/log/messagefile.

<strong>mysql>helpsearch_string</strong>

<strong>mysql>helpme</strong><br/><strong>Nothingfound</strong><br/><strong>Pleasetrytorun'helpcontents'froalistofallaccessibletopics</strong>

Ifsearch_stringmatchesmultiplecontentsofatopic,thenitshowsalistofmatchingtopicitems.Atopiccanalsobeusedassearch_stringandlooksfortheentryforthetopic.Italsocontainsthewildcardcharacter%and_,whichhavethesamemeaningformatchingoperationsperformedbytheLIKEoperator.

<strong>shell>mysqldb_name<text_file</strong>

<strong>shell>mysql<text_file</strong>

<strong>mysql>sourcefile_name</strong>

Byusingthe--verboseoption,eachstatementgetsdisplayedjustbeforetheresultproducedbyit.

<strong>shell>mysqladmin[options]command[command-arg][command[command-org]]...</strong>

mysqladminsupportsplentyofprogramcommands,startingfromcreatedb_nametocreatenewdatabasewithnamedb_name,debugtogetdebuginformation,dropdb_nametodropadatabase,flush-xxxx,wherexxxxcanbereplacedwithlogs,hosts,privileges,status,tables,threads,andsoon.killidtokilltheserverthreadormultiplethreads,passwordnew_passwordtosetanewpassword,pingtochecktheserver'savailability,shutdowntostoptheserver,start-slavetostartreplicationonaslaveserver,stop-slavetostopreplicationontheslaveserver,variablestodisplayserversystemvariablesandtheirrespectivevalues.

mysqladminstatuscommandsgiveresultswithvaluesofuptime,threads,questions,slowqueries,opens,andflushtableswithrelevantinformation.

Alongwiththecommandlist,thereareoptionsthatcomeinhandywhenretrievingspecificinformationfromtheserver.Suchinformationcanberetrievedusingthemysqladmin--helpcommand.

<strong>shell>mysqlcheck[options]db_name[tbl_name...]</strong><br/><strong>shell>mysqlcheck[options]--databasesdb_name...</strong><br/><strong>shell>mysqlcheck[options]--all-databases</strong>

mysqlcheckhasaspecialfeaturethatisthedefaultbehaviorofcheckingtables.Itcanbechangedbyrenamingthebinary,suchasrenamingmysqlchecktothemysqlrepairprogrambycreatingacopyofmysqlcheckandaddingasymboliclinktomysqlcheck,afterwhichmysqlrepaircanrepairtablesinstead.Thisalsoworkswiththemysqlanalyzeandmysqloptimizeoptionstomakethemthedefaultoperationforthemysqlcheckcommand.

Similartootheradministeringprograms,thisprogramalsohavemanyoptionsthatcanbeusedtogetspecificinformation,andbyusingthemysqlcheck--helpcommand,alistofoptionscanberetrieved.

mysqldump - a database backupprogramThis program is a utility program used for making a logical backup bygenerating a set of SQL statements which can be executed to reproduce theoriginaldatabasetabledataandobjectdefinition.Itdumpsoneormoredatabasefor backup or may transfer to another SQL server. It can also generate dataoutputindifferentformats,suchasCSV,XML,orotherdelimitedtextfiles.

<strong>shell>mysqldump[options]db_name[tbl_name...]</strong><br/><strong>shell>mysqldump[options]--databasesdb_name...</strong><br/><strong>shell>mysqldump[options]--all-databases</strong>

Therearemorethan25optionsavailableformodifyingtheoperationofthemysqldumpcommandandtheycanberetrievedbyusingthemysqldump--helpcommand.Specificmodificationscanbeusedinthiscommandfordebuggingoptions,helpoptions,connectionoptions,DDLoptions,andsoon,basedonyourrequirement.

<strong>shell>mysqlimport[options]db_nametextfile1[textfile2...]</strong>

OptionsforthecommandscanbespecifiedontheCLIorinthe[mysqlimport]or[client]groupoftheoptionsfileasperyourrequirement.Itprovidedoptionstoretrieveandmodifyimportoperationsforthedata,suchasusingadifferentdelimiterformat,debugging,forcingthepathofafile,providingdefaultvalues,ignoringandlockingtables,andsoon,whichcanberetrievedbyusingthemysqlimport--helpcommand.

<strong>shell>mysqlpump--all-databases</strong>

<strong><spanclass="tokenprompt">shell></span><spanclass="tokencommand">mysqlpump</span><spanclass="tokenconstant">--include-databases</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>db1,db2</span><spanclass="tokenconstant">--exclude-tables</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>db1.t1,db2.t2</span></strong>

Theprecedingcommanddumpsdatabasesdb1anddb2butitwillexcludetablet1fromthedatabasedb1andtablet2fromthedatabasedb2.

mysqlpumpusesparallelismtoachievetheconcurrentprocessinganditcanbebetweendatabasesorwithinadatabase.--default-parallelism=NspecifiesthedefaultnumberofthreadsusedinthequeuecreatedbytheprogramandthevalueofNis2bydefault.--parallel-schemas=[N:]db_listsetsuptheprocessingqueueasperthedatabasenamelistprovided.Thus,additionalqueuesandthenumberofthreadscanbecontrolled.

mysqlpumpdoesnotdumpperformance_schema,nbdinfo,orsysbydefaultbutcandosobyspecifyingthe--include-databasesoptionandsimilarlyitalsodoesnotdumptheINFORMATION_SCHEMA.

mysqlsh-theMySQLShellThe advanced command line client and editor for MySQL is the very wellknownMySQLShell.IthascapabilitiesforscriptinginPythonandJavaScript.WhenconnectedtotheMySQLserverusingtheXProtocol,theXDevAPIcanwork with documents and relational data. It includes the AdminAPI, whichenablesyoutoworkwithanInnoDBcluster.

MySQLShell hasmanyoptions associatedwith it, but the important ones arelistedasfollows:

--port=port_num,-Pport_num

TheTCP/IPportnumbertousewithport_num.Thedefaultportis33060.

--node

CreatesanodesessiontoasingleserverusingtheXProtocolandisdeprecatedin8.0.3.

--js

StartsJavaScriptmode.

--py

StartsPythonmode.

--sql

StartsSQLmode.

--sqlc

StartsSQLmodeinClassicSession.

--sqln

StartsSQLmodeinNodeSession.

--sqlx

StartsSQLmodebycreatinganXprotocolconnection.

--ssl*

Optionsbeginningwith--ssl specifyconnecting theserverusingSSLandalsofinding certificates and SSL keys. It works the same way as for theMySQLServerand itacceptsSSLoptions :--ssl-crl,--ssl-crlpath,--ssl-mode,--ssl-ca, --ssl-capath,--ssl-cert,--ssl-cipher,--ssl-key,--tls-version.

Otheroptionsnotlistedcanberetrievedusingthemysqlsh--helpcommand.

mysqlshow - showing database,table,andcolumninformationThis is the client mainly used to quickly check which databases, their tables,columns,andindexesexistornot. Itprovidesaninterfacetosomeof theSQLSHOWstatements.

Theexecutionsyntaxforthecommandisasfollows:

shell>mysqlshow[options][db_name[tbl_name[col_name]]]

By executing the preceding command, you will get information about thedatabases,tables,orcolumnsforwhichyouhaveprivileges:

AlistofdatabasesisshownifnodatabaseisgivenAlistofallmatchingtablesinthedatabaseisshownifnotableisgivenA list of all the matching columns and column types in the table isdisplayedifnocolumnisgiven

Inthepreviouscommandexecution,ifyouusedtheSQLwildcardcharacter(*,?,%,_), thennamesthatarematchedby thewildcardaredisplayed.*and?wildcardcharacters,ifgiven,areconvertedinto SQL % and _ wildcard characters. It might create confusionwhen trying to display a columnwith a table or a column namewith_inthename,itdisplaysonlynamesmatchingthepattern,butit can be fixed easily by adding an extra % at the end of thecommandlineasaseparateargument.

Theprogramhasmanyoptionstogetthedesiredinformationbyusingspecificoptionarguments.Afewoftheimportantonesaregivenasfollows:

--character-sets-dir=dir_name

Specifiesthedirectorynamewherecharactersetsareinstalled.

--compress,-C

Ifboththeclientandtheserversupportcompression,alltheinformationsentiscompressed.

--enable-cleartext-plugin

Enablesthecleartextauthenticationplugin.

--get--server-public-key

Requests the RSA public key from the server required for key-pair-basedpassword exchange. Also, if a client connects to the server using a secureconnection,RSA-basedpasswordexchangeisnotneededandisignored.

--keys,-k

Displaystableindexes.

--ssl*

Specifies options starting with --ssl to connect to the server using an SSLconnectionbyusingcertificatesandSSLkeys.

Manyotheroptionsnotlistedherecanberetrievedbyusingthemysqlshow--helpcommand.

<strong>shell>mysqlslap[options]</strong>

<strong>mysqlslap<spanclass="tokenconstant">--delimiter</span>=<spanclass="tokenatrule">";"</span><spanclass="tokenconstant">--create</span>=<spanclass="tokenatrule">"CREATETABLEt(iint);INSERTINTOtVALUES(21)"</span><spanclass="tokenconstant">--query</span>=<spanclass="tokenatrule">"SELECT*FROMt"</span><spanclass="tokenconstant">--concurrency</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=2</span>0</span><spanclass="tokenconstant">--iterations</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=1</span>00</span></strong>

<strong>mysqlslap<spanclass="tokenconstant">--concurrency</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=7</span></span><spanclass="tokenconstant">--iterations</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>20</span><spanclass="tokenconstant">--number-int-cols</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>2</span><spanclass="tokenconstant">--number-char-cols</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=2</span></span><spanclass="tokenproperty">--auto-generate-sql</span></strong>

Here,mysqlslapwillbuildastatementwithatableoftwoINTcolumnsandtwoVARCHARcolumnsandsevenclientsquerying20timestoeachofthem.Italsosupportsspecifyingstatementfilesforcreatingandqueryingseparatelyandrunningtheloadtest.Itprovidesmanysuchalterationsinloadtestingexecutionwithoptionsthatyoucancheckbyexecutingthemysqlslap--helpcommandonthecommandline.

MySQL8administrativeprogramsThis section describes different administrative programs along with someutilities that will help in doing administration operations such as performingchecksum,compressionandextraction,andsoon.

ibdsdi - InnoDB tablespace SDIextractionutilityThis is a utility program that extracts serialized dictionary information fromInnoDB tablespace files. Serivalized dictionary information that is SDI datawillalwaysbepresent inallpersistent InnoDB tablespace files. It canbe runon file-per-tableontablespacefilesandgeneraltablespacefiles,systemtablespacefiles,anddatadictionarytablespacefiles,butusingatemporarytablespaceorundoingatablespaceisnotsupported.

ibd2sdicanbeusedwhiletheserverisofflineoratruntime.ItreadsuncommitteddataofSDIfromaspecifiedtablespaceandundoeslogsandredoeslogsthatarenotaccessible.

Executionfortheidb2sdiwilllooklikethefollowingcommandline:

shell>ibd2sdi[options]file_name1[file_name2file_name3...]

ibd2sdi also supports multiple tablespaces but does not run on more that onetablespace at a time as the InnoDB system tablespace. Specifying each file willworkasfollows:

shell>ibd2sdiibdata1ibdata2

ibd2sdioutputsSDIdatainJSONformat.

Therearemanyoptionsavailablefortheprogramwhichcanberetrievedusingtheibd2sdi--helpcommand.

<strong>shell>innochecksum[options]file_name</strong>

innochecksumcommandalsohasfewoptionstodisplayinformationofthepagesbeingverifiedandcanberetrievedwithinnochecksum--helpcommand.

myisam_ftdump - displaying full-textindexutilityThis is a utility for displaying information about MyISAM tables and FULLTEXTindexes.Itwillscananddumptheentireindex,whichcanbeaslowandlengthyprocess.Iftheserverisalreadyrunning,thenyouneedtomakesuretoinsertaFLUSHTABLESstatementfirst.

Executionforthemyisam_ftdumpcommandwilllooklikethefollowingcodeblock:shell>myisam_ftdump[options]<table_name><index_num>

Inthepreviousexample,table_nameshouldbethenameoftheMyISAMtablewiththe.MYIindexextension.

Suppose the test database has a table named mytexttable with the followingdefinition:CREATETABLEmytexttable ( id INTNOTNULL, txt TEXTNOTNULL,PRIMARYKEY(id),FULLTEXT(txt))ENGINE=MyISAM;

Theindexcreatedonidis0onFULLTEXTindexandonthetxtitis1.Iftheworkingdirectory is the test database directory, then execute myisam_ftdump as follows:shell>myisam_ftdumpmytexttable1

myisam_ftdump can also be used to generate a list of index entries in order offrequencyofoccurrenceasfollows(thefirstlineinWindowsandthesecondlineforUnixsystems):shell>myisam_ftdump-cmytexttable1|sort/Rshell>myisam_ftdump-cmytexttable1|sort-r

myisam_ftdump also has several options that can be retrieved by using themyisam_ftdump--helpcommand.

myisamchk - MyISAM table-maintenanceutility

myisamchk is a command line tool for getting information about database tables,checking,repairing,

andoptimizingnon-partitionedMyISAMtables.ItworkswithMyISAMtables.

CHECK_TABLEandREPAIR_TABLEstatementscanalsobeusedtocheckandrepairMyISAMtables.

Executionforthemyisamchkcommandisshowninthefollowingcodeblock:

shell>myisamchk[OPTIONS]tables[.MYI|.MYD]

Beforerunningthemyisamchkcommand,youmustmakesurethatanyotherprogramisnotusingthetables.Otherwise,

itwilldisplaywarningmessagesaying:warning:clientsareusingor haven't closed the table properly. To do this effectively, shutdowntheMySQLserverorlockalltablesbeingused

bymyisamchk.

Thisprogramhasmanyoptionstoperformtablemaintenance.

<strong>shell>myisamlog[options][file_name[tbl_name]...]</strong><spanclass="tokenpunctuation"><br/></span>

Thedefaultoperationismarkedasanupdateandifrecoveryisdoneallwrites,updates,anddeletesaredoneanderrorsarecountedonly.myisam.logisthedefaultlogfilename.

Theprogramhassomeoptionsusedtospecifyoffsets,recovery,openanumberoffiles,andafewmore,whichcanberetrievedbyusingthemyisamlog--helpcommand.

myisampack - generatingcompressed, read-only MyISAMtablesThisisautilityprogramthatcompressesMyISAMtables.Itcompresseseachofthecolumnsintablesseparatelyandcompressesthedatafilebyabout40%to70%.

MySQL preferably uses the mmap() function to do memory mapping oncompressedtables;otherwise,itusesnormalread/writefileoperations.

myisampackdoesnotsupportpartitionedtables.

Oncetablesarepacked,theybecomereadonly.

Stopping theserverand thengoingforcompress tables issafeway toperformthecompressoperation.

Executionsyntaxformyisampack looks like the followingblockon thecommandline:

shell>myisampack[options]file_name...

Specifytheindexfilenamewithorwithoutthe.MYIfileandalsoaddthepathnameifyouarenotinthedatabasedirectory.

Aftercompressingatablewithmyisampack,youshouldusemyisampack-rqtorebuildthe indexesof thecompressed table. It supports someof thecommonoptions,such as versioning, debugging, and so on along with specific compressionchecks,suchas--test,--backup,--join=big_tbl_name,--silent,andsoon.Ifyouwanttocheckindetail,youcanexecutethemyisampack--verbose--helpcommandonthecommandline.

mysql_config_editor - MySQLconfigurationutilityThisisautilitytostoreandupdatetheauthenticationcredentialsinanencryptedloginpathfilewiththename.mylogin.cnf.

Forfurtherdetails,usethemysql_config_editor--verbose--helpcommandtoexecuteonthecommandline.

<strong>shell>mysqlbinlog[options]log_file...</strong>

Thereareseveraloptionsthatmodifytheformatoftheoutputandusageofmysqlbinlog.Theyarelistedasfollows:

Itcanbeconvertedintoahexdumpformatthatcontainsbyteposition,eventtimestamp,andtypeoftheeventthatoccurredItalsoprovidesaroweventdisplayformatthatdisplaysdatamodificationsinformationintheformofpseudo-SQLstatementsItisalsousedformakingabackupofthebinarylogfilesbyprovidingtherequiredoptionvalues,suchasapathforthebackupfileandtypeorformatoftheoutputinthebackupWhenyouconnecttotheMySQLserverusingmysqlbinlog,itprovidesaspecificserverIDtoidentifyitselfandrequestsbinarylogfilesfromtheserver

Theprogramhassomecommonoptionswhicharenotmentionedbutcanberetrievedusingthemysqlbinlog--helpcommand.

<strong>shell>mysqldumpslow[options][log_file...]</strong>

Youcanmodifytheoutputbyusingsomeoptions,suchaslimitingthenumberofresult-tN,whereNisthenumberofqueryresultstobedisplayed.-sstandsforsorttypebyquerytime,locktime,orbyrowscount,and-rforreversingthesortorder.

MySQL8environmentvariables

Inthissection,wewilllookatthenumberofenvironmentvariablesthatareused

directly or indirectly for different MySQL programs, changing their behaviorwith

theuseofenvironmentvariables.

Optionsprovidedonthecommandlinetakeprecedenceovervaluesspecifiedinthe

option files and on the environment variables, and similarly values in optionstake

precedenceover the environmentvariables; so inmost cases, it ispreferred touse

anoptionfileinsteadofenvironmentvariablestomodifythebehavior.

The following is the list of environment variables and descriptions for thevariable:

CXX:ThenameofyourC++compilerforrunningCMake

CC:ThenameofyourCcompilerforrunningCMake

DBI_USER:ThedefaultusernameforPerlDBI

DBI_TRACE:TraceoptionsforPerlDBI

HOME:Thedefaultpathforthemysqlhistoryfileis$HOME/.mysql_history

LD_RUN_PATH:Usedtospecifythelocationoflibmysqlclient.so.

LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN:Enables the mysql_clear_passwordauthenticationplugin

LIBMYSQL_PLUGIN_DIR:Directoryinwhichtolookforclientplugins

LIBMYSQL_PLUGINS:Clientpluginstopreload

MYSQL_DEBUG:Debugstraceoptionswhendebugging

MYSQL_GROUP_SUFFIX:optionalgroupsuffixvalue(suchasspecifying--defaults-group-suffix)

MYSQL_HISTFILE: The path to the mysql history file; if this variable is set, itsvalueoverridesthedefaultfor$HOME/.mysql_history

MYSQL_HISTIGNORE:Patternsspecifyingstatementsthatmysqlshouldnotlogto$HOME/.mysql_history,orsyslogif--syslogisgiven

MYSQL_HOME:Thepath to thedirectory inwhich theserver-specificmy.cnf fileresides

MYSQL_HOST:Thepath to thedirectory inwhich theserver-specificmy.cnf fileresides

MYSQL_PWD: The default password when connecting to mysqld; using this isinsecure

MYSQL_TCP_PORT:DefaultTCP/IPportnumber

MYSQL_TEST_LOGIN_FILE:Nameofthe.mylogin.cnfloginpathfile

PATH:UsedbytheshelltofindMySQLprograms

PKG_CONFIG_PATH:Locationofthemysqlclient.pcpkg-configfile

TMPDIR:Directoryinwhichtemporaryfilesarecreated

TZ:Thisshouldbesettoyourlocaltimezone

UMASK:User-filecreationmodewhencreatingfiles

UMASK_DIR:User-directorycreationmodewhencreatingdirectories

USER:DefaultusernameonWindowswhenconnectingtomysqld

MYSQL_TEST_LOGIN_FILE is the pathname for the login path file that is created bymysql_config_editor.

TheUMASKandUMASK_DIRvariablesareusedasmodesinsteadofmasks.

It is necessary to set PKG_CONFIG_PATH if using pkg-config for building MySQLprograms.

MySQLGUItoolsTherearemanyMySQLGUItoolsavailableforperformingvariousoperations,startingwithcreatingdatabasestoperformingdailyadministrationtasks.

MySQLWorkbench

MySQLWorkbench is a graphical tool to work with the MySQL server anddatabases.It

fully supportsMySQLversions 5.1 and above. In this section,wewill brieflydiscuss

thecapabilitiesofMySQLWorkbench.

FivemainfunctionprovidedbyMySQLWorkbenchareasfollows:

SQL development: This creates and manages database connections,andconfigurationforconnection

parameters. It executes SQL statements using the built-in SQL editoranditreplaces

thestandaloneapplicationquerybrowserprovidedearlier.

Datamodeling: This createsmodels of database schema graphically,

withreverseandforwardengineering

betweentwodifferentschemaaswellasonalivedatabase.Itprovidesacomprehensive

table editor, with easy to use facilities for editing tables, columns,triggers,indexes,

options,inserts,partitioning,routines,views,andprivileges.

Serveradministration:Thiscreates,maintains,andadministersserverinstances.

Data migration: This allows migration from PostgreSQL, SQLite,SybaseASE,MicrosoftSQLServer,

and other relational databasemanagement system objects, tables, anddatatoMySQL.

It also facilitates migration from earlier versions to the latest releaseversion.

MySQLEnterprisesupport:Thisprovidesenterpriselevelsupportfortheproduct;forexample,MySQLEnterprise

backup,MySQLAudit.

MySQL workbench is available in two different editions, theCommercialEditionand

the Community Edition. The Community Edition is providedwithoutanycost.TheCommercial

Edition provides additional features, such as databasedocumentationgeneration.

MySQLNotifierMySQL Notifier is a simple tool used to monitor and adjust the status oflocal/remoteMySQL server instances. It is an indicator which is placed in asystemtray.ItisinstalledwiththeMySQLinstalleritself.

MySQLNotifier acts as quick launcherwith list actions clubbed together thatcanbeactandmonitoredveryeasilyfromthesystemtrayitselfandalongwiththat it keeps monitoring based on specified interval and notifies on statuschange.

MySQLNotifierusage

MySQLNotifierstaysinsystemtrayandprovidesoneclickoptionforMySQLstatusandmaintenance.FollowingsareimportantusageofMySQLNotifier.

MySQLNotifierprovidesstart,stop,andrestartofMySQLSeverinstances

MySQL Notifier configures MySQL server services and automaticallydetectsandaddsnewMySQLserverservices

MySQLNotifiermonitorsbothlocalandremoteMySQLinstances

Summary

In this chapter, we dived deep into the ocean of commands that are used foralmostalltheactivitiesfortheMySQL

server, starting with installation, server start up, client programs toadministrative

programs, and several utility programs to cater for different purposes on aroutine

basis for database administration. The chapter also provided a workingknowledgeof

makingdatabasebackupsand importing thedatabaseswithorwithoutspecifictables.

ThenextchapterwillfocusonMySQL8datatypesindetail.Itwillcategorizedata

types based on their content types and itwill describe properties in detail foreach

of the categories and storage level details that should be kept inmind duringtable

andcolumndesign.

MySQL8DataTypes

In the previous chapter, we learned how to use MySQL 8 command-lineprogramsandutilitiestoperformvariousoperationsontheMySQL8database.Itisalwaysgoodtohaveholdoncommand-linetools.Itprovidesflexibilitytowork innon-GUIenvironments.Thefocusof thischapter isdata types. Isn't itfascinatingtoknowthetypeofdatatheprogramminglanguagesupportsorthestorage engine can store? It is a fundamental feature of any programminglanguageordatabase.Atthesametime,itisthemostignoredtopic,aswell.Themajority of programmers don't spend enough time assessing the storagerequirements for variables used in a piece of code. Actually, it is extremelyimportant to understand the basic and custom data types that the databasesupports,whichiswhythischapterexists.

Thefollowingisalistoftopicstobecoveredinthischapter:

OverviewofMySQL8datatypes

Numericdatatypes

Dateandtimedatatypes

Stringdatatypes

JSONdatatype

Storagerequirementsfordatatypes

Choosingtherightdatatypeforcolumns

OverviewofMySQL8datatypes

All standard SQL data types are supported in MySQL. These data types areclassifiedinafewcategories,suchasnumerictypes,stringtypes,dateandtimetypes,andtheJSONdatatype.Whenweassignadatatypetoacolumn,certainconventionsmustbefollowed.TheseconventionsarenecessaryforMySQLtoallowvaluestobestoredinacolumn:

Mdenotesthemaximumdisplaywidthforintegertypes.Forfloatingpointandfixedpointtypes,itisthetotalnumberofdigitsthatcanbestored.Forstring types, it is the maximum length. The maximum value alloweddependsonthedatatype.

D is applicable to floating points and fixed point types. It denotes thenumberofdigitsafterthedecimalpoint.Themaximumallowedvalueis30,butmustbelessthanorequaltoM-2.

fsp is applicable to date and time types. It denotes the fractional secondsprecision,whichmeansthenumberofdigitsfollowingthedecimalpointforthefractionalpartofseconds.

This overview is brief so that the detailed descriptions of the features of eachdatatypecanbecoveredintopicstofollow.

NumericdatatypesTheMySQL8numericdatatypesincludeintegerorexactdatatypes,decimalorapproximatedatatypes,andbitdatatypes.

Bydefault,REALdatatypevaluesarestoredasDOUBLE.IfwehavesettheREAL_AS_FLOATflagonMySQL,REALdatatypevaluesarestoredasFLOAT.FLOAToccupieslessspacecomparedtoDOUBLE.

<strong>CREATETABLEemployees</strong><br/><strong>(salaryINTEGER(5)UNSIGNED);</strong>

<strong>CREATETABLEemployees</strong><br/><strong>(idINT(255));</strong>

<strong>CREATETABLEdocuments</strong><br/><strong>(document_noINT(5)ZEROFILL);</strong>

Wespecifiedthevaluetobestoredas111;itwillbestoredas00111ifweprovidedtheZEROFILLoption.

FixedpointtypesFixed point types represent numbers with a fixed number of digits after thedecimalor radixpoint.MySQLhas DECIMALandNUMERIC as fixedpoint, or exact,value data types.These values are stored in a binary format. Fixed point datatypes are useful, especially in storing monetary values in multiplication anddivisionoperations.Thevalue of a fixedpoint data type is an integer numberscaledbyaspecificfactor,accordingtothetype.Forexample,thevalueof1.11canberepresentedinfixedpointas111,withascalingfactorof1/100.Similarly,1,110,000canberepresentedas1110,withascalingfactorof1000.

ThefollowingcodeblockdemonstratesthedeclarationofaDECIMALdatatype:

CREATETABLEtaxes

(tax_rateDECIMAL(3,2));

Intheprecedingexample,3istheprecisionand2isthescale.Anexamplevaluecouldbe4.65,where4istheprecisionand65isthescale:

Precision:DenotesthenumberofsignificantdigitsstoredforthevaluesScale:Representsthenumberofdigitsafterthedecimalpoint

Precisionandscaledefinetherangeofvaluesthatcanbestoredinthecolumn.So,intheprecedingcolumndeclaration,tax_ratecanstorevaluesfallingbetween-9.99and9.99.

ThesyntaxfordefiningtheDECIMALtypeinstandardSQLisasfollows:

DECIMAL(M)

InMySQL,thisisequivalentto:

DECIMAL(M,0)

DeclaringacolumnwithDECIMALisequivalenttoDECIMAL(M,0)inMySQL.

InMySQL,10isthedefaultvalueforM,ifit'snotprovided.

Themaximumnumberof digits supported for the DECIMAL type is 65, includingprecisionandscale.Wecan limit thenumberofdigitsonvalueswhichcanbeenteredforacolumnbyusingprecisionandscale.Ifauserentersavaluewithalarger number of digits than permitted in scale, the valuewill be truncated tomatchthepermittedscale.

DECIMAL is often considered tobe an alternative to DOUBLE or FLOAT. Asmentionedearlier, DECIMAL numbers are an exact representation of REAL numbers inmathematics. The only problem with the DECIMAL data type is that it occupiesmuch more space, even for small numbers. For example, to store a value of0.000003,thecolumndeclarationshouldhavethedatatypedefinedasDECIMAL(7,6).

Ifthescaleis0,thecolumnvaluesdon'thavedecimalpointsorfractionalvalues.

FloatingpointtypesFloatingpointnumbersrepresentrealnumbersincomputing.Realnumbersareusefulformeasuringcontinuousvalues,suchasweight,height,orspeed.

MySQLhas twofloatingpointdata types forstoringapproximatevalues:FLOATandDOUBLE.

For floating point numbers, precision is an important factor. Precision definesthemeasureofaccuracy.MySQLsupportssingleprecisionanddoubleprecisionfloating point numbers. It consumes four bytes to store a single precisionfloatingpointnumberwiththeFLOATdatatype,whereasitconsumeseightbytestostoreadoubleprecisionfloatingpointnumberwiththeDOUBLEdatatype.

In MySQL, REAL is a synonym for DOUBLE PRECISION. As mentioned earlier, ifREAL_AS_FLOAT isenabled,acolumndefinedwiththeREALdatatypewillbetreatedsimilarlytoFLOAT.

TheprecedingdescriptiondepictsFLOATorDOUBLEassimilartoDECIMAL.No,itisnot.There isahugedifference.Asdescribedearlier, fixedpointdata typessuchasDECIMAL or NUMERIC can store exact values, up to the maximum digit after thedecimal point, whereas floating point data types, such as FLOAT or DOUBLE, storeapproximatevalues.Thevaluesstoredaredetailedenough,butnotcompletelyaccurate.Thereremainsaminorinaccuracy.

Let'sunderstandthisthroughthefollowingcodeexample:

mysql>CREATETABLEtyped_numbers(idTINYINT,float_valuesFLOAT,decimal_valuesDECIMAL(3,2));

mysql>INSERTINTOtyped_numbersVALUES(1,1.1,1.1),(2,1.1,1.1),(3,1.1,1.1);

mysql>SELECT*FROMtyped_numbers;

+------+--------------+------------------+

|id|float_values|decimal_values|

+------+--------------+------------------+

|1|1.1|1.10|

|2|1.1|1.10|

|3|1.1|1.10|

+------+--------------+------------------+

mysql>SELECTSUM(float_values),SUM(decimal_values)FROMtyped_numbers;

+--------------------+---------------------+

|SUM(float_values)|SUM(decimal_values)|

+--------------------+---------------------+

|3.3000000715255737|3.30|

+--------------------+---------------------+

Intheprecedingexample:

1. WecreatedatablecontainingFLOATandDECIMALtypecolumns.2. We inserted the same values in two columns, named float_values and

decimal_values.3. Weexecutedaselectquerytofetchthesumofstoredvalues.

Thoughthesamevalues,theoutputisdifferent.Thesumofdecimal_valueslooksmoreprecisecompared to thatof thefloat_values.Thesumoffloat_values looksless precise. This is because of internal rounding performed by the MySQLenginefor floating point data types,which results in the approximation storedvalue.

Standard SQL has a provision for specifying precision while defining a FLOATcolumn. The precision is in bits specified following the keyword FLOAT withinparenthesis.MySQLalsosupportsspecifyingprecisionvaluesforFLOATorDOUBLE,buttheprecisionisusedtodeterminethesize:

Precisionfrom0to23resultsina4bytesingleprecisionFLOATcolumnPrecisionfrom24to53resultsinan8bytedoubleprecisionDOUBLEcolumn

ThefollowingisanexampleofFLOATcolumndeclarationattributes:

FLOAT(M,D)

where,

M-numberofdigitsintotal

D-numberofdigitsmaybeafterthedecimalpoint

So,thecolumndefinedasthefollowingwillstoreavaluesuchas99.99:

FLOAT(4,2)

While storing floating point values,MySQLperforms rounding. So, the valueinserted as 99.09 into a FLOAT(4, 2) column may be stored as 99.01 as anapproximateresult.

Though the floating point column definition supports specifyingprecision, it is advisable to use FLOAT or DOUBLE PRECISION with no

precisionornumberofdigits,soastotakeadvantageofmaximumflexibilityandportability.

<strong>mysql>CREATETABLEtemp(idINT,col1DOUBLE,col2DOUBLE);</strong><br/><br/><strong>mysql>INSERTINTOtempVALUES(1,5.30,2.30),(1,-3.00,0.00),</strong><br/><strong>(2,0.10,-10.00),(2,-15.20,4.00),(2,0.00,-7.10),</strong><br/><strong>(3,0.00,2.30),(3,0.00,0.00);</strong><br/><br/><strong>mysql>SELECTid,SUM(col1)asv1,SUM(col2)asv2FROMtemp</strong><br/><strong>GROUPBYidHAVINGv1<>v2;</strong><br/><strong>+------+--------+--------+</strong><br/><strong>|id|v1|v2|</strong><br/><strong>+------+--------+--------+</strong><br/><strong>|1|2.3|2.3|</strong><br/><strong>|2|-15.1|-13.1|</strong><br/><strong>|3|0.0|2.3|</strong><br/><strong>+------+--------+--------+</strong>

Intheprecedingexample,itseemsthatthefirsttworowsintheoutputhavesimilarnumbers.Itispossiblethattheymightnotbe,inthecaseoffloatingpointtypes.Ifwewanttoensure,intheprecedingcase,thatsimilar-lookingvaluesareconsidered,wehavetocomparethedifferenceagainstapredefinednumberwithprecision.Forexample,intheprecedingcase,ifwemodifytheHAVINGclausetochecktheconditionABS(v1-v2)>0.1,itwillreturntheexpectedoutput.

Asinterpretationoffloatingpointnumbersisplatformdependent,ifwetrytoinsertavaluewhichisoutsideoftherangeoffloatingpointdatatypesupportedvalues,itmayinsert+-infor+-0.

BitvaluetypeHave you ever come across a requirement to store binary representations ofnumbers?Canyouthinkofsuchusecases?Onesuchusecaseistostoreweeklyworkingdaysinformationforayear.Wewilltouchbaseonthisexamplelaterinthesection.

TheBITdatatypeisusedtostorebinarybitsorgroupsofbitvalues.ItisalsooneoftheoptionstostoreBoolean,yes/noor0/1values.

TheBITtypecolumncanbedefinedas:

column_nameBIT

or

column_nameBIT(m)

wherem=numberofbitstobestored

ForaBITdatatype,mcanvaryfrom1to64.Supplyingmisoptional.Thedefaultvalueformis1.

ThefollowingisanexampleofhowaBITcolumncanbedefined:

CREATETABLEworking_days(

yearINT,

weekINT,

daysBIT(7),

PRIMARYKEY(year,week));

AftertheBITdatatypecolumndeclaration,nextisstoringbitvaluesinacolumn.The bit values are a combination of zeros (0s) and ones (1s). The b'value'notationisusedtospecifybitvalues.

Thefollowingaretheexamplesofhowtostore11and55inaBITcolumn:

CREATETABLEbit_values(valBIT(7));

INSERTINTObit_valuesVALUES(b'1011');

INSERTINTObit_valuesVALUES(b'110111');

WhathappensifthevaluestoredintheBITcolumnislessthanthenumberofbits(m)specifiedinthecolumndefinition?MySQLwillpadthevaluewith0sontheleft of the number. So, for the preceding example, the values stored will be

0001011and0110111,respectively.

HowdowedefineaBITcolumntostoreboolean_values?Thefollowingcodeblockshowsthat:

CREATETABLEboolean_values(valueBIT(1));

or

CREATETABLEboolean_values(valueBIT);

INSERTINTOboolean_valuesVALUES(b'0');

INSERTINTOboolean_valuesVALUES(b'1');

<strong>mysql>SELECTb'1010110',CHARSET(b'1010110');</strong><br/><strong>+--------------+----------------------+</strong><br/><strong>|b'1010110'|CHARSET(b'1010110')|</strong><br/><strong>+--------------+----------------------+</strong><br/><strong>|V|binary|</strong><br/><strong>+--------------+----------------------+</strong><br/><br/><strong>mysql>SELECT0b1100100,CHARSET(0b1100100);</strong><br/><strong>+--------------+----------------------+</strong><br/><strong>|0b1100100|CHARSET(0b1100100)|</strong><br/><strong>+--------------+----------------------+</strong><br/><strong>|d|binary|</strong><br/><strong>+--------------+----------------------+</strong>

PracticalusesofBITLet'scontinuewiththeworkingdaysperweekinayearexample.Pleaserefertotheworking_daystableschemaprovidedearlier.

HowcanwespecifythattheMondayandFridayintheweek4 intheyear2017are non-working days? The following is the INSERT query for this: INSERTINTOworking_daysVALUES(2017,4,0111011);

Ifwe fetch the working_days records using the SELECT query, the following is theoutput:

mysql>SELECTyear,week,daysFROMworking_days;

+--------+---------+--------+

|year|week|days|

+--------+---------+--------+

|2017|4|59|

+--------+---------+--------+

In theprecedingoutput, thedays, thoughbeingofbitdata types,showintegervalues.Howcanweshowbitvaluesintheoutput?

TheansweristheBIN()MySQLfunction.Thefunctionconvertsanintegervaluetoitsbinaryrepresentation:

mysql>SELECTyear,week,BIN(days)FROMworking_days;

+--------+---------+------------+

|year|week|days|

+--------+---------+------------+

|2017|4|111011|

+--------+---------+------------+

Asyoucan see, the leadingzeros are removed from thedays' bit value in theoutput.Toaccomplishtherepresentationintheoutput,ontopoftheBINfunction,wecanusetheLPADMySQLfunction:

mysql>SELECTyear,week,LPAD(BIN(days),7,'0')FROMworking_days;

+--------+---------+------------+

|year|week|days|

+--------+---------+------------+

|2017|4|0111011|

+--------+---------+------------+

TypeattributesAs shown earlier, while defining integer columns, we can also specify anoptionaldisplaywidthattribute.Forexample,INT(5)indicatesanintegernumberwithadisplaywidthof5digits.WhenthiscolumnisusedintheSELECTquery,theoutputwilldisplaythenumberleftpaddedwithspaces.So,ifthevaluestoredintheINT(5)columnis123,thenitwillbedisplayedas__123.The_willbeaspaceintheactualoutput.

However,thedisplaywidthdoesn'tlimittherangeofvalueswhichcanbestoredintheINT(5)column.Thequestionthenarises:Whatifwestoreavalueforwhichthedisplaywidth is larger than thedisplaywidthspecified?Thedisplaywidthdoesn't prevent values wider than the display width of a column from beingdisplayed correctly. So, values wider than the column display width aredisplayedinfullwidth,usingmorethanthenumberofdigitsspecifiedwiththedisplaywidth.

Asmentionedearlier,MySQLcolumndefinitionprovidesanoptionalattributecalled ZEROFILL. This optional attribute, when specified, replaces left paddedspaceswithzeros.Forexample,foracolumndefinedasthefollowing,avalueof82isretrievedas00082:

INT(5)ZEROFILL

This optional attribute is useful where the proper formatting of numbers isimportant.

TheZEROFILLattribute is ignoredwhen thecolumnvalue isused inexpressionsorinaUNIONquery.

MySQLcreatestemporarytableswhencomplicatedjoinsareusedinaquerytostoreintermediateresults.Insuchacase,wemayfaceissuesifwespecifiedacolumnwithdisplaywidth.Inthesecases,MySQLconsidersthatthedatavaluesfitwithinthedisplaywidth.

Another important attribute is UNSIGNED.The UNSIGNED attribute permits only non-

negative values to be stored in the column. It is also useful when we needsupportforalargerrangeofvaluestobestoredwiththesamedatatype.

UNSIGNEDisalsosupportedbyfloatingpointtypesandfixedpointtypes.

If we specify a ZEROFILL attribute for a column, UNSIGNED isautomaticallyaddedtothecolumn.

Another important attribute for integer and floating point columns isAUTO_INCREMENT. When we insert a NULL value in the column defined with theAUTO_INCREMENTattribute,MySQLstoresvalue+1insteadofNULL.Avalueof0willbetreated the sameas that of a NULL value, unless the NO_AUTO_VALUE_ON_ZEROmode isenabled.Here,thevalueisthelargestvaluestoredinthecolumn.ItisextremelyimportantthatthecolumnisdefinedasNOTNULL.Otherwise,theNULLvaluewillbestoredasNULL,eventhoughtheAUTO_INCREMENTattributeisprovided.

Overflowhandling

Whenanout-of-rangevalueisstoredinthenumerictypecolumninMySQL,thevaluestoreddependsontheMySQLmode:

Ifstrictmode is enabled,MySQLwillnot accept thevalueand throwanerror.Theinsertoperationfails.

If restrictive modes are enabled, the value is clipped by MySQL to anappropriatevalue,andthatiswhatisstoredinthecolumn.

DateandtimedatatypesDATE,TIME,DATETIME,TIMESTAMP,andYEAR formthegroupofdateandtimedatatypesforstoringtemporalvalues.Eachofthesetypeshasarangeofpermittedvalues.Apartfromthepermittedvalues,aspecialzerovaluecanalsobeusedtospecifyaninvalidvaluewhichMySQLcannotrepresent.Thezerovaluecanbe00-00-0000.MySQLallowsthisvaluetobestoredinadatecolumn.ThisissometimesmoreconvenientthanstoringNULLvalues.

Thefollowingarethegeneralconsiderationswemusttakecareofwhileworkingwithdateandtimetypes.

ThewayMySQLtreatsstorageandretrievaloperationsfordateortimetypesisdifferent in the context of the format.Basically, for a date or time type valuestoredinthetable,MySQLretrievesvalues inastandardoutput format. In thecaseofinputtingadateor timetypevalue,MySQLattemptstoapplydifferentformatsonthesuppliedinputvalue.So,itisexpectedthatthesuppliedvalueisvalid,orunexpectedresultsmayoccurifusedvaluesinunsupportedformats.

ThoughMySQLcaninterpret inputvalueswithseveraldifferentformats,partsof the date valuemust be supplied in a year-month-day format. For example,2017-10-22or16-02-14.

Supplying a two-digit year creates ambiguity forMySQL to interpret the yearbecause of the unknown century. The following are the rules that must befollowed,usingwhichMySQLinterpretstwo-digityearvalues:

Yearvaluesbetween70-99areconvertedto1970-1999Yearvaluesbetween00-69areconvertedto2000-2069

Itispossibletoconvertavaluefromonetemporaltypetoanothertemporaltypefollowingcertainrules.Wewilldiscusstheseruleslaterinthechapter.

Ifthedateortimevalueisusedinanumericcontext,MySQLwillautomaticallyconvertthevaluetoanumber.

Wehaveoneinterestingusecase.Wewanttodevelopanauditlogfeaturewherewestoreeveryuser-enteredvalue.Supposethatinoneofthedatefields,theuserenteredan invaliddate,2017-02-31.Will thisbe stored in theaudit log table?Certainly not. How do we complete the feature, then? MySQL has theALLOW_INVALID_DATESmode.Ifenabled,itwillallowinvaliddatestobestored.Withthismodeenabled,MySQLverifies that themonth is in the rangeof1-12anddayisintherangeof1-31.

AsODBCcannothandlezerovaluesfordateortime,suchvaluesusedthroughConnector/ODBCareconvertedtoNULL.

Followingtableshowszerovaluesfordifferentdatatypes:

DataType ZeroValueDATE 0000-00-00TIME 00:00:00

DATETIME 0000-00-0000:00:00TIMESTAMP 0000-00-0000:00:00

YEAR 0000

Reference:https://dev.mysql.com/doc/refman/8.0/en/date-and-time-types.html

Thepreceding table shows zero values fordifferent temporaldata types.Thesearespecialvalues,astheseareallowedbyMySQLandareveryusefulincertaincases.Wecanalsospecifyzerovaluesusing'0'or0.MySQLhasaninterestingmodeconfiguration:NO_ZERO_DATE.Ifthisconfigurationisenabled,MySQLshowsawarningifthetemporaltypehasavaluewiththedateaszero.

DATE, DATETIME, andTIMESTAMPtypesThis section describes the most commonly used MySQL date and time datatypes: DATE, DATETIME, and TIMESTAMP. This section explains the similarities anddifferencesbetweenthesedatatypes.

TheDATEdatatypeissuitablewhenthevalueswewishtostorehaveadatepart,but the timepart ismissing.ThestandardMySQLdate format isYYYY-MM-DD.Thedatevaluesare retrievedanddisplayed in the standard formatunlessDATEfunctionsareapplied.TheMySQLsupportedrangeofvaluesis1000-01-01to 9999-12-31. Supported, here, means the values may work, but there is noguarantee.ThesameisthecasefortheDATETIMEdatatype.

TheDATETIMEdatatypeissuitableforvaluescontainingdateandtimeparts.ThestandardMySQLDATETIMEformatisYYYY-MM-DDHH:MM:SS.Thesupportedrangeofvaluesis1000-01-0100:00:00to9999-12-3123:59:59.

Similar toDATETIME, the TIMESTAMP data type is also suitable for values containingdateandtimeparts.However,therangeofvaluessupportedbytheTIMESTAMPdatatypeis1970-01-0100:00:01UTCto2038-01-1903:14:07UTC.

Thoughtheylooksimilar,theDATETIMEandTIMESTAMPdatatypesdiffersignificantly:

TheTIMESTAMPdata typerequires4bytes tostoredateand timevalues.TheDATETIMEdatatyperequires5bytestostoredateandtimevalues.TIMESTAMPcanstorevaluestill2038-01-1903:14:07UTC.Ifwewishtostorevaluesbeyond2038,theDATETIMEdatatypeshouldbeused.TIMESTAMPconsidersUTCasthetimezonewhilestoringvalues.DATETIMEstoresvalueswithouttimezoneconsideration.

Let'suseanexampletounderstandthedifferencebetweenDATETIMEandTIMESTAMPwithinthecontextoftime_zone.

Supposetheinitialtime_zonevalueissetto+00:00:

SETtime_zone='+00:00';

Let'screateatablecalleddatetime_temp.Thetablehastwocolumns;oneisDATETIMEandanotherisofthetypeTIMESTAMP.Wewillstorethesamedateandtimevaluesinbothcolumns.WiththehelpoftheSELECTquery,wewilltrytounderstandhowtherepresentationsdifferinoutput:

mysql>CREATETABLEdatetime_temp(

tsTIMESTAMP,

dtDATETIME);

mysql>INSERTINTOdatetime_temp

VALUES(NOW(),NOW());

mysql>SELECTts,dtFROMdatetime_temp;

+------------------------+-------------------------+

>|ts|dt|

+------------------------+-------------------------+

|2017-10-1418:10:25|2017-10-1418:10:25|

+------------------------+-------------------------+

Intheprecedingexample,NOW()istheMySQLfunctionwhichreturnsthecurrentdateandtimevalues.Lookingattheoutput,itseemsthatboththeTIMESTAMPandDATETIMErepresentationsaresame.Itisbecausethetime_zonevalueissettoUTC.Bydefault,TIMESTAMPshowsthedatetimevalueconsideringtheUTCtime_zone.Ontheotherpart,DATETIMEshowsdatetimewithoutatime_zone.

Let'schangethetime_zoneandobservetheoutput:

mysql>SETtime_zone='+03:00';

mysql>SELECTts,dtFROMdatetime_temp;

+------------------------+-------------------------+

|ts|dt|

+------------------------+-------------------------+

|2017-10-1421:10:25|2017-10-1418:10:25|

+------------------------+-------------------------+

Lookingattheoutput,itisclearthattheTIMESTAMPconsidersthetime_zonevaluesetinMySQL.So,theTIMESTAMPvaluegotadjustedwhenwechangedthetimezone.DATETIMEisn'timpacted,sotheoutputisnotchanged,evenafterchangingthetimezone.

IfTIMESTAMPisusedtostoredateandtimevalues,wemustconsideritseriously when migrating data to a different server located in adifferenttimezone.

If higher precision for the time value is required, DATETIME and TIMESTAMP can

include trailing fractional seconds as small asmicroseconds (sixdigits).So, ifwe insert adate timevaluewithamicrosecondsvalue, itwillbe stored in thedatabase. The format, including the fractional part, is YYYY-MM-DDHH:MM:SS[.fraction], and the range is from 1000-01-01 00:00:00.000000 to9999-12-3123:59:59.999999.The rangeforTIMESTAMP, including the fraction, is1970-01-0100:00:01.000000to2038-01-1903:14:07.999999.

ThefractionalpartisseparatedfromthetimevaluebyadecimalpointbecauseMySQLdoesn'trecognizeanyotherdelimiterforfractionalseconds.

DateandtimevaluesstoredwiththeTIMESTAMPdata typeareconvertedfromtheserver'stimezonetoUTCforstorageandfromUTCtotheserver'stimezoneforretrieval.IfwestoredaTIMESTAMPvalueandthenchangedtheserver's timezoneandretrievedthevalue,theretrievedvaluewouldbedifferentfromtheonewestored.

ThefollowingisthelistofpropertiesofdatevalueinterpretationinMySQL:

MySQL supports a relaxed format for values specified as string. In arelaxed format, any punctuation character can be used as the delimiterbetweendatepartsortimeparts.Thisisalittlebitconfusing.Forexample,avalue10:11:12mightlooklikeatimevaluebecauseoftheuseof:,butisinterpretedasa2010-11-12date.The only recognized delimiter between the rest of the time part and thefractionalsecondspartisthedecimalpoint.It is expected that month and day values are valid. With strict modedisabled,invaliddatesareconvertedtorespectivezerovaluesandawarningmessageisshown.TIMESTAMPvaluesthatincludezerointhedayormonthcolumnarenotavaliddate.Theexceptiontothisruleisthezerovalue.

IfMySQL is runwith MAXDBmode enabled, TIMESTAMP is identical toDATETIME. If this mode is enabled at the time of table creation,TIMESTAMPvaluesareconvertedtoDATETIME.

<strong>mysql>SET@dt=NOW();</strong>

<strong>mysql>SELECT@dt;</strong><br/><strong>+---------------------+</strong><br/><strong>|@dt|</strong><br/><strong>+---------------------+</strong><br/><strong>|2017-10-1513:43:17|</strong><br/><strong>+---------------------+</strong>

<strong>mysql>SELECTDATE(@dt);</strong><br/><strong>+------------------+</strong><br/><strong>|DATE(@dt)|</strong><br/><strong>+------------------+</strong><br/><strong>|2017-10-15|</strong><br/><strong>+------------------+</strong>

<strong>mysql>SELECTTIME(@dt);</strong><br/><strong>+------------------+</strong><br/><strong>|TIME(@dt)|</strong><br/><strong>+------------------+</strong><br/><strong>|13:43:17|</strong><br/><strong>+------------------+</strong>

<strong>mysql>SELECT</strong><br/><strong>HOUR(@dt),</strong><br/><strong>MINUTE(@dt),</strong><br/><strong>SECOND(@dt),</strong><br/><strong>DAY(@dt),</strong><br/><strong>WEEK(@dt),</strong><br/><strong>MONTH(@dt),</strong><br/><strong>QUARTER(@dt),</strong><br/><strong>YEAR(@dt);</strong><br/><strong>+-----------+-------------+-------------+---------+----------+</strong><br/><strong>|HOUR(@dt)|MINUTE(@dt)|SECOND(@dt)|DAY(@dt)|WEEK(@dt)|</strong><br/><strong>+-----------+-------------+-------------+---------+----------+</strong><br/><strong>+------------+--------------+-----------+</strong><br/><strong>|MONTH(@dt)|QUARTER(@dt)|YEAR(@dt)|</strong><br/><strong>+------------+--------------+-----------+</strong><br/><strong>+-----------+-------------+-------------+---------+----------+</strong><br/><strong>|13|43|17|15|41|</strong><br/><strong>+-----------+-------------+-------------+---------+----------+</strong><br/><strong>+------------+--------------+----------

-+</strong><br/><strong>|10|4|2017|</strong><br/><strong>+------------+--------------+-----------+</strong>

TIMEtypeMySQL DATETIME or TIMESTAMP data types are used to represent specific times atparticular dates. How about storing only the time of the day or the timedifferencebetweentwoevents?MySQL'sTIMEdatatypeservesthepurpose.

The standardMySQL format for storing or displaying TIME data type values isHH:MM:SS. The time value represents the time of the day, which is less than 24hours, but the TIME data type, asmentioned earlier, can also be used to storedelapsed time or time difference between two events. So, the TIME column canstorevaluesgreaterthan24hours.

TheMySQLTIMEcolumnisdefinedasfollows:

column_nameTIME;

TherangeofvaluesthatcanbestoredintheTIMEdatatypecolumnis-838:59:59to838:59:59.

The MySQL TIME column can also store the fractional seconds part up tomicroseconds (six digits), similar to the DATETIME column. Considering thefractionalsecondprecision, therangeofvaluesvaries from-838:59:59.000000to838:59:59.00000.

TheMySQLTIMEcolumncanalsohaveanoptionalvalue:

column_nameTIME(N);

whereNrepresentsnumberoffractionalpart,whichisupto6digits.

The TIME value usually takes 3 bytes for storage. In the case of the TIME valueincluding fractional secondprecision, itwill requireadditionalbytes,basedonthenumberoffractionalsecondprecision.

The following table shows the number of additional bytes required to storefractional-secondprecision:

FractionalSecondPrecision Storage(bytes)

0 0

1,2 1

3,4 2

5,6 3

MySQLsupportsabbreviatedvaluesfortheTIMEcolumn.TherearetwodistinctwaysforMySQLtointerpretabbreviatedvalues:

Iftheabbreviatedvaluehasacolon(:),MySQLinterpretsitastimeoftheday.Forexample,11:12isinterpretedas11:12:00andnotas00:11:12.If theabbreviatedvaluedoesn'thaveacolon(:),MySQLassumes that thetworightmostdigitsrepresentseconds.Thismeansthevalueis interpretedaselapsedtime,ratherthantimeoftheday.Forexample, '1214'and1214areinterpretedbyMySQLas00:12:14.

The decimal point is the only delimiter accepted by MySQL to separatefractionalsecondprecisionfromtherestofthetimevalueparts.

MySQL, by default, clips the values that lie outside of the permitted range ofvalues to the closest endpoint of the range. For example, -880:00:00 and880:00:00 are stored as -838:59:59 and 838:59:59. Invalid TIME values areconverted to00:00:00.As00:00:00 itself is avalid TIME value, it is difficult toknowifthevalue00:00:00wasstoredintentionally,orconvertedfromaninvalidTIMEvalue.

MySQLacceptsstringandnumericvaluesastheTIMEvalues.

<strong>mysql>SELECT</strong><br/><strong>CURRENT_TIME()AS'CUR_TIME',</strong><br/><strong>ADDTIME(CURRENT_TIME(),020000)AS'ADDTIME',</strong><br/><strong>SUBTIME(CURRENT_TIME(),020000)AS'SUBTIME';</strong><br/><br/><strong>+----------+-----------+-----------+</strong><br/><strong>|CUR_TIME|ADDTIME|SUBTIME|</strong><br/><strong>+----------+-----------+-----------+</strong><br/><strong>|10:12:34|12:12:34|08:12:34|</strong><br/><strong>+----------+-----------+-----------+</strong>

TheUTC_TIME()functioncanbeusedtofetchtheUTCtime.

<strong>manufacturing_yearYEAR</strong><br/><strong>or</strong><br/><strong>manufacturing_yearYEAR(4)</strong>

OnenotablethingisthatearlierMySQLversionssupportedtheYEAR(2)typecolumndeclaration.ThesupportforYEAR(2)hasbeendiscontinuedfromMySQL8.ItispossiblethatwemightwanttoupgradetheolderMySQLdatabasetotheMySQL8database.Inalatersection,wewillexplainthemigrationdetailsfromYEAR(2)toYEAR(4).

MySQLrepresentsYEARvaluesinaYYYYformat.Therangeofvaluesisfrom1901to2155and0000.

ThefollowingisthelistofformatssupportedforinputtingYEARvalues:

Fourdigitnumberfrom1901to2155.Fourdigitstringfrom1901to2155.Oneortwodigitnumberwiththerangeof0to99.YEARvaluesfrom1to69areconvertedto2001to2069andfrom70to99areconvertedto1970to1999.Oneortwodigitstringwiththerangeof0to99.YEARvaluesfrom1to69areconvertedto2001to2069andfrom70to99areconvertedto1970to1999.Insertinganumeric0hasadisplayvalueof0000andaninternalvalueof0000.Ifwewanttoinsert0andwantittobeinterpretedas2000,weshouldspecifyitasastring0or00.TheresultofafunctionthatreturnsanacceptablevalueYEARcontext,forexample,NOW().

MySQLconvertsinvalidYEARvaluesto0000.

<strong>mysql>CREATETABLEtemp(yearYEAR(2));</strong><br/><strong>ERROR1818(HY000):SupportsonlyYEARorYEAR(4)column.</strong>

TheALTERTABLEquery,whichrebuildsthetable,willautomaticallyconvertYEAR(2)toYEAR(4).TheYEAR(2)column,afterupgradingthedatabasetotheMySQL8database,remainsasYEAR(2),butthequeriesgiveerrors.

TherearemultiplewaystomigratefromYEAR(2)toYEAR(4):

UsingtheALTERTABLEquerywithFORCEattributeconvertstheYEAR(2)columntoYEAR(4).Itdoesn'tconvertthevalues,though.IftheALTERTABLEqueryisappliedtoareplicationmaster,thereplicationslaveswillreplicatetheALTERTABLEstatement.So,thechangewillbeavailableonallthereplicationnodes.Usingbinaryupgrade,withoutdumpingorreloadingdata,isanotherwayofupgradingYEAR(2)toYEAR(4).Runningmysql_upgradesubsequentlyexecutesREPAIR_TABLEandconvertsYEAR(2)toYEAR(4)withoutchangingvalues.Similartothepreviousalternative,thiswillbereplicatedinreplicationslavesifitisappliedtoareplicationmaster.

Animportantthingtonoteisthatwhileupgrading,wemustnotdumptheYEAR(2)datawithmysqldumpandreloadthedumpfileafterupgrading.ThismethodhasthepotentialtochangetheYEAR(2)valuessignificantly.

BeforeYEAR(2)toYEAR(4)migration,applicationcodemustbereviewedfor:

CodethatselectstheYEARvalueintwodigits.Codethatdoesn'thandlenumeric0insertions.Inserting0into

YEAR(2)resultsin2000,whereasinserting0intoYEAR(4)resultsinto0000.

StringdatatypesWhichis themostwidelyrequiredanduseddata typeforrepresentingvalues?Stringorcharacterdatatypes;it'seasy,right?MySQLsupportsawiderangeofstringdata types to fulfill different storage requirements. Stringdata types arecategorized into two categories: fixed length and variable length. CHAR, VARCHAR,BINARY, VARBINARY, BLOB, TEXT, ENUM, and SET are the MySQL-supported string datatypes. The storage requirement for each data type is different and will beexplainedlaterinaseparatesection.

CHARandVARCHARdatatypesTheCHARdatatypeisafixed-lengthstringdatatypeinMySQL.TheCHARdatatypeis often declaredwith amaximumnumber of characters that can be stored asfollows:

dataCHAR(20);

In the preceding example, the data column can store string values that arecapableofstoringmaximumcharacters.

CHARandVARCHARaresimilarinmanyways,withcertaindifferences.TheCHARdatatype ispreferred if the stringvalues tobe storedareof fixed size. ItwillgivebetterperformancecomparedtoifVARCHARisusedforfixedsizestrings.

Thelengthsvaryfrom0to255.ThevalueintheCHARcolumncannotexceedthemaximumlengthdeclaredatthetimeoftablecreation.Ifthelengthofthestringislessthanthemaximumallowedlength,MySQLaddspaddingontherighttothe length specified.At the timeof retrieval, trailing spaces are removed.Thefollowingisanexample:mysql>CREATETABLEchar_temp(dataCHAR(3));

mysql>INSERTINTOchar_temp(data)VALUES('abc'),('a');

mysql>SELECTdata,LENGTH(data)FROMchar_temp;+-------+--------------+|data|LENGTH(data)|+-------+--------------+|abc|3|+-------+--------------+|a|2|+-------+--------------+

Aswecanobserveintheprecedingexample,thesecondrecordwasinsertedas'

a',butintheoutput,thetrailingspaceisremoved.So,thelengthisdisplayedtobe2insteadof3.

MostMySQLcollationshaveapadattribute.Itdetermineshowtrailingspacesare treated for comparison of non-binary strings. There are two types ofcollations:PADSPACEandNOPAD. IncaseofPADSPACEcollation, trailingspacesarenot considered in comparison. Strings are comparedwithout regard to trailingspaces.

In the case of NO PAD collation, the trailing spaces are treated as any othercharacter.Thefollowingisanexample:mysql>CREATETABLEemployees(emp_nameCHAR(10));

mysql>INSERTINTOemployeesVALUES('Jack');

mysql> SELECT emp_name = 'Jack', emp_name = 'Jack ' FROMemployees;+-------------------+--------------------+|emp_name='Jack'|emp_name='Jack'|+-------------------+--------------------+|1|1|+-------------------+--------------------+mysql>SELECTemp_nameLIKE'Jack',emp_nameLIKE'Jack'FROMemployees;+----------------------+------------------------+|emp_nameLIKE'Jack'|emp_nameLIKE'Jack'|+----------------------+------------------------+|1|0|+----------------------+------------------------+

LIKE is a MySQL operator used for comparison in the WHERE clause. It isspecificallyusedforpatternsearchinginastring.TrailingspacesaresignificantwhencomparingstringvalueswiththeLIKEoperator.

IfPAD_CHAR_TO_FULL_LENGTHmodeisenabled,atthetimeofretrieval,thetrailingspaceswillnotberemoved.

The MySQL VARCHAR data type is a variable length string data type with amaximumlengthofupto65,535characters.VARCHARvaluesarestoredbyMySQLasaoneortwobytelengthprefix,alongwithactualdata.Theactualmaximumlengthof a VARCHAR is subject to themaximum row size,which is 65,536 bytessharedamongallcolumns.

IftheVARCHARvaluerequireslessthan255bytes,onebyteisusedfordetermininglengthprefix.Ifthevaluerequiresmorethan255bytes,twobytesareusedfordetermininglengthprefix.

IfMySQLstrictmodeisenabledandavaluetobeinsertedintheCHARorVARCHARcolumnvalueexceedsthemaximumlength,anerrorwillbegenerated.Ifstrictmode is disabled, the valuewill be truncated to themaximum allowed lengthwithawarning.

UnlikeintheCHARdatatype,valuestobestoredinVARCHARarenotpadded.Also,trailingspacesarenotremovedwhenthevaluesareretrieved.

BINARY and VARBINARY datatypesAnothersetofMySQLstringdatatypesisBINARYandVARBINARY.ThesearesimilartoCHAR and VARCHAR data types.An important difference between CHAR/VARCHAR andBINARY/VARBINARY is that BINARY/VARBINARY data types contain binary strings thancharacter strings. BINARY/VARBINARY uses binary character sets and collation.BINARY/VARBINARY are different from CHAR BINARY and VARCHAR BINARY data types. Thebasicdifferenceliesinthecharactersetandcollationreferredto.

Themaximumlength forpermittedvalues is similar to thatofCHARandVARCHAR.The only difference is that the length of BINARY and VARBINARY is in bytes, ratherthancharacters.

HowwouldMySQLcomparebinaryvalues?Theansweristhatthecomparisonhappensbasedonthenumericvaluesforthebytesinthevalues.

Similar to CHAR/VARCHAR data types, the values are truncated if the length of thevalue exceeds the column length, and awarning is generated. This is if strictmodeisnotenabled.Ifstrictmodeisenabled,anerrorisgenerated.

BINARY values are right-padded with the pad value 0x00 (zero bytes) to thespecifiedcolumnlength.Thepadvalueisaddedoninsert,butnotrailingbytesareremovedonretrieval.WhilecomparingBINARYvalues,allbytesareconsideredsignificant.ThisappliestoORDERBYandDISTINCToperators,aswell.Zerobytesandspaces are different when compared with 0x00 < space. The following is anexampleofinsertingabinaryvalue:

mysql>CREATETABLEtemp(

dataBINARY(3));

mysql>INSERTINTOtemp(data)VALUES('a');

In this case, 'a' becomes 'a \0' on insertion. 'a\0' is converted to 'a\0\0'. Onretrieval,valuesremainunchanged.

VARBINARYisavariablelengthstringdatatype.UnlikeBINARY,forVARBINARY,padding

isnotaddedoninsertionandbytesarenotstrippedonretrieval.SimilartoBINARY,allbytesaresignificantincomparisonforVARBINARY.

If the table has a unique index on columns, insertion of values in the columndifferingonlyinnumberoftrailingpadbyteswillgiveaduplicate-keyerror.Forexample,ifsuchacolumncontains'a'andwetrytoinsert'a\0',itwillcauseaduplicate-keyerror.

ThefollowingexampleexplainsthepaddingofBINARYvaluesincomparison:

mysql>CREATETABLEbin_temp(dataBINARY(3));

mysql>INSERTINTObin_temp(data)VALUES('c');

mysql>SELECTdata='c',data='c\0\0'frombin_temp;

+------------+-------------------+

|data='c'|data='c\0\0'|

+------------+-------------------+

|0|1|

+------------+-------------------+

In the case that it is required to retrieve the same value as specified withoutpadding,VARBINARYispreferable.

Ifthevalueretrievedmustbethesameasthevaluespecifiedforstoragewithnopadding, it might be preferable to use VARBINARY or one of the BLOB data typesinstead.

BLOBandTEXTdatatypesInwhatsituationcouldweberequiredtostoredatainaBinaryLargeObject(BLOB)column?Anyidea?Storingafileorimage,yousaid?Itispartiallytrue.Beforewe make a decision to store the images or files in a database or filesystem,weneedtoassessthesituation.Ifthefilesarestoredinafilesystemandmigratedovertoanotheroperatingsystem,itispossiblethatfilepointerscouldgetcorrupted.Itwillrequireadditionaleffortstofixthefilepointers.Insuchacase, storing files in a database is preferable. However, it might impactperformanceifwestorealargecloggedfileorimagedatainthedatabase.

BLOBisMySQL'ssolutiontostoringlargebinaryinformationofvariablelengths.MySQL has four BLOB types: TINYBLOB, BLOB, MEDIUMBLOB, and LONGBLOB. The onlydifferenceamongthesedatatypesisthemaximumlengthofvalueswecanstore.Thestoragerequirementsfor thesedata typesareexplainedin latersectionsofthechapter.

Similar to BLOB, TEXT data types are TINYTEXT, TEXT, MEDIUMTEXT, and LONGTEXT. Thesehave maximum lengths and storage requirements similar to that of BLOB datatypes.

Like BINARY data types, BLOB values are stored as byte strings and have binarycharacter setsandcollation.Comparisonsandsortingaredoneon thenumericvaluesofthecolumnvalues.TEXTvaluesarestoredasnon-binarystrings.

InthecaseofBLOBorTEXTdatatypes,ifthevaluecontainsexcesstrailingspaces,MySQL truncates with a warning, regardless of the MySQL mode. MySQLdoesn't pad BLOB or TEXT column values on insertion and doesn't strip bytes onretrieval.

ForaTEXTcolumnwhichisindexed,theindexcomparisonsaddtrailingspacesaspadding at the end of the values. So, a duplicate-key error may occur oninsertioniftheonlydifferencebetweenanexistingTEXTvalueandtheTEXTvaluetobeinsertedisinthetrailingspaces.BLOBcanberegardedasVARBINARYandTEXTcanberegardedasVARCHAR,withnorestrictiononthelengthofthevalues.

ThefollowingarethedifferencesbetweenVARBINARY,VARCHARandBLOB,TEXT:

When creating indexes on BLOB or TEXT columns, we must specify indexprefixlengthBLOBandTEXTcannothavedefaultvalues

BLOBorTEXTvaluesarerepresentedinternallyasobjectswithseparateallocations,unlikeotherdatatypes,forwhichthestorageisallocatedoncepercolumn.

ENUMdatatype

MySQL provides a data type for which lists of permitted values can bepredefinedwhenthetableiscreated.ThedatatypeisENUM.Ifwewanttorestricttheuser from insertingvaluesoutside a rangeof values,we shoulddefine thecolumn of data type ENUM. MySQL encodes the user input string values intonumbersforENUMdatatypes.

ENUMprovidesthefollowingmentionedbenefits:

Compactdatastorage

Readablequeriesandoutput

The following is an example that showcases when ENUM is useful: mysql>CREATETABLEsubjects(nameVARCHAR(40),streamENUM('arts','commerce','science'));

mysql> INSERT INTO subjects (name, stream) VALUES

('biology','science'),('statistics','commerce'),('history','arts');

mysql>SELECTname,streamFROMsubjectsWHEREstream='commerce';

+------------+----------+

|name|stream|

+------------+----------+

|statistics|commerce|

+------------+----------+

mysql>UPDATEsubjectsSETstream='science'WHEREstream='commerce';

ENUMvaluesrequireonebyteofstorage.Storingonemillionsuchrecordsinthistablewouldrequireonemillionbytesofstorage,opposedtothesixmillionbytesrequiredbytheVARCHARcolumn.

Thefollowingareimportantlimitationstoconsider:

ENUMvaluesarestoredinternallyasnumbers.So,iftheENUMvalueslooklikenumbers,literalvaluesmaymixupwiththeirinternalindexnumbers.

UsingENUM columns in ORDERBY clauses requires extra care. ENUM values areassignedindexnumbersbasedontheorderoflisting.ENUMvaluesaresortedbasedontheirindexnumbers.So,itisimportanttomakesurethattheENUMvalues list is in alphabetical order. Also, the column should be sortedlexicallythanbyindexnumbers.

TheENUMvaluemustbeaquotedstringliteral.

Each ENUM value has an index beginning with 1. The index of the emptystringorerrorvalue is0.Wecan find invalidENUMvaluesbyquerying thetablewithenum_column_value=0intheWHEREclause.TheindexofNULLvalueisNULL.IndexreferstothepositionofavaluewithintheENUMlistofvalues.

MySQL automatically removes trailing spaces from ENUM member valueswhen a table is created. Upon retrieval, values from an ENUM column aredisplayed in the case used in the columndefinition. If a number is to bestoredintheENUMcolumn,thenumberistreatedasanindexintothepossiblevalues.Thevaluestoredis theENUMvaluewith that index. In thecaseofaquoted numeric value, it is still interpreted as an index if there is no

matchingstringinthelistofenumeratedvalues.

If an ENUM column is declared to contain NULL values, the NULL value isconsideredavalidvalueforthecolumnandNULLbecomesthedefaultvalue.IfNULLisnotallowed,thefirstENUMvaluebecomesthedefaultvalue.

IfENUMvaluesareusedinanumericcontext,theindexisused.Thefollowingisan example query to use ENUM values in a numeric context:mysql> SELECTstream+1FROMsubjects;+--------------+|stream+1|+--------------+|4||3||2|+--------------+

SETdatatypeMySQLSETisadatatypewhichcanhavezeroormorevalues.Apermittedlistofvaluesisspecifiedatthetimeoftablecreation.Eachvaluemustbefromwithinthelistofpermittedvalues.Multiplesetmembersarespecifiedbyacomma(,)separated listofvalues.ASET canhaveamaximumof64distinctmembers. Ifstrictmodeisenabled,anerrorisgeneratedifduplicatevaluesarefoundinthecolumndefinition.

ItmustbetakencarethatSETmembervaluesdonotcontaincommas;otherwise,theyareinterpretedasSETmemberseparators.

A column specified as SET('yes', 'no') NOT NULL can have any of the followingvalues:

''

'yes'

'no'

'yes,no'

Trailing spaces are removed automatically from SET member values. Uponretrieval,SETcolumnvaluesaredisplayedusingthelettercasewhichwasusedinthecolumndefinition.

The following is an example of inserting values in the SET data type:mysql>CREATETABLEtemp(hobbiesSET('Travel','Sports','FineDining','Dancing'));

mysql>INSERTINTOtemp(hobbies)VALUES(9);

TheSETvaluesarestoredintheMySQLtableasabitmapinwhicheachelementis represented by one bit. In the preceding case, each element in the SET isassigned a bit. If the row has a given element, the associated bitwill be one.Becauseof thisapproach,eachelementhasanassociateddecimalvalue.Also,becauseof the bitmap, though there are only four values, SET will occupy onebyte.Thefollowingisthetableexplainingthis:

Element SETvalue Decimalvalue

Travel 00000001 1

Sports 00000010 2

FineDining 00000100 4

Dancing 00001000 8

MultipleSETelementscanberepresentedbyaddingtheirdecimalvalues.Intheprecedingcase,thedecimalvalue9isinterpretedasTravel,Dancing.

TheSETdatatypeisnotsocommonlyused.Thisisbecausealthoughitisastringdatatype,itisabitcomplexinimplementation.Thevaluesthatcanbestoredarelimitedto64elements.WecannotaddcommasaspartofSETvalues,becauseacommaisastandardSETvalueseparator.Fromadatabasedesignpointofview,usingSETmeansthedatabaseisnotnormalized.

JSONdatatypeJSONstandsforJavaScriptObjectNotation.Supposethatwewanttostoreuserpreferencesfor aweb application in the database.Usually,wemay choose tocreateaseparatetablewithid,user_id,key,valuefields.Thismayworkwellforasmall number of users, but in the case of thousands of users, the cost ofmaintenance is unaffordable compared to the value it adds to the webapplication.

InMySQL, we can utilize the JSON data type for this requirement.MySQLsupports thenative JSONdata type,which enables efficient storage for JSONdocuments.MySQLsupportsautomaticvalidationofJSONdocumentsstoredinthe JSONcolumn.Trying to store invalid JSONdocumentsproduces an error.JSONdocuments stored inJSONcolumnsareconverted toan internal format.Theformatisbinary,andstructuredtoenabletheservertolookupsubojbectsornestedvaluesdirectly,bykeyorarrayindex,withoutreadingothervalues.

A JSON column cannot have a default value. The JSON data type requiressimilar storage to that of LONGTEXT or LONGBLOB. JSON columns are not indexeddirectly,unlikeotherstringdatatypes.

ThefollowingisanexampleofinsertingJSONvaluesinatable:

mysql>CREATETABLEusers(

user_idINTUNSIGNEDNOTNULL,

preferencesJSONNOTNULL);

mysql>INSERTINTOusers(user_id,preferences)

VALUES(1,'{"page_size":10,"hobbies":{"sports":1}}');

mysql>SELECTpreferencesFROMusers;

+---------------------------------------------------------+

|preferences|

+---------------------------------------------------------+

|{"hobbies":{"sports":1},"page_size":10}|

+---------------------------------------------------------+

Intheprecedingexample,wehaveformattedtheJSONvalue.Asanalternative,wecan also use the built-in JSON_OBJECT function.The function accepts a list ofkey/valuepairsandreturnsaJSONobject.Anexamplefollows:

mysql>INSERTINTOusers(user_id,preferences)

VALUES(2,JSON_OBJECT("page_size",1,"network",JSON_ARRAY("GSM","CDMA","WIFI")));

TheprecedingINSERTquerywill insert theJSONvalue{"page_size":1,"network":["GSM","CDMA","WIFI"]}.WecanalsousenestedJSON_OBJECTfunctions.TheJSON_ARRAYfunctionreturnsaJSONarraywhenpassedasetofvalues.

Ifthesamekeyisspecifiedmultipletimes,onlythefirstkey/valuepairwillberetained. In thecaseof theJSONdata type, theobjectkeysaresortedand thetrailing space between the key/value pairs is removed. The keys in the JSONobjectmustbestrings.

InsertingaJSONvalueinaJSONcolumnsucceedsonlyiftheJSONdocumentis valid. In the case that the JSON document is invalid,MySQL produces anerror.

MySQLhas onemore important anduseful functionwhichoperates on JSONvalues. The JSON_MERGE function takes multiple JSON objects and produces asingle,aggregateobject.

TheJSON_TYPE function takesa JSONasanargumentand tries toparse it intoaJSONvalue.Itreturnsthevalue'sJSONtypeifitisvalidandproducesanerrorifotherwise.

PartialupdatesofJSONvalues

WhatshouldwedoifwewanttoupdateavalueinaJSONdocumentstoredinaJSON

datatype?Oneoftheapproachesistoremovetheolddocumentandinsertanewdocument, with updates. The approach doesn't seem good, right?MySQL 8.0supportspartial,inplaceupdateofaJSONdocumentstoredinaJSONdatatypecolumn. The optimizer requires that an update must meet the followingconditions:

ThecolumnmustbeofJSONtype.

Oneofthreefunctions,JSON_SET(),JSON_REPLACE()orJSON_REMOVE(),canbeusedto update the column. MySQL doesn't permit direct assignment of thecolumnvalueasapartialupdate.

The input column and target column must be the same. For example, astatement such as UPDATE temp SET col1 = JSON_SET(col2, 'one', 10) cannot beperformedasapartialupdate.

Thechangesonlyupdate existing arraysor objects, andnonewelementsareaddedtotheparentobjectorarray.

Thereplacementvaluemustnotbelargerthanthevaluebeingreplaced.

Storage requirements for datatypes

This section explains storage requirements for different data types inMySQL.The storage requirements depend on different factors. The storage enginesrepresentdatatypesandstorerawdatadifferently.

Atablehasamaximumrowsizeof65,535bytes,evenif thestorageengineiscapableofsupportinglargerrows.BLOBandTEXTdatatypesareexcluded.

Thefollowingtableexplainsthestoragedetailsfornumericdatatypes:

DataType

Storagerequired

TINYINT 1byteSMALLINT 2bytes

MEDIUMINT 3bytes

INT,INTEGER 4bytesBIGINT 8bytes

FLOAT(p)

4bytesif0<=p<=24,

8bytesif25<=p<=53

FLOAT 4bytes

DOUBLE[precision],REAL 8bytes

DECIMAL(M,D),NUMERIC(M,D) VariesBIT(M) Approximately(M+7)/8bytes

Reference:https://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html

The following table explains the storage requirements for DATE and TIME datatypes:

DataType

StorageRequired

YEAR 1byteDATE 3bytesTIME 3bytes+fractionalsecondsstorageDATETIME 5bytes+fractionalsecondsstorageTIMESTAMP 4bytes+fractionalsecondsstorage

The following table explains the storage required for fractional secondsprecision:

FractionalSecondsPrecision

StorageRequired

0 0bytes

1,2 1byte

3,4 2bytes

5,6 3bytes

Thefollowingtableexplainsstoragerequirementsforstringdatatypes:

DataType

StorageRequired

CHAR(M)

M×wbytes,0<=M<=255,wherewisthenumberofbytesrequiredforthemaximum-lengthcharacterinthecharacterset

BINARY(M) Mbytes,0<=M<=255

VARCHAR(M),VARBINARY(M)

L+1bytesifthecolumnvaluesrequire0−255bytes,L+2bytesifthevaluesmayrequire

morethan255bytes

TINYBLOB,TINYTEXT L+1bytes,whereL<28

BLOB,TEXT L+2bytes,whereL<216

MEDIUMBLOB,MEDIUMTEXT L+3bytes,whereL<224

LONGBLOB,LONGTEXT L+4bytes,whereL<232

ENUM('value1','value2',...) 1or2bytes,dependingonthenumberofenumerationvalues(65,535valuesmaximum)

SET('value1','value2',...)1,2,3,4,or8bytes,dependingonthenumberofsetmembers(64membersmaximum)

Reference:https://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html

Inthecaseofstringdatatypes,variablelengthstringsarestoredusingthelengthof the value and the length prefix. The length prefix varies from one to fourbytes,dependingonthedatatype.

Storage requirements for theJSONdata typearesimilar to thatofLONGBLOBandLONGTEXT.However,astheJSONdocumentsarestoredinbinaryrepresentations,itimposesanoverheadinstoringJSONdocuments.

Choosing the right data type forcolumn

Asageneralpractice,weshouldusethemostprecisetypeforstoringdata.Forexample, a CHAR data type should be used to store a string value that varies inlengthfrom1to255characters.AnotherexampleisthatMEDIUMINTUNSIGNEDshouldbeusedtostorenumbersrangingfrom1to99999.

Basic operations such as addition, subtraction, multiplication, and division withDECIMALdataareperformedwiththeprecisionof65decimaldigits.

Basedon the importanceofaccuracyorspeed,useofFLOATorDOUBLE shouldbechosen.FixedpointvaluesstoredinBIGINTcanbeusedforhigherprecision.

Thesearegeneralguidelines,butthedecisiontousetherightdatatypeshouldbemade based on the detailed characteristics explained separately for each datatypeintheearliersections.

Summary

It was an interesting chapter with important content to learn, right? In thischapter, we understood the significance of data types in MySQL. We sawdifferentcategories inwhichMySQLdata typesareclassified.Welearnedandunderstoodthecharacteristicsandspecificationsofeachdatatypeindepth.Wealso learnedMySQL datamanipulation functions and understood some of theMySQL settings and modes. In the later section of the chapter, we learnedstorage requirements of data types. Finally, we learned general guidelines forchoosingtherightdatatypes.

Moving on to next chapter,wewill learnMySQL databasemanagement. Thechapter will focus on server administration, understanding the basic buildingblocksoftheMySQL

server,suchasthedatadictionary,systemdatabase,andsoon.Thechapterwillexplain how we can run multiple server instances on a single machine andMySQLrolesandpermissions.

MySQL8DatabaseManagement

In the previous chapter,we learned aboutMySQL 8 data types, explaining indetailwhich data types are available and how they are categorized. There arevarious properties associated with each of these data types, and the storagecapacityvarieswitheachtype.Thepreviouschapteralsoprovidedyouwithanin-depthunderstandingofMySQL

8 data types. Now its time to get some practical knowledge on MySQL 8administrativefeatures.Isn'titinterestingtoknowmoreabouttheadministrativefeaturesofMySQL

8, how configuration will be done for it, and much more? It's extremelyimportant for an administrator to have detailed knowledge on howMySQL 8worksforglobalization,howlogsaremaintained,andhowtoenhancecapabilityoftheserver.Now,let'sstartwithsomefundamentalconcepts.

Wewillcoverthefollowtopicsinthischapter:

MySQL8serveradministration

Datadirectory

Thesystemdatabase

Runningmultipleinstancesonasinglemachine

Componentsandpluginmanagement

Rolesandpermissions

Cachingtechniques

Globalization

MySQL8serverlogs

MySQL8serveradministrationTherearemanyoperatingparametersthatavailablewithMySQL8,andamongthem all the required parameters are set by default during the installationprocess. After installation, you are allowed to change the option file byremoving or adding a comment sign (#) at the start of the line of a specificparametersetting.Theuserisalsoallowedtosetparametersatruntimebyusingcommandlineargumentsortheoptionfile.

<strong>mysqld--help</strong>

<strong>mysqld–verbose--help</strong>

ServerSystemvariable:TheMySQLservermanagesmanysystemvariables.MySQLprovidesthedefaultvalueforeachsystemvariable.Systemvariablescanbesetusingthecommandlineorcanbedefinedintheoptionfile.MySQL8hastheflexibilitytochangethesevariablesatruntimewithoutserverstartorstop.Formoredetailsreferto:https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html.Serverstatusvariable:TheMySQLserverusesmanystatusvariablestoprovideinformationaboutitsoperation.Formoredetailsreferto:https://dev.mysql.com/doc/refman/8.0/en/server-status-variables.html.

ServerSQLmodesMySQL 8 provides different modes that will affectMySQL support and datavalidation checks. This option makes it easier for the user to useMySQL indifferentenvironments.

TosetdifferentmodesMySQLprovidesthesql_modesystemvariablewhichcanbesetateitheraglobalorsessionlevel.Refertothefollowingpointsindetailtounderstandmodes:

<strong>SETGLOBALsql_mode='modes';</strong><br/><strong>SETSESSIONsql_mode='modes';</strong>

<strong>SELECT@@GLOBAL.sql_mode;</strong><br/><strong>SELECT@@SESSION.sql_mode;</strong>

TheavailableSQLmodes

ThissectiondescribesalltheavailableSQLmodes.Outofthem,thefirstthreeare

themostimportantSQLmodes:

ANSI:Thismodeisusedtochangesyntaxandbehavior,bymakingitclosertostandardSQL.

STRICT_TRANS_TABLES:As thenameimplies, thismodeisrelated to transactionandit ismainlyusedfortransactionalstorageengines.Whenthismodeisenablefornontransactionaltables,MySQL8willconvertinvalidvaluestotheclosestvalidvalueandinserttheadjustedvalueintothecolumn.Ifthevalueismissing,thenMySQL8willinsertanimplicitdefaultvaluerelatedto thecolumn'sdata type. In thiscase,MySQL8willgenerateawarningmessage instead of an error message, and continue with the statementexecutionwithoutbreaking it.In thecaseof transactional tables,however,MySQL8givesanerrorandwillbreaksexecution.

TRADITIONAL: This mode generally behaves like traditional SQL database

system.Itindicatesgiveerrorinsteadofawarningwhenanincorrectvalueinsertedintothecolumn.

ALLOW_INVALID_DATES: This mode checks only the month range and the daterangeofthedatevalue.Inotherwords,themonthrangemustbebetween1to12anddaterangemustbebetween1to31.ThismodeisapplicableforDATEandDATETIMEdatatypesandnotfortimestampdatatype.

ANSI_QUOTES:Used toconsider" asan identifierquotecharacter insteadofastringquotecharacter.When thismode isenabled,youcannotusedoublequotationtoquotestringliteral.

ERROR_FOR_DIVISION_BY_ZERO:Used tohandle thecaseofdivisionbyzero.ThismodeoutputalsodependsonstrictSQLmodestatus:

Ifthismodeisnotenabled,divisionbyzeroinsertsNULLandproducesnowarning.

If thismode is enabled, division by zero inserts NULL and produces awarning.

Ifthismodeandstrictmodeareenabled,divisionbyzeroproducesanerror,unlessIGNORE isgivenaswell.ForINSERTIGNOREandUPDATEIGNORE,divisionbyzeroinsertsNULLandproducesawarning.

HIGH_NOT_PRECEDENCE: Thismode is used to set a high precedence for the NOToperator. For example, when the mode is enabled the expression NOT a

BETWEENbANDcisparsedasNOT(aBETWEENbANDc)insteadof(NOTa)BETWEENbANDc.

IGNORE_SPACE:Thismodeappliestobuilt-infunctionsratherthanuserdefinedfunctionsorstoredprocedures.

NO_AUTO_CREATE_USER: This mode is used to prevent GRANT statements byautomaticallycreatingnewuseraccountsunlessauthenticationinformationisspecified.

NO_AUTO_VALUE_ON_ZERO:Thismodeisusedforautoincrementalcolumns.When0 is foundMySQLcreatesanewsequencenumber for the field, and thatwillcreateproblemswhenyouareloadingdump.Enablethismodebeforereloadingdumptoresolvethisproblem.

NO_BACKSLASH_ESCAPES:Ifthismodeisenabled,backslashbecomesanordinarycharacter.

NO_DIR_IN_CREATE:ThisoptionisusefulforslavereplicationserverswheretheINDEXDIRECTORYandDATADIRECTORYdirectivesareignoredontablecreation.

NO_ENGINE_SUBSTITUTION: Used to provide substitution of the default storageengine.When thismode is enabledand thedesiredengine isunavailable,MySQLgivesanerrorandatableisnotcreated.

NO_FIELD_OPTIONS:This indicates,don'tprintMySQLspecificcolumnoptionsintheoutputofSHOW_CREATE_TABLE.

NO_KEY_OPTIONS: This indicates, don't printMySQL specific index options intheoutputofSHOW_CREATE_TABLE.

NO_TABLE_OPTIONS:Thisindicates,don'tprintMySQLspecifictableoptionsintheoutputofSHOW_CREATE_TABLE.

NO_UNSIGNED_SUBTRACTION: When this mode is enabled, it makes sure thatsubtractionresultmustbeasignedvalueeventhoughanyoftheoperandisunsigned.

NO_ZERO_DATE:Theeffectofthismodedependsonthestrictmodeasdefinedbelow:

If it is not enabled, 0000-00-00 is allowed andMySQLproduces nowarningoninsertion

Ifthismodeisenabled,0000-00-00isallowedandMySQLrecordsawarning

If both this mode and strict mode are enabled, 0000-00-00 is notallowedandMySQL

producesanerroroninsertion

NO_ZERO_IN_DATE: This mode effect is also depending on the strict mode asdefinedbelow:

If it is not enabled, dates with zero parts are allowed and MySQLproducesnowarningoninsertion

Ifthismodeisenabled,dateswithzeropartsareallowedandproduceawarning

Ifthismodeandstrictmodeareenabled,dateswithzeropartsarenotallowedandMySQLproducesanerror

ONLY_FULL_GROUP_BY: If this mode is enabled, MySQL will reject queries inwhich select list, order by list, and the HAVING condition refer to nonaggregatedcolumns.

PAD_CHAR_TO_FULL_LENGTH:ThismodeisappliedonthecolumnwhosedatatypeissetasCHAR.Whenthismodeisenabled,MySQLretrievescolumnvaluesbypaddingtotheirfulllength.

PIPES_AS_CONCAT:Whenthismodeissetasenabled||willbeconsideredasastringconcatenationoperatorinsteadofOR.

REAL_AS_FLOAT:Bydefault,MySQL8willconsiderREALasasynonymofDOUBLE,butwhen this flag is enabledMySQLwill consider REAL as a synonymofFLOAT.

STRICT_ALL_TABLES:Inthismodeinvaliddatavaluesarerejected.

TIME_TRUNCATE_FRACTIONAL:ThismodeindicatesiftruncationisallowedonTIME,DATE, and TIMESTAMP columns or not. The default behavior is to performroundingonthevaluesinsteadoftruncation.

CombinationSQLmodes

MySQL8alsoprovidessomespecialmodesascombinationsofmodevalues:

ANSI: It includes the effects of the REAL_AS_FLOAT, PIPES_AS_CONCAT, ANSI_QUOTES,IGNORE_SPACE,andONLY_FULL_GROUP_BYmodes.

DB2: It includes the effects of the PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,andNO_FIELD_OPTIONSmodes.

MAXDB: It includes the effects of PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_FIELD_OPTIONS,andNO_AUTO_CREATE_USER.

MSSQL: It includes the effects of PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,andNO_FIELD_OPTIONS.

MYSQL323:ItincludestheeffectsoftheMYSQL323andHIGH_NOT_PRECEDENCEmodes.

MYSQL40:ItincludestheeffectsoftheMYSQL40andHIGH_NOT_PRECEDENCEmodes.

ORACLE: It includes the effects of the PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE,

NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_FIELD_OPTIONS,andNO_AUTO_CREATE_USERmodes.

POSTGRESQL:ItincludestheeffectofthePIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,andNO_FIELD_OPTIONSmodes.

TRADITIONAL: It includes theeffectsof the STRICT_TRANS_TABLES, STRICT_ALL_TABLES,NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER andNO_ENGINE_SUBSTITUTIONmodes.

<strong>ALTERTABLE</strong><br/><strong>CREATETABLE</strong><br/><strong>CREATETABLE...SELECT</strong><br/><strong>DELETE(bothsingletableandmultipletable)</strong><br/><strong>INSERT</strong><br/><strong>LOADDATA</strong><br/><strong>LOADXML</strong><br/><strong>SELECTSLEEP()</strong><br/><strong>UPDATE(bothsingletableandmultipletable)</strong>

Youcangoto:https://dev.mysql.com/doc/refman/8.0/en/sql-mode.htmlforadetailedlistoftheerrorsassociatedwithstrictSQLmodeinMySQL.

TheIGNOREkeyword

MySQLprovidesan IGNORE keywordwhich is optional for statement execution.TheIGNORE keyword is used todowngrade errors towarnings and applicable toseveralstatements.

Formultiplerowstatements,theIGNOREkeywordallowsyoutoskiptheparticularrow,insteadofaborting.ThefollowingstatementssupporttheIGNOREkeyword:

CREATE TABLE ... SELECT: Individual CREATE and SELECT statements do not havesupport on this keyword, but when we insert into the table using SELECTstatement, rows that duplicate an existing row on a unique key value arediscarded.

DELETE:IfthisstatementexecuteswiththeIGNOREoptionMySQLavoiderrorsoccurredduringexecution.

INSERT:Duplicate values in unique key and data conversion issueswill behandled by this keyword during row insertion. MySQL will insert theclosestpossiblevaluesintothecolumnandignoretheerror.

LOADDATAandLOADXML:AtthetimeofloadingdataifduplicationisfoundthestatementwilldiscarditandcontinueinsertionfortheremainingdataiftheIGNOREkeywordisdefined.

UPDATE: In cases of duplicate key conflict on unique key during statementexecution,MySQL

willupdatethecolumnwiththeclosestidentifiedvalues.

TheIGNOREkeywordalsoappliesonsomespecificerrors,listedhere:https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html.

IPv6support

MySQL8providessupportforIPv6,withthefollowingcapabilities:

MySQL server will accept TCP/IP connections from clients with IPv6connectivity

MySQL 8 account names permit IPv6 addresses, which enables DBA tospecifyprivileges

fortheclientsthatareconnectedwithserver,usingIPv6

The IPv6 functions enable conversions between string and internal IPv6addressformats,

andcheckingwhetherthevaluesrepresentavalidIPv6addressornot

<strong>mysql-urootmysql<fill_help_tables.sql</strong>

Atthetimeofinstallationprocesscontentinitializationoccurs.Incaseofupgradingitwillbeperformso;executetheabovecommandmanually.

Theservershutdownprocess

Theservershutdownprocessperformsthefollowingsteps:

1. Theshutdownprocessisinitiated:Thereareseveralwaystoinitializetheshutdown process. Execute the mysqladmin shutdown commandwhich can beexecutedonanyplatform.Therearesomesystemspecificwaystoinitializethe shutdownprocess; for example,Unix based systemswill start to shutdown when it receives a SIGTERM signal. In the same way, Windowbasedsystemswillstarttoshutdownwhentheservicemanagertellsthemto.

2. The server creates a shutdown thread if necessary: Based on theshutdowninitializationprocess,theserverwilldecidetocreatenewthreador not. If it is requested by the client, a new threadwill be created. If asignalisreceived,thentheservermightcreateathreadoritmighthandleitby itself. If the server tries to create a separate thread for the shutdownprocessandanerroroccurs,thenitproducesthefollowingmessageintheerrorlog:

Error:Can'tcreatethreadtokillserver

3. Theserverstopsacceptingnewconnections:Whentheshutdownactivityisinitiated,theserverwillstopacceptingnewconnectionrequests,usingahandlerofnetwork interfaces.TheserverwillbeusingWindows featuressuchasnamedpipe,TCP/IPport,theUnixsocketfile,andsharedmemoryonWindowsinordertolistentonewconnectionrequests.

4. The server terminates current activity: Once the shut down process isinitialized, the serverwill start tobreak theconnectionwith theclient. Inthenormal scenario, the connection threadswill diequickly,but theoneswhichareworkingorareinanongoingactivitystagewilltakealongtimetodie.Soifathreadisexecutingopentransactionsandifitgetsrollbackinthemiddleofexecutionthentheusermightgetonlypartiallyupdateddata.Ontheotherhand,ifthethreadisworkingonatransaction,thentheserverwillwaituntilthetransactioniscompleted.Also,theusercanterminatetheongoingtransactionbyexecutingtheKILLQUERYorKILLCONNECTIONstatements.

5. Theservershutsdownorclosesstorageengines:Inthisphase,theserverflushes the cache and closes all the open tables.Here, the storage engineperformsall theactionsnecessaryfor tables.InnoDB flushes itsbufferpool,writesthecurrentLSNintotablespaceandterminatesitsthread.TheMyISAMflushesthependingindex.

6. Theserverexits:Inthisphase,theserverwillprovideoneofthefollowingvaluestothemanagementprocesses:

0=successfultermination(norestartdone)

1=unsuccessfultermination(norestartdone)

2=unsuccessfultermination(restartdone)

Datadirectory

ThedatadirectoryistheplacewhereMySQL8storesalltheinformationthatismanagedbyitself.Eachsub-directoryofthedatadirectoryrepresentsadatabasedirectory and its related data.All theMySQL installations have the followingstandarddatabases:

The sys directory: This represents the sys schema, which contains theobjectsusefulforthePerformanceSchemainformationinterpretation.

The performance schema directory: This directory is used to observe theinternalexecutionoftheMySQLserveratrun-time.

The mysql directory: This directory is related to the MySQL systemdatabase,whichcontainsthedatadictionarytableandthesystemtables.Itcontains the information that is required by theMySQL server once it isrunning.

ThesystemdatabaseThe system databasemainly contains the data dictionary tables that stores theobject'smetadataandsystemtables forotheroperationalpurposes.Thesystemdatabasecontainsanumberofsystemtables.Wewilllearnmoreabouttheminthecomingsections.

DatadictionarytablesThe data dictionary tables contains themetadata about data objects. Tables ofthis directory are invisible and are not read by general SQL queries suchasSELECT,SHOWTABLES,INFORMATION_SCHEMA.TABLES,andsoon.MySQLmainlyexposesthemetadatausingtheINFORMATION_SCHEMAoption.

GrantsystemtablesThese tables are used to manage and provide grant information of users,database and relevant privileges. MySQL 8 uses grant tables as transactionaltables,notnontransactional(MyISAM,forexample)tables,soalltheoperationsonthe transaction are either completed or failed; no partial case will be madepossible.

Objectinformationsystemtables

These tables contains information related to the stored programs, componentsandserver-sideplugins.Thefollowingmaintablesareusedtostoreinformation:

Component:Worksasaregistryfortheserver.TheMySQL8serverloadsallthecomponentslistedbythistableonserverstartup.

Func: This table contains information related to all the user-definedfunctions (UDF). MySQL 8 will load all the UDFs listed in this tableduringserverstartup.

Plugin: Contains the information related to the server-side plugins. TheMySQL8serverloadsalltheavailablepluginsduringstartup.

LogsystemtablesThesetablesareusefulforloggingandusingcsvstorageengines.Forexample,thefunctionsgeneral_logandslow_log.

The server-side help systemtablesThese tables are useful to store help information. The following tables areavailableinthiscategory:

help_category:Providesinformationaboutthehelpcategorieshelp_keyword:Provideskeywordsassociatedwithhelptopicshelp_relation:Helpsinmappingsbetweenhelpkeywordsandtopicshelp_topic:Helptopiccontents

Timezonesystemtables

Thesetablesareusefultostoretimezoneinformation.Thefollowingtablesareavailableinthiscategory:

time_zone:ProvidesthetimezoneIDsandwhethertheyuseleapseconds

time_zone_leap_second:Willcomeinhandywhenleapsecondsoccur

time_zone_name:HelpsinmappingsbetweentimezoneIDsandnames

time_zone_transitionandtime_zone_transition_type:Timezonedescriptions

Replicationsystemtables

These tables are useful to support the replication feature. It helps to storereplication related information when it is configured to as mentioned infollowingtables.Thefollowingtablesareavailableinthiscategory:

gtid_executed:UsedforcreatingthetableforstoringGTIDvalues

ndb_binlog_index: Provides the binary log information for MySQL Clusterreplication

slave_master_info, slave_relay_log_info and slave_worker_info: Used to storereplicationinformationonslaveservers

OptimizersystemtablesThis tables are useful for optimizer. The following tables are available in thiscategory:

innodb_index_statsandinnodb_table_stats:UsedforgettingtheInnoDBpersistentoptimizerstatisticsserver_cost: Contains the optimizer cost estimates for general serveroperations.engine_cost: Contains the estimates for operations specific to particularstorageengines

Other miscellaneous systemtables

Tables that don't fall into the above-mentioned categories fall under thiscategory.

Thefollowingtablesareavailableinthiscategory:

servers:UsedbytheFEDERATEDstorageengine

innodb_dynamic_metadata: Used by the InnoDB storage engine to store fastchanging tablemetadata such as auto-increment counter values and indextreecorruptionflags

Youcanlearnmoreaboutthedifferentsystemtablesat:https://dev.mysql.com/doc/refman/8.0/en/system-database.html.

Running multiple instances on asinglemachine

There might be some situations where you are required to install multipleinstances on a single machine. It may be to check the performance of twodifferentversions,orperhapsthereisaneedtomanagetwoseparatedatabasesondifferentMySQLinstances.

The reason can be anything, but MySQL allows user to execute multipleinstances on the same machine by providing different configuration values.MySQL8allowsuserstoconfigureparametersbymakinguseofthecommandline,optionfile,orbysettingenvironmentvariables.TheprimaryresourceusedbyMySQL 8 for this is the data directory and itmust be unique for the twoinstances. We can define the value for the same using the --datadir=dir_namefunction.Apartfromthedatadirectory,wewillalsoconfigureuniquevaluesforthefollowingoptionsaswell:

--port=port_num

--socket={file_name|pipe_name}

--shared-memory-base-name=name

--pid-file=file_name

--general_log_file=file_name

--log-bin[=file_name]

--slow_query_log_file=file_name

--log-error[=file_name]

--tmpdir=dir_name

Setting up multiple datadirectories

As described above, each of theMySQL instancesmust have a separate datadirectory.

Theuserisallowedtodefineseparatedirectoriesusingthefollowingmethods:

Create a new data directory: In this method we must follow the sameprocedure which was defined in Chapter 2, Installing and UpgradingMySQL. For Microsoft Windows, when we install MySQL 8 from Ziparchives,copyitsdatadirectorytothelocationwhereyouwanttosetupthenewinstance.InthecaseofanMSIpackagealongwiththedatadirectory,create a pristine template data directory named data under the installationdirectory.Oncetheinstallationiscomplete,copythedatadirectorytosetupadditionalinstances.

Copyanexistingdatadirectory:Inthismethod,wewillcopyanexistinginstance's data directory to the new instance's data directory. To copy anexistingdirectory,performthefollowingsteps:

1. StoptheexistingMySQLinstance.Makesureit'scleanlyshutdownsothatnopendingchangesareavailableinthedisk.

2. Copythedatadirectorytothenewlocation.

3. Copythemy.cnformy.inioptionfileusedbytheexistinginstancetothenewlocation.

4. Modify the new option as per the new instance. Make sure all theuniqueconfigurationsaredoneproperly.

5. Startthenewinstancewiththenewoptionfile.

<strong>[mysqld]</strong><br/><strong>basedir=C:/mysql-5.5.5</strong><br/><strong>port=3307</strong><br/><strong>enable-named-pipe</strong><br/><strong>socket=mypipe1</strong>

<strong>[mysqld]</strong><br/><strong>basedir=C:/mysql-8.0.1</strong><br/><strong>port=3308</strong><br/><strong>enable-named-pipe</strong><br/><strong>socket=mypipe2</strong>

<strong>C:\>C:\mysql-5.5.5\bin\mysqld--installmysqld1--<br/>defaults-file=C:\my-opts1.cnf</strong><br/><strong>C:\>C:\mysql-8.0.1\bin\mysqld--installmysqld2--<br/>defaults-file=C:\my-opts2.cnf</strong>

<strong>C:\>C:\mysql-5.5.9\bin\mysqld--installmysqld1</strong><br/><strong>C:\>C:\mysql-8.0.4\bin\mysqld--installmysqld2</strong>

<strong>C:\>NETSTARTmysqld1</strong><br/><strong>C:\>NETSTARTmysqld2</strong>

<strong>SELECT*FROMmysql.component;</strong>

MySQL8serverpluginsMySQL 8 server has a plugin API that enables the creation of servercomponents.WithMySQL8, you have the flexibility of installing a plugin atruntimeoratstartup.

InthefollowingtopicswewilllearnaboutthelifecycleoftheMySQL8serverplugins.

Installingtheplugins

The loadingof thepluginsvarieswith their typesandcharacteristics.Togetaclearerpictureofthis,let'sgothoughthefollowing:

Built-in plugins: The server knows about the built-in plugins and loadsthemautomatically,onstartup.

Theuserisallowedtochangethestateofpluginsbyanyoftheiractivationstatuses,whichwillbediscussedinthefollowingsection.

Pluginsregistered in the mysql.plugin systemtable:On startupMySQL8serverwillloadallthepluginswhichareregisteredinthemysql.plugintable.Iftheserverisstartedwiththe--skip-grant-tablesoption,theserverwillnotloadthepluginslistedthere.

Plugins named with command-line options: MySQL 8 provides the --plugin-load, --plugin-load-add, and --early-plugin-load options for loadingplugins with the command line. The --plugin-load and --plugin-load-add

options load the plugins on server startup after the built-in plugins areinstalled.

But,we can use the --early-plugin-load option to load the plugins, prior toinitializationofbuilt-inpluginsandstorageengines.

Plugins installed with the INSTALL PLUGIN statement: This is a permanentpluginregistrationoption,whichwillregistertheplugininformationinthemysql.plugin table. It will also load all the plugins available in the pluginlibrary.

Activateplugin

To control the state (like the activation or deactivation) of plugins,MySQL 8providesthefollowingoptions:

--plugin_name=OFF:Disables the named plugin. Some of the built-in plugins,suchastheasmysql_native_passwordplugin,arenotaffectedbythiscommand.

--plugin_name[=ON]: This command enables the specified plugin. If plugininitialization failed during startup MySQL 8 will start with the plugindisabled.

--plugin_name=FORCE:Thisisthesameastheabovecommand,excepttheserverdoesnotstart.Thismeansthatitforcestheservertostartwiththepluginifitismentionedonstartup.

--plugin_name=FORCE_PLUS_PERMANENT: The same as the FORCE option, butadditionallypreventsthepluginfrombeingunloadedatruntime.

UninstallpluginMySQL 8 uses the UNINSTALL PLUGIN statement to uninstall the plugin, withoutconsideringwhether itwas installed during the runtime or at startup. But thisstatementwillnotallowustouninstallthebuilt-inpluginsandthepluginsthatwere installedby the--plugin_name=FORCE_PLUS_PERMANENToption.This statement justunloads the plugin and removes it from the mysql.plugin table, so it requiresadditionaldeleteprivilegesonthemysql.plugintable.

<strong>SELECT*FROMinformation_schema.PLUGINS;</strong>

<strong>SHOWPLUGINS;</strong>

Themysql.plugintablecontainsdetailsregardingallthepluginswhichwereregisteredbytheINSTALLPLUGINfunction.

<strong>CREATEROLEhrdepartment;</strong><br/><strong>grantallonhr_employeetohrdepartment;</strong>

Theabovecodewillhelpustocreatethehrdepartmentroleandgrantallthenecessaryaccesstoit.ThistopicwillbecoveredindetailedinChapter11,Security.

CachingtechniquesCache is a mechanism used to improve performance. MySQL uses severalstrategiestocacheinformationinthebuffer.MySQL8makeuseofthecacheatthe storage engine level to handle its operations. It also applies the cache inpreparedstatementsandstoredprogramstoimproveperformance.MySQL8hasintroduced various system level variables to manage cache, such asbinlog_stmt_cache_size, daemon_memcached_enable_binlog, daemon_memcached_w_batch_size,host_cache_size, and many more. We will cover caching in detail in Chapter 12,OptimizingMySQL8.

GlobalizationGlobalization is a feature which provides multi-language support for anapplication, such as enabling the use of native languages. It ismuch easier tounderstandmessagesinourownnativelanguagethanotherlanguages,right?Toachievethis,globalizationcomesintothepicture.Usingglobalizationausercanstore,retrieveandupdatedataintomanylanguages.Therearecertainparametersthataretobeconsideredinglobalization.Wewilldiscussthemindetail inthefollowingsections.

Charactersets

Beforegoingintodetailaboutcharactersetsitisrequiredtounderstandwhata

charactersetactuallyis,aswellasitsrelatedterms,right?Let'sstartwiththe

termitself;thecharactersetisasetofsymbolsandencoding.Anotherimportant

term related to character set is collation, the set of rules used for comparingcharacters.Let'stakeasimpleexampletounderstand

thecharactersetsandcollation.Considertwoalphabets,PandQ,andassignanumbertoeach,sothatP=1andQ=2.Now,assumePisasymboland1isitsencoding.Here,thecombinationofboththelettersand

theirencodingisknownasthecharacterset.Nowsuppose,wewanttocomparethese

values; the simplestway isby referring theencodingvalues.With this as1 isless

than 2 we can sayP is less thanQ which is known as collation. This is thesimplestexampletounderstandcharacter

sets and collation, but in real lifewe havemany characters, including specialcharacters,

andinthesamewaycollationshavemanyrules.

<strong>mysql>showcharacterset;</strong><br/><strong>+----------+---------------------------------+---------------------+--------+</strong><br/><strong>|Charset|Description|Defaultcollation|Maxlen|</strong><br/><strong>+----------+---------------------------------+---------------------+--------+</strong><br/><strong>|armscii8|ARMSCII-8Armenian|armscii8_general_ci|1|</strong><br/><strong>|ascii|USASCII|ascii_general_ci|1|</strong><br/><strong>|big5|Big5TraditionalChinese|big5_chinese_ci|2|.........</strong><br/><strong>.........</strong><br/><strong>+----------+---------------------------------+---------------------+--------+</strong><br/><strong>41rowsinset(0.01sec)</strong>

<strong>mysql>SHOWCOLLATIONWHERECharset='ascii';</strong><br/><strong>+------------------+---------+----+---------+----------+---------+---------------+</strong><br/><strong>|Collation|Charset|Id|Default|Compiled|Sortlen|Pad_attribute|</strong><br/><strong>+------------------+---------+----+---------+----------+---------+---------------+</strong><br/><strong>|ascii_bin|ascii|65||Yes|1|PADSPACE|</strong><br/><strong>|ascii_general_ci|ascii|11|Yes|Yes|1|PADSPACE|</strong><br/><strong>+------------------+---------+----+---------+----------+---------+---------------+</strong><br/><strong>2rowsinset(0.00sec)</strong>

<strong>CREATETABLEemployee(</strong><br/><strong>firstnameCHAR(10)CHARACTERSETlatin1,</strong><br/><strong>lastnameCHAR(10)CHARACTERSETascii</strong><br/><strong>);</strong><br/><br/><strong>INSERTINTOemployeeVALUES('Mona','Singh');</strong><br/><br/><strong>selectconcat(firstname,lastname)fromemployee;</strong><br/><strong>+----------------------------+</strong><br/><strong>|concat(firstname,lastname)|</strong><br/><strong>+----------------------------+</strong><br/><strong>|MonaSingh|</strong><br/>

<strong>+----------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong>

UTF-8formetadata:Metadatameansthedataaboutthedata.Intermsofdatabasewecansaythatanythingthatdescribesdatabaseobjectsisknownasmetadata.Forexample:columnnames,usernames,andmanymore.MySQLfollowsthebelowtworulesformetadata:

Includeallcharactersinallthelanguagesformetadata;thisenablesausertousehisownlanguageforcolumnnameandtablename.Manageonecommoncharactersetforallmetadata.Otherwise,theSHOWandSELECTstatementsfortablesinINFORMATION_SCHEMAwillnotworkproperly.

TofollowtheaboverulesMySQL8storesmetadataintotheUnicodeformat.ConsiderthatMySQLfunctionssuchasUSER(),CURRENT_USER(),SESSION_USER(),SYSTEM_USER(),DATABASE(),andVERSION()havetheUTF-8charactersetbydefault.MySQL8serverhasdefinedcharacter_set_systemtospecifycharactersetsformetadata.MakesurethatthestorageofmetadatainUnicodedoesnotmeanthatcolumnheadersandtheDESCRIBEfunctionwillreturnvaluesintheformofthemetadatacharacterset.Itworksasperthecharacter_set_resultssystemvariable.

Addingthecharacterset

ThissectiondescribeshowtoaddcharactersetsinMySQL8.Thismethodmayvarybasedonthecharactersettype-itmightbesimpleorcomplexdependingonthecharactertype.ThefollowingfourstepsarerequiredforaddingcharactersetsintoMySQL8:

1. Add a <charset> element for MYSET to the sql/share/charsets/Index.xml file. Forthesyntax,referthealreadydefinedfilefortheothercharacterset.

2. Inthisstep,theprocessisdifferentforsimpleandcomplexcharactersets.For simple character sets, create a configuration file, MYSET.xml, in thesql/share/charsets directory to describe the character set properties. In thecaseofcomplexcharactersets,theCsourcefileisrequired.Forexample,create the ctype-MYSET.c type in the strings directory. For each <collation>element,providectype-MYSET.cfile.

3. Modifytheconfigurationinformation:

1. Editmysys/charset-def.c,andregisterthecollationsforthenewcharacterset.Addtheselinestothedeclarationsection:

#ifdefHAVE_CHARSET_MYSET

externCHARSET_INFOmy_charset_MYSET_general_ci;

externCHARSET_INFOmy_charset_MYSET_bin;

#endif

Add these lines to the registration section: #ifdefHAVE_CHARSET_MYSETadd_compiled_collation(&my_charset_MYSET_general_ci);add_compiled_collation(&my_charset_MYSET_bin);#endif

2. Ifthecharactersetusesctype-MYSET.c,editstrings/CMakeLists.txtandaddctype-MYSET.ctothedefinitionoftheSTRINGS_SOURCESvariable.

3. Editcmake/character_sets.cmakewiththefollowingchanges:

AddMYSETtothevalueofwithCHARSETS_AVAILABLEinalphabeticorder.

AddMYSET tothevalueofCHARSETS_COMPLEXinalphabeticorder.Thisis needed even for simple character sets, or CMake will notrecognizeDDEFAULT_CHARSET=MYSET.

4. Reconfigure,recompile,andtest.

ConfiguringthecharactersetsMySQL 8 provides the --character-set-server and --collation-server options toconfigurethecharactersets.Thedefaultcharactersethasbeenchanged

fromlatin1 toUTF8.UTF8 is thedominatingcharacter set, though ithadn'tbeenadefaultoneinpriorversions

ofMySQL.Withthesechangesgloballyaccepted,charactersetsandcollationsarenowbasedonUTF8;oneofthecommonreasonsisbecausetherearearound21differentlanguagessupported

byUTF8,whichmakessystemsprovidemultilingualsupport.Beforeconfiguringcollation,

refertothecollationlistavailableathttps://dev.mysql.com/doc/refman/8.0/en/show-collation.html.

<strong>SETlc_messages='fr_FR';</strong>

<strong>mysqld--lc_messages_dir=/usr/share/mysql--lc_messages=nl_NL</strong>

IfthemessagefiledoesnotexistunderthedirectorythenMySQL8willignorethevalueofthelc_messagesvariableandconsiderthevalueofthelc_messages_dirvariableasalocationinwhichtolook.IftheMySQL8serverdoesnotfindthemessagefile,thenitshowsamessageintheerrorlogfileandusesEnglishforthemessages.

<strong>mysql>SETGLOBALtime_zone=timezone;</strong>

<strong>mysql>SETtime_zone=timezone;</strong>

Thissessionvariableaffectsthedisplayandstorageofzonespecificvalues.Forexample,valuesreturnedbytheNOW()andCURTIME()functions.Ontheotherhand,thisvariabledoesnotaffectsvalueswhicharedisplayedandstoredinUTCformat,suchaswiththeUTC_TIMESTAMP()function.

<strong>mysql>SETNAMES'utf8';</strong><br/><strong>QueryOK,0rowsaffected(0.09sec)</strong><br/><br/><strong>mysql>SELECT@@lc_time_names;</strong><br/><strong>+-----------------+</strong><br/><strong>|@@lc_time_names|</strong><br/><strong>+-----------------+</strong><br/><strong>|en_US|</strong><br/><strong>+-----------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SELECTDAYNAME('2010-01-01'),MONTHNAME('2010-01-01');</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>|DAYNAME('2010-01-01')|MONTHNAME('2010-01-01')|</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>|Friday|January|</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SELECTDATE_FORMAT('2010-01-01','%W%a%M%b');</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>|DATE_FORMAT('2010-01-01','%W%a%M%b')|</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>|FridayFriJanuaryJan|</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SETlc_time_names='nl_NL';</strong><br/><strong>QueryOK,0rowsaffected(0.00sec)</strong><br/><br/><strong>mysql>SELECT@@lc_time_names;</strong><br/><strong>+-----------------+</strong><br/><strong>|@@lc_time_names|</strong><br/><strong>+-----------------+</strong><br/><strong>|nl_NL|</strong><br/><strong>+-----------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SELECTDAYNAME('2010-01-01'),MONTHNAME('2010-01-01');</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>|DAYNAME('2010-01-01')|MONTHNAME('2010-01-01')|</strong><br/><strong>+-----------------------+------------------

-------+</strong><br/><strong>|vrijdag|januari|</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SELECTDATE_FORMAT('2010-01-01','%W%a%M%b');</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>|DATE_FORMAT('2010-01-01','%W%a%M%b')|</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>|vrijdagvrjanuarijan|</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong></strong>

<strong>SHOWCREATETABLEmysql.general_log;</strong>

<strong>SHOWCREATETABLEmysql.slow_log;</strong>

Theerrorlog

Thislogisusedtorecorddiagnosticmessageslikeerror,warningsandnotesthatoccur from the startup of MySQL 8 through till its end. MySQL 8 providesvariousconfigurationsandcomponentsforuserstogeneratelogfilesaspertheirrequirements.Whenwe startwriting into files some basics questions come tomind;whatdowewrite?Howdowewrite it?Wheredowewrite it to?Let'sstartwithfirstquestion.MySQL8

usesthelog_error_verbositysystemvariableandassignsthebelowfilteringoptionstodecidewhattypeofmessagesshouldbewrittenintotheerrorlogfile:

ErrorOnly

ErrorsandWarnings

Errors,WarningsandNotes

TowriteatthedestinationplaceMySQLusesthebelowformatwherethetimestamp depends on the log_timestamps system variable: timestamp thread_id[severity]message

Afterwritinglogfiles,thefirstquestionthatcomestomindis,howdoweflushtheselogs?Forthat,MySQL8providesthreeways;FLUSHERRORLOGS,FLUSHLOGS,ormysqladminflush-logs.Thesecommandswillcloseandreopenthelogfiletowhichitiswriting.Whenwetalkabouthowtowriteandwheretowrite,therearesomanythingstounderstand.

Componentconfiguration

MySQL 8 uses the log_error_services system variable to control error logcomponents. It allows users to define multiple components by semicolons,separatedfor theexecution.Here,componentswillbeexecuted in theorder inwhichtheyaredefined.Theuserisallowedtochangethevaluesofthisvariablewiththefollowingconstraints:

INSTALLCOMPONENT:Toenableanylogcomponentwemustfirstinstallitusingthiscommand,andthenusethecomponentbylistingitinlog_error_servicessystem variable. Follow the following commands to add thelog_sink_syseventlogcomponent:

INSTALLCOMPONENT'file://component_log_sink_syseventlog';

SETGLOBALlog_error_services='log_filter_internal;

log_sink_syseventlog';

Afterexecutionof the installationcommandMySQL8will register thecomponent into the mysql.component system table tomake it available forloadingoneachstartup.

UNINSTALL COMPONENT: To disable any of the log components, first remove itfromthelog_error_servicessystemvariablelistandthenuninstallitwiththiscommand.Executethebelowcommandtouninstallacomponent:

UNINSTALLCOMPONENT'file://component_log_sink_syseventlog';

Toenableerrorlogcomponentsoneachstartup,defineitinthemy.cnffileoruseSET_PERSIST.Whenwedefineitinmy.cnfittakeseffectfromthenextrestart, whereas SET_PERSIST will give an immediate effect. Use thefollowingcommandforSET_PERSIST:SETPERSISTlog_error_services='log_filter_internal;log_sink_internal;log_sink_json';

MySQL8alsoallowsuserstowriteerrorlogsintosystemlogs:forMicrosoft,considerEventlog,andforUnixbasedsystems,considersyslog.Toenableerrorloggingintosystemlogfibf,configurelog_filter_internalandthesystemlogwriterlog_sink_syseventlog componentsand follow the same instructionsexplainabove.AnotherwayistowriteaJSONstringintothelogfileconfigurationlog_sink_jsoncomponent.AninterestingpointaboutaJSONwriteristhatitwillmanagefilenamingconventionsbyaddingNN(two-digitnumbers).Forexample,considerfilenamesasfile_name.00.json,file_name.01.json,andsoforth.

Default error log destinationconfigurationErrorlogscanbewrittenintologfilesoronconsole.Thissectiondescribeshowtoconfigurethedestinationoferrorlogondifferentenvironments.

Default error log destination onWindows

--console: If this is given then the console will be considered the defaultdestination.--console takesprecedenceover--log-error incaseswherebotharedefined.Ifthedefaultlocationisconsole,thenMySQL8

serversetsthelog_errorvariable'svalueasstderror.

--log-error: If this is not given, or given without naming a file, then thedefault file name is host_name.err and the file will be created in the datadirectoryunlessthe--pid-fileoptionisspecified.Ifthefilenameisspecifiedin–pid-file option, then the naming conventionwould be aPID file basenamewithasuffixof.errinthedatadirectory.

Default error log destination onUnixandUnix-LikesystemsAll the abovementioned scenarios inMicrosoftWindowswill bemanagedbythe–log_erroroptioninUnixsystems.

--log-error:Ifthisisnotgiventhenthedefaultdestinationistheconsole.Ifnofilenameisgiven,thenaswithWindowsitwillcreateafileinthedatadirectorywith the host_name.err name.The user is allowed to specify –log-errorinanoptionfileunderthemysqldormysqld_safesections.

<strong>SETGLOBALgeneral_log='OFF';</strong>

Oncethelogisdisabled,renamethelogfileandenablethelogagainwiththeONoption.Similarly,toenableordisablethelogatruntimeforparticularconnectionsusethesessionsql_log_offvariablewiththeONorOFFoption.Onemoreoptionisaligningwiththegenerallogfile,thatis,--log-output.Byusingthisoption,wecanspecifythedestinationoflogoutput;itdoesnotmeanlogsareenabled.

Thethreefollowingdifferentoptionsareavailablewiththiscommand:

TABLE:LogtotablesFILE:LogtofilesNONE:Donotlogintotablesorfiles.NONE,ifpresent,takesprecedenceoveranyotherspecifiers.

Ifthe--log-outputoptionisomitted,thenthedefaultvalueisfile.

<strong>mysql>SETGLOBALbinlog_format='STATEMENT';</strong><br/><strong>mysql>SETSESSIONbinlog_format='STATEMENT';</strong>

Therearetwoexceptionalcaseswherewecannotchangeformat:

WithinastoredprocedureorfunctionIncaseswheretherowbasedformatissetandtemporarytableisopen

MySQL8hasthe--binlog-row-event-max-sizevariabletocontrolthesizeofthebinarylogfileintermsofbytes.Assignasavaluetothisvariableamultipleof256;thedefaultvalueofthisoptionis8192.IndividualstorageenginesofMySQL8havetheirowncapabilitiesforlogging.Ifastorageenginesupportsrowbasedlogging,thenitisknownasrow-loggingcapable,andifastorageenginesupportsstatementbasedloggingthenitisknownasstatement-loggingcapable.Refertothebelowtableformoreinformationonstorageengineloggingcapabilities.

Storageengine

Rowloggingsupported Statementloggingsupported

ARCHIVE Yes YesBLACKHOLE Yes YesCSV Yes YesEXAMPLE Yes NoFEDERATED Yes YesHEAP Yes Yes

InnoDB Yes YeswhenthetransactionisolationlevelisREPEATABLE,READ,orSERIALIZABLE;Nootherwise.

MyISAM Yes YesMERGE Yes YesNDB Yes No

Asdescribeinthissectionthebinarylogwillworkbasedontypesofstatementlikesafe,unsafe,orbinaryinjected,ontheloggingformatsuchasROW,STATEMENT,orMIXED,andwiththeloggingcapabilitiesofstorageengineslikerowcapable,statementcapable,both,orneither.Tounderstandallthepossiblecasesofbinaryloggingrefertothetablegiveninthislink:https://dev.mysql.com/doc/refman/8.0/en/binary-log-mixed.html.

Theslowquerylog

Slow query logs are used to record SQL statements that takes long time toexecute.

MySQL8hasdefinedthefollowingtwosystemvariablesfortimeconfigurationofslowquery:

long_query_time:Thisisusedtodefinetheidealtimeforqueryexecution.IfaSQL statement takes longer than this time, then it is considered a slowqueryandastatementisrecordedintothelogfile.Thedefaultvalueis10seconds.

min_examined_row_limit:Thisistheminimumtimerequiredfortheexecutionofeachquery.Thedefaultvalueis0seconds.

MySQL8will not consider the initial time of acquiring a lock into executiontimeandwill returnslowquery logs intoafileonceall locksarereleasedandqueryexecutioniscompleted.WhenMySQL8isstarted,slowqueryloggingisdisabledbydefault;tostartthislogusetheslow_query_log[={0|1}]command,where0 indicates slow query log is disabled and 1 or without argument is used to

enabled it. To log administrative statements and querieswithout indexing, usethe log_slow_admin_statements and log_queries_not_using_indexes variables. Here,administrativestatementsincludeALTERTABLE,ANALYZETABLE,CHECKTABLE,CREATEINDEX,DROPINDEX,OPTIMIZETABLE, and REPAIRTABLE.MySQL8 allowsusers to specify thenameofthelogfileusing--slow_query_log_file=file_namecommand.Ifnofilenameis specified, thenMySQL8willcreatea filewith the host_name-slow.lognamingconventioninthedatadirectory.Towriteminimalinformationintothislogfileusethe--log-short-formatoption.

AlltheabovedescribedparametersarecontrolledbyMySQL8inthefollowingsequence:

1. The query must either not be an administrative statement, orlog_slow_admin_statementsmustbeenabled

2. The query must have taken at least long_query_timeseconds, orlog_queries_not_using_indexesmust be enabled and the querymust have usednoindexesforrowlookups

3. Thequerymusthaveexaminedatleastmin_examined_row_limitrows

4. The query must not be suppressed according to thelog_throttle_queries_not_using_indexessetting

The--log-outputoptionisalsoavailableforthislogfile,andhasthesameimplementationandeffectasthegeneralpurposelog.

TheDDLlog

Asnameimplies,thislogfileisusedtorecordalltheDDLstatementexecutionrelated details.MySQL8 uses this log file to recover from crashes that occurduring themetadataoperationexecution.Let's takeoneexample tounderstandthescenarios:

Drop table t1, t2: We must be sure that both the t1 and t2 tables aredropped

WhenweexecuteanyDDLstatement,arecordoftheseoperationsiswrittenintotheddl_log.logfileundertheMySQL8datadirectory.Thisfileisabinaryfileandnotinhumanreadableformat.Theuserisnotallowedtoupdatethecontentsofthislogfile.

Metadata statements recording is not required in normal execution ofMySQLserver;enableitonlyifitisrequired.

<strong>SETGLOBALgeneral_log='OFF';</strong>

<strong>SETGLOBALgeneral_log='ON';</strong>

Summary

ThiswasaninterestingchapterforanyMySQL8user,wasn'tit?InthischapterweunderstoodhowMySQL8managesdifferentlogfilesandwhichlogfiletouseatwhattime.Atthesametimewealsocoveredmanyoftheadministrativefeatures,suchasglobalization,systemdatadatabase,andcomponentandpluginconfiguration,andexplainedhowtorunmultipleinstancesonasinglemachine.Thelaterpartofthechaptercoveredlogmaintenance.

Moving on to the next chapter, we will provide you with information aboutstorage engines, such aswhat the different types of storage engine are,whichone to use for your application, and how to create our own custom storageengineforMySQL8.

MySQL8StorageEngines

In the previous chapter, we learned about setting up a new system, datadictionary,andsystemdatabase.Detailedinformationwasprovidedoncachingtechniques, globalization, different types of components, and pluginconfiguration,alongwithseveraltypesoflogfileswhichareveryimportantforadministration.

ThischaptergivesdetailedinformationonMySQL8storageengines.ItexplainstheInnoDB storageengineand its features indetailandalsoprovidesapracticalguideline on custom storage engine creation and how tomake it pluggable sothatitcanbeinstalledinMySQL8.Thetopicsthatwewillbecoveringinthischapterareasfollows:

Overviewofstorageengines

Severaltypesofstorageengines

TheInnoDBstorageengine

Creatingacustomstorageengine

Overviewofstorageengines

StorageenginesareMySQLcomponentsforhandlingtheSQLoperationsusedin different types of tables. MySQL storage engines are designed to managedifferenttypesoftasksindifferenttypesofenvironments.Itisveryimportanttoknow and choose which storage engine is best suited for the system orapplication requirements. In following sections, wewill get to know in detailaboutthetypesofstorageengines,thedefaultstorageengine,andthecreationofcustomstorageengines.

Letusgothroughandseewhythestorageengineisaveryimportantcomponentindatabases, includingMySQL8.Storageenginesworkwithdatabaseenginestoperformvarioustypesoftasksindifferentenvironments.Theyexecutecreate,read,update, anddeleteoperations in the formof statementsondata from thedatabase. It looksquitesimplewhenyouprovide theENGINEparameterwith thecreate table statement but there is configuration for plenty of operations to bedoneonthedataforeachoftherequestssentviaSQLstatements.

Itismuchmorethanjustpersistingdata-theenginetakescareoffeaturessuchas storage limits, transactions, locking granularity/level, multi-versionconcurrencycontrol,geospatialdata types,geospatial indexing,B-tree indexes,T-tree indexes, Hash indexes, full-text search indexes, clustered indexes, data

caches, index caches, compressed data, encrypted data, cluster databases,replication, foreignkeys,backup,querycaches,andupdatingstatistics for thedatadictionary.

MySQL storage enginearchitecture

The MySQL storage engine's pluggable architecture allows a databaseprofessionalto

select any storage engine for the specialization required in any particularapplication.

TheMySQLStorageenginearchitectureprovidesaneasyapplicationmodelandAPIwith

the consistency that isolates the database administrator and the applicationprogrammer

from all the low-level implementation details underlying at the storage level.Thus,

the application always works above different storage engines' differentcapabilities.

Itprovidesstandardmanagementandsupport services thatarecommonforall

underlying

storageengines.

Storage engines perform activities on the data that is persisted at the physicalserver

level. Such modular and efficient architecture provides solutions to specificneeds

of any particular application, such as transaction processing, high availabilitysituations,

or data warehousing, and at the same time has the advantage of independentinterfaces

andservicesfromtheunderlyingstorageengines.

The database administrator and the application programmer interact with theMySQL

database by Connector APIs and services on top of the storage engines. Theapplication

isshieldedbytheMySQLserverarchitecturefromthedetailedlevelcomplexityof

the storage engines by providing easy to use APIs that are consistent andapplicable

onallthestorageengines.Iftheapplicationrequireschangesintheunderlying

storageengine,orifoneormorestorageenginesareaddedtosupporttheneedsof

the application, nomajor codingorprocess changes are required toget thingsworking.

Severaltypesofstorageengine

Now we know the importance of storage engines and critical decisions toidentifywhich storageengines touse fromplentyof storageengines availablefor MySQL 8. Let us take a look at what is available and with whichspecifications. InnoDB is the name that first entered your thoughtswhen youstartedthinkingofstorageengines,right?

InnoDBisthedefaultandmostgeneral-purposestorageengineinMySQL8andit is recommendedbyOracle touse for tablesaswellas forspecialusecases.TheMySQL

server has a pluggable storage engine architecture that enables storage engineloadingaswellasunloadingfromthealreadyrunningMySQLserver.

To identify which storage engines your server supports is made very easy inMySQL

8.Weonly have to go to theMySQL shell or prompt and use the SHOWENGINESstatement.Hitthestatementwhenpromptedandresultwillbethelistofengineswith a few columns, such as Engine, Support, Transactions, Savepoints, andComment.

Values in Support column, DEFAULT,YES, and NO, indicate that a storageengineisavailableandcurrentlysetasthedefaultstorage.

Overview of the InnoDB storageengineInnoDB is the default and most general-purpose storage engine in MySQL 8,providinghighreliabilityandhighperformance.

If you have not configured a different default storage engine, then issuing theSQL

statementCREATETABLEwithouttheENGINE=clausecreatesa tablewith thestorageengineInnoDBasthedefaultengineinMySQL8.

The featuresandadvantagesofferedby theInnoDB storageengineareexplainedlaterintheTheInnoDBstorageenginesection.

Customstorageengine

storageenginearchitectureinMySQL5.1andallthelaterversionsandMySQL8havetakenadvantageoftheflexiblestorageenginearchitecture.

Thestorageenginepluggablearchitectureprovides thecapability tocreateandaddnewstorageengineswithoutrecompilationoftheserver,addingdirectlytoarunning MySQL server. The architecture makes it very easy to develop anddeploynewstorageenginestoMySQL8.

We will develop a new storage engine by using the pluggable feature of theMySQLstorageenginearchitectureintheupcomingCreatingacustomstorageenginesection.

Severaltypesofstorageengines

Inthissection,wewilltakeacloserlookatthewidelyusedstorageenginesthatare supported byMySQL8.But before checking on them, let us see how thestorage engine architecture has made it pluggable and provided flexibility toenableusingmultiplestorageenginesinthesameschemaorserver.

ThefollowingisthelistofstorageenginessupportedinMySQL8:

InnoDB: The default storage engine for MySQL 8. It is an ACID compliant(transaction-safe) storage engine that has commit, roll back, and crash-recoveryforprotectingtheuserdataandreferential-integrityconstraints tomaintaindataintegrity,andmuchmore.

MyISAM:Thestorageenginewithtableshavingasmallfootprint.Ithastable-level locking and so is mostly used in read-only or read-mostly dataworkloads,suchasindatawarehousingandwebconfigurations.

Memory:ThestorageenginepreviouslyknownastheHEAPengine.ItkeepsdatainRAM,whichprovidesfasterdataaccess,mostlyusedinquicklookupsofnon-criticaldataenvironments.

CSV:Thestorageenginewithtablesascomma-separatedvaluesintextfilesandtables.

TheyarenotindexedandaremostlyusedforimportinganddumpingdatainCSVformat.

Archive:Thestorageenginecomprisescompact,unindexedtables, intendedtostoreandretrieveahugeamountofhistorical,archived,orsecurityauditdata.

Blackhole: The storage engine with tables that can be used for replicationconfiguration.Aqueryalwaysreturnsanemptyset.DMLSQLstatementsaresenttoslaveservers.Itacceptsdatabutdataisnotstored,suchasinaUnix/dev/nulldeviceuse.

Merge:ThestorageengineprovidesthecapabilitytologicallygroupaseriesofsimilarMyISAM tablesand refer to themasoneobject insteadofseparatetable.

Federated:ThestorageenginethatcanlinkmanyseparatephysicalMySQLservers into one logical database. It is ideal for datamarts or distributedenvironments.

Example: The storage engine that does nothing but works as a stub. It isprimarilyusedby thedeveloperswho illustratehow tobeginwritingnewstorageenginesintheMySQLsourcecode.

MySQL does not restrict using the same storage engine for anentire server or schema; instead, specifying the engine at tablelevelmakesitflexiblebasedonthetypeofdataandtheusecaseoftheapplication.

<strong>mysql>INSTALLPLUGINMyExampleSONAME'MyExample.so';</strong>

<strong>mysql>UNINSTALLPLUGINMyExample;</strong>

The common database serverlayer

The MySQL pluggable storage engine is responsible for executing I/Ooperationsontheactualdataandalso tocater to thespecificapplicationneedsthat includes enabling and enforcing the required features whenever required.Using a specific or single storage engine will more likely result in moreefficiency and higher database performance because the engine enables thefeatures only needed for a particular application, and resulting in less systemoverheadonthedatabase.

A storage engine supports the following unique infrastructure components orkeys:

Concurrency:Someapplicationshavegranular lock levels (such as row-level locks) requirements more than others. Overall performance andoverhead due to locking can be affected by choosing the right/wronglockingstrategyandthisalsoincludesmulti-versionconcurrencycontrolorsnapshotreadcapabilities.

Transaction support: Very well-defined requirements exist, such as ACIDcomplianceandmoreiftheapplicationneedstransactions.

Referentialintegrity:TheservercanenforcerelationaldatabasereferentialintegrityusingDDL-definedforeignkeysifrequired..

Physicalstorage:Thisincludeseverythingfromthepagesizeoftablesandindexesandalsoincludestheformatusedforstoringdataonaphysicaldiskaswell.

Indexsupport:This includes indexingstrategiesbasedon theapplicationneeds,aseachofthestorageengineshavetheirownindexingmethods.

Memorycaches:Thecachingstrategiesbasedontheapplicationneeds,aseach of the storage engines have their own caching methods along withcommonmemorycachesacrossallthestorageengines.

Performance aids: This involves bulk insert handing, database checkpointing,multiple I/O threads for parallel operations, thread concurrency,andmore.

Miscellaneous target features: This may includes support for securityrestrictionsoncertaindatamanipulationoperations,geospatialoperations,andothersimilarfeatures.

Each of the preceding infrastructure components are designed to support aspecific set of features for a particular application's needs and so it is veryimportanttounderstandtheapplicationrequirementverycarefullyandselecttheright storage engine, as it may impact on the overall system efficiency andperformance.

<strong>CREATETABLEtable1(i1INT)ENGINE=INNODB;</strong><br/><strong>CREATETABLEtable3(i3INT)ENGINE=MEMORY;</strong><br/><strong>ALTERTABLEtable3ENGINE=InnoDB;</strong>

<strong>SETdefault_storage_engine=MEMORY;</strong>

ThedefaultstorageengineforTEMPORARYtablesusingCREATETEMPORARYTABLEcanbesetseparatelybysettingthedefault_tmp_storage_enginevariableateitherstartuporruntime.

<strong>--myisam-recover-options=mode</strong>

ThisoptionwillsetthemodeintheautomaticrecoveryofcrashedtablesinMyISAM.

SpacesneededforkeysinMyISAM,B-TreeindexesareusedbyMyISAMtablesandspacecompressionisusedinStringindexes.Ifastringisthefirstpartoftheindexthenprefixcompressionisalsodone,whichoverallmakestheindexfilesizesmaller.Theprefixcompressionhelpsifmanystringshaveasimilarprefix.ByusingthetableoptionPACK_KEYS=1inMyISAMtables,prefixcompressioncanalsobeappliedonthenumbersiftherearemanynumberswithasimilarprefix.

PartitioningisnotsupportedforMyISAMtablesinMySQL8.0.

SomeoftheimportanttablescharacteristicsforMyISAMtablesareasfollows:

Alldatavaluesstoredhavethelowbytefirstorder,whichmakesthedataindependentofmachineandoperatingsystemsAllnumerickeyvaluesarestoredwithhighbytefirstorder,whichpermitsbetterindexcompressionMyISAMtableislimitedwith(232)2(1.844E+19)rowsMyISAMtableislimitedtoamaximumnumberof64indexespertableMyISAMtablecolumnsislimitedtoamaximumnumberof16columnsperindexConcurrentinsertsaresupportedinMyISAM,ifatabledoesnothaveanyfreeblocksinthemiddleofthedatafilesTEXTandBLOBtypecolumnscanalsobeindexedinMyISAMInindexedcolumns,NULLvaluesarepermittedEachofthecolumnscanhaveadifferentcharactersetItalsosupportforatrueVARCHARtypecolumnwithastarting

lengthstoredof1or2byte,tableswithVARCHARcolumnswithafixedordynamicrowlength,andUNIQUEconstraintswithanarbitrarylengthMyISAMtablestorageformats:ThefollowingthreedifferenttypesofstorageformatslistedaresupportedinMyISAM:

Statictable:ThedefaultformatforthetablesintheMyISAMstorageengine,withfixed-sizedcolumnsDynamictable:Asthenamesuggests,theformatthatcontainsvariablesizedcolumns,includingVARCHAR,BLOB,orTEXTCompressedtable:Thetableformatforkeepingread-onlydataandcompressedformatsinMyISAMstorageenginetables

Thefirsttwoformats,fixedanddynamic,arechosenautomaticallybasedonthecolumntypeused.Thecompressedformatcanbecreatedbyusingthemyisampackutility.

MyISAMtableproblems:Thefileformathasbeenextensivelytestedbutsomecircumstancesarisethatresultincorrupteddatabasetables.Letuslookatsuchcircumstancesandthewaytorecoverthosetables.

Wecouldgetcorruptedtablesintheeventofanyofthefollowingevents:

IfthemysqldprocessiskilledinthemiddleofawriteIfthereisanunexpectedcomputershutdownIfthereisanyhardwarefailureIfatableisbeingmodifiedatthesametimebytheMySQLserverandanexternalprogram,suchasmyisamchkTheMySQLorMyISAMcodehasasoftwarebug

CheckthehealthofthetablewiththeCHECKTABLEstatementandattempttorepairanycorruptedMyISAMtablebyusingtheREPAIRTABLEstatement.

ThereisalsopossibleissueyougetwithMyISAMtablesandthatistablesarenotbeingclosedproperly.Inordertoidentifyifthetableisclosedproperlyornot,eachMyISAMindexfilekeepsacounterintheheader.Thecountercanbeincorrectunderthefollowingcircumstances:

IfatableiscopiedwithoutissuingLOCKTABLESandFLUSHTABLES

MySQLcrashedbeforethefinalcloseduringanupdatemysqldisusingthetableandatthesametimeitismodifiedbyanotherprogram:myisamcheck--recoverormyisamchk--update-state

TheMEMORYstorageengine

TheMEMORYstorageengine,alsopreviouslyknownastheHEAPengine,keepsdatainRAM,whichprovidesfasterdataaccess.Itismostlyusedinquicklookupsofnon-criticaldataenvironments.Itcreatesspecial-purposetableswithcontentsstoredinmemorybut thedata isvulnerable to crashes,poweroutages, andhardwareissues. Therefore, these tables are used in temporary work areas or possiblyusingreadonlydatathatiscachedafterthedataispulledfromothertables.

You should choose whether use MEMORY or NDB Cluster. You should check if theapplicationisrequiredforimportant,highlyavailable,orfrequentlyupdateddataandconsiderwhetherNDBCluster is thebetterchoiceornot.NDBClusterprovidesthe same features as the MEMORY engine, butwith higher performance levels andadditionalfeaturesnotprovidedbyMEMORYengine.Theseinclude:

Low contention between clients by multiple thread operations and row-levellocking

Scalabilitywithstatementsmixes,includingwrites

Datadurability;itsupportsoptionaldisk-backedoperations

Shared-nothing architecture, providingmultiple-host operationswithout asinglepointoffailure,enabling99.999%availabilityfortheapplication

Automaticdatadistributionsacrossnodes

Supportforvariablelengthdatatypes,includingBLOBandTEXT

PartitioningisnotsupportedinMEMORYtables.

Performancedependsonhowbusytheserverisandtheeffectofsinglethreadexecutionwithtablelockoverheadduringupdatesprocessing.ThetablelockingduringupdatesprocessingcausesaslowdownofconcurrentusagefrommultiplesessionsonMEMORYtables.

MEMORYtablescharacteristics:Tabledefinitionsarestoredon theMySQLdata dictionary and do not create any files on the disk. The following are thetablefeaturehighlights:

100%dynamichashingforinsertsandspaceisallocatedinsmallblocks.

Noextrakeyspaceoroverflowareaorextraspaceforfreelistsisrequired.Reuseofdeletedrowswhennewrecordsinsertedbyputtingrowsinlinkedlists.

Fixed length row-storage format, VARCHAR , is stored with fixed length.CannotstoreBLOBorTEXTcolumns.

AUTO_INCREMENTcolumnsaresupported.

IndexinginHASHandBTREEtypesaresupportedbytheMEMORYstorageengine.MEMORYtableshaveamaximumof64indexespertable,amaximumof16columnsperindexandamaximumkeylengthof3,072bytes.MEMORYtablesalsocanhavenon-uniquekeys.

User createdand temporary tables: Internal temporary tables are createdbythe server on the fly while processing queries. Two types of tables differ instorageconversion,wheretheMEMORYtablesarenotsubjecttoconversion:

Whenaninternaltemporarytablebecomestoolarge,itisconvertedtoon-diskstoragebytheserverautomatically

UsercreatedMEMORYtablesareneverconvertedbytheserver

Dataloadingcanbeperformedusingthe--init-fileoption,usingINSERTINTO...SELECTorLOADDATAINFILEstatementsfromanypersistencedatasourceifrequired.

TheCSVstorageengine

This storage engine stores data in the formof comma-separatedvalues in textfiles.

The engine is always compiled into theMySQL server and the source can beexaminedfromthestorage/csvdirectoryofyourMySQLdistribution.

Thedatafilecreatedbytheserverbeginswiththegiventableandtheextensionof.CSV.Thedata file isaplain text filecontainingdata in thecomma-separatevaluesformat.

TheMySQLservercreatesacorrespondingmetafilealongwithaCSVtablethatstores information about the state of the table and the count for the rows thatexists in the table. The metafile is also stored with the table name at thebeginningwiththe.CSMextension.

RepairingandcheckingCSVtables:ThestorageenginesupportsCHECKandREPAIRstatementstoverifyandpossiblyrepairadamagedCSVtable.YoucanusetheCHECKTABLEstatementtoverifyorvalidatethetableandusetheREPAIRTABLEstatementtorepairatablethatcopiesvalidrowsfromanexistingCSV

datafileandreplacesanexistingfilewithnewlycopied/recoveredrows.

Duringrepair,onlyrowsfromtheCSVdatafiletothefirstdamagedrowgetscopiedtothenewtableorcopieddatafile.

Therestoftherowsafterthedamagedrowgetsremovedfromthetable,includingvalidrows,soIsuggestthatyoutakeenoughbackupofthedatafilepriortoproceedingwiththerepair.

Indexing or partitioning is not supported in the CSV storage engine and all thetablescreatedwiththeCSVstorageenginemusthavetheNOTNULLattributeonallthecolumns.

TheARCHIVEstorageengine

TheARCHIVEstorageenginecreatesspecial-purposetablesthatareusedforstoringhugeamountsofunindexeddatawithaverysmallfootprint.

WhentheARCHIVEtableiscreated,itbeginswiththetablenameandendswiththe.ARZextension.Duringoptimizationoperations,afilewithan.ARNextensionmayappear.

The AUTO_INCREMENT column attribute is supported by the engine. It also supportsINSERT,REPLACE,SELECT,andBLOBcolumns(allbutspatialdatatypes)butitdoesnotsupportDELETE,UPDATE,ORDER,orBYoperations.

PartitioningisnotsupportedbytheARCHIVEstorageengine:

Storage:Theengineuseslosslessdatacompressionwithzlibandtherowsget compressed as inserted. It supports the CHECK TABLE operation. Severaltypesofinsertionareusedintheengine:

INSERT statementsends rows intoacompressionbuffer,and thebuffergetsflushedasnecessary.

InsertioninthecompressionbufferisprotectedbythelockandflushwillonlyoccurifSELECTisrequested.

Oncecompletedabulkbuffercanbeseen.Itcanonlybeseenifanyother inserts occur at the same time.Here flushwill not occur uponSELECT,unlesswhileloadinganynormalinsert.

Retrieval:Afterretrieval,rowsgetsuncompressedasrequestedanditdoesnot use any rowcache.A complete table scan is performed for the SELECToperation:

SELECT checks howmany rows are available currently and reads onlythatnumberofrows.

Itisperformedasaconsistentreadoperation.

ThenumberofrowsreportedbySHOWTABLESTATUSisalwaysaccuratefortheARCHIVEtables.

Use OPTIMIZE TABLE or REPAIR TABLE operations to achieve bettercompression.

TheBLACKHOLEstorageengine

TheBLACKHOLE storage engine acts as a black hole. It accepts data but does notstoreitandaqueryalwaysreturnsanemptyresult.

TheserveronlyaddsthetabledefinitionintheglobaldatadictionarywhenyoucreateaBLACKHOLEtableandnofilesareassociatedwiththetable.

AllkindsofindexingissupportedintheBLACKHOLEstorageengineandsothesamecanbeincludedinthetabledefinition.

PartitioningisnotsupportedintheBLACKHOLEstorageengine.

Insertiontothetabledoesnotstoreanydatabutifbinaryloggingisenabledforstatements, then the statements are logged and replicated to the slave servers.Suchamechanismisusefulasafilterorrepeater.

TheBLACKHOLEstorageenginehasthefollowingpossibleuses:

Dumpfilesyntaxverification

Overhead measurement using binary logging enabled or disabled with aBLACKHOLEperformancecomparison

Itcanalsobeusedforfindinganyperformancebottlenecks,exceptforthestorageengineitself

Autoincrementcolumns:Astheengineisano-opengine,itwillnotincrementany field values but it has implications in the replication, which can be veryimportant.Considerascenariothathasthefollowingconditions:

1. ThemasterserverhasaBLOCKHOLE tablewithanautoincrementfieldwithaprimarykey

2. ThesametableexistsontheslaveserverbutusestheMyISAMengine

3. Insertionisperformedintothemasterserver'stablewithoutsettinganyautoincrementvalueintheINSERTstatementorusingtheSETINSERT_IDstatement

In the preceding scenario, the replicationwill fail on the primary key columnwithaduplicateentry.

TheMERGEstorageengine

The MERGE storage engine, known also as the MRG_MyISAM engine, is collection ofsimilartablesthatcanbeusedasonetableinstead.Here,"similar"meansthatallthetableshavesimilarcolumndatatypesandindexinginformation.

Itisnotpossibletomergetableswiththecolumnslistedinadifferentorderortohavethesamedata types inrespectivecolumnsorhaveindexinginadifferentorder.

Thefollowingisthelistofdifferencesintablesthatwillnotrestrictamerge:

Namesofrespectivecolumnsandindexescanbedifferent.

Commentsinbetweentables,columns,andindexescanbedifferent.

AVG_ROW_LENGTH,MAX_ROWS,orPACK_KEYStableoptionscanbedifferent.

WhenaMERGEtableiscreated,MySQLalsocreatesa.MRGfileonthediskwiththenamesofunderlyingMyISAM tablesbeingusedasone.Theformatof the table isstoredintheMySQLdatadictionaryandtheunderlyingtablesdonotrequireto

beinthesamedatabaseastheMERGEtable.

HavingprivilegesareamustforSELECT,UPDATE,andDELETEontheMyISAMtablesthatare being mapped with the MERGE table and so SELECT, INSERT, UPDATE, and DELETEstatementsontheMERGEtablecanbeused.

Executing the DROP TABLE statement on the MERGE table will drop only thespecificationfortheMERGEandnothingisimpactedontheunderlyingtables.

UsingMERGEtableshasthefollowingsecurityissues.IftheuserhasaccesstotheMyISAMtablet1,thentheusercancreatetheMERGEtablem1thatcanaccesst1.Now,iftheuser'sprivilegesonthetablet1arerevoked, the user can still continue accessing table t1 by usingtablem1.

TheFEDERATEDstorageengine

TheFEDERATEDstorageenginecanlinkmanyseparatephysicalMySQLserversintoonelogicaldatabase

and so it can let you access data from a remoteMySQL serverwithout usingeither

replicationorclustertechnology.

WhenwequerytothelocalFEDERATEDtable,thatautomaticallypullsthedatafromtheremotefederatedtablesandthe

dataisnotrequiredtobestoredonlocaltables.

TheFEDERATEDstorageengineisnotsupportedbydefaultintheMySQLserverbutstartingtheserver

withthe--federatedoptionwillenabletheFEDERATEDengineoption.

WhentheFEDERATEDtableiscreatedthetabledefinitionisthesameasothertables,butthephysical

storage of the associated data is handled on the remote server instead. TheFEDERATEDtableconsistsofthefollowingtwoelements:

Aremoteserverwithadatabasetableconsistingofatabledefinitionandtheassociatedtabledata.

ThistypeoftablecanbeanysupportedbytheremoteserverthatincludesMyISAMorInnoDBaswell.

A local server with a database table consisting of a table definition thesameastherespective

table on the remote server. The table definition is stored in the datadictionary

andnoassociateddatafileonthelocalserverisstored.Instead,inadditionto

the table definition, it keeps a connection string that is pointing to theremote

tableitself.

The following is the flowof information between the local and remote serverwhen

aSQLstatementisexecutedontheFEDERATEDtable:

1. The engine checks each of the columns the table has and builds anappropriateSQL

statementthatreferstotheremotetable.

2. TheMySQLclientAPIisusedforsendingtheSQLstatementtotheremoteserver.

3. Thestatementisprocessedbytheremoteserverandtherespectiveresultisretrieved

bythelocalserver.

TheEXAMPLEstorageengineTheEXAMPLEstorageengineisonlyastubengineandthepurposeoftheengineistoprovideexamplesintheMySQLsourcecode,whichhelpsdeveloperstowritenewstorageengines.

ToworkwiththeEXAMPLEenginesourcecode,lookatthestorage/exampledirectoryoftheMySQLsourcecodedistributiondownload.

NofilesarecreatedifthetableiscreatedwiththeEXAMPLEengine.DatacannotbestoredintheEXAMPLEengineanditreturnsemptyresults.

IndexingandpartitioningisnotsupportedintheEXAMPLEstorageengine.

TheInnoDBstorageengine

InnoDB is themost general-purpose storage engine and is the default engine inMySQL8,providinghighreliabilityandhighperformance.

ThefollowingarethekeyadvantagesofferedbytheInnoDBstorageengine:

Its DML operations follows the ACID model and transactions have commit,rollback,andcrash-recoveryfeaturestoprotectuserdata

Oracle-style gives consistent reads and row level locking increases theperformanceofmulti-userconcurrency

EachInnoDB table has a primary key index, known as the clustered index,thatarrangesdataonthediskinordertooptimizequeriesbasedonprimarykeyandminimizesI/Oduringprimarykeylookups

By supporting foreign keys, inserts, deletes, and updates are checked,ensuring consistency across different tables in order to maintain dataintegrity

ThefollowingarethekeybenefitsofusingInnoDBtables:

If the server crashesdue to anyhardwareor software issue, regardlessofwhat changeswere being processed in the server at that time, you're notrequired to do anything special after restarting the server. It has a crashrecoverysystemthattakescareofchangesthatwerecommittedduringthecrash of the server. It will go to those changes and start where theprocessingwasleftoff.

The engine has it's own buffer pool used for caching table and indexingdata to memory based on data accessed. Frequently used data is fetcheddirectly from the cache memory and so it speeds up processing. Indedicated servers, it takes up to 80%of physicalmemory assigned to beusedinthebufferpool.

Splittingrelateddata to tablesusingforeignkeysetupenforcesreferentialintegritywhichprevents inserting anyunrelateddata to a secondary tablewithouttherespectivedataintheprimarytable.

In case of corrupt data in thememory or disk, the checksummechanismgivesanalertaboutthecorruptdatabeforewegettouseit.

Change buffering automatically optimizes Insert, Update, and Delete. InnoDBalsoallowsconcurrentreadandwriteaccesstothesametableandcachingdatachangestostreamlinethediskI/O.

When the same data rows are accessed from the table repeatedly, theAdaptive Hash Index feature makes the lookups faster and givesperformancebenefits.

Compressionisallowedontablesandassociatedindexes.

Monitoring on internal workings and performance details of the storageengineiseasybyqueryingINFORMATION_SCHEMAorPerformanceSchematables.

Now let us look at each of the areas of the storage engine where InnoDB isenhancedoroptimizedtoprovideveryefficientandenhancedperformance.

ACIDmodelThe ACID model is a group of database design principles with an emphasis onreliability,whichismostimportantformissioncriticalapplicationsandbusinessdata.

MySQLhascomponentssuchastheInnoDBstorageenginethatcloselyadheretothe ACIDmodel. Therefore, data is safe and not corrupted, even in exceptionalcasesofhardwaremalfunctionsorsoftwarecrashes.

WithMySQL8,InnoDB supportsatomicDDL, ensuring that the DDL operations arefullycommittedorrolledback,eveniftheserverishaltedwhileperformingtheoperation.NowDDLlogscanbewrittentothemysql.innodb_ddl_logconfigurationforthedatadictionary tables,enabling theinnodb_print_ddl_logsconfigurationoptiontoprintDDLrecoverylogstostderr.

MultiversioningInnoDBisamultiversionedstorageengine.Thatmeans ithas thecapability tokeep old versions of changed row data information and support transnationalfeatures, such as concurrency and roll back. The information is stored in thetablespace,datastructure,andnamedrollbacksegment.

Internally,foreachoftherowsgettingstoredinthedatabase,InnoDBcreatesthreefields: 6-byte DB_TRX_ID, 7-byte DB_ROLL_PTR (called a roll pointer) and 6-byteDB_ROW_ID. With these fields, InnoDB creates clustered indexes to keep theinformationofchangedrowdatainthedatabase.

Architecture

Inthissection,wewillgiveabriefintroductiontothemajorcomponentsoftheInnoDBarchitecture:

Buffer pool: Area of main memory where tables and indexing data arecachedtospeedupprocessing

Changebuffer: Special data structurewhere changes to secondary indexpagesarecached

Adaptive Hash Index: Enables in-memory database, such as lookups,operations on systemswith balanced and appropriate combinations of thebufferpool'smemoryandworkload

Redologbuffer:Memoryareawheredataisheldtobewrittenontheredolog

System tablespace: Storage area where the doublewrite buffer, undo logs,and thechangebuffer,prior to theMySQL8datadictionary information,arestored

Doublewritebuffer:Storageareainthesystemtablespacewherepagesarewrittenthatareflushedfromthebufferpool

Undologs:Collectionofundo log recordswhichareassociatedwithanysingletransaction

File-per-table tablespaces: Single-table tablespace added to its own datafile

General tablespaces: Shared tablespace created by the CREATE TABLESPACE

syntax

Undotablespace:Oneormorefileswithundologs

Temporarytablespace:Utilizedfornon-compressedtemporarytablesandtheirrelatedobjects

Redo log: Disk-based data structure used for correcting incompletetransactiondataduringcrashrecovery

With MySQL 8, the InnoDB storage engine utilizes the global MySQL datadictionaryandnotitsownstorageengine-specificdatadictionary.

Lockingandtransactionmodel

ThissectiongivesbriefinformationonlockingusedbyInnoDBandthetransactionmodelimplementedbyInnoDB.InnoDBusesthefollowingdifferentlocktypes:

Sharedandexclusivelocks:Twotypesofstandardrow-level lockingareimplemented. A shared lock allows you to read a row to differenttransactions;anexclusivelockholdstoupdateordeletearowanddoesnotallowyoutoevenreadtherowtoanydifferenttransaction.

Intention locks:Table level locks to supportmultiple granularity lockingbywhichInnoDBpracticallymaintainsthecoexistenceofrow-levellocksandentiretable-levellocks.

Record locks: Index record lock that prevents any other transaction toinsert,update,ordeletetherecord.

Gaplocks:Lockappliesonagap(range)betweenindexrecords.

Next-keylocks:Combinationofindexrecordlockplusgaplockonthegapfortheprecedingindexrecord.

Insert intention locks:Typeof gap lockwhich is set by INSERToperationjustbeforetherowinsertion.

AUTO-INClocks: Special table-level lock for inserting recordswith theAUTO_INCREMENTcolumn.

Predicate locks for spatial indexes: Lock on spatial index, enablingsupportforisolationlevelsintableswithspatialindexes.

The goal of following the transaction model is to unite traditional two-phaselocking with the best of the multiversioning database properties. Row-levellockingisperformedandqueriesarerunwithnonlockingconsistentreads.InnoDBtakescareoftransactionisolationlevels,autocommit,rollbackandcommit,andlocking reads. It allows nonlocking consistent reads as applicable. InnoDB alsouses a mechanism to avoid phantom rows and a configuration to supportautomaticdeadlockdetection.

Configuration

Thissectionprovidesbrief informationabout theconfigurationandproceduresusedintheInnoDBinitializationstartupfordifferentInnoDBcomponents:

InnoDBstartupconfiguration:This involvesspecifyingstartupoptions, logfileconfiguration,storageconsiderations,systemtablespacedatafiles,undotablespaces,temporarytablespaces,pagesizes,andmemoryconfigurations

InnoDB forread-onlyoperation:This enables aMySQL instance for read-only operation, using the --innodb-read-only=1 option,which is very helpfulwhenusingread-onlymediasuchasCDorDVD

InnoDBbufferpoolconfiguration:Configuresthebufferpoolsize,multipleinstances,flushing,andmonitoring

InnoDB change buffering: Configures the change buffer options forsecondaryindexcaching

Thread concurrency for InnoDB: Concurrent thread count limitconfiguration

ThenumberofbackgroundInnoDBI/Othreads:ConfiguresthenumberofbackgroundthreadsservicingI/Oread/writeoperationsondatapages

Using asynchronous I/O on Linux: A configuration to use nativeasynchronousI/OsubsytemsonLinux

TheInnoDBmaster thread I/Orate:Configuresoverall I/Ocapacity for amasterthreadworkinginthebackground,responsibleformultipletasks

Spin lock polling: Configures a spin wait delay period to control themaximumdelayforfrequentpollingbetweenmultiplethreadsrequestingtoacquiremutexesorrw-locks

InnoDBpurgescheduling:Configurespurgethreadsforapplicablescalability

Optimizer statistics for InnoDB: Configures persistent and non-persistentoptimizerstatisticsparameters

Themergethresholdforindexpages:ConfiguresMERGE_THRESHOLDtoreducemerge-splitbehavior

Enabling automatic configuration for a dedicated MySQL Server:Configures the dedicated server option --innodb_dedicated_server , whichmakesautomaticconfigurationforthebufferpoolsizeandlogfilesize

Tablespaces

ThissectionprovidesbriefinformationontablespacesandoperationsrelatedtotablespacesperformedinInnoDB:

ResizingtheInnoDB systemtablespace: Increasinganddecreasing thesizeof the system tablespace with configuration while starting/restarting theMySQLserver.

Changing the number or size of InnoDB redo log files: Configuresinnodb_log_files_in_group and innodb_log_file_size values respectively in my.cnfpriortostarting/restartingtheMySQLserver.

Usingrawdiskpartitionsforthesystemtablespace:Configurestherawdiskpartitionstobeusedasdatafilesinthesystemtablespace.

InnoDBFile-Per-Tabletablespaces:Thefeatureinnodb_file_per_table enabledbydefaultwhichensuresthateachofthetablesandassociatedindexesarestoredinaseparate.idbdatafile.

Configuringundotablespaces:Aconfigurationtosetthenumberofundo

tablespaceswhereanundologresides.

Truncatingundotablespaces:Configuresinnodb_undo_log_truncate toenabletruncating undo tablespace files exceeding themaximum limit defined ininnodb_max_undo_log_size.

InnoDB general tablespaces: A shared tablespace created using the CREATETABLESPACEstatement.Itissimilartoasystemtablespace.

InnoDB tablespaceencryption:Support fordataencryption in tablesstoredas file-per-table tablespaces which use the AES block-based encryptionalgorithm.

Tablesandindexes

This section provides brief information on InnoDB tables and indexes and theirrelatedoperations:

CreatingInnoDBtables:CreatestablesusingtheCREATETABLEstatement.

ThephysicalrowstructureofanInnoDB table:Dependson the specifiedrow format during the table creation. If not specified, uses the default,DYNAMIC.

Moving or copying InnoDB tables: Different techniques for moving orcopyingsomeorallInnoDBtablestoadifferentinstanceorserver.

Converting tables from MyISAM to InnoDB: Considers guidelines and tipswhile converting MyISAM tables to InnoDB tables, except a partitioned table,whichisnotsupportedwithMySQL8.

AUTO_INCREMENThandlinginInnoDB:ConfiguresthemodeforAUTO_INCREMENTwiththe innodb_autoinc_lock_mode parameter as 0,1, and 2 for traditional,consecutive, or interleaved, respectively, where interleaved is the default

modefromMySQL8.

LimitsonInnoDBtables:Atablecancontainamaximumof1,017columns,amaximumof64secondaryindexes,andseveralotherlimitsdefinedbasedonthepagesize,tablesize,anddata-rowformats.

Clustered and secondary indexes: InnoDB uses a special index called aclusteredindex.Therestoftheindexesarecalledsecondaryindexes.

ThephysicalstructureofInnoDBindex:Forspatialindexes,InnoDBusestheR-treedatastructure,aspecializeddatastructure.Forrestoftheindexes,theB-treedatastructureisused.

Sorted index builds: Bulk load when creating or rebuilding indexes forinserts. They are known as sorted index builds, and are not supported inspatialindexes.

InnoDBFULLTEXTindexes:Createdfortext-basedcolumns-char,varchar,ortexttype.Theyhelptospeedupqueriesandsearchingoperations.

<strong>mysql>SHOWTABLESFROMINFORMATION_SCHEMALIKE'INNODB%';</strong>

Tablesaboutcompression:Thenumberofcompressionoperationsandtheamountoftimespentforcompression-relatedinformationprovidedintheINNODB_CMPandINNODB_CMP_RESETtables.MemoryallocationduringcompressionisprovidedintheINNODB_CMPMEMandINNODB_CMPMEM_RESETtables.Transactionandlockinginformation:INNODB_TRXhasinformationontransactionscurrentlyexecutingandthedata_locksanddata_lock_waitstablesfromthePerformanceSchematablegiveinformationaboutthelocks.Schemaobjecttables:ThisprovidesmetadatainformationabouttheInnoDBschemaobjects.FULLTEXTindextables:ThisprovidesmetadatainformationaboutFULLTEXTindexes.Bufferpooltables:Thisprovidesstatusinformationandmetadataaboutthepagesinthebufferpool.Metricstable:Thisprovidesperformanceandresourcerelatedinformation.Temporarytableinformationtable:Thisprovidesmetadatainformationaboutallusersandsystem-createdtemporarytablescurrentlyactiveinanInnoDBinstance.RetrievingInnoDBtablespacemetadata:ThisprovidesmetadatainformationaboutallthetypesoftablespacesinanInnoDBinstance.

Anewview,INNODB_TABLESPACES_BRIEF,hasbeenaddedtoprovidethename,path,flag,space,andspacetypedata.

Anewtable,INNODB_CACHED_INDEXES,hasbeenaddedtoprovidethenumberofindexpagescachedinthebufferpoolforeachindex.

Memcachedplugin

MySQL8providesyouwiththeInnoDBmemcachedpluginnameddaemon_memcached,which can help us in managing data easily. It will automatically store andretrieve data from InnoDB tables and provide get, set, and incr operations thatremoveperformanceoverheadbyskippingSQLparsing,whichspeedsupdataoperations. The memcached plugin uses the integrated memcached daemon thatautomaticallyretrievesandstoresdatafromandtotheInnoDBtable,enablingtheMySQLservertosenddataquicklytothekey-valuestore.

ThefollowingarethemajorbenefitsofusingtheInnoDBmemcachedplugin:

Accesses theInnoDB storageenginedirectly, reducingparsingandplanningSQLoverhead

memcached uses the same process space as the MySQL server, reducingnetworkoverhead

Datawrittenorrequestedinthememcachedprotocolistransparentlywrittenorqueried from InnoDB tables, reducing having to go through SQL layeroverhead

Simplifiesapplicationlogicbyautomaticallytransferingbetweendiskandmemory

TheMySQLdatabasestoresdatasothatitisprotectedagainstcorruption,crashes,oroutages

Ensures high availability using the daemon_memcached plugin on the masterserverandMySQLreplicationincombination

Repeated data requests are cached using the InnoDB buffer pool, providinghighspeedprocessing

As the data is stored in the InnoDB tables, thedata consistency is enforcedautomatically

The InnoDB memcached plugin supports multiple get operations (fetching multiplekey/valuepairsinasinglememcachedquery)andrangequeries.

Creatingacustomstorageengine

MySQLABintroducedpluggablestorageenginearchitectureinMySQL5.1andalllater

versions, including MySQL 8, have taken advantage of the flexible storageenginearchitecture.

Thestorageenginepluggablearchitectureprovides thecapability tocreateandadd

newstorageengineswithoutrecompilingtheserver,addingdirectlytoarunningMySQL

server.Thearchitecturemakes itveryeasy todevelopanddeploynewstorageengines

toMySQL8.

When developing new storage engine, it is required to take care of all thecomponents

that work for and with storage engines. These include installation handlers,

operations

ontablesuchascreating,opening,andclosing,DML,indexing,andsoon.

Inthissection,wewillcoverhowyoucanstartdevelopinganewstorageengineon

ahigh-levelbasiswithreferenceto theMySQLdocumentationprovidedinthedevelopment

community. The creation of a custom storage engine requires a workingknowledgeofdevelopmentwithCandCPP,andcompilationwithcmakeandVisualStudio.

Creating storage engine sourcefilesTheeasiestwaytoimplementanewstorageengineistobeginbycopyingandmodifying

theEXAMPLEstorageengine.Thefilesha_example.ccandha_example.hcanbefoundinthestorage/exampledirectoryoftheMySQLsourcedistribution.

Whencopying the files,change thenames fromha_example.ccandha_example.h tosomethingappropriatetoyourstorageengine,suchasha_foo.ccandha_foo.h.

Afteryouhavecopiedandrenamedthefiles,youmustreplaceall instancesofEXAMPLEandexamplewiththenameofyourstorageengine.

Adding engine-specific variablesandparametersAplugincanimplementstatusandsystemvariablesandinthissectionwehavecovered the changes to variables and parameters with appropriate values anddatatypes.

Theserverplugininterfaceenablespluginstoexposestatusandsystemvariablesusingthestatus_varsandsystem_varsmembersofthegeneralplugindescriptor.

status_varsisamemberofthegeneralplugindescriptor.Ifthevalueisnot0,thenitpointstoanarrayofthest_mysql_show_varstructurewhereeachofthemdescribeone statusvariable followedby a structurewith all themembers set to0.Thedefinition for the st_mysql_show_var structure is as follows: structst_mysql_show_var{constchar*name;char*value;enumenum_mysql_show_typetype;};

Whenthepluginisinstalled,thepluginnameandthenamevaluearejoinedwithanunderscoretoformthenamedisplayedbytheSHOWSTATUSstatement.

Thefollowinglistshowsthepermissiblestatusvariabletypevaluesandwhatthecorrespondingvariableshouldbe:

SHOW_BOOL:ThisisapointertothebooleanvariableSHOW_INT:ThisisapointertotheintegervariableSHOW_LONG:ThisisapointertothelongintegervariableSHOW_LONGLONG:ThisisapointertothelonglongintegervariableSHOW_CHAR:ThisisaStringindexSHOW_CHAR_PTR:ThisisapointertoStringindexesSHOW_ARRAY:Thisisapointertoanotherst_mysql_show_vararraySHOW_FUNC:ThisisapointertoafunctionSHOW_DOUBLE:Thisisapointertoadouble

Allsessionandglobalsystemvariableshavetobepublishedtomysqldbeforetheyare used.This is precisely done by constructing a NULL terminated arrayof thevariablesandlinkingtoitinthepluginpublicinterface.

All mutable and plugin system variables are stored internally in the HASH

structure.

The display of the server command-line help text is generated by compilingDYNAMIC_ARRAY of all the relevant variables, sorting and iterating through them todisplayeachoption.

During the plugin installation process, the server processes command-lineoptions,immediatelyafterthepluginhasbeensuccessfullyloadedbuttheplugininitializationfunctionisyettobecalled.

Pluginsloadedatruntimedonotbenefitfromanyconfigurationoptionsandmusthave usable defaults. Once they are installed, they are loaded at mysqld

initialization timeandconfigurationoptionscanbesetat thecommand lineorwithinmy.cnf.

Thethdparametershouldbeconsideredasread-onlyinplugins.

<strong>handlertonexample_hton={</strong><br/><strong>"EXAMPLE",/*Nameofthestorageengine*/</strong><br/><strong>SHOW_OPTION_YES,/*Itshouldbedisplayedinoptionsornot*/</strong><br/><strong>"Examplestorageengine",/*Descriptionofthestorageengine*/</strong><br/><strong>DB_TYPE_EXAMPLE_DB,/*Typeofstorageengineitshouldreferto*/</strong><br/><strong>NULL,/*Initializehandlerton*/</strong><br/><strong>0,/*slotavailable*/</strong><br/><strong>0,/*definesavepointsize.*/</strong><br/><strong>NULL,/*handleclose_connection*/</strong><br/><strong>NULL,/*handlesavepoint*/</strong><br/><strong>NULL,/*handlerollbacktosavepoint*/</strong><br/><strong>NULL,/*handlereleasesavepoint*/</strong><br/><strong>NULL,/*handlecommit*/</strong><br/><strong>NULL,/*handlerollback*/</strong><br/><strong>NULL,/*handleprepare*/</strong><br/><strong>NULL,/*handlerecover*/</strong><br/><strong>NULL,/*handlecommit_by_xid*/</strong><br/><strong>NULL,/*handlerollback_by_xid*/</strong><br/><strong>NULL,/*handlecreate_cursor_read_view*/</strong><br/><strong>NULL,/*handleset_cursor_read_view*/</strong><br/><strong>NULL,/*handleclose_cursor_read_view*/</strong><br/><strong>example_create_handler,/*Createanewhandlerinstance*/</strong><br/><strong>NULL,/*handledropdatabase*/</strong><br/><strong>NULL,/*handlepaniccall*/</strong><br/><strong>NULL,/*handlereleasetemporarylatches*/</strong><br/><strong>NULL,/*UpdaterelevantStatistics*/</strong><br/><strong>NULL,/*StartConsistentSnapshotforreference*/</strong><br/><strong>NULL,/*handleflushlogs*/</strong><br/><strong>NULL,/*handleshowstatus*/</strong><br/><strong>NULL,/*handlereplicationReportSenttoBinlog*/</strong><br/><strong>HTON_CAN_RECREATE</strong><br/><strong>};</strong>

Thereare30handlertonelements,onlyfewofwhicharemandatory.

<strong>statichandler*tina_create_handler(TABLE*table);</strong>

<strong>statichandler*myisam_create_handler(TABLE*table)</strong><br/><strong>{</strong><br/><strong>returnnewha_myisam(table);</strong><br/><strong>}</strong>

<strong>constchar**ha_tina::bas_ext()const</strong><br/><strong>{</strong><br/><strong>returnha_tina_exts;</strong><br/><strong>}</strong>

Byprovidingextensioninformation,youcanalsoskipimplementingDROPTABLEfunctionality,astheMySQLserverwillimplementthesamebyclosingthetableanddeletingallfileswiththeextensionsspecified.

<strong>virtualintcreate(constchar*name,TABLE*form,HA_CREATE_INFO*info)=0;</strong>

<strong>typedefstructst_ha_create_information</strong><br/><strong>{</strong><br/><strong>CHARSET_INFO*table_charset,*default_table_charset;/*charsetintable*/</strong><br/><strong>LEX_STRINGconnect_string;/*connectionstring*/</strong><br/><strong>constchar*comment,*password;/*storingcommentsandpasswordvalues*/</strong><br/><strong>constchar*data_file_name,*index_file_name;/*dataandindexfilenames*/</strong><br/><strong>constchar*alias;/*valuepointerforalias*/</strong><br/><strong>ulonglongmax_rows,min_rows;</strong><br/><strong>ulonglongauto_increment_value;</strong><br/><strong>ulongtable_options;</strong><br/><strong>ulongavg_row_length;</strong><br/><strong>ulongraid_chunksize;</strong><br/><strong>ulongused_fields;</strong><br/><strong>SQL_LISTmerge_list;</strong><br/><strong>enumdb_typedb_type;/*valuefordb_type*/</strong><br/><strong>enumrow_typerow_type;/*valueforrow_type*/</strong><br/><strong>uintnull_bits;/*NULLbitsspecifiedatstartofrecord*/</strong><br/><strong>uintoptions;/*ORofHA_CREATE_optionsspecification*/</strong><br/><strong>uintraid_type,raid_chunks;/*raidtypeandchunksinfo*/</strong><br/><strong>uintmerge_insert_method;</strong><br/><strong>uintextra_size;/*lengthofextradatasegments*/</strong><br/><strong>booltable_existed;/*1increateiftableexisted*/</strong><br/><strong>boolfrm_only;/*1ifnoha_create_table()*/</strong><br/><strong>boolvarchar;/*1iftablehasaVARCHAR*/</strong><br/><strong>}HA_CREATE_INFO;</strong>

Storageenginescanignorethecontentsof*infoand*formbecausethecreationandtheinitializationofthedatafilesisonlyreally

requiredwhenusedbythestorageengine.

<strong>intopen(constchar*name,intmode,inttest_if_locked);</strong>

<strong>#defineHA_OPEN_ABORT_IF_LOCKED0/*default*/</strong><br/><strong>#defineHA_OPEN_WAIT_IF_LOCKED1/*waitiftableislocked*/</strong><br/><strong>#defineHA_OPEN_IGNORE_IF_LOCKED2/*ignoreiflocked*/</strong><br/><strong>#defineHA_OPEN_TMP_TABLE4/*Tableisatemptable*/</strong><br/><strong>#defineHA_OPEN_DELAY_KEY_WRITE8/*Don'tupdateindex*/</strong><br/><strong>#defineHA_OPEN_ABORT_IF_CRASHED16</strong><br/><strong>#defineHA_OPEN_FOR_REPAIR32/*openevenifcrashedwithrepair*/</strong>

Thetypicalstorageenginewillimplementsomeformofsharedaccesscontrolinordertopreventfilecorruptioninamulti-threadedenvironment.Forexample,seetheget_share()andfree_share()methodsofsql/example/ha_tina.ccforimplementingfilelocking.

<strong>ha_tina::store_lock</strong><br/><strong>ha_tina::external_lock</strong><br/><strong>ha_tina::info</strong><br/><strong>ha_tina::rnd_init</strong><br/><strong>ha_tina::extra-ENUMHA_EXTRA_CACHECacherecordinHA_rrnd()</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::extra-ENUMHA_EXTRA_NO_CACHEEndcachingofrecords(def)</strong><br/><strong>ha_tina::external_lock</strong><br/><strong>ha_tina::extra-ENUMHA_EXTRA_RESETResetdatabasetoafteropen</strong>

Thefollowingmethodscanbeimplementedtotakecareofspecificoperations:

Implementingthestore_lock():Thismethodcanmodifythelocklevel,ignoringoraddinglocksformanytablesImplementingtheexternal_lock():ThismethodiscalledwhentheLOCKTABLESstatementisissuedImplementingthernd_init():ThismethodisusedintablescanningforresettingcountersandpointersatthestartofatableImplementingtheinfo(uinfflag):ThismethodisusedtoprovideextratableinformationtotheoptimizerImplementingtheextra():Thismethodisusedtoprovideextrahintsinformationtothestorageengine

Implementingthernd_next():ThismethodiscalledoneachrowofscanninguntilEOFisreachedorthesearchconditionissatisfied

<strong>intha_tina::close(void)</strong><br/><strong>{</strong><br/><strong>DBUG_ENTER("ha_tina::close");</strong><br/><strong>DBUG_RETURN(free_share(share));</strong><br/><strong>}</strong>

Storageenginesusetheirownsharemanagementsystems.Theyshouldusetherequiredmethodsinordertoremovethehandlerinstancefromthesharefortherespectivetableopenedintheirhandler.

Ifyourstorageengineiscompiledasasharedobject,duringloadingifyougetanerrorsuchasundefinedsymbol:_ZTI7handler,thenmakesureyoucompileandlinkyourextensionusingthesameflagsastheserveruses.TheusualreasonforthiserroristhatLDFLAGSaremissingthe-fno-rttioption.

Reference for advanced customstorageengineWe have gone through the preceding sections in detail, giving high-levelinformation

for custom storage engine components and the required changes. Forimplementing INSERT, UPDATE, DELETE, indexing, and so on, in a custom storageengine, requires a working knowledge of development with C/CPP andcompilation with cmake and Visual Studio. For advanced development for thecustomstorageengines,pleaserefertothedetailed

informationgivenathttps://dev.mysql.com/doc/internals/en/custom-engine.html

Summary

Bynow,youhavelearnedthedifferentdatabaseenginesavailableinMySQL8andwelearnedwhyweshouldcareaboutstorageenginesandavailablestorageengineoptions inMySQL8.Wecovered indetail theInnoDB storageengineandrelatedimportantfeaturesalreadyprovidedwithintheInnoDBstorageengine.Nowyou are practically able to create a custom storage engine as per the systemrequirement andmake it pluggable inMySQL 8.An important aspectwas tochooseasuitablestorageengineforyoursystem,whichiscovereddetail.

Inthenextchapter,youwilllearnabouthowindexingworksinMySQL8,thenewfeaturesintroducedrelatedtoindexing,thedifferenttypesofindexing,andhowtouseindexingonyourtables.Alongwiththat,acomparisonwillalsobeprovided along with in-depth knowledge of various ways of indeximplementation.

IndexinginMySQL8

In the previous chapter,we learned about storage engines.Nowwe are awarewhat types of storage engines are available and which ones to use for ourrequirements. The previous chapter also covered the InnoDB storage engine indetail, along with other storage engine information. It also described how todefineacustomstorageengineforuse,withapracticalexample.Nowit'stimeto understand one more important functionality of MySQL 8 and that is,indexing.

Wewill cover different types of indexeswith their functionalities,whichwillencourage you to use indexes and provided youwith guidance on how to usethem.So,yourjourneyintoindexeshasstarted!Let'sgo.

Wewillcoverthefollowingtopicsinthischapter:

Anoverviewonindexing

Column-levelindexing

B-Treeindexes

Hashindexes

Indexextensions

Usinganoptimizerforindexes

Invisibleanddescendingindexes

Anoverviewonindexing

TodefineanindexonatableisthebestwaytoimprovetheperformanceoftheSELECT operation. An index acts like a pointer for the table rows and permitsqueriesto

quicklypoint tomatching rowsbasedon theWHERE condition.MySQL8allowsyoutocreateindexesonallthedatatypes.Althoughindexing

providesgoodperformanceonqueries,itisrecommendtodefineitintheproperway,

becauseunnecessaryindexeswastespaceandtime(forMySQL8tofindwhichindex

is best to use). In addition to that, indexes also add costs to INSERT, UPDATE, andDELETE operations, because during these operations,MySQL8will update eachindex.

Aswedescribedpreviously,anindexisadatastructurethatimprovesthespeedof

operations.Basedonthestructure,anindexisbifurcatedintotwomajorforms—

aclustered

indexandanon-clusteredindex:

Clustered index: A clustered index defines the order in which data isphysicallystoredinatable.

Therefore,onlyoneclusteredindexisallowedpertable.Itgreatlyincreasesthe

speedofretrievalwhendataisretrievedinasequentialmanner,eitherinthesame

order or in reverse order. A clustered index also provides betterperformancewhen

arangeofitemsareselected.Aprimarykeyisdefinedasaclusteredindex.

Non-clustered index: A non-clustered index doesn't define the order inwhichdataisphysicallystored.

Thismeansanon-clusteredindexisstoredinoneplace,anddataisstoredinanother

place.Therefore,morethanonenon-clusteredindexisallowedpertable.Itrefers

tonon-primarykeys.

Asweknow,theprimarykeyrepresentsthecolumn,orsetofcolumns,whichismost

widelyusedforfetchingrecordsfromthe table.Theprimarykeyhasan indexassociated

withitandisusedforfastqueryperformance.Itprovidescomparativelyfasterperformance

becauseaprimarykeydoesnotallowaNULLvalue,sonocheckisrequiredonNULLvalues. It is recommended that if your table does not have a columnor set ofcolumns

to define as a primary key, then you define one auto increment field as theprimary

key for better performance. On the other hand, if your table contains manycolumns

andthereisaneedtoexecuteaquerywithacombinationofmultiplecolumns,then

it is advisable to the less frequently used data and transfer in onto a separatetable.

Relatealltheseparatetableswithprimaryandforeignkeyreferences,whichwill

helpyouinmanagingdata,andqueryretrievalprovidesyougoodperformance.

<strong>SELECTMIN(key_part2),MAX(key_part2)FROMtble_nameWHERE<br/>key_part1=10;</strong>

SQL commands related toindexesMySQL 8 provides two main commands related to indexes. We will discussthesecommandsinthefollowingsections.

<strong>CREATE[UNIQUE|FULLTEXT|SPATIAL]INDEXindex_name</strong><br/><strong>[index_type]</strong><br/><strong>ONtbl_name(index_col_name,...)</strong><br/><strong>[index_option]</strong><br/><strong>[algorithm_option|lock_option]...</strong><br/><strong>index_col_name:</strong><br/><strong>col_name[(length)][ASC|DESC]</strong><br/><strong>index_option:</strong><br/><strong>KEY_BLOCK_SIZE[=]value</strong><br/><strong>|index_type</strong><br/><strong>|WITHPARSERparser_name</strong><br/><strong>|COMMENT'string'</strong><br/><strong>|{VISIBLE|INVISIBLE}</strong><br/><strong>index_type:</strong><br/><strong>USING{BTREE|HASH}</strong><br/><strong>algorithm_option:</strong><br/><strong>ALGORITHM[=]{DEFAULT|INPLACE|COPY}</strong><br/><strong>lock_option:</strong><br/><strong>LOCK[=]{DEFAULT|NONE|SHARED|EXCLUSIVE}</strong>

Usingcol_name(length)syntax,theuserisabletospecifyanindexprefixlength,whichwillconsideronlyaspecifiednumberofcharactersfromthestringvalue.Atthetimeofdefining,theprefixconsidersthefollowingpoints:

TheprefixisoptionalforCHAR,VARCHAR,BINARY,andVARBINARYcolumnindexesTheprefixmustbespecifiedinthecaseofBLOBandTEXTcolumnindexesMySQL8willconsiderprefixesasanumberofcharactersfornon-binarystringtypes(CHAR,VARCHAR,TEXT)andanumberofbytesforbinarytypes(BINARY,VARBINARY,BLOB)Theprefixisnotallowedforthespatialcolumns

Adetailedexampleoftheprefixoptionisdescribedlaterinthischapter,undertheColumnindexessection.AUNIQUEindexisa

constraintwhichindicatesthatallthevaluesintheindexwillbeunique.Ifyoutrytoaddvalueswhichalreadyexist,thenMySQL8displaysanerror.AlltypesofstorageengineswithaUNIQUEindexpermitmultiplenullvalues.InthecaseofprefixeswhenyouuseNULLvalues,makesurecolumnvaluesareuniquewithintheprefixes.Ifanindexprefixexceedsitssize,thenMySQL8willhandletheindexasfollows:

Foranon-uniqueindex:IfthestrictSQLmodeisenabled,thenMySQL8willthrowanerror,andifthestrictmodeisdisabled,thentheindexlengthisreducedtothemaximumcolumndatatypesizeandwillproduceawarning.Forauniqueindex:Inthiscase,MySQL8producesanerrorregardlessoftheSQLmode,becauseitmightbreaktheuniquenessofthecolumn.Thismeansyouhavedefinedacolumnwith25lengthandtriedtodefineanindexonthesamecolumnwithaprefixlength27;thenMySQL8throwsanerror.

Spatialindexcharacteristics

MySQL8followsthefollowingrulesforspatialindexcharacteristics:

ItisonlyavailableforInnoDBandMyISAMstorageengines;ifyoutrytouseitforotherstorageengines,thenMySQL8givesanerror.

ANULLvalueisnotallowedforindexedcolumns.

A prefix attribute is not allowed for this column. Full width will beconsideredfortheindex.

<strong>createtableemployee(idint(11)notnull,namevarchar(50));</strong><br/><strong>CREATEINDEXemp_name_indexONemployee(name)USINGBTREE;</strong>

<strong>CREATEINDEXname_indexONemployee(name)COMMENT<br/>'MERGE_THRESHOLD=40';</strong>

Ifthepage-fullpercentageforanindexislessthantheMERGE_THRESHOLDvalue,thentheInnoDBstorageenginewillmergetheindexpagewithaneighboringindexpage.

VISIBLE,INVISIBLE:Thisparameterdefinestheindexvisibility.Bydefault,allindexesarevisible.Theoptimizerwillnotuseaninvisibleindexduringtheoptimizationprocess.

TheALGORITHMandLOCKattributeswillhaveanimpactwhenyoutrytousethetableforreadingorwriting,andwhilesimultaneouslymodifyingitsindexing.

<strong>DROPINDEXindex_nameONtbl_name</strong><br/><strong>[algorithm_option|lock_option]...</strong><br/><strong>algorithm_option:</strong><br/><strong>ALGORITHM[=]{DEFAULT|INPLACE|COPY}</strong><br/><strong>lock_option:</strong><br/><strong>LOCK[=]{DEFAULT|NONE|SHARED|EXCLUSIVE}</strong>

<strong><spanclass="tokenkeyword">DROP</span><spanclass="tokenkeyword">INDEX</span>name_index<spanclass="tokenkeyword">ONemployee;</span></strong>

<strong>CREATETABLEgeom_data(dataGEOMETRYNOTNULL,SPATIALINDEX(data));</strong>

<strong>SELECTDISTINCTST_SRID(column_name)FROMtable_name;</strong>

Ifthequeryreturnsmultiplerows,thenitindicatesthatthecolumncontainsmixedSRIDs.Ifso,changethecontentsofthecolumnforthesameSRIDvalue.DefineanexplicitSRIDforthecolumn.RecreatetheSPATIALINDEX.

InnoDB and MyISAM indexstatisticscollection

MySQL 8 will consider table statistics based on the value group, which isnothing but a set of rows with the same prefix values. The storage enginecollectsstatisticsrelatedtothetable,whichareusedbytheoptimizer.Fromtheoptimizationperspective,averagevaluegroupsizeisanimportantstatistic.Iftheaveragevalueofthegroupsizeincreases,thentheindexisnotmeaningful.

So, it's better to target a small number of rows for each index. This can beachievedbytablecardinality,whichisnothingbutanumberofvaluegroup.ForInnoDB and MyISAM tables, MySQL 8 provides control on statistics by themyisam_stats_methodand theinnodb_stats_method systemvariables.The followingarethepossiblevaluesforthesevariables:

nulls_ignored:ItindicatesthatNULLvaluesareignored

nulls_equal:ItindicatesallNULLvaluesareidentical

nulls_unequal:ItindicatesallNULLvaluesarenotidentical

The innodb_stats_method system variable has a global value, while themyisam_stats_methodsystemvariablehasbothglobalandsessionvalues.Whenwesetaglobalvalueofavariable,itwillaffectstatisticscollectionfortablesfromthe corresponding storage engine. In the case of session value statistics,collectionisavailableonlyforthecurrentclientconnection.Thismeansthatyouhave to regenerate a table's statistics for the other client on the same tablewithoutaffectingotherclients,andneedtosetitinasessionvalue.ToregenerateMyISAMstatistics,useeitherofthesemethods:

Executethemyisamchk--stats_method=method_name--analyzecommand

Make changes to the table tomake its statistics out of date, and then setmyisam_stats_methodandissueanANALYZETABLEstatement

Therearesomepointsthatmustbeconsideredbeforeusingthesetwovariables:

These variables are available only for InnoDB and MyISAM tables. For otherstorageengines,onlyonemethodisavailabletocollecttablestatistics,anditisveryneartothenulls_equalmethod.

MySQL8providesawaytogeneratestatisticsforatableexplicitly,butthisisnot always thecase.Sometimes,MySQL8mayalsogenerate statisticsautomatically,ifitisrequired.Forexample,inthecaseofanyoperations,ifsomeoftheSQL

statementsmodified table data, thenMySQL 8will automatically collectstatistics.

Considerbulkinsertordeleteoperations.

Wecannotsaywhichmethodwasusedtogeneratestatisticsforatable.

Column-levelindexingMySQL 8 allows you to create an index on a single column, as well as onmultiplecolumns.

Themaximumnumberofindexespertableandmaximumindexlengthdependonthestorageengine.Mostly,all thestorageenginesallowat least16indexesper table and total index lengthsof at least256bytes,butmostof the storageenginespermithigherlimits.

ColumnindexesThisisthemostcommonwaytodefineanindexwhereonlyasinglecolumnisinvolved.

MySQL8storesacopyofcolumnvaluesinadatastructuresothatrowscanbeaccessed

quickly.MySQL8usesaB-Treedatastructuretoenablevaluestobeaccessedquickly.TheB-Treeexecutionwill

workbasedonoperators,suchas=,<,>,BETWEEN,IN,andmanymore,whichweredefined in thewhere condition.Youcangetdetailson theB-Treedata structureanditsexecutionin

thenexttopic.Wewilldiscussthecharactersofcolumnindexesinthecomingsections.

<strong>CREATETABLEperson(personal_dataTEXT,INDEX(personal_data(8)));</strong>

Thiscommandcreatesanindexonthepersonal_datacolumnbyconsideringthefirsteightcharacters.Theprefixlengthvaries,basedonthestorageengine.TheInnoDBstorageengineallowsuptoa767byteprefixlengthforREDUNDANTorCOMPACTrowformats,whileinthecaseofDYNAMICorCOMPRESSEDrowformats,itallowsupto3072bytes.InthecaseoftheMyISAMstorageengine,theprefixcanbedefinedforupto1,000bytes.

Prefixlengthwillbemeasuredinbytesforbinarystringtypes,suchasBINARY,VARBINARY,andBLOB,whileinthecaseofnon-binarystringtypes,itwillbeconsideredasanumberofcharacters.

FULLTEXTindexes

Asthenameimplies,FULLTEXT indexesallowCHAR,VARCHAR,andTEXTcolumnsonly.This index is supported by InnoDB and MyISAM storage engines. In this type,indexingwilltakeplaceonanentirecolumn,ratherthanaprefixlength.MySQL8evaluates the full text expressionduring theoptimizationphaseof thequeryexecution. Before making estimations, optimization evaluates the full textexpressionintheprocessofdevelopinganexecutionplan.Asaresult,theEXPLAINqueryoffulltextisslowerthannon-fulltextqueries.Fulltextqueriesareusefulinthefollowingscenarios:

WhenaFULLTEXTqueryreturnseitheradocumentIDoradocumentIDandsearchrank

When a FULLTEXT query sorts the matching rows by descending order andusesaLIMITclausetofetchNnumberofrows,applyonlyasingleORDERBYclauseindescendingorder,anddon'tuseaWHEREclauseinitforoptimization

When a FULLTEXT query fetches a COUNT(*) value from the rowswithout anyadditional WHERE clauses, apply the WHERE clause as WHERE MATCH(text) AGAINST('other_text'),withoutany>0comparisonoperator

SpatialIndexesMySQL 8 allows you to create indexes on spatial data types. The InnoDB andMyISAM storage engines support R-Tree for spatial data, while other storageengines useB-Tree. SinceMySQL 5.7, spatial indexes are supported in MyISAMandInnoDBdatabaseengines.

Indexes in the MEMORY storageengineMemorystorageenginessupportbothHASHindexesandB-Treeindexes,butHASHindexesare,bydefault,setfortheMEMORYstorageengine.

<strong>CREATETABLEEmployee(</strong><br/><strong>idINTNOTNULL,</strong><br/><strong>lastnamevarchar(50)notnull,</strong><br/><strong>firstnamevarchar(50)notnull,</strong><br/><strong>PRIMARYKEY(id),</strong><br/><strong>INDEXname(lastname,firstname)</strong><br/><strong>);</strong>

<strong>SELECT*FROMEmployeeWHERElastname='Shah';</strong><br/><strong>SELECT*FROMEmployeeWHERElastname='Shah'ANDfirstname='Mona';</strong><br/><strong>SELECT*FROMEmployeeWHERElastname='Shah'AND(firstname='Michael'ORfirstname='Mona');</strong><br/><strong>SELECT*FROMEmployeeWHERElastname='Shah'ANDfirstname>='M'ANDfirstname<'N';</strong>

<strong>SELECT*FROMEmployeeWHEREfirstname='Mona';</strong><br/><strong>SELECT*FROMEmployeeWHERElastname='Shah'ORfirstname='Mona';</strong>

Remember,inthecaseofamultiple-columnindex,anyleft-mostprefixoftheindexcanbeusedbytheoptimizerforsearchingrows.Forexample,iftheindexwasdefinedonthreecolumnsinsequence,column1,column2,andcolumn3,thenyoucanuseindexedcapabilitieson(column1,column2,column3),(column1),(column1,column2),bydefiningitintheWHEREclause.

<strong>CREATETABLEEmployee(</strong><br/><strong>lastnamevarchar(50)notnull,</strong><br/><strong>firstnamevarchar(50)notnull,</strong><br/><strong>dobdatenotnull,</strong><br/><strong>genderchar(1)notnull,</strong><br/><strong>key(lastname,firstname,dob)</strong><br/><strong>);</strong>

Asperthetabledefinitionindexwillcontainsvaluesincombinationofthreecolumnsfirstname,lastname,anddob.Itwillsortvaluesaspertheordergivenpreviously;thismeansthatifsomeemployeeshavesimilarnames,thentheywillbesortedbytheirbirthdates.ConsiderthefollowingtypesofquerieswhichwillbenefitfromtheB-Treeindex:

Matchtherangeofvalues:FindstheemployeeswhoselastnamesarePatelandGupta.Matchthefullvalues:FindsemployeewhosenameisMohanPatelandwasbornon28/11/1981.Matchleftmostprefix:Findsallemployeewithalastname.Theseuseonlythefirstcolumnintheindex.Matchwithcolumnprefix:FindstheemployeeswhoselastnamesbeginwithM.Itusesonlythefirstcolumnintheindex.Matchonepartexactlyandwithrangeofanothercolumn:FindstheemployeewhoselastnameisPatelandwhosefirstnamestartswithA.

ThefollowingarequerieswhereB-Treeisnotuseful:

Don'tuseanyconditionafterarangecondition:Forexample,youhaveputtheWHEREconditionlastname='Patel'andfirstnamelike‘A%'anddob='28/11/1981'.Here,onlythefirsttwocolumnsareconsideredfortheindex,becauseLIKEisarangecondition.

Don'tskipanyofthecolumnsdefinedintheindex:ThismeansyouarenotallowedtouselastnameanddobtofindanemployeewherefirstnameismissingintheWHEREcondition.Lookupdoesn'tstartwithleft-mostsideofindexedcolumns:Forexample,theindexwillnotworkifyoufindtheemployeewhosefirstnameisMohanandwhosedobisonacertaindate.Inthisquery,definedcolumnsarenotleft-mostintheindex.Inthesameway,theindexdoesnotworkinthecasewhereyoufindtheemployeewhoselastnameendswithsomething.

Hashindex

It is very difficult to find a single value from a large database by followingcompletetreetraversalswithmultiplelevels.Toovercomethisproblem,MySQLhasprovidedanother index type,which isknownasahash index.This indexcreatesahashtableratherthanatree,whichisveryflatinstructurecomparedtoaB-Treeindex.Hashingmainlyuseshashfunctionstogeneratetheaddressesofdata.Twoimportanttermsrelatedtohashingare:

Hashfunction:Themappingfunctionwhichwillbeusefultomapsearch-keyswiththeaddresswhereactualrecordsarestored.

Bucket:Abucketisaunitofstoragewhereahashindexstoresthedata.Abucket indicates a complete disk block, which will store one or morerecords.

Along with the hashing mechanism, a hash index has some specialcharacteristics,describedasfollows:

Thewholekeyisusedtosearchtherow.WhileinthecaseofaB-Tree,onlytheleft-mostprefixofthekeyisusedtofindrows.

TheoptimizerwillnotuseahashindextospeedupORDERBYoperations.Inotherwords,thisindexisneverusedtofindthenextentry.

Hashindexesareusedforequalitycomparisonbyusing=or<=>operators.Itwillneverusecomparisonoperatorswhichwillreturnarangeofvalues.

Forexample,the<(lessthan)operator.

The range optimizer cannot actually gauge howmany rows are availablebetweenthetwovalues.And,ifweuseahash-indexedMEMORYtableinsteadofInnoDBorMyISAM,thenitmayaffectthequeriesaswell.

<strong>SEToptimizer_switch='use_index_extensions=off';</strong>

<strong>CREATETABLEtable1(</strong><br/><strong>c1INTNOTNULLDEFAULT0,</strong><br/><strong>c2INTNOTNULLDEFAULT0,</strong><br/><strong>d1DATEDEFAULTNULL,</strong><br/><strong>PRIMARYKEY(c1,c2),</strong><br/><strong>INDEXkey1(d1)</strong><br/><strong>)ENGINE=InnoDB;</strong><br/><br/><strong>--Insertvaluesintotable</strong><br/><strong>INSERTINTOtable1VALUES</strong><br/><strong>(1,1,'1990-01-01'),(1,2,'1991-01-01'),</strong><br/><strong>(1,3,'1992-01-01'),(1,4,'1993-01-01'),</strong><br/><strong>(1,5,'1994-01-01'),(2,1,'1990-01-01'),</strong><br/><strong>(2,2,'1991-01-01'),(2,3,'1992-01-01'),</strong><br/><strong>(2,4,'1993-01-01'),(2,5,'1994-01-01'),</strong><br/><strong>(3,1,'1990-01-01'),(3,2,'1991-01-01'),</strong><br/><strong>(3,3,'1992-01-01'),(3,4,'1993-01-01'),</strong><br/><strong>(3,5,'1994-01-01'),(4,1,'1990-01-01'),</strong><br/><strong>(4,2,'1991-01-01'),(4,3,'1992-01-01'),</strong><br/><strong>(4,4,'1993-01-01'),(4,5,'1994-01-01'),</strong><br/><strong>(5,1,'1990-01-01'),(5,2,'1991-01-01'),</strong><br/><strong>(5,3,'1992-01-01'),(5,4,'1993-01-01'),</strong><br/><strong>(5,5,'1994-01-01');</strong>

<strong>--Indexextensionissetasoff</strong><br/><strong>SEToptimizer_switch='use_index_extensions=off';</strong><br/><br/><strong>--Executeselectquerywithexplain</strong><br/><strong>EXPLAINSELECTCOUNT(*)FROMtable1WHEREc1=3ANDd1='1992-01-01';</strong><br/><br/><strong>--Outputofexplainquery</strong><br/><strong>***************************1.row***************************</strong><br/><strong>id:

1</strong><br/><strong>select_type:SIMPLE</strong><br/><strong>table:table1</strong><br/><strong>type:ref</strong><br/><strong>possible_keys:PRIMARY,key1</strong><br/><strong>key:PRIMARY</strong><br/><strong>key_len:4</strong><br/><strong>ref:const</strong><br/><strong>rows:5</strong><br/><strong>Extra:Usingwhere</strong>

<strong>--Indexextensionissetason</strong><br/><strong>SEToptimizer_switch='use_index_extensions=on';</strong><br/><br/><strong>--Executeselectquerywithexplain</strong><br/><strong>EXPLAINSELECTCOUNT(*)FROMtable1WHEREc1=3ANDd1='1992-01-01';</strong><br/><br/><strong>--Outputofexplainquery</strong><br/><strong>***************************1.row***************************</strong><br/><strong>id:1</strong><br/><strong>select_type:SIMPLE</strong><br/><strong>table:table1</strong><br/><strong>type:ref</strong><br/><strong>possible_keys:PRIMARY,key1</strong><br/><strong>key:key1</strong><br/><strong>key_len:8</strong><br/><strong>ref:const,const</strong><br/><strong>rows:1</strong><br/><strong>Extra:Usingindex</strong>

Now,wewillcheckthedifferencebetweenthesetwoapproaches:

Thekey_lenvaluechangesfrom4bytesto8bytes,whichindicatesthatkeylookupsuseboththecolumnsd1andc1,notonlyd1.Therefvaluechangesfrom(const)to(const,const)whichindicateskeylookupusestwokeypartsinsteadofone.Therowscountchangesfrom5to1,whichindicatesthatInnoDBrequiresfewerrowsthanthefirstapproachtoproducetheresult.TheExtravaluechangesfromUsingwheretoUsingindex.Itindicatesthattherowscanbereadbyusingonlytheindex,

withoutconsultinganyothercolumnsinthedatarow.

<strong>CREATETABLEt1(c1INT,c2INTAS(c1+1)STORED,INDEX(c2));</strong>

<strong>SELECT*FROMt1WHEREc1+1>100;</strong>

<strong>mysql>explainSELECT*FROMt1WHEREc1+1>100;</strong><br/><strong>***************************1.row***************************</strong><br/><strong>id:1</strong><br/><strong>select_type:SIMPLE</strong><br/><strong>table:t1</strong><br/><strong>partitions:NULL</strong><br/><strong>type:range</strong><br/><strong>possible_keys:c2</strong><br/><strong>key:c2</strong><br/><strong>key_len:5</strong><br/><strong>ref:NULL</strong><br/><strong>rows:1</strong><br/><strong>filtered:100.00</strong><br/><strong>Extra:Usingindexcondition</strong>

<strong>nameTEXTAS(JSON_EXTRACT(emp,'$.name'))STORED</strong>

<strong>nameTEXTAS(JSON_UNQUOTE(JSON_EXTRACT(emp,'$.name')))STORED</strong>

<strong>c2INTAS(c1)STOREDincolumndefinition.</strong>

Useaindexhintiftheoptimizertriestousethewrongindex,whichwilldisableditandforcetheoptimizertouseadifferentchoice

Invisibleanddescendingindexes

Theinvisibleindexisaspecialfeaturewhichwillmarkanindexasunavailablefortheoptimizer.MySQL

8 will maintain invisible indexes and keep them up-to-date when data ismodified.

This will apply on indexes other than primary key. As we know, indexes arevisible by default; we have to make them invisible explicitly at the time ofcreation, or by using the alter command. MySQL 8 provides the VISIBLE andINVISIBLEkeywordstomaintainindexvisibility.Adescendingindexisthemethodofstoringkeyvaluesindescendingorder.Adescendingindexismoreefficient,asitcanbescannedintheforwardorder.Let'sseetheseindexesindetail,withexamples.

InvisibleindexAsmention previously, an invisible index is not used by the optimizer. Thenwhatistheuseofthisindex?Thisquestioncomesintoourmind,right?Wewillexplaintoyousomeoftheusecasesforinvisibleindexes:

Whenmanyindexesaredefined,butyouarenotsurewhichindexisnotinuse. In this case, you can make an index invisible and check theperformance impact. If it has an impact, then you can make that indexvisibleonanimmediatebasis.Special cases where only one query is using an index. In this case, aninvisibleindexisagoodsolution.

In the following example,wewill create an invisible index using CREATE TABLE,CREATEINDEX,orALTERTABLEcommands:

CREATETABLE`employee`(

`id`int(11)NOTNULLAUTO_INCREMENT,

`department_id`int(11),

`salary`int(11),

PRIMARYKEY(`id`)

)ENGINE=InnoDB;

CREATEINDEXidx1ONemployee(department_id)INVISIBLE;

ALTERTABLEemployeeADDINDEXidx2(salary)INVISIBLE;

Tochangethevisibilityoftheindexes,usethefollowingcommands:

ALTERTABLEemployeeALTERINDEXidx1VISIBLE;

ALTERTABLEemployeeALTERINDEXidx1INVISIBLE;

Togetinformationaboutanindex,executeINFORMATION_SCHEMA.STATISTICStableorSHOWINDEXcommandsinthefollowingways:

mysql>SELECT*FROMinformation_schema.statisticsWHEREis_visible='NO';

***************************1.row***************************

TABLE_CATALOG:def

TABLE_SCHEMA:db1

TABLE_NAME:employee

NON_UNIQUE:1

INDEX_SCHEMA:db1

INDEX_NAME:idx1

SEQ_IN_INDEX:1

COLUMN_NAME:department_id

COLLATION:A

CARDINALITY:0

SUB_PART:NULL

PACKED:NULL

NULLABLE:YES

INDEX_TYPE:BTREE

COMMENT:

INDEX_COMMENT:

IS_VISIBLE:NO

mysql>SELECTINDEX_NAME,IS_VISIBLEFROMINFORMATION_SCHEMA.STATISTICS

->WHERETABLE_SCHEMA='db1'ANDTABLE_NAME='employee';

+------------+------------+

|INDEX_NAME|IS_VISIBLE|

+------------+------------+

|idx1|NO|

|idx2|NO|

|PRIMARY|YES|

+------------+------------+

mysql>SHOWINDEXESFROMemployee;

***************************1.row***************************

Table:employee

Non_unique:1

Key_name:idx1

Seq_in_index:1

Column_name:department_id

Collation:A

Cardinality:0

Sub_part:NULL

Packed:NULL

Null:YES

Index_type:BTREE

Comment:

Index_comment:

Visible:NO

MySQL 8 provides a use_invisible_indexes flag in the optimizer_switch systemvariable tocontrol invisible indexesusedbythequeryoptimizer.If thisflag ison, then the optimizer uses invisible indexes in execution plan construction,while if the flag is off, the optimizer ignores invisible indexes. MySQL 8providesa facility to use an implicit primary key if you have defined a UNIQUEindexonaNOTNULLcolumn.Onceyoudefine the indexonthisfield,MySQL8doesnotallowyoutomakeitinvisible.Inordertounderstandthisscenario,let'stake one example with the following table. Let's try to execute the followingcommandtomaketheidx1indexinvisible:

CREATETABLEtable2(

field1INTNOTNULL,

field2INTNOTNULL,

UNIQUEidx1(field1)

)ENGINE=InnoDB;

Theserverwillnowgiveanerror,asshowninthefollowingcommands:

mysql>ALTERTABLEtable2ALTERINDEXidx1INVISIBLE;

ERROR3522(HY000):Aprimarykeyindexcannotbeinvisible

Nowlet'saddtheprimarykeyintothetable,usingthefollowingcommand:

ALTERTABLEtable2ADDPRIMARYKEY(field2);

Now,wewill try tomaketheidex1 invisible.This time, theserverallowsit,asshowninthefollowingcommands:

mysql>ALTERTABLEtable2ALTERINDEXidx1INVISIBLE;

QueryOK,0rowsaffected(0.06sec)

Records:0Duplicates:0Warnings:0

<strong>CREATETABLEt1(</strong><br/><strong>aINT,bINT,</strong><br/><strong>INDEXidx1(aASC,bASC),</strong><br/><strong>INDEXidx2(aASC,bDESC),</strong><br/><strong>INDEXidx3(aDESC,bASC),</strong><br/><strong>INDEXidx4(aDESC,bDESC)</strong><br/><strong>);</strong>

<strong>ORDERBYaASC,bASC--optimizercanuseidx1</strong><br/><strong>ORDERBYaDESC,bDESC--optimizercanuseidx4</strong><br/><strong>ORDERBYaASC,bDESC--optimizercanuseidx2</strong><br/><strong>ORDERBYaDESC,bASC--optimizercanuseidx3</strong>

<strong>Query1:SELECT*FROMt1ORDERBYaDESC;</strong><br/><strong>Query2:SELECT*FROMt1ORDERBYaASC;</strong><br/><strong>Query3:SELECT*FROMt1ORDERBYaDESC,bASC;</strong><br/><strong>Query4:SELECT*FROMt1ORDERBYaASC,bDESC;</strong><br/><strong>Query5:SELECT*FROMt1ORDERBYaDESC,bDESC;</strong><br/><strong>Query6:SELECT*FROMt1ORDERBYaASC,bASC;</strong>

ThefollowingstatisticalgraphisprovidedbyMySQL8on10millionrows,withrespecttothepreviouslymentionedqueries:

Reference:https://mysqlserverteam.com/mysql-8-0-labs-descending-indexes-in-mysql/

Therearesomeimportantpointsthatyoushouldrememberatthetimeofusingdescendingindexes,whichareasfollows:

Allthedatatypessupportedbyascendingindexesarealsosupportedbydescendingindexes.DescendingindexesaresupportedforBTREEbutnotforHASH,FULLTEXT,andSPATIALindexes.IfyoutrytouseASCandDESCkeywordsexplicitlyforHASH,FULLTEXT,andSPATIALindexes,thenMySQL8willgenerateanerror.DescendingindexesonlysupportInnoDBstorageengines,buttheInnoDBSQLparserdoesnotusedescendingindexes.Changebufferingisnotsupportedforsecondaryindexesiftheprimarykeyincludesadescendingindex.DISTINCTcanuseanyindex,includingadescendingkey,butforMIN()/MAX(),nodescendingkeypartsareused.Bothnon-generatedandgeneratedcolumnsallowuseofdescendingindexes.

Summary

Everythingbecomesveryinterestingwhenyouareawareofhowitworks,right?Wehopeyouhavefoundthesamethingaboutindexinginthischapter.Wehavecovered very useful informationwhichwill help you to define indexes on theright columns to get better performance. In addition to that, we have alsodescribedvarioustypesofindexingwiththeirstoragestructures.

Inthenextchapter,wewillprovideyouwithinformationaboutreplication.Wewillexplaintheconfigurationandimplementationofreplicationindetail.

ReplicationinMySQL8

Inthepreviouschapter,wediveddeepintoMySQL8indexing.Indexesareanimportantentityforanydatabasemanagementsystem.TheyhelpimproveSQLquery performance by limiting the number of records to be visited. Databaseadministrators working on performance improvement must be aware of thisimportant technique. This chapter explains in detail the types of indexes andtheiradvantages.Thischapteralsoexplainshowindexingworks inMySQL8.It'sgoingtobeaprettyinformativechapter!

Moving further along the same line, in this chapter, wewill discuss databasereplication.

Howmucharewealreadyawareaboutdatabasereplication?Itdoesn'tactuallymatter.

This chapter covers insightful details about database replication. If you havepriorknowledgeofdatabasereplication,thischapterwilladdtoit.Ifyouhaveonly just heard about it for the first time, you will find every detail that isrequired to make it work in this chapter. So, are we ready to jump in? Thefollowingisalistoftopicsthatwewillbecoveringinthischapter:

Overviewofreplication

Configuringreplication

Implementingreplication

Groupreplicationversusclustering

Replicationsolutions

OverviewofreplicationWewillwalkthroughthebasicsofdatabasereplicationinthissection.Wewillunderstandwhat replication is, theadvantages itprovides,and thescenarios inwhichreplicationcanbebeneficial.

<strong><spanclass="tokenconstant">GTID</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>source_id:transaction_id</span></strong>

<strong>1A22AF74<spanclass="tokenoperator">-</span>17AC<spanclass="tokenoperator">-</span>E111<spanclass="tokenoperator">-</span>393E<spanclass="tokenoperator">-</span>80C49AB653A2:1</strong>

Thesequencenumberfortransactionscommittedstartswith1.Itcanneverbe0.

AGTID-basedMySQLreplicationmethodistransactionalandsothisiswhyitismorereliablethanabinarylogfile-basedreplicationmethod.GTIDguaranteesthereplicationaccuracyandconsistencybetweenmasterandslavedatabasesaslongasallthetransactionscommittedonmasterdatabaseservershavealsobeenappliedonalloftheslavedatabaseservers.

Asmentionedearlier,MySQLdatabasereplicationisusuallyasynchronous.However,MySQL8supportsdifferenttypesofsynchronizationforreplication.Theusualmethodofsynchronizationisasynchronous.Itmeansoneserveractsasamasterdatabaseserver.Itwritesalleventstoabinarylogfile.Otherdatabaseserversactasslaves.Slavedatabaseserversreadandcopyposition-basedeventrecordswithinbinarylogfilesfromthemasterdatabaseserver.So,itisalwaysfromamasterdatabaseservertoaslavedatabaseserver.MySQL8alsosupportssemisynchronoussynchronizationmethods.Insemisynchronousmethodsofreplication,anytransactioncommittedonamasterdatabaseserverisblockeduntilthemasterdatabaseserverreceivesacknowledgementfromatleastoneoftheslavedatabaseserversthatithasreceivedandloggedthetransactionevent.Delayedreplicationisanotherreplicationmethodsupportedby

MySQL8.Indelayedreplication,slavedatabaseserversintentionallylogthetransactioneventbehindmasterdatabaseserversbysomeamountoftime.

AdvantagesofMySQLreplication

Aswe arenow familiarwithwhatMySQLdatabase replication is, it's time toassessiftheaddedcomplexityofmaintainingmultipledatabaseserversisworthitornot.

TheadvantagesofMySQL8databasereplicationareasfollows:

1. Scale out solutions: As described earlier, usually web applications areread-heavy applications. The read operations aremuch higher in numberthan the write operations. The applications provide features that requireheavy,complexSQLqueries tobeexecutedonthedatabaseserver.Thesearenotthequeriesthattakemillisecondstoexecute.Suchcomplexqueriesmaytakeafewsecondstominutesforexecution.Executionofsuchqueriesputupheavy loadondatabaseserver. Insuchcases, it isalwaysbetter tohave such read operations performed on a separate database server thanmasterdatabaseservers.

Write database operations will always be performed on master databaseservers.Doyouknowwhy?It'sbecauseittriggersdatabasemodifications.Events of these modifications must be written to binary log files for

replicationsynchronizationbyslaveservers.

Also, the synchronization is from master to slaves. So, if we performedwritedatabaseoperationsonslaves,thosewillneverbeavailableonmasterdatabaseservers.Thisapproachimprovesperformanceofwriteoperationswith increased speed for read operations as the read operations areperformedacrossthenumberofslaveservers.

2. DataSecurity:Security,ingeneral,isanimportantfeaturethateverywebapplicationneeds.

The security can on at an application layer or on a database layer. Datasecurityprotectsagainstlossofdata.Datasecurityisachievedbybackingup a database on a regular basis. If replication is not set up, backing upproduction databases requires the application to be put on maintenancemode. This is required because simultaneous access to a database by anapplicationandthebackupprocessmaycorruptthedata.

Withreplicationinplace,wecanuseoneoftheslavesforbackup.Astheslavedatabaseserverisalwaysinsynchronizationwiththemasterdatabaseserver,wecanbackuptheslavedatabaseserver.Forthat,wecanmaketheslavedatabaseserverstopreplicatingfromamasterdatabaseserverwhilethebackupprocessisrunning.Thisdoesn'trequirethewebapplicationtostop using amaster database server. In fact, it doesn't impact themasterdatabaseserverinanyway.Anotherdatasecurityaspectistoproviderole-basedaccesstoproductionormasterdatabaseservers.Wecanhaveonlyafewroleswhocanaccessthemasterdatabaseserverfromthebackend.Therest of the users or roles have access to the slave database server. Thisreducestheriskofaccidentaldatalossbecauseofhumanerror.

3. Analytics: Analytics and reporting are always important features for adatabasebackedapplication.

These features require fetching of information from a database on afrequent basis so that analysis on the data can be performed. If databasereplication issetup,wecanfetch thedata requiredforanalytics fromtheslave database server without affecting the master database serverperformance.

4. Long distance data distribution: It is a common requirement forapplication developers to replicate production data on local developmentenvironments. In a database replication enabled infrastructure, a slavedatabase server can be used to prepare database copy on a developmentdatabaseserverwithoutconstantaccesstothemasterdatabaseserver.

ConfiguringreplicationIn this section, we will learn configuration for different types of MySQL 8replication methods. It includes step by step instructions for setting up andconfiguringreplication.

Binarylogfilebasedreplication

OneofthemostcommontraditionalmethodsofMySQLdatabasereplicationisthebinarylogfilepositionmethod.Thissectionfocusesonconfigurationofthebinary log file position method of replication. Before we jump into theconfiguration section, itwouldbegood to revise andunderstand thebasics ofbinarylogpositionbasedreplication.

Asdescribedearlier,oneoftheMySQLdatabaseserversactsasmasterandtherestoftheMySQLdatabaseserversbecomeslaves.Themasterdatabaseserveristheoriginforthedatabasechanges.Themasterdatabaseserverwriteseventsbasedonupdatesorchangestothedatabaseinthebinarylogfile.Theformatofthe information recordbeingwritten in the binary log file varies basedon thedatabasechangebeingrecorded.

MySQL REPLICATION SLAVE database servers are configured so that they read thebinarylogeventsfromthemasterdatabaseserver.Slavesexecutetheeventsonlocaldatabasebinarylogfiles.

Thiswayslavessynchronizethedatabasewiththemasterdatabase.Whenslavedatabaseserversreadthebinarylogfilefromthemasterdatabaseserver,slavesgetanentirecopyofthebinarylogfile.Oncethebinarylogfileisreceived,itis

up to the slaves todecidewhichstatements toexecuteon the slavebinary logfile.Itispossibletospecifythatallstatementsfromthemasterdatabaseserverbinarylogfileshouldbeexecutedontheslavedatabaseserversbinarylogfile.Itisalsopossibletoprocesseventsfilteredbyparticulardatabasesortables.

Onlyslavedatabaseserverscanbeconfiguredtofiltereventsfrommasterdatabaseserverlogfiles.Itisnotpossibletoconfigurethemasterdatabaseservertologonlyspecificevents.

MySQL 8 provides a system variable that helps identify the database serveruniquely.

AllthedatabaseserversparticipatinginMySQLreplicationmustbeconfiguredtohaveauniqueID.Eachoftheslavedatabaseserversmustbeconfiguredwiththemasterdatabaseserverhostname,logfilename,andpositionwithinthelogfile.Once set up, it is possible tomodify these details fromwithin aMySQLsessionusingtheCHANGEMASTERTOstatementexecutedontheslavedatabaseserver.

Whentheslavedatabaseserverreadstheinformationfromthemasterdatabasebinary log file, it keeps track of a record of the binary log coordinates. Thebinary log coordinates consists of the filename and position within the file,whichisreadandprocessedfromthemasterdatabaseserver.Theefficiencyofslavedatabaseserversreadingthebinarylogfilefromthemasterdatabaseserveris very high because multiple slave database servers can be connected to themasterdatabaseserverandprocessdifferentpartsofthebinarylogfilefromthemasterdatabaseserver.Themasterdatabaseserveroperationsremainunaffectedbecause the connecting and disconnecting of slave database servers from themasterdatabaseserveriscontrolledbyslavesthemselves.Asmentionedearlier,eachslavedatabaseserverkeepstrackofthecurrentpositionwithinthebinarylog file. So, it is possible for the slave database server to disconnect andreconnect with the master database server and resume the binary log fileprocessing.

A number of methods for setting up database replication are available inMySQL.Theexactmethod for replicationdependson ifdataalreadyexists inthedatabaseandhowreplicationisbeingsetup.EachofthefollowingsectionsareastepforconfiguringMySQLreplication.

ReplicationmasterconfigurationBeforewesetupthereplicationmasterdatabaseserver,itmustbeensuredthatthedatabaseserverhasauniqueIDestablishedandbinaryloggingisenabled.Itmay be required to restart the database server after these configurations aremade.ThemasterdatabaseserverbinarylogisthebasisforMySQL8databasereplication.

Toenablebinarylogging,thelog_binsystemvariableshouldbesettoON.BinaryloggingisenabledforaMySQLdatabaseserverbydefault. Ifmysqld isused toinitialize the data directory manually with a --initialize or --initialize-insecureoption, the binary logging is disabled by default. It has to be enabled byspecifyingthe--log-binoption.The--log-binoptionspecifiesthebasenametobeusedforthebinarylogfiles.

Ifthefilenameisnotspecifiedwiththestartupoption,thebinarylogfilenameswillbe setbasedon thedatabase serverhostname. It is recommended that thebinarylogfilename isspecifiedwith the--log-binoption. If the logfilename isspecifiedwith --log_bin=old_host_name-bin, the log filenamewill be retained evenafterthedatabaseserverhostischanged.

Tosetupthemasterdatabaseserver,opentheMySQLconfigurationfileonthemasterdatabaseserver:

sudovim/etc/mysql/my.cnf

Intheconfigurationfile,makethefollowingchanges.

Firstofall,findthesectionthatbindstheservertolocalhost:

bind-address=127.0.0.1

ReplacethelocalIPaddresswiththeactualdatabaseserverIPaddress.ThisstepisimportantbecausetheslavescanaccessthemasterdatabaseserverusingthepublicIPaddressofthemasterdatabaseserver:

bind-address=175.100.170.1

The following changes are required to configure a unique ID for the masterdatabase server. It also includes the configuration required for setting up themasterbinarylogfile:

[mysqld]

log-bin=/var/log/mysql/mysql-bin.log

server-id=1

Now,let'sconfigurethedatabasetobereplicatedontheslavedatabaseservers.Ifmorethanonedatabaseisrequiredtobereplicatedonslavedatabaseservers,repeatthefollowinglinemultipletimes:

binlog_do_db=database_master_one

binlog_do_db=database_master_two

Once these changes are done, restart the database server using the followingcommand:

sudoservicemysqlrestart

Now, we have the master database server set up. The next step is to grantprivilegestotheslaveuserasfollows:

mysql>mysql-uroot-p

mysql>CREATEUSER'slaveone'@'%'IDENTIFIEDBY'password';

mysql>GRANTREPLICATIONSLAVEON*.*TO'slaveone'@'%'IDENTIFIEDBY'password';

Theprecedingcommandscreatestheslaveuser,grantsprivilegesonthemasterdatabaseserver,andflushesdatabasecachedprivileges.

Now,wehavetobackupthedatabasethatwewanttoreplicate.Wewillbackupthedatabaseusingthemysqldumpcommand.Thisdatabasewillbeusedforcreatingtheslavedatabase.Themasterstatusoutputdisplaysthenameofthebinarylogfilename,currentposition,andthenameofthedatabasetobereplicated:

mysql>USEdatabase_master_one;

mysql>FLUSHTABLESWITHREADLOCK;

mysql>SHOWMASTERSTATUS;

+------------------+----------+---------------------+------------------+

|File|Position|Binlog_Do_DB|Binlog_Ignore_DB|

+------------------+----------+---------------------+------------------+

|mysql-bin.000001|102|database_master_one||

+------------------+----------+---------------------+------------------+

1rowinset(0.00sec)

mysqldump-uroot-pdatabase_master_one>database_master_one_dump.sql

Beforewe take the database backup using the mysqldump command,we have tolock the database to check the current position.This informationwill be usedlatertosetuptheslavedatabaseserver.

After the database dump is taken, the database should be unlocked using thefollowingcommands:

mysql>UNLOCKTABLES;

mysql>QUIT;

Weare donewith all the configuration required to set up a replicationmasterdatabaseserverandmakeitaccessiblebytheREPLICATIONSLAVEdatabaseservers.

Thefollowingoptionshaveanimpactonthemasterdatabaseserversetup:

1. innodb_flush_log_at_trx_commit=1 and sync_binlog=1 options should be set toachieve higher durability and consistency. The options can be set in themy.cnfconfigurationfile.

2. The skip-networking optionmust not be enabled. If it is enabled, the slavecannotcommunicatewiththemasteranddatabasereplicationfails.

REPLICATION SLAVEconfigurationSimilar to themaster database server, each slave database servermust have auniqueID.Oncesetup,thiswillrequiredatabaseserverrestart:

[mysqld]

server-id=2

Forsettingupmultipleslavedatabaseservers,auniquenonzeroserver-idmustbe configured that is different from that ofmaster or anyother slavedatabaseservers.Binaryloggingonaslavedatabaseserverisnotrequiredforreplicationtobesetup.Ifenabled,abinarylogfileonaslavedatabaseservercanbeusedfordatabasebackupsandcrashrecovery.

Now,createanewdatabasethatwillbecomethereplicaofthemasterdatabaseand import the database from the database dump prepared from the masterdatabaseasfollows:

mysql>CREATEDATABASEdatabase_slave_one;

mysql>QUIT;

#mysql-uroot-pdatabase_slave_one&lt;/path/to/database_master_one_dump.sql

Now,wehavetoconfigureafewotheroptionsinthemy.cnffile.Likethebinarylog, a relay log consists of numbered files with database change events ascontentsof the file. It alsocontainsan index file thathas thenamesofall theused relay log files.The followingconfigurations set the relay log file, binarylogfile,andnameoftheslavedatabase,whichisareplicaofthemasterdatabaseasfollows:

relay-log=/var/log/mysql/mysql-relay-bin.log

log_bin=/var/log/mysql/mysql-bin.log

binlog_do_db=database_slave_one

A database server restart is required after this configuration change. The nextstepistoenableslavereplicationfromwithinMySQLshellprompt.Executethefollowingcommand toset themasterdatabase informationrequiredby theslavedatabaseserver:

mysql>CHANGEMASTERTOMASTER_HOST='12.34.56.789',MASTER_USER='slaveone',MASTER_PASSWORD='password',MASTER_LOG_FILE='mysql-bin.000001',MASTER_LOG_POS=103;

Asafinalstep,activatetheslaveserver:

mysql>STARTSLAVE;

If the binary logging is enabled on the slave database server, a slave canparticipateinacomplexreplicationstrategy.Insuchareplicationsetup,databaseserverA acts as amaster for database server B.B acts as a slave to the Amasterdatabaseserver.Now,BinturncanactasamasterdatabaseserverfortheCslavedatabaseserver.Somethinglikethiscanbeseenasfollows:

A->B->C

AddingslavestoreplicationIt is possible to add a new slave database server to an existing replicationconfiguration.Thisdoesn'trequirethemasterdatabaseservertobestopped.Theapproach shouldbe tomake a copy of an existing slave database server.Oncecopied,wehavetomodifythevalueforaserver-idconfigurationoption.

Thefollowinginstructionssetupanewslavedatabasetoanexistingreplicationconfiguration. First, an existing slave database server should be shut down asfollows:

mysql>mysqladminshutdown

Now,adatadirectoryfromtheexistingslaveshouldbecopiedtothenewslavedatabase server.Alongwith the data directory, binary logs and relay log filesmustbecopiedaswell.Itisrecommendedtousethesamevaluefor--relay-logforthenewslavedatabaseserverasthatoftheexistingslavedatabaseserver.

Ifthemasterinfoandrelayloginforepositoriesusefilesthenthosefilesmustbecopied from an existing slave database server to a new slave database server.These files hold a master's current binary log coordinates and a slave's relaylogs.

Now,starttheexistingslaveserverthatwasstoppedearlier.

Now,weshouldbeabletostartthenewslavedatabaseserver.Wemusthavetheuniqueserver-idconfiguredbeforestartingthenewslaveserver,ifitisnotsetupalready.

<strong>GTID=source_id:transaction_id</strong>

<strong>gtid_set:</strong><br/><strong>uuid_set[,uuid_set]...</strong><br/><strong>|''</strong><br/><br/><strong>uuid_set:</strong><br/><strong>uuid:interval[:interval]...</strong><br/><br/><strong>uuid:</strong><br/><strong>hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh</strong><br/><br/><strong>h:</strong><br/><strong>[0-9|A-F]</strong><br/><br/><strong>interval:</strong><br/><strong>n[-n]</strong><br/><strong>(n>=1)</strong>

<strong>[mysqld]</strong><br/><strong>server-id=1</strong><br/><strong>log-bin=mysql-bin</strong><br/><strong>binlog_format=ROW</strong><br/><strong>gtid_mode=on</strong><br/><strong>enforce_gtid_consistency</strong><br/><strong>log_slave_updates</strong>

<strong>>CREATEUSER'slaveuser'@'%'IDENTIFIEDBY'password';</strong><br/><strong>>GRANTREPLICATIONSLAVEON*.*TO'slaveuser'@'%'IDENTIFIED<br/>BY'password';</strong><br/><strong>>mysqldump-uroot-pdatabaseName>databaseName.sql</strong>

<strong>>mysql-uroot-pdatabaseName&lt;/path/to/databaseName.sql</strong>

<strong>[mysqld]</strong><br/><strong>server_id=2</strong><br/><strong>log_bin=mysql-bin</strong><br/><strong>binlog_format=ROW</strong><br/><strong>skip_slave_start</strong><br/><strong>gtid_mode=on</strong><br/><strong>enforce_gtid_consistency</strong><br/><strong>log_slave_updates</strong>

<strong>sudoservicemysqlrestart</strong>

<strong>>CHANGEMASTERTOMASTER_HOST='170.110.117.12',MASTER_PORT=3306,</strong><br/><strong>MASTER_USER='slaveuser',MASTER_PASSWORD='password',MASTER_AUTO_POSITION=1;</strong>

<strong>STARTSLAVE;</strong>

<strong>----GTIDstateatthebeginningofthebackup--</strong><br/><strong>mysql>SET@@GLOBAL.GTID_PURGED='b9b4712a-df64-11e3-b391-60672090eb04:1-7';</strong>

MySQLmulti-sourcereplicationThissectionfocusesonreplicatingfrommultipleimmediatemastersinparallel.The method is known as multi-source replication. With multi-sourcereplication,aREPLICATIONSLAVE receives transactionsfrommultiplesourcesat thesametime.AchanneliscreatedbyaREPLICATIONSLAVEforeachmasterfromwhichitshouldreceivetransactions.

Themulti-source replication configuration requires at least twomasters and aslavetobeconfigured.ThemasterscanbeconfiguredusingbinarylogpositionbasedreplicationorGTID-basedreplication.ReplicationrepositoriesarestoredinFILEorTABLEbasedrepositories.ATABLEbasedrepositoryiscrashsafe.MySQLmulti-sourcereplicationrequiresaTABLEbasedrepository.TherearetwowaystosetupaTABLErepository.

One is to start mysqld with options as follows: mysqld —master-info-repostiory=TABLE&&–relay-log-info-repository=TABLE

Another preferred way of doing this is to modify the my.cnf file as follows:[mysqld]master-info-repository=TABLErelay-log-info-repository=TABLE

ItispossibletomodifyanexistingREPLICATIONSLAVEthatisusingaFILErepositoryto use a TABLE repository. The following commands convert the existingrepositoriesdynamically:STOPSLAVE;SETGLOBALmaster_info_repository='TABLE';SETGLOBALrelay_log_info_repository='TABLE';

The following commands can be used to add a new GTID-based replicationmaster to an existing multi-source REPLICATION SLAVE. It adds a master to theexisting slave channel: CHANGE MASTER TOMASTER_HOST='newmaster', MASTER_USER='masteruser',MASTER_PORT=3451, MASTER_PASSWORD='password',MASTER_AUTO_POSITION=1FORCHANNEL'master-1';

Thefollowingcommandscanbeusedtoaddanewbinarylogfilepositionbasedreplicationmastertoanexistingmulti-sourceREPLICATIONSLAVE.Itaddsamastertothe existing slave channel: CHANGE MASTER TOMASTER_HOST='newmaster', MASTER_USER='masteruser',MASTER_PORT=3451, MASTER_PASSWORD='password'MASTER_LOG_FILE='master1-bin.000006', MASTER_LOG_POS=628FORCHANNEL'master-1';

ThefollowingcommandsSTART/STOP/RESETalltheconfiguredreplicationchannels:STARTSLAVEthread_types;--TostartallchannelsSTOPSLAVEthread_types;--TostopallchannelsRESETSLAVEthread_types;--Toresetallchannels

The following commands START/STOP/RESET a named channel using a FOR CHANNEL

clause:STARTSLAVEthread_typesFORCHANNELchannel;STOPSLAVEthread_typesFORCHANNELchannel;RESETSLAVEthread_typesFORCHANNELchannel;

<strong>mysql>SHOWSLAVESTATUS\G</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Slave_IO_State:Waitingformastertosendevent</strong><br/><strong>Master_Host:master1</strong><br/><strong>Master_User:root</strong><br/><strong>Master_Port:3306</strong><br/><strong>Connect_Retry:60</strong><br/><strong>Master_Log_File:mysql-bin.000004</strong><br/><strong>Read_Master_Log_Pos:931</strong><br/><strong>Relay_Log_File:slave1-relay-bin.000056</strong><br/><strong>Relay_Log_Pos:950</strong><br/><strong>Relay_Master_Log_File:mysql-bin.000004</strong><br/><strong>Slave_IO_Running:Yes</strong><br/><strong>Slave_SQL_Running:Yes</strong><br/><strong>Replicate_Do_DB:</strong><br/><strong>Replicate_Ignore_DB:</strong><br/><strong>Replicate_Do_Table:</strong><br/><strong>Replicate_Ignore_Table:</strong><br/><strong>Replicate_Wild_Do_Table:</strong><br/><strong>Replicate_Wild_Ignore_Table:</strong><br/><strong>Last_Errno:0</strong><br/><strong>Last_Error:</strong><br/><strong>Skip_Counter:0</strong><br/><strong>Exec_Master_Log_Pos:931</strong><br/><strong>Relay_Log_Space:1365</strong><br/><strong>Until_Condition:None</strong><br/><strong>Until_Log_File:</strong><br/><strong>Until_Log_Pos:0</strong><br/><strong>Master_SSL_Allowed:No</strong><br/><strong>Master_SSL_CA_File:</strong><br/><strong>Master_SSL_CA_Path:</strong><br/><strong>Master_SSL_Cert:</strong><br/><strong>Master_SSL_Cipher:</strong><br/><strong>Master_SSL_Key:</strong><br/><strong>Seconds_Behind_Master:0</strong><br/><strong>Master_SSL_Verify_Server_Cert:No</strong><br/><strong>Last_IO_Errno:0</strong><br/><strong>Last_IO_Error:

</strong><br/><strong>Last_SQL_Errno:0</strong><br/><strong>Last_SQL_Error:</strong><br/><strong>Replicate_Ignore_Server_Ids:0</strong>

<strong>mysql>SHOWPROCESSLIST\G;</strong><br/><strong>***************************4.row***************************</strong><br/><strong>Id:10</strong><br/><strong>User:root</strong><br/><strong>Host:slave1:58371</strong><br/><strong>db:NULL</strong><br/><strong>Command:BinlogDump</strong><br/><strong>Time:777</strong><br/><strong>State:Hassentallbinlogtoslave;waitingforbinlogtobeupdated</strong><br/><strong>Info:NULL</strong>

<strong>mysql>SHOWSLAVEHOSTS;</strong><br/><strong>+-----------+--------+------+-------------------+-----------+</strong><br/><strong>|Server_id|Host|Port|Rpl_recovery_rank|Master_id|</strong><br/><strong>+-----------+--------+------+-------------------+-----------+</strong><br/><strong>|10|slave1|3306|0|1|</strong><br/><strong>+-----------+--------+------+-------------------+-----------+</strong><br/><strong>1rowinset(0.00sec)</strong>

<strong>mysql>STOPSLAVE;</strong><br/><strong>mysql>STARTSLAVE;</strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">STOP</span><spanclass="tokenkeyword">SLAVE</span><spanclass="tokenkeyword">IO_THREAD</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">STOP</span><spanclass="tokenkeyword">SLAVE</span><spanclass="tokenkeyword">SQL_THREAD</span></strong><spanclass="tokenpunctuation"><strong>;</strong><br/><br/><strong><span

class="tokenprompt">mysql></span><spanclass="tokenkeyword">START</span><spanclass="tokenkeyword">SLAVE</span><spanclass="tokenkeyword">IO_THREAD</span>;<br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">START</span><spanclass="tokenkeyword">SLAVE</span><spanclass="tokenkeyword">SQL_THREAD</span>;</strong><br/></span>

Implementingreplication

Thebasisforreplicationisthatthemasterdatabaseserverkeepstrackofallthechanges taking place on the master database. The changes are tracked in thebinary log files in the form of events since the server was started. SELECToperationsarenotrecordedastheymodifyneitherthedatabasenorthecontents.EachoftheREPLICATIONSLAVEpullacopyofthebinarylogfilefrommasterinsteadofamasterdatabasepushingthelogfiletotheslave.Theslaveinturnexecutesthe events as it is read from the master's binary log file. This maintains theconsistencybetween master and slave servers. InMySQL replication, each slavefunctions independently from master and other slave servers. So, it is up to theslave to request the master's binary log file at a convenient time withoutimpactingthemasterorslavefunctioning.

The focus for this section of the chapter is onMySQL replication details.Wehavealreadyunderstoodthebasics,whichwillhelpusunderstandtheindepthdetails.

Replicationformats

As we already know by now,MySQL replication works based on replicatingeventsfromthemasterservergeneratedbinarylogs.Later,theseeventsarereadandprocessedbytheslave.Whatwedonotyetknowistheformatinwhichtheevents are recorded in binary log files.Replication formats is the emphasis ofthissection.

When the events are recorded in the master's binary log files, the replicationformat used depends on the binary log format used. Basically, two binaryloggingformatsexist:statementbasedandrowbased.

Withstatement-basedbinarylogging,SQLstatementsarewritteninthemaster'sbinarylogfile.ReplicationontheslaveworksbyexecutingtheSQLstatementson the slave database. This approach is called statement-based replication. ItcorrespondswiththeMySQLstatement-basedbinaryloggingformat.

ThiswastheonlytraditionalformatthatexisteduntilMySQLversions5.1.4andearlier.

With row-based binary logging, the events written in the master binary logindicatehow individual table rows changed.Replication in this caseworksbythe slave copying the events representing changes to the table rows. This is

calledrow-basedreplication.

Row-basedloggingisthedefaultMySQLreplicationmethod.

MySQLsupportsconfiguration tomixstatement-basedandrow-based logging.Thedecisiontousetheloggingformatdependsonthechangebeinglogged.Thisis known as mixed-format logging. Statement-based logging is the defaultformatwhenmixed-formatloggingisused.Basedonthetypeofstatementsandstorageenginebeingused, thelogautomaticallyswitchestorow-basedformat.Replication based on the mixed logging format is known as mixed-formatreplication.

Thebinlog_format systemvariablecontrols the logging formatused ina runningMySQL server. SYSTEM_VARIABLES_ADMIN or SUPER privileges are required to set thebinlog_formatsystemvariableatasessionorglobalscope.

Statement-based versus row-basedreplication

In the earlier section,we learned three different logging formats. Each one ofthese has its own advantages and disadvantages. In usual cases,mixed formatshouldprovidethebestcombinationofintegrityandperformance.However,toachieve the best performance from either statement-based or row-basedreplication, the advantages and disadvantages described in this section arehelpful.

Statement-basedreplicationisatraditionalandproventechniqueincomparisonwithrow-basedreplication.Thenumberofrecordsoreventsrecordedinthelogfiles is smaller. If a statement impactsmany rows, only one statementwill bewrittentothebinarylogfile.Incaseofrow-basedreplication,arecordwillbeenteredforevery tablerowmodified thoughaspartof thesinglestatement. Inessence,thismeansstatement-basedreplicationrequiresmuchlessstoragespaceforlogfiles.Italsomeansbackingupandrestoringorreplicatingtheeventsismuchquicker.

Alongwiththeadvantagesdescribedpreviously,statement-basedreplicationhasdisadvantagesaswell.AsthereplicationworksbasedontheSQLstatements,it

is possible that not all the statements thatmodify data can be replicatedwithstatement-basedreplication.

Afewexamplesaredescribedasfollows:

SQLstatementsdependonauser-definedfunctionthatisnon-deterministicwhenthevaluereturnedbysuchuser-definedfunctionsdependonfactorsotherthantheparameterssuppliedtoit.

UPDATEandDELETEstatementswithaLIMITclausewithoutanORDERBYclause isnon-deterministic as it is possible that theordermayhave changedwhilereplicating.

FORUPDATEorFORSHARE locking read statements that useNOWAIT or SKIP LOCKEDoptions.

User-definedfunctionsmustbeappliedontheslavedatabases.

SQLstatementsusingfunctionssuchasLOAD_FILE(),UUID(),USER(),UUID_SHORT(),FOUND_ROWS(), SYSDATE(), GET_LOCK(), and so on cannot be replicated properlyusingstatement-basedreplication.

INSERTorSELECTstatementsrequirehighernumberofrowlevellocks.

UPDATEwithtablescanrequireslockingahighernumberofrows.

Complex SQL statements must be evaluated and executed on the slavedatabaseserverbeforerowsareinsertedorupdated.

Let's seeadvantagesprovidedby row-based replication.Row-based replicationis the safest form of replication because instead of depending on SQLstatements, itdependsonthevaluesstoredin the tablerows.So,everychangecanbereplicated. It requires fewer rowlocks incaseINSERT...SELECT statements.UPDATEandDELETEstatementswithWHEREclausesthatdonotusekeysrequirefewerrowlevellocks.

Themajordisadvantagewithrow-basedreplicationisthatitgeneratesmoredata

thatmustbelogged.Withstatement-basedreplication,oneDMLSQLstatementis sufficient for logging though it modifies many rows. In case of row-basedreplication, it requires logging for every row that changed.Thebinary log filegrowsveryquicklywithrow-basedreplication.IttakeslongertimetoreplicatedeterministicuserdefinedfunctionsthatgeneratelargeBLOBvalues.

<strong>mysql>SHOWPROCESSLIST\G</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Id:2</strong><br/><strong>User:root</strong><br/><strong>Host:localhost:32931</strong><br/><strong>db:NULL</strong><br/><strong>Command:BinlogDump</strong><br/><strong>Time:94</strong><br/><strong>State:Hassentallbinlogtoslave;waitingforbinlogtobeupdated</strong><br/><strong>Info:NULL</strong>

<strong>mysql>SHOWPROCESSLIST\G</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Id:10</strong><br/><strong>User:systemuser</strong><br/><strong>Host:</strong><br/><strong>db:NULL</strong><br/><strong>Command:Connect</strong><br/><strong>Time:11</strong><br/><strong>State:Waitingformastertosendevent</strong><br/><strong>Info:NULL</strong><br/><strong>***************************2.row***************************</strong><br/><strong>Id:11</strong><br/><strong>User:systemuser</strong><br/><strong>Host:</strong><br/><strong>db:NULL</strong><br/><strong>Command:Connect</strong><br/><strong>Time:11</strong><br/><strong>State:Hasreadallrelaylog;waitingfortheslaveI/Othreadtoupdateit</strong><br/><strong>Info:NULL</strong>

Intheoutput,thread10istheslave'sI/Othreadandthread11istheslave'sSQLthread.TheI/Othreadiswaitingforthemaster'sbinlogdumpthreadtosendbinarylogcontents.TheSQLthreadhasreadallthestatementsloggedintheslaverelaylogs.FromtheTimecolumn,itcanbedeterminedhowslowtheslaveisrunningbehindthe

master.

Replicationchannels

Areplicationchannelisapathoftransactionflowfromamastertoaslave.Thissection explains how channels can be used in replication. TheMySQL serverautomatically creates a default channelwith the name as "" (empty string) onstartup.Thedefaultchannelisalwayspresentandcan'tbecreatedordestroyedby the user. Replication statements work on the default channel if no otherchannel is created. This section describes statements that are applied toreplicationchannelswhenthereexistsatleastonenamedchannel.

Inmulti-source replication, the slave database server opens multiple channels,oneforeachmaster.Eachchannelhas itsownrelaylogandSQLthreads.Thereplicationchannelhasahostnameandportassociation.

Multiplechannelscanbeassignedtothesamehostnameandportcombination.A maximum of 256 channels can be added to one slave in a multi-sourcereplication topology inMySQL8.The channelmust have a nonempty uniquename.

TheFORCHANNELclauseisusedwithvariousMySQLstatementsforthereplicationoperationstobeperformedonindividualchannels.Theclausecanbeappliedtothefollowingstatements:

CHANGEMASTERTO

STARTSLAVE

STOPSLAVE

RESETSLAVE

SHOWRELAYLOGEVENTS

FLUSHRELAYLOGS

SHOWSLAVESTATUS

Apartfromthese,thefollowingfunctionshaveanadditionalchannelparameter:

MASTER_POS_WAIT()

WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS()

Formulti-sourcereplicationtoworkcorrectly,thefollowingstartupoptionsmustbeconfigured:

--relay-log-info-repository:Asdescribedearlier, thismustbe set to TABLE formulti-source replication. In MySQL 8, the FILE option is deprecated andTABLEisthedefaultoption.

--master-info-repository:ThismustbesettoTABLE.

--log-slave-updates:Transactionsreceivedfromthemasterarewrittentothebinarylogs.

--relay-log-purge:Eachchannelpurgesitsownrelaylogsautomatically.

--slave-transaction-retries:SQLthreadsofallchannelsretrytransactions.

--skip-slave-start:Noreplicationthreadsstartonanychannels.

--slave-skip-errors: Execution continues and errors are skipped for allchannels.

--max-relay-log-size=size: The relay log file is rotated after reaching themaximumsize.

--relay-log-space-limit=size:Upperlimitfortotalsizeofallrelaylogsforeachindividualchannel.

--slave-parallel-workers=value:Numberofslaveparallelworkersperchannel.

--slave-checkpoint-group:WaitingtimebyI/Othread.

--relay-log-index=filename:Eachchannel'srelaylogindexfilename.

--relay-log=filename:Eachchannel'srelaylogfilename.

--slave-net-timeout=N:EachchannelwaitsforNsecondstocheckforbrokenconnection.

--slave-skip-counter=N:EachchannelskipsNeventsfromthemaster.

Replicationrelayandstatuslogs

TheREPLICATIONSLAVEservercreateslogsthatholdthebinarylogeventssentfromthemasterdatabase

servertotheslavedatabaseserver.Theinformationisrecordedaboutthecurrent

statusandlocationintherelaylog.Threetypesoflogsareusedinthisprocess:

1. Relaylog:Therelayloghaseventssentfromthemaster'sbinarylog.Theeventsarewritten

byaslave'sI/Othread.Eventsfromtheslave'srelaylogareexecutedontheslave

bytheslave'sSQLthread.

2. Master info log: The master info log has information about status andcurrentconfigurationforthe

slave's connection to themaster database server.The informationheldby

themaster

infologincludeshostname,logincredentials,andcoordinatesindicatingaslave's

position on reading themaster's binary log.These logs arewritten to themysql.slave_master_infotable.

3. Relayloginfolog:Therelayloginfologstoresinformationregardingtheexecutionpointwithinthe

slave's relay log. The relay log info log is written in amysql.slave_relay_log_infotable.

No attempt should be made to insert or update rows in theslave_master_info or slave_relay_log_info tables manually. This maycauseunexpectedbehavior.ItisnotsupportedinMySQL

replication.

Theslave relay logconsistsof an index file alongwitha setofnumbered logfiles.

Theindexfilecontainsthenamesofallrelaylogfiles.MySQLdatadirectoryis

the default location for the relay log files. The relay log file indicates anindividually

numbered file containing events. Whereas the relay log denotes the set ofnumbered

relaylogfilesandanindexfilecollectively.Theformatforrelaylogfilesis

the same as that of the binary log files. The index filename for relay log ishost_name-relay-bin.indexbydefault for thedefaultchannelandhost_name-relay-bin-channel.index for non-default replication channels. The default locations for therelaylogfile

andrelaylogindexfilecanbeoverriddenwiththe--relay-logand--relay-log-index

serverstartupoptions.Iftheslave'shostnameischangedafterreplicationhas

been set up and the slave uses default host-based relay log filenames, it canthrow

errors such as Failed to open the relay log andCould not find target logduringrelayloginitialization.Thismayfailthereplication.Sucherrorsmaybe

avoided by using --relay-log and --relay-log-index options to specify relay logfilenamesexplicitly.Usingtheseoptionsonslavesetup

willmakethenamesindependentoftheserver'shostname.

Evaluating replication filteringrules

Thissectionfocusesonthefilteringrulesandhowserversevaluatetheserules.

Basically,if themasterdoesn't logthestatement,theslavedoesn'treplicatethestatement. If the master logs the statement in its binary log file, the slavereceives the statement. However, it is up to the slave database server if itprocessesthestatementorignoresit.Optionsareavailableforthemasterserverto controlwhich databases and tables should be replicated on the slaves. Therecommendedwayistousefiltersontheslavetocontroltheeventsthataretobeexecutedontheslavedatabaseserver.Thedecisionaboutwhethertoexecuteor ignore the statements received from the master are made based on the --replicate-* options used when the slave was started. Once the slave server isstarted, the CHANGE REPLICATION FILTER statement can be used to set the optionsdynamically.

All replication filtering options follow the same rules for casesensitivity as names of databases and tables, including thelower_case_table_namessystemvariable.

Groupreplication

Thissectionof thechapterexplainswhatgroupreplicationis,settingupgroupreplication, configure andmonitor group replication. Basically,MySQL groupreplication is a plugin that enables us to create elastic, highly-available, fault-tolerantreplicationtopologies.

Thepurposeofthegroupreplicationistocreateafaulttolerantsystem.Tocreatea fault tolerant system, the components should be made redundant. Thecomponent should be removed without impacting the way system operates.There are challenges in setting up such a system. The complexity of such asystem is of a different level. Replicated databases require maintenance andadministration of several servers instead of just one. The servers cooperatetogether to create a group, which raises the problems related to networkpartitioning and split-brain scenarios. So, the ultimate challenge is to haveagreementfrommultipleserversonthestateofthesystemanddataaftereverychangeappliedonthesystem.Thismeansthattheserversneedtooperateasadistributedstatemachine.

MySQL group replication can provide such a distributed state machinereplicationwithstrongcoordinationbetweenservers.Theserversthatbelongtothesamegroupcoordinatethemselvesautomatically.Inagroup,onlyoneserver

accepts updates at a time.The election of primary is done automatically.Thismodeisknownassingle-primarymode.

MySQLprovidesagroupmembershipservice,whichisresponsibleforkeepingtheviewofthegroupconsistentandavailableforallservers.Theviewiskeptupdatedwhen the servers join or leave the group. In case, any of the serversleaves the group unexpectedly, the failure detection mechanism notifies thegroupaboutviewchange.

Thisbehaviorisautomatic.

Themajorityofthegroupmembershavetoagreeontheorderofthetransactiontocommitintheglobalsequenceoftransactions.Itisuptotheindividualservertodecidewhetherthetransactionshouldbecommittedoraborted,butallserversmake the same decision. The system does not proceed until themembers areunable to reach toagreementasa resultofsplitdue tonetworkpartition.Thismeansthesystemhasbuilt-in,automatic,split-brainprotectionmechanism.AllthisisdonebyGroupCommunicationSystem(GCS)protocols.Itprovidesafailure detection mechanism, group membership service, safe and completelyorderedmessagedelivery.The implementationof thePaxosalgorithmisat thecoreofthistechnology,whichactsasthegroupcommunicationengine.

Primary-secondary replicationversusgroupreplicationThissectionfocusesonsomebackgrounddetailsofhowreplicationworks.ThiswillbeusefulinunderstandingtherequirementsforgroupreplicationandhowitisdifferentfromtheclassicasynchronousMySQLreplication.

The following figure showcases how traditional asynchronous primary-secondaryreplicationworks.Theprimaryisthemasterandthesecondaryisoneormoreslavesconnectedtothemaster,asshowninthefollowingfigure:

Figure1.MySQLAsynchronousReplication

MySQLalsosupportssemisynchronousreplication inwhich themaster waitsfor at least one of the slaves to acknowledge the transaction receipt:

Figure2.MySQLSemisynchronousReplication

Thebluearrowsinthefiguresindicatethemessagespassedbetweenserversandtheclientapplication.

With group replication, a communication layer is provided that guaranteesatomicmessages and total ordermessage delivery. All read-write transactionsare committed only after they are approved by the group. The read onlytransactionsarecommittedimmediatelyasitdoesnotneedcoordination.So,ingroupreplication,thedecisiontocommitatransactionornotisnotunilateralbytheoriginatingserver.Whenthetransactionisreadyforcommit,theoriginatingserver broadcasts the write values and corresponding write set. All serversreceivethesamesetoftransactionsinthesameorder.So,allserversapplythesame transactions in the same order. This way all servers remain consistentwithinthegroup:

Figure3.MySQLGroupReplicationProtocol

GroupreplicationconfigurationThissectionfocusesonconfiguringgroupreplication.

Firstofall,openthemy.cnfconfigurationfileandaddthefollowingentriesinthemysqldsectionasfollows:

[mysqld]

gtid_mode=ON

enforce_gtid_consistency=ON

master_info_repository=TABLE

relay_log_info_repository=TABLE

binlog_checksum=NONE

log_slave_updates=ON

log_bin=binlog

binlog_format=ROW

transaction_write_set_extraction=XXHASH64

loose-group_replication_bootstrap_group=OFF

loose-group_replication_start_on_boot=OFF

loose-group_replication_ssl_mode=REQUIRED

loose-group_replication_recovery_use_ssl=1

These are general configurations related to global transaction IDs and binaryloggingrequiredforgroupreplication.

Thenextstepistosetupgroupreplicationconfigurations.Theseconfigurationsinclude group UUID, group members white listing, and indicating seedmembers:

#Sharedreplicationgroupconfiguration

loose-group_replication_group_name="929ce641-538d-415d-8164-ca00181be227"

loose-group_replication_ip_whitelist="177.110.117.1,177.110.117.2,177.110.117.3"

loose-group_replication_group_seeds="177.110.117.1:33061,177.110.117.2:33061,177.110.117.3:33061"

...Choosing

The following configuration is required for deciding whether a single-mastergroupormulti-mastergroupistobesetup.Forenablingthemulti-mastergroup,uncomment

loose-group_replication_single_primary_modeand

loose-group_replication_enforce_update_everywhere_checks directives. It will set up amulti-masterormulti-primarygroup:

...

#SingleorMulti-primarymode?Uncommentthesetwolines

#formulti-primarymode,whereanyhostcanacceptwrites

loose-group_replication_single_primary_mode=OFF

loose-group_replication_enforce_update_everywhere_checks=ON

Itmustbeensuredthattheseconfigurationsarethesameonalltheservers.AnychangestotheseconfigurationsrequireMySQLgroupstoberestarted.

Thefollowingconfigurationsaredifferentoneachoftheserversinthegroup:

...

#Hostspecificreplicationconfiguration

server_id=1

bind-address="177.110.117.1"

report_host="177.110.117.1"

loose-group_replication_local_address="177.110.117.1:33061"

Theserver-idmustbeuniqueacrossall servers in thegroup.Theport33061 isthe one used bymembers to coordinate for group replication.MySQL serverrestartisrequiredafterthesechangesaremade.

Ifnotalreadydone,wehavetoallowaccesstotheseportsusingthefollowingcommands:

sudoufwallow33061

sudoufwallow3306

The next step is to create a replication user and enable the replication plugin.Thereplicationuserisrequiredforeachservertoestablishgroupreplication.Weneedtoturnbinaryloggingoffduringthereplicationusercreationprocessastheuserwillbedifferentforeachserver,asshowninthefollowingblock:

SETSQL_LOG_BIN=0;

CREATEUSER'mysql_user'@'%'IDENTIFIEDBY'password'REQUIRESSL;

GRANTREPLICATIONSLAVEON*.*TO'mysql_user'@'%';

FLUSHPRIVILEGES;

SETSQL_LOG_BIN=1;

Now,use CHANGE MASTER TO to configure the server to use the credentials for thegroup_replication_recoverychannel:

CHANGEMASTERTOMASTER_USER='mysql_user',MASTER_PASSWORD='password'FORCHANNEL'group_replication_recovery';

Now,weareallset toinstall theplugin.Connect totheserverandexecutethefollowingcommand:

INSTALLPLUGINgroup_replicationSONAME'group_replication.so';

Usethefollowingstatementtoverifyifthepluginisactivatedornot:

SHOWPLUGINS;

Thenextstepistostartupthegroup.Executethefollowingstatementsononememberofthegroup:

SETGLOBALgroup_replication_bootstrap_group=ON;

STARTGROUP_REPLICATION;

SETGLOBALgroup_replication_bootstrap_group=OFF;

Now,wecanstartgroupreplicationonanotherserver:

STARTGROUP_REPLICATION;

WecancheckthegroupmemberslistusingthefollowingSQLquery:

mysql>SELECT*FROMperformance_schema.replication_group_members;

+---------------------------+--------------------------------------+

|CHANNEL_NAME|MEMBER_ID|

+---------------------------+--------------------------------------+

|group_replication_applier|13324ab7-1b01-11e7-9dd1-22b78adaa992|

|group_replication_applier|1ae4b211-1b01-11e7-9d89-ceb93e1d5494|

|group_replication_applier|157b597a-1b01-11e7-9d83-566a6de6dfef|

+---------------------------+--------------------------------------+

+---------------+-------------+--------------+

|MEMBER_HOST|MEMBER_PORT|MEMBER_STATE|

+---------------+-------------+--------------+

|177.110.117.1|3306|ONLINE|

|177.110.117.2|3306|ONLINE|

|177.110.117.3|3306|ONLINE|

+---------------+-------------+--------------+

3rowsinset(0.01sec)

Groupreplicationusecases

The MySQL group replication feature provides a way to build fault tolerantsystemsby replicating the stateof the system throughout a setof servers.Thegroup replication system stays available as long as themajority of servers arefunctioning even if some of the servers fail. Server failures are tracked by agroupmembershipservice.

Thegroupmembership service relieson thedistributed failuredetector,whichsignalsifanyserverleavesthegroup,voluntarilyorduetounexpectedhalt.Thedistributedrecoveryprocedureensuresthatwhentheserversjointhegroup,theyarebroughtup todateautomatically.Therefore,continuousdatabaseservice isguaranteed with MySQL group replication. There is one problem though.Although the database service is available, the clients connected to itmust beredirected to a different serverwhen the server crashes.The group replicationdoes not attempt to resolve it. It should be dealt with by a connector, loadbalancer,router,orsomeothermiddleware.

ThefollowingarethetypicalusecasesofMySQLgroupreplication:

1. Elastic replication: Group replication is suitable for fluid environments

where the number of servers grow or shrink dynamically withminimumsideeffects.Theexampleiscloud-baseddatabaseservices.

2. Highly available shards: MySQL group replication can be used toimplement highly available write scale-out shards where each replicationgroupmapstooneshard.

3. Alternative to master-slave replication: Group replication can be ananswer to contention problems arising in certain situations with singlemasterserverreplication.

4. Autonomic systems: MySQL group replication can be deployed for theautomationbuiltintothereplicationprotocol.

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">UPDATE</span>performance_schema<spanclass="tokenpunctuation">.</span>setup_instruments<spanclass="tokenkeyword">SET</span>ENABLED<spanclass="tokenoperator">=</span><spanclass="tokenstring">'YES'</span><spanclass="tokenkeyword">WHERE</span><spanclass="tokenkeyword">NAME</span><spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'stage/sql/Applyingbatchofrowchanges%'</span><spanclass="tokenpunctuation">;</span></strong>

TheMySQL8replicationprocesscanworkeventhoughthesourcetableonthemasterandthedestinationtableontheslaveusesdifferentenginetypes.Thedefault_storage_enginesystemvariableisnotreplicated.Thisisahugeadvantageinreplicationwhereindifferentenginetypescanbeusedfordifferentreplicationscenarios.Anexampleisascale-outscenariowherewewantallreadoperationstobeperformedontheslavedatabaseserver,whereasallwriteoperationsshouldbeperformedonthemasterdatabaseserver.Insuchacase,wecanuseatransactionalInnoDBengineonthemasterandanon-transactionalMyISAMenginetypeontheslavedatabaseserver.

Consideranexampleofanorganizationthatwantstodistributesalesdatatodifferentdepartmentstospreadtheloadfordataanalysis.MySQLreplicationcanbeusedtohaveasinglemasterreplicatedifferentdatabasestodifferentslaves.Thiscanbeachievedbylimitingthebinarylogstatementsbyusingthe--replicate-wild-do-tableconfigurationoptiononeachslave.

OnceMySQLreplicationissetup,asthenumberofslavesconnectedtothemasterincrease,theloadalsoincreases.Thenetworkloadonthemasteralsoincreasesaseachslaveissupposedtoreceiveafullcopyofthebinarylogs.Themasterdatabaseserverisalsobusy

processingrequests.Inthisscenario,itbecomesnecessarytoimprovetheperformance.Oneofthesolutionstoimproveperformanceistocreateadeeperreplicationstructurethatenablesreplicationofamastertoonlyoneslave.Therestoftheslavesconnecttotheprimaryslavefortheiroperations.

Summary

In this chapter,we learned insightfuldetails aboutMySQL8 replication,whatreplicationis,andhowithelpssolvespecificproblems.Wealsolearnedhowtosetupstatement-basedandrow-basedreplicationtypes.Alongtheway,wealsolearnedaboutthesystemvariablesandserverstartupoptionsforreplication.Inthelaterpartofthechapter,wediveddeepintogroupreplicationandhowitisdifferent from the traditional method of MySQL replication.We also learnedlogging and replication formats. Last but not least, we learned differentreplicationsolutionsinbrief.Wecoveredalotofstuff,huh?

It'snowtimetomoveontoournextchapter,wherewewillbesettingupseveraltypesofpartitioning,andexploringtheselectionofpartitioning,andpruningofpartitioning.Italsoexplainshowtocopewithrestrictionsandlimitationswhilepartitioning. The reader will be able to understandwhich type of partitioningsuitsasituationaspertherequirement.

PartitioninginMySQL8

In the previous chapter, replication inMySQL8was explained.This includeddetailed explanations of replication, configuration, and implementation. Thechapter also explained group replication versus clustering, and covered thereplicationapproachasasolution.

Inthischapter,wewilldopartitioninginMySQL8.Partitioningistheconceptof managing and maintaining data with specific operations with severaloperators,anddefiningrulestocontroloverpartitioning.Basically,itprovidesaconfigurationhookformanagingtheunderlyingdatafilesinaspecifiedway.

Wewillcoverthefollowingtopicsonpartitioning:

Overviewofpartitioning

Typesofpartitioning

Partitionmanagement

Partitionselection

Partitionpruning

Restrictionandlimitationinpartitioning

<strong>CREATETABLEtp(tp_idINT,amtDECIMAL(5,2),trx_dateDATE)</strong><br/><strong>ENGINE=INNODB</strong><br/><strong>PARTITIONBYHASH(MONTH(trx_date))</strong><br/><strong>PARTITIONS4;</strong>

Partitioningisapplicableonallindexesandalldataofthetable.Itisnotapplicableoneitherindexesordata,andviceversaisalsonotapplicable.Itcanbeapplicableonbothindexesanddatatogetheranditcannotbeappliedonpartofthetable.Theprecedingtabletphasnouniqueorprimarykeysdefinedbutingeneralpracticeweusuallyhaveprimarykeys,uniquekeys,orbothaspartofthetable,andpartitioningcolumnchoicedependsuponthesekeysifanyofthemispresent.Thepartitioningcolumnchoiceisgivenindetailinthepartitioningkeys,primarykeys,anduniquekeyssection.Tosimplifytheconceptofpartitioningtheexamplesgivenmaynotincludethesekeys.

Typesofpartitioning

SeveraltypesofpartitioningaresupportedinMySQL8,listedasfollows:

RANGE Partitioning: Assigns rows to partitions from the column values thatcomebetweenthegivenrangeofvalues

LIST Partitioning: Assigns rows to partitions from the column values thatmatcheswithoneofthegivensetofvalues

COLUMNSPartitioning:AssignsrowstopartitionswithmultiplecolumnvalueswitheitherRANGEorLISTpartitioning

HASH Partitioning: Assigns partition based on user specified expressionsevaluatedoncolumnvalues

KEYPartitioning: In addition to HASH partitioning, allows the use ofmultiplecolumnvalues

Subpartitioning: In addition to partitioning, allows further division inpartitionedtables,alsoknownascompositepartitioning

Differentrowsofthetablecanbeassignedtodifferentphysicalpartitions;thisisknownashorizontalpartitioning.Differentcolumnsofthetablecanbeassignedtodifferentphysicalpartitions;thisisknownasverticalpartitioning.MySQL8

currentlysupportshorizontalpartitioning.

FortheLIST,RANGE,andLINEARHASH typesofpartitioning,thevalueofpartitioningcolumns isgiven to thepartitioning function.Thepartitioning function returnsan integer value that is the partition number in which the record should bestored.Thepartitionfunctionmustbenonrandomandnonconstant.Thepartitionfunctioncannotcontainqueriesandcanuse theSQLexpressionwhich returnseitheranintegerorNULL,wheretheintegerasintvalmustfollowtheexpression-MAXVALUE<=intval<=MAXVALUE.Here,-MAXVALUErepresentsthelowerlimitandMAXVALUEistheupperlimitfortheintegertypevalue.

Thestorageenginemustbethesameforallpartitionsofthesametable,howeverthereisnorestrictiononusingdifferentstorageenginesfordifferentpartitionedtablesinthesamedatabaseorMySQLserver.

PartitioningmanagementTherearedifferentways touseSQL statements tomodifypartitioned tablesandperform operations such as add, redefine, merge, drop, or split existingpartitionedtables.InformationaboutpartitionedtablesandpartitionscanalsobeobtainedwithSQLstatements.

MIN_ROWS and MAX_ROWS can be used to configure the maximum and minimumnumberofrowsandcanbestoredinthepartitiontable.

PartitionselectionandpruningExplicit selectionof partition and subpartition is alsoprovided. It enables rowmatching to the conditionsgiven in thewhereclause. Inpartition, thepruningconceptsdescribeddonotscanpartitionswherenopossiblematchingvaluescanbe present, and are applied using queries, whereas partition selection isapplicableforbothqueriesandmanyoftheDMLstatements.

Restrictions and limitations inpartitioning

Stored procedures or functions, user defined functions or plugins, and uservariables or declaredvariables are restricted in partitioning expressions.Therearealsoseveralrestrictionsandlimitationsapplicabletopartitioninggiveninthedetailedsection.

Seethefollowinglistforsomeoftheadvantagesofpartitioning:

Partitioningfacilitatesstoringmoredatainonetablethancanbeheldonafilesystempartitionorsingledisk.

Datathathasbecomeuselesscanberemovedeasilybydroppingapartitionorpartitionsthatonlycontaintheuselessdata.Insomecaseswherespecificdata is required to be added separately, this can be done easily withpartitioninginsingleormultiplepartitionsbasedonthespecifiedrule.

Queryoptimizationthatoccursautomaticallybasedonpartitioneddatabynotsearchingfordatainpartitionsthatarenotapplicableasperthewhere

condition.

Inadditiontopartitionpruning,partitionselectionissupportedexplicitlybywhichwhereclauseisappliedonaspecifiedpartitionormultiplepartitions.

Greater query throughput is achieved by separating data search intomultipledisks.

Typesofpartitioning

In this section,youwillunderstanddifferent typesofpartitioningandalso thepurposeofusingspecificpartitioning.ThefollowingisalistofthepartitioningtypesthatareavailableinMySQL8:

RANGEpartitioning

LISTpartitioning

COLUMNSpartitioning

HASHpartitioning

KEYpartitioning

Subpartitioning

In addition to the above list, we will also see NULLhandling in MySQL 8Partitioningindetailedsection.

Averycommonusecase fordatabasepartitioning is segregatingdatabydate.MySQL

8 does not support date partitioning, which some database systems provideexplicitly, but it is easy to create partitioning schemes with date, time, ordatetime columns, or that are based on date/time related expressions thatevaluatevaluesfromthesecolumntypes.

You can use the date, time, or datetime types as column values for partitioncolumnswithoutanymodificationsifusingKEYorLINEARKEYpartitioning,whereasinotherpartitioningtypesanexpressiongivingbackanintegerorNULLvalue isrequired.

Irrespective of which type of partitioning you use, partitions always getnumbered automatically with an integer number in sequence of the partitionscreated. If, for example, the table uses four partitions, they are numbered as0,1,2,and3foreachofthepartitionsaspercreationsequence.

Whenyouspecifynumbersofpartitions,itmustbeevaluatedtoapositive,nonzero integer without any leading zeros. Decimal fractions are not allowed aspartitionnumbers.

Names of partitions are not case-sensitive and should follow conventions orrules just like other MySQL identifiers such as tables. The options used inpartitiondefinitionarealreadyprovidedbytheCREATETABLEsyntax.

Now,let'slookatpartitionindetailandexamineeachofthetypestolearnhowtheyaredifferenttoeachother.

RANGEpartitioningInthistypeofpartitioning,asthenamestates,RANGEisgiveninanexpressionthatevaluateswhetheravalueliesinthegivenrangeornot.RangesaredefinedwiththeVALUESLESSTHANoperatorandtheyshouldnotbeoverlappingandcontiguous.

For thenext fewexamples, supposeweare creating a tableholding employeepersonalrecordsfor25foodstores.Thestoresarenumberedfrom1to25andisachainof25foodstores,asshowninthefollowingblock:

CREATETABLEemployee(

employee_idINTNOTNULL,

first_nameVARCHAR(30),

last_nameVARCHAR(30),

hired_dateDATENOTNULLDEFAULT'1990-01-01',

termination_dateDATENOTNULLDEFAULT'9999-12-31',

job_codeINTNOTNULL,

store_idINTNOTNULL

);

Nowlet'sdopartitioningofthetable,soyoucanpartitionthetablebyrangeasperyourneed.Supposeyouconsiderusingdivision tosplit thedata fivewayswiththestore_idrangeforpartitioning.Forthis,thetablecreationdefinitionwilllookasfollows:

CREATETABLEemployee(

employee_idINTNOTNULL,

first_nameVARCHAR(30),

last_nameVARCHAR(30),

hired_dateDATENOTNULLDEFAULT'1990-01-01',

termination_dateDATENOTNULLDEFAULT'9999-12-31',

job_codeINTNOTNULL,

store_idINTNOTNULL

)

PARTITIONBYRANGE(store_id)(

PARTITIONp0VALUESLESSTHAN(6),

PARTITIONp1VALUESLESSTHAN(11),

PARTITIONp2VALUESLESSTHAN(16),

PARTITIONp3VALUESLESSTHAN(21),

PARTITIONp4VALUESLESSTHAN(26)

);

So, as per the above partitioning scheme, all inserted rows that contain theemployeesworking at stores 1 to 5 are stored in the p0 partition, employeesworkingatstores6to10arestoredinthep1partition,andsoon.Ifyoutakealookatthepartitiondefinition,partitionsareorderedfromthelowesttohighest

store_id column values, and the PARTITION BY RANGE syntax looks similar toprogrammingstatementsif…elseif…statements,doesn'tit?

Well,youarethinkingaboutwhatwillhappenifarecordcomeswithstore_id26;this would result in an error as the server does not knowwhere to place therecord.Therearetwowaystokeepthiserrorfromoccurring:

1. ByusingtheIGNOREkeywordwiththeINSERTstatement.2. ByusingMAXVALUEinsteadofaspecifiedrange(26).

Andyes,ofcourse,youcanextendthelimitsbyusingtheALTERTABLEstatementtoaddnewpartitionsforstores26-30,30-35,andsoon.

Similar to store_id, you could also partition the table based on the job codes -based on the range of column values. Suppose if 5 digit codes are used formanagementpositions,4digitcodesareusedforofficeandsupportpersonnel,and3digitcodesareusedforregularworkers, thenthepartitiontablecreationdefinitionwouldbeasfollows:

CREATETABLEemployee(

employee_idINTNOTNULL,

first_nameVARCHAR(30),

last_nameVARCHAR(30),

hired_dateDATENOTNULLDEFAULT'1990-01-01',

termination_dateDATENOTNULLDEFAULT'9999-12-31',

job_codeINTNOTNULL,

store_idINTNOTNULL

)

PARTITIONBYRANGE(job_code)(

PARTITIONp0VALUESLESSTHAN(1000),

PARTITIONp1VALUESLESSTHAN(10000),

PARTITIONp2VALUESLESSTHAN(100000)

);

You can also specify partitioning with one of the two columns of date typevalues.Supposeyouwish topartitionbasedon theyear eachof the employeejoined - soby thevalueof YEAR(hired_date).Now the table definitionwill be asfollows:

CREATETABLEemployee(

employee_idINTNOTNULL,

first_nameVARCHAR(30),

last_nameVARCHAR(30),

hired_dateDATENOTNULLDEFAULT'1990-01-01',

termination_dateDATENOTNULLDEFAULT'9999-12-31',

job_codeINTNOTNULL,

store_idINTNOTNULL

)

PARTITIONBYRANGE(YEAR(hired_date))(

PARTITIONp0VALUESLESSTHAN(1996),

PARTITIONp1VALUESLESSTHAN(2001),

PARTITIONp2VALUESLESSTHAN(2006),

PARTITIONp3VALUESLESSTHANMAXVALUE

);

Accordingtothisscheme,allemployeesrecordedhiredbefore1996willbestoredinthepartitionp0,thenrecordswithahiredatebefore2001willbestoredinthepartitionp1,recordsbetween2001and2006inp2,andtherestoftherecordswillbestoredinpartitionp3.

Partition schemes based on time intervals can be implemented using thefollowingtwooptions:

1. PartitionthetablebyRANGEanduseafunctionoperatingonthedate,timeordatetime column values to return an integer value for the partitioningexpression

2. Partition the table by RANGE COLUMN and use the date, time, or datetimecolumnsasthepartitioncolumn

RANGE COLUMN is supported in MySQL 8 and is described in detail in the COLUMNPARTITIONINGsection.

LISTpartitioningAsthenamestates,LISTpartitioninguses lists for tablepartitioning.The list iscomma separated integer values defined while partitioning with VALUES IN

(value_list);here,value_listreferstocommaseparatedintegerliterals.

LIST partitioning is similar to RANGE partitioning in many ways, but there aredifferences.Theoperatorusedineachpartitioningisdifferent.Theoperatorusesa list of comma separated values to bematchedwith the column value or thepartitionexpressionevaluatingtointegervalue.

Consideringtheemployeetableasanexample,thebasicdefinitionforthetableusingthecreatetablesyntaxwillbeasfollows:

CREATETABLEemployee(

employee_idINTNOTNULL,

first_nameVARCHAR(30),

last_nameVARCHAR(30),

hired_dateDATENOTNULLDEFAULT'1990-01-01',

termination_dateDATENOTNULLDEFAULT'9999-12-31',

job_codeINTNOTNULL,

store_idINTNOTNULL

);

Supposeyouwish todistribute these25foodstoresamongfivezones—North,South, East, West, and Central, with the store ID numbers (1,2,11,12,21,22),(3,4,13,14,23,24), (5,6,15,16,25), (7,8,17,18), and (9,10,19,20) respectively forthezones.

Partitioningthetablewiththezoneslistwillprovidethefollowingdefinitionfortablepartition:

CREATETABLEemployee(

employee_idINTNOTNULL,

first_nameVARCHAR(30),

last_nameVARCHAR(30),

hired_dateDATENOTNULLDEFAULT'1990-01-01',

termination_dateDATENOTNULLDEFAULT'9999-12-31',

job_codeINTNOTNULL,

store_idINTNOTNULL

)

PARTITIONBYLIST(store_id)(

PARTITIONpNorthVALUESIN(1,2,11,12,21,22),

PARTITIONpSouthVALUESIN(3,4,13,14,23,24),

PARTITIONpEastVALUESIN(5,6,15,16,25),

PARTITIONpWestVALUESIN(7,8,17,18),

PARTITIONpCentralVALUESIN(9,10,19,20)

);

Asyoucanseeintheprecedingstatement,partitioningperzonesmeansitwillbeeasytoupdaterecordsforstoresbasedonzoneswithinparticularpartitions.Supposetheorganizationsoldthewestzonetoanothercompany;thenyoumightneedtoremoveallemployeerecordsfromthewestzoneusingthepWestpartitioninquery.ExecutingALTERTABLEemployeeTRUNCATEPARTITIONpWestwouldbemucheasierandefficient thantheDELETEstatementDELETEfromemployeewherestore_idIN(7,8,17,18);also,youcanusetheDROPstatementforemployeerecordsremoval-ALTER TABLE employee DROP PARTITION pWest. Along with the previous statementexecution you will also remove the pWest PARTITION from the table partitiondefinition,andwillneedtousetheALTERstatementagaintoaddthepWestPARTITIONandrestorethepartitiontableschemeyouhadearlier.

SimilartoRANGEpartitioning,youcanalsouseLISTpartitioningusinghashorkeyto produce composite partitioning,which is also known assubpartitioning. Youwill get to know more details on subpartitioning as a dedicated section forsubpartitioningfollows.

In LIST partitioning there is no catch-all mechanism such as MAXVALUE that cancontainallpossiblevalues.Instead,youhavetomanagetheexpectedvalueslistinthevalues_listitself,otherwisetheINSERTstatementwillresultinanerrorwherethetablehasnopartitionforvalue9,asinthefollowingexample:

CREATETABLEtpl(

cl1INT,

cl2INT

)

PARTITIONBYLIST(cl1)(

PARTITIONp0VALUESIN(1,3,4,5),

PARTITIONp1VALUESIN(2,6,7,8)

);

INSERTINTOtplVALUES(9,5);

Asyoucansee in theprecedingINSERT statement,value9 isnotpartof the listgivenduringpartitionschemaandsothereisanerror.Ifyouusemultiplevalueinsert statements, the same error can result in failure for all inserts and norecordswillbeinserted;insteadusetheIGNOREkeywordtoavoidsucherrors,asinthefollowingINSERTstatementexample:

INSERTIGNOREINTOtplVALUES(1,2),(3,4),(5,6),(7,8),(9,11);

COLUMNSpartitioning

Asthenamesuggests,thistypeofpartitioningusescolumnsthemselves.Wecanusetwoversionsofcolumnpartitioning.OneisRANGECOLUMNandtheotherisLISTCOLUMN. In addition to both RANGE COLUMN and LIST COLUMN partitioning, MySQL 8supports using non-integer types of column that can be used to define valuerangesorlistvalues.Thelistofpermitteddatatypesareasfollows:

AllcolumntypesofINT,BIGINT,MEDIUMINT,SMALLINT,andTINYINTaresupportedfortheRANGEandLISTpartitioningcolumns,butothernumericcolumntypessuchasFLOATorDECIMALarenotsupported

DATEandDATETIMEaresupportedbutothercolumntypesrelating todateandtimearenotsupportedaspartitioningcolumns

ThestringcolumntypesBINARY,VARBINARY,CHARandVARCHARaresupportedbuttheTEXTandBLOBcolumntypesarenotsupportedaspartitioningcolumns

Now,let'sseeRANGECOLUMNpartitioningandLISTCOLUMNpartitioningindetailonebyone.

<strong>CREATETABLEtable_name</strong><br/><strong>PARTITIONBYRANGECOLUMNS(column_list)(</strong><br/><strong>PARTITIONpartition_nameVALUESLESSTHAN(value_list)[,</strong><br/><strong>PARTITIONpartition_nameVALUESLESSTHAN(value_list)][,</strong><br/><strong>...]</strong><br/><strong>)</strong><br/><strong>column_list:</strong><br/><strong>column_name[,column_name][,...]</strong><br/><strong>value_list:</strong><br/><strong>value[,value][,...]</strong>

<strong>CREATETABLEtrc(</strong><br/><strong>pINT,<br/>qINT,<br/>rCHAR(3),<br/>sINT<br/>)</strong><br/><strong>PARTITIONBYRANGECOLUMNS(p,s,r)(</strong><br/><strong>PARTITIONp0VALUESLESSTHAN(5,10,'ppp'),</strong><br/><strong>PARTITIONp1VALUESLESSTHAN(10,20,'sss'),</strong><br/><strong>PARTITIONp2VALUESLESSTHAN(15,30,'rrr'),</strong><br/><strong>PARTITIONp3VALUESLESSTHAN(MAXVALUE,MAXVALUE,MAXVALUE)</strong><br/><strong>);</strong>

<strong>INSERTINTOtrcVALUES(5,9,'aaa',2),(5,10,'bbb',4),(5,12,'ccc',6);</strong>

<strong>CREATETABLEcustomer_z(</strong><br/><strong>first_nameVARCHAR(30),<br/>last_nameVARCHAR(30),<br/>street_1VARCHAR(35),<br/>street_2VARCHAR(35),<br/>cityVARCHAR(15),<br/>renewalDATE<br/>)</strong><br/><strong>PARTITIONBYLISTCOLUMNS(city)(</strong><br/><strong>PARTITIONpZone_1VALUESIN('Ahmedabad','Surat','Mumbai'),</strong><br/><strong>PARTITIONpZone_2VALUESIN('Delhi','Gurgaon','Punjab'),</strong><br/><strong>PARTITIONpZone_3VALUESIN('Kolkata','Mizoram','Hyderabad'),</strong><br/><strong>PARTITIONpZone_4VALUESIN('Bangalore','Chennai','Kochi')</strong><br/><br/><strong>);</strong>

SimilartoRANGECOLUMNpartitioning,itisnotrequiredtoprovideanyexpressionintheCOLUMNS()clausethatconvertsthecolumnvaluetoanintegerliteral,andnothingotherthanthelistofcolumnnamesitselfispermitted.

<strong>CREATETABLEemployee(</strong><br/><strong>employee_idINTNOTNULL,</strong><br/><strong>first_nameVARCHAR(30),</strong><br/><strong>last_nameVARCHAR(30),</strong><br/><strong>hired_dateDATENOTNULLDEFAULT'1990-01-01',</strong><br/><strong>termination_dateDATENOTNULLDEFAULT'9999-12-31',</strong><br/><strong>job_codeINTNOTNULL,</strong><br/><strong>store_idINTNOTNULL</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(store_id)</strong><br/><strong>PARTITIONS4;</strong>

Intheabovestatement,ifyouexcludethePARTITIONSclause,thenumberofpartitionsautomaticallydefaultstoone.

<strong>CREATETABLEemployee(</strong><br/><strong>employee_idINTNOTNULL,</strong><br/><strong>first_nameVARCHAR(30),</strong><br/><strong>last_nameVARCHAR(30),</strong><br/><strong>hired_dateDATENOTNULLDEFAULT'1990-01-01',</strong><br/><strong>termination_dateDATENOTNULLDEFAULT'9999-12-31',</strong><br/><strong>job_codeINTNOTNULL,</strong><br/><strong>store_idINTNOTNULL</strong><br/><strong>)</strong><br/><strong>PARTITIONBYLINEARHASH(YEAR(hired_date))</strong><br/><strong>PARTITIONS4;</strong>

Anadvantageofusinglinearhashingisfasterpartitioningoperations,andadisadvantageislessevendatadistributioncomparedtoregularhashingpartitioning.

<strong>CREATETABLEtk1(</strong><br/><strong>tk1_idINTNOTNULLPRIMARYKEY,<br/>noteVARCHAR(50)<br/>)<br/>PARTITIONBYKEY()<br/>PARTITIONS2;</strong>

<strong>CREATETABLEtk2(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2CHAR(10),</strong><br/><strong>cl3DATE</strong><br/><strong>)</strong><br/><strong>PARTITIONBYLINEARKEY(cl1)</strong><br/><strong>PARTITIONS3;</strong>

Asyoucanseeintheprecedingexamplestatement,similartoHASHpartitioning,KEYpartitioningalsosupportsLINEARKEYpartitioningandhasthesameeffectasLINEARHASHpartitioning.

<strong>CREATETABLEtrs(trs_idINT,soldDATE)</strong><br/><strong>PARTITIONBYRANGE(YEAR(sold))<br/>SUBPARTITIONBYHASH(TO_DAYS(sold))<br/>SUBPARTITIONS2(<br/>PARTITIONp0VALUESLESSTHAN(1991),<br/>PARTITIONp1VALUESLESSTHAN(2001),<br/>PARTITIONp2VALUESLESSTHANMAXVALUE<br/>);</strong>

Asyoucanseeintheprecedingexamplestatement,tabletrshasthreeRANGEpartitionsandeachofthepartitionsp0,p1,p2isfurtherdividedintotwomoresubpartitions.Effectively,theentiretableisdividedintosixpartitions.

SubpartitioningispossibleontablespartitionedusingRANGEorLISTpartitioning,andsubpartitioningcanusetheKEYorHASHpartitioningtypes.Thesyntaxrulesforsubpartitioningarethesameasinregularpartitioning,withtheexceptiontospecifythedefaultcolumninKEYpartitioningasitdoesnottakethecolumnautomaticallyforsubpartitioning.

Thefollowingisalistofpointstoconsiderwhenusingsubpartitioning:

NumberofpartitionsmustbesameforeachofthepartitionsdefinedNamemustbespecifiedwiththeSUBPARTITIONINGclauseorspecifyadefaultoptioninsteadNamesspecifiedforsubpartitioningmustbeuniqueacrossthetable

HandlingNULLinpartitioning

There is nothing specific toMySQL 8 that disallows NULL in partitioning as acolumn value, partitioning expression, or the value from the user-definedexpression. Even if NULL is permitted as a value ,the value returned from theexpression must be an integer and so MySQL 8 has implementation forpartitioningsuchthatittreatsNULLaslessthananynon-NULLvalueasdoneintheORDERBYclause.

BehaviorforNULLhandlingvariesamongdifferenttypesofpartitioning:

HandlingNULLinRANGEpartitioning:IfaNULLvaluecontainedinthecolumnisinserted,therowwillbeinsertedinthelowestpartitionspecifiedinrange

Handling NULL with LIST partitioning: If the table has a partitioningdefinitionwithLISTpartitioningand itspartitionsaredefinedwithavaluelistthatexplicitlyspecifiesNULLasavalueinvalue_list,theninsertionwillbesuccessful; otherwise, it will give an error table that has no partitionspecifiedforNULL

Handling NULL with HASH and KEY partitioning: NULL is handled differently

when table partitioning is defined with HASH or KEY partitioning, and if apartitionexpressionreturnsNULLitiswrappedwithzerovalue.Sothatbasedonpartitioningtheinsertionoperationwillsuccessfullyinserttherecordtopartitionbeingzero.

Partitionmanagement

There are plenty ofways to use SQL statements in order tomodify partitionedtables—youcandrop,add,merge,split,orredefinepartitionswiththeALTERTABLEstatement. There are also ways to retrieve partitioned tables and partitioninformation.

Wewillseeeachoftheseinthefollowingsections:

RANGEandLISTpartitionmanagement

HASHandKEYpartitionmanagement

Partitionmaintenance

Obtainpartitioninformation

RANGE and LIST partitionmanagementPartitionaddinganddroppingishandledinasimilarwayfortheRANGEandLISTpartition types.A tablepartitionedby RANGE or LIST partitioning canbedroppedusingtheALTERTABLEstatementwiththeDROPPARTITIONoptionavailable.

Make sureyouhave the DROP privilegebefore executing the ALTER TABLE ... DROPPARTITION statement. DROP PARTITION will delete all the data and also remove thepartitionfromthetablepartitiondefinition.

The following example illustrates the DROP PARTITION option with the ALTER TABLEstatement:

SET@@SQL_MODE='';

CREATETABLEemployee(

idINTNOTNULLAUTO_INCREMENTPRIMARYKEY,

first_nameVARCHAR(25)NOTNULL,

last_nameVARCHAR(25)NOTNULL,

store_idINTNOTNULL,

department_idINTNOTNULL

)

PARTITIONBYRANGE(id)(

PARTITIONp0VALUESLESSTHAN(5),

PARTITIONp1VALUESLESSTHAN(10),

PARTITIONp2VALUESLESSTHAN(15),

PARTITIONp3VALUESLESSTHANMAXVALUE

);

INSERTINTOemployeeVALUES

('','Chintan','Mehta',3,2),('','Bhumil','Raval',1,2),

('','Subhash','Shah',3,4),('','Siva','Stark',2,4),

('','Chintan','Gajjar',1,1),('','Mansi','Panchal',2,3),

('','Hetal','Oza',2,1),('','Parag','Patel',3,1),

('','Pooja','Shah',1,3),('','Samir','Bhatt',2,4),

('','Pritesh','Shah',1,4),('','Jaymin','Patel',3,2),

('','Ruchek','Shah',1,2),('','Chandni','Patel',3,3),

('','Mittal','Patel',2,3),('','Shailesh','Patel',2,2),

('','Krutika','Dave',3,3),('','Dinesh','Patel',3,2);

ALTERTABLEemployeeDROPPARTITIONp2;

In theprecedingstatement,afterexecuting theALTER TABLE employee DROP PARTITIONp2;statement,youcanseethatalldataisremovedfrompartitionp2.Incaseyouwant to remove all the data but also need to keep the table definition and thepartitioningscheme,youcanusetheTRUNCATEPARTITIONoptiontoachieveasimilar

result.

InordertoaddnewLISTorRANGEpartitionstoexistingpartitionedtablesyoucanusetheALTERTABLE...ADDPARTITIONstatement.

ByusingtheSHOWCREATETABLEstatementyoucanverifyandseeiftheALTERTABLEstatement has the desired effect on the table definition and the partitioningschema.

<strong>CREATETABLEclient(</strong><br/><strong>client_idINT,</strong><br/><strong>first_nameVARCHAR(25),</strong><br/><strong>last_nameVARCHAR(25),</strong><br/><strong>signedDATE</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(MONTH(signed))</strong><br/><strong>PARTITIONS12;</strong>

<strong>ALTERTABLEclientCOALESCEPARTITION8;</strong>

Intheprecedingstatementthenumber8representsthenumberofpartitionstoberemovedfromthetable.Youcannotremovemorepartitionsthanalreadyexistinthetablepartitioningschema.Similarly,youcanaddmorepartitionsusingtheALTERTABLE...ADDPARTITIONstatement.

<strong>ALTERTABLEtrpREBUILDPARTITIONp0,p1,p2;</strong>

<strong>ALTERTABLEtopOPTIMIZEPARTITIONp0,p1,p2;</strong>

<strong>ALTERTABLEtapANALYZEPARTITIONp1,p2;</strong>

<strong>ALTERTABLEtrpREPAIRPARTITIONp3;</strong>

<strong>ALTERTABLEtcpCHECKPARTITIONp0;</strong>

ThereisanoptiontouseALLinsteadofaspecificpartition,specifiedinallaboveoptions,inordertoperformtheoperationonallthepartitions.

<strong>SHOWCREATETABLEemployee;</strong>

Theoutputfromtheprecedingstatementhasseparateinformationforpartitioningschema,includingcommoninformationforthetableschema.

Similarly,youcanretrieveinformationaboutpartitioningfromtheINFORMATION_SCHEMA.PARTITIONStable.

TheEXPLAINoptiongivesalotofinformationonpartitioningwithcolumn.Forexampleitgivesnumberofrowsobtainedfromthequeryspecifictopartitions.Thepartitionwouldbesearchedasperthequerystatement.Italsogivesinformationaboutkeys.

EXPLAINisalsousedtogetinformationfromnonpartitionedtables.Itdoesnotgiveanyerroriftherearenopartitions,butgivesaNULLvalueinthepartitionscolumn.

PartitionselectionandpruningInthissection,youwillseehowpartitioningcanoptimizeSQLstatementsclauseexecution with the optimizer known as partition pruning, and the use of SQLstatements to effectively use partition data for selection and performmodificationoperationsonthepartitioning.

PartitionpruningPartitionpruningisrelatedtotheoptimizationconcept inpartition.Inpartitionpruning the concept described as Do not scan partitions where no possiblematchingvaluescanbepresentisappliedbasedonthequerystatements.

Supposethereisapartitionedtable,tp1,createdwiththefollowingstatement:

CREATETABLEtp1(

first_nameVARCHAR(30)NOTNULL,

last_nameVARCHAR(30)NOTNULL,

zone_codeTINYINTUNSIGNEDNOTNULL,

dojDATENOTNULL

)

PARTITIONBYRANGE(zone_code)(

PARTITIONp0VALUESLESSTHAN(65),

PARTITIONp1VALUESLESSTHAN(129),

PARTITIONp2VALUESLESSTHAN(193),

PARTITIONp3VALUESLESSTHANMAXVALUE

);

Intheprecedingexampletabletp1,supposeyouwanttoretrievearesultfromthefollowingSELECTstatement:

SELECTfirst_name,last_name,dojfromtp1wherezone_code>126ANDzone_code<131;

Now,youcanseefromtheprecedingstatementthattherearenorowsthathavedatainpartitionsp0orp3asperthestatement,soweonlyneedtosearchthedatainp1orp2formatchingcriteria.So,bylimitingthesearch,itispossibletospendlesstimeandeffortmatchingandsearchingforthedatathroughallthepartitionsinthetable.Thiscuttingawayoftheunmatchedpartitionsisknownaspruning.

The optimizer can make use of partition pruning for performing the queryexecution much faster compared to nonpartitioned tables that have the sameschema,data,andquerystatements.

TheoptimizercandopruninginthefollowingcasesbasedontheWHEREconditionreduction:

partition_columnIN(constant1,constant2,...,contantN)

partition_column=constant

Inthefirstcase, theoptimizerevaluates thepartitioningexpressionforeachofthe values in the list and creates a list of partitions that are matched duringevaluation,andthenscanningorsearchingisperformedonlyonthepartitionsinthispartitionlist.

In the second case, the optimizer only evaluates the partitioning expressionbased on the given constant or specific value and determines which partitioncontainsthevalue,andsearchingorscanningisperformedonlyonthispartition.Therecanbeuseofanotherarithmeticcomparisoninsteadofequalsforthistypeofcase.

Currently,pruningisnotsupportedonINSERTstatementsbutissupportedinSELECT,UPDATE,andDELETEstatements.

Pruning is also applicable to short rangeswhere theoptimizer can convert therangesintoanequivalentlistofvalues.Theoptimizercanbeappliedwhenthepartitioning expression consists of equality or range that can be reduced toequalitiessetorifanincreasingordecreasingrelationshipisrepresentedbythepartitioningexpression.

Pruning can also applicable to the column types of DATE or DATETIME if thepartitioningusestheTO_DAYS()orYEAR()function,andalsoapplicableifsuchtablesusetheTO_SECONDS()functionintheirpartitioningexpression.

Supposeyouhaveatable,tp2,asperthefollowingstatement:

CREATETABLEtp2(

first_nameVARCHAR(30)NOTNULL,

last_nameVARCHAR(30)NOTNULL,

zone_codeTINYINTUNSIGNEDNOTNULL,

dojDATENOTNULL

)

PARTITIONBYRANGE(YEAR(doj))(

PARTITIONp0VALUESLESSTHAN(1971),

PARTITIONp1VALUESLESSTHAN(1976),

PARTITIONp2VALUESLESSTHAN(1981),

PARTITIONp3VALUESLESSTHAN(1986),

PARTITIONp4VALUESLESSTHAN(1991),

PARTITIONp5VALUESLESSTHAN(1996),

PARTITIONp6VALUESLESSTHAN(2001),

PARTITIONp7VALUESLESSTHAN(2006),

PARTITIONp8VALUESLESSTHANMAXVALUE

);

Now, in the preceding statement the following statements can benefit from

partitionpruning:

SELECT*FROMtp2WHEREdoj='1982-06-24';

UPDATEtp2SETregion_code=8WHEREdojBETWEEN'1991-02-16'AND'1997-04-26';

DELETEFROMtp2WHEREdoj>='1984-06-22'ANDdoj<='1999-06-22';

Forthelaststatement,theoptimizercanactasfollows:

1. Finding thepartition thathas the lowendof therangeasYEAR('1984-06-22')givesthevalue1984,foundinthep3partition.

2. FindingthepartitionthathasthehighendoftherangeasYEAR('1999-06-22')givesthevalue1999,foundinthep5partition.

3. Scan only the above two identified partitions and any partitions that liebetweenthem.

So, in the abovementioned case thepartitions tobe scannedare p3,p4, and p5only,andtherestofthepartitionscanbeignoredwhilematching.

The preceding examples use RANGE partitioning but partition pruning is alsoapplicableonothertypesofpartitioningaswell.Supposeyouhavethetabletp3schemaasperthefollowingstatement:

CREATETABLEtp3(

first_nameVARCHAR(30)NOTNULL,

last_nameVARCHAR(30)NOTNULL,

zone_codeTINYINTUNSIGNEDNOTNULL,

descriptionVARCHAR(250),

dojDATENOTNULL

)

PARTITIONBYLIST(zone_code)(

PARTITIONp0VALUESIN(1,3),

PARTITIONp1VALUESIN(2,5,8),

PARTITIONp2VALUESIN(4,9),

PARTITIONp3VALUESIN(6,7,10)

);

Fortheprecedingtableschema,considerifthisstatementSELECT*FROMtp3WHEREzone_codeBETWEEN1AND3istobeexecuted.Theoptimizerdetermineswhichofthepartitionscanhavethevalues1,2,and3andfindsp1andp0,soitskipstherestofthepartitionsp3andp2.

Column values with a constant can be pruned, as in the following examplestatement:

UPDATEtp3setdescription='ThisisdescriptionforZone5'WHEREzone_code=5;

The optimization is performed onlywhen the size of the range issmallerthanthenumberofpartitions.

PartitionselectionExplicitselectionofpartitionandsubpartitionisalsosupportedandthisenablesrowmatchingtoconditionsgiveninthewhereclause-thisisknownaspartitionselection. It is very similar to partition pruning as only specific partitions arescannedformatching,butdiffersinthefollowingtwokeyaspects:

ThepartitionstobescannedarespecifiedbytheissuerofthestatementandarenotautomaticsuchaswithpartitionpruningThe partition pruning is limited to queries, whereas partition selectionsupportsbothqueriesandanumberofDMLstatements

SQLstatementssupportedforexplicitpartitionselectionarelistedasfollows:

INSERT

SELECT

UPDATE

REPLACE

LOADDATA

LOADXML

DELETE

The following syntax with the PARTITION option is used for explicit partitionselection:

PARTITION(partition_names)

partition_names:

partition_name,...

Theprecedingoptionisalwaysfollowedbythetablestructureortableschemaitbelongs to. partition_names stands for the list of comma separated names ofpartitions or subpartitions that will be used in partitioning. Partition andsubpartitionnamesinpartition_namescanbeinanyorderorevenoverlapbuteachname from the list must be the existing partition or subpartition name of thespecific table, otherwise the statement will fail with the error messagepartition_namedoesn'texist.

IfthePARTITIONoptionisused,onlylistedpartitionsandsubpartitionsarecheckedformatching rows.PARTITION option can also be used in the SELECT statement toretrieverowsbelongingtoanygivenpartition.

Supposeyouhavethetableemployeecreatedwiththefollowingstatements:

SET@@SQL_MODE='';

CREATETABLEemployee(

idINTNOTNULLAUTO_INCREMENTPRIMARYKEY,

first_nameVARCHAR(25)NOTNULL,

last_nameVARCHAR(25)NOTNULL,

store_idINTNOTNULL,

department_idINTNOTNULL

)

PARTITIONBYRANGE(id)(

PARTITIONp0VALUESLESSTHAN(5),

PARTITIONp1VALUESLESSTHAN(10),

PARTITIONp2VALUESLESSTHAN(15),

PARTITIONp3VALUESLESSTHANMAXVALUE

);

INSERTINTOemployeeVALUES

('','Chintan','Mehta',3,2),('','Bhumil','Raval',1,2),

('','Subhash','Shah',3,4),('','Siva','Stark',2,4),

('','Chintan','Gajjar',1,1),('','Mansi','Panchal',2,3),

('','Hetal','Oza',2,1),('','Parag','Patel',3,1),

('','Pooja','Shah',1,3),('','Samir','Bhatt',2,4),

('','Pritesh','Shah',1,4),('','Jaymin','Patel',3,2),

('','Ruchek','Shah',1,2),('','Chandni','Patel',3,3),

('','Mittal','Patel',2,3),('','Shailesh','Patel',2,2),

('','Krutika','Dave',3,3),('','Dinesh','Patel',3,2);

Now,ifyoucheckwithpartitionp1,youseethefollowingoutputasrowsaddedinpartitionp1:

mysql>SELECT*FROMemployeePARTITION(p1);

+----+-----------+------------+----------+---------------+

|id|last_name|last_name|store_id|department_id|

+----+-----------+------------+----------+---------------+

|5|Chintan|Gajjar|1|1|

|6|Mansi|Panchal|2|3|

|7|Hetal|Oza|2|1|

|8|Parag|Patel|3|1|

|9|Pooja|Shah|1|3|

+----+-----------+------------+----------+---------------+

5rowsinset(0.00sec)

IfyouusethisstatementSELECT*FROMemployeeWHEREidBETWEEN5AND9;,itwillgivethesameoutput.

Inordertoretrieverowsfrommultiplepartitionsyoucanuseacommaseparatedlistofpartitionnames.Forexample,SELECT * FROM employee PARTITION (p1,p2), willresult in all the rows from partitions p1 and p2 and exclude the remainingpartitions.

Any supported partitioning types can be used using partitioning selectionstatements.MySQL8automaticallyaddspartitionnameswhenatableiscreatedwiththeLINEARHASHorLINEARKEYpartitioningtypesspecifiedwithoutanynames,and this is also applicable to subpartitions aswell.While executing the SELECTstatementonthistableyoucanspecifypartitionnamesgeneratedbyMySQL8forpartitionspecificdataretrieval.

ThePARTITION option is also applicableon the SELECT statement for the INSERT...SELECTstatement,bywhichwecaninsertdataretrievedfromspecificpartitionsorsubpartitionsaswell.

ThePARTITIONoptionisalsoapplicableontheSELECTstatementwithjoinqueriesontableswithspecificpartitionorsubpartitiondata.

Restrictions and limitations inpartitioningIn this section, you will see the restrictions and limitations in MySQL 8partitioning, covering prohibited constructs, performance considerations, andlimitation aspects related to storage engines and functions in detail, to gainoptimumbenefitsfromthetablepartitioning.

<strong>CREATETABLEtk1(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1,cl2)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl3)</strong><br/><strong>PARTITIONS4;</strong><br/><br/><strong>CREATETABLEtk2(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1),</strong><br/><strong>UNIQUEKEY(cl3)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl1+cl3)</strong><br/><strong>PARTITIONS4;</strong>

<strong>CREATETABLEtk1(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1,cl2,cl3)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl3)</strong><br/><strong>PARTITIONS4;</strong><br/><br/><strong>CREATETABLEtk2(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1,cl3)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl1+cl3)</strong><br/><strong>PARTITIONS4;</strong>

<strong>CREATETABLEtk4(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2INTNOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4

INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1,cl3),</strong><br/><strong>UNIQUEKEY(cl2,cl4)</strong><br/><strong>);</strong>

<strong>CREATETABLEtk5(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>PRIMARYKEY(cl1,cl2)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl3)</strong><br/><strong>PARTITIONS4;</strong><br/><br/><strong>CREATETABLEtk6(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>PRIMARYKEY(cl1,cl3),</strong><br/><strong>UNIQUEKEY(cl2)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(YEAR(cl2))</strong><br/><strong>PARTITIONS4;</strong>

<strong>CREATETABLEtk7(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>PRIMARYKEY(cl1,cl2)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl1+YEAR(cl2))</strong><br/><strong>PARTITIONS4;</strong><br/><br/><strong>CREATETABLEtk8(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>PRIMARYKEY(cl1,cl2,cl4),</strong><br/><strong>UNIQUEKEY(cl2,cl1)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl1+YEAR(cl2))</strong><br/><strong>PARTITIONS4;</strong>

Ifthetabledoesnothaveauniquekeyorprimarykeythentherestrictionisnotapplicable,andanycolumnorcolumnscanbeusedinthepartitioningexpressionaspercompatiblecolumntypesforthepartitioningtype.AllaboverestrictionsarealsoapplicabletotheALTERTABLEstatementsaswell.

PartitioninglimitationsrelatingtostorageenginesPartitioning support is not provided by MySQL server but from the storageengine's

ownornativepartitioninghandlerinMySQL8.InMySQL8,theInnoDBstorageengineonlyprovidesanativepartitioninghandlerandsothepartitioned

tablecreationisnotapplicablewithanyotherstorageengine.

ALTER TABLE ... OPTIMIZE PARTITION doesnotwork correctlywith the InnoDB storageengine, so instead use the ALTER TABLE ... REBUILD PARTITION and ALTER TABLE ...

ANALYZEPARTITIONoperationsforsuchtables.

Partitioninglimitationsrelatingtofunctions

In partitioning expressions only the following listed MySQL functions areallowedinMySQL8:

ABS():Itprovidesanabsolutevalueforthegivenargument

CEILING(): It provides the smallest integer number possible for the givenargument

DAY():Itprovidesthedayofthemonthforthegivendate

DAYOFMONTH():ItprovidesthedayofthemonthforthegivendatesameasDAY()

DAYOFWEEK():Itprovidestheweekdaynumberforthegivendate

DAYOFYEAR():Itprovidesthedayoftheyearforthegivendate

DATEDIFF():Itprovidesthenumberofdaysbetweentwogivendates

EXTRACT():Itprovidespartofthegivenargument

FLOOR():Itprovidesthelargestintegervaluepossibleforthegivenargument

HOUR():Itprovidesthehourfromthegivenargument

MICROSECOND():Itprovidesthemicrosecondsfromthegivenargument

MINUTE():Itprovidestheminutefromthegivenargument

MOD(): It performs theModulo operation and provides the remainder of NdividedbyMwhereMOD(N,M)

MONTH():Itprovidesthemonthfromthegivenargument

QUARTER():Itprovidesthequarterfromthegivenargument

SECOND():Itprovidesthesecondfromthegivenargument

TIME_TO_SEC():Itprovidesthesecondfromthegiventimevalueargument

TO_DAYS():Itprovidesthenumberofdaysfromyear0forthegivenargument

TO_SECONDS():Itprovidesthenumberofsecondsfromtheyear0forthegivenargument

UNIX_TIMESTAMP()(withTIMESTAMPcolumns): Itprovides thesecondssince '1970-01-0100:00:00'UTCforthegivenargument

WEEKDAY():Itprovidestheweekdayindexforthegivenargument

YEAR():Itprovidestheyearforthegivenargument

YEARWEEK():Itprovidestheyearandweekforthegivenargument

PartitionpruningsupportstheTO_DAYS(),TO_SECONDS(),TO_YEAR(),andUNIX_TIMESTAMP()functionsinMySQL8.

Summary

Inthischapter,welearnedaboutdifferenttypesofpartitioningandtheneedforpartitions. We also covered detailed information on managing all types ofpartitions.

Welearnedaboutpartitionpruningandselectionofpartitionswhichisusedbythe optimizer. We also discussed applicable limitations and restrictions toconsiderwhileusingpartitioning.

Inthenextchapter,youwilllearnhowtodoscalinginMySQL8,anddiscovercommonchallengesfacedwhenprovidingscalabilityinMySQL8.Youwillalsolearn how to make the MySQL server highly available and achieve highavailability.

MySQL 8 – Scalability and HighAvailability

In this chapter, we will cover the following important topics for MySQL 8scalabilityandhighavailability:

OverviewofscalabilityandhighavailabilityinMySQL8

ScalingMySQL8

ChallengesinscalingMySQL8

Achievinghighavailability

Beforewemoveontothedetails,let'shaveanoverviewofscalabilityandhighavailabilityinMySQL8

Overview of scalability and highavailabilityinMySQL8

In any type of application, be it mobile, web portals, websites, social, e-commerce, enterprise or cloud applications, data is the core portion of thebusiness.Dataavailability isconsideredanutmostconcernforanybusinessororganization.Datalossoranydowntimeofanapplicationcanresultinaheavylossintermsofmoneyandalsoimpactthecreditofthecompanyinthemarket.

Ifweconsideranexampleofanonlineshoppingsitewhichhasanicelycoveredmarket in a specific area, with customers and good business credit. If thisbusiness faced an issue with data loss or any application server or databaseserver downtime, itwould impact thewhole business.Many customerswouldlosefaithinthebusinessandalsothebusinesswouldsufferalossbothintermsoffinanceandcredit.

Thereisnosingleformulathatcanprovideasolution.Differentbusinesseshavetheirownapplicationrequirements,businessneeds,distinctprocesses,differentinfrastructure in different locations, and operational competencies. In thesecircumstances,technologyplaysamajorroleinachievinghighavailability.

Aspertherequirementsofscalabilityandhighavailability,MySQLcanbeusedfor various applications, and as per need it is capable of overcoming failures,includingfailuresofMySQL,failuresfromtheoperatingsystem,oranyplannedmaintenance activity thatmay impact availability. Scalability in simple terms,that has the capability to distribute database load and application queriesbetweenMySQLservers.

Theattributesthatmatterwhenchoosingtherightsolutionforhighavailabilitydepend on to what extent the system can be called highly available, as suchrequirements vary from system to system. For smaller applications,where theuser load is not expected to be very high, setting up the replication or clusterenvironment can result in very high cost. In such cases, providing the correctconfigurationoftheMySQLcanalsobeenoughtoreduceapplicationload.

The following sections briefly describe the primary solutions supported byMySQL8

forhighavailability.

MySQLreplicationMySQL replication allows data from one server to be replicated onto themultipleMySQL

servers. MySQL replication provides master-slave design, so that one of theservers from the group acts as amasterwherewrite operations are performedfromtheapplicationand then themasterservercopies thedata to themultipleslaveservers.ReplicationisawellestablishedsolutionforhighavailabilityandisusedbythesocialgiantssuchasFacebook,Twitter,andsoon.

MySQLclusterThisisanotherpopularhighavailabilitysolutionforMySQL.Clusteringenablesdata to be replicated tomultipleMySQL serverswith automated sharing. It isdesignedforbetteravailabilityandthroughput.

OracleMySQLcloudserviceOracleMySQLcloudserviceprovidesanefficientmeanstohelpbuildasecure,cost-effective MySQL database as a service for applications used in modernworld.Itprovestobescalableandcost-efficientwithlessresourceutilizationformanagingtheservicewhencomparedtoon-premises.

MySQLwiththeSolariscluster

The sun Solaris cluster provided by the MySQL data service provides amechanism for orderly startup and shutdown, faultmonitoring, and automaticfailoveroftheMySQL

service.ThefollowingMySQLcomponentsareprotectedbythesunclusterHAfortheMySQLdataservice.

There are some further options available using third-party solutions. Eacharchitecture that is used to achieve highly available database services isdifferentiatedbythelevelsofuptimethateachoffers.Thesearchitecturescanbegroupedintothreemaincategories:

Datareplication

Clusteredandvirtualizedsystems

Geographically-replicatedclusters

Basedonthebestanswertothequestion,youcanselecttherightoptionforyour

application with optimal cost and a highly available solution. This discussiongivesusafairoverviewofMySQL8'shighavailability.

ScalingMySQL8

Scalability is theability todistribute the loadofanyapplicationqueriesacrossvariousMySQL instances.For somecases, it is unpredictable thatdata cannotexceed up to some limit or the number of users will not go out of bounds.Scalabledatabaseswouldbeapreferablesolutionsothat,atanypoint,wecanmeetunexpecteddemandsofscale.MySQLisarewardingdatabasesystemforits scalability, which can scale horizontally and vertically; in terms of data,distributionofclientqueriesacrossvariousMySQLinstancesisquitefeasible.ItisprettyeasytoaddhorsepowertotheMySQLclustertohandletheload.

The requirements for achievingHigh Availability (HA) and scalability mayvary from system to system.Each system requires a different configuration inorder to achieve these abilities. There are many questions that come to mindwhen we think about scaling in MySQL, and while we perform scalingoperationsinMySQL:

Whyisscalingrequired?

WhataretheadvantagesofscalinginMySQL?

Whatpointsneed toputacross inourmindswhenweperformscaling inMySQL?

Howwillscalingwork?

Isitsecurefordata-doesitprovidesuretyofdatasecurity?

Plusmanymore...

Let's takea real timeexample tounderstandwhyweneed scaling inMySQL.Wehave anonline e-commercewebsite that has covered a smallmarket,withlimitedusersandlimitedhitsonthewebsite,withasingledatabaseserver.Thebusiness isgrowingupnicely; theperformanceof thebusiness iscontinuouslyincreasingandtheusercountisincreasing,andwithoursingledatabaseserverall requests and performance cannot be scaled at all time. This may possiblyresult ina servercrashand thebusinessmight face loss in termsofprofit andcreditinthemarket.Toavoidsuchasituation,scalabilitywillperformamajorpart.Ifanyrequestfromacustomerfailsduetoanyreason,orifthenodegoesdown, the other node will take care of it quickly and give the appropriateresponsetothecustomer.

Scaling is required for the continuous increase in performance of databaseresponsetimeandtoimprovetheproductivityoftheproduct.Itwillhelptheendproductintermsofdatascalability,performance,andbetterresults.ClusterandreplicationarebothkeyfeaturesinMySQLthatcanbeleveragedforscaling.

ScalingusingclusterBasicclusterarchitectureisdividedintofourdifferentlayers:

ClientnodeApplicationnodeManagementnodeDatanode

Theseareshowninthefollowingimage:

ClientnodeTheclientnodeisanenduserorapplicationthatsendsarequestforanyqueryinterms of read data or write data from a different device, such as a computer,mobile,tablet,andsoon.

Applicationnode

The application node ismeant to provide the bridge between the logic of theapplication and the nodes containing the data in MySQL. Applications canaccess the data, which is stored in theMySQL cluster, by SQL, with one ormanyMySQL servers using the function of SQL. In the application we havemultipletechnologiesfromwhereweconnecttotheMySQLserver.WeconnectMySQLserverwithstandardtheMySQLconnectors,whichgivesustheabilitytoconnectwithawiderangeofaccesstechnologies.

Asanotheroption,wehaveNDBAPI;ahighperformanceinterfacethatcanbeused to control real-time user experiences and provide better throughput. InMySQLwehaveNDBAPI,whichaddsalayeradditionallytoNoSQLinterfacesthatconsistcapabilitytoaccesstheclusterdirectly.Applicationnodescanfetchdatafromallthedatanodes,sotheonlycauseoffailurecanbetheunavailabilityofapplicationservices,astheapplicationcanusealldatanodestoperformdatamanipulation.

ManagementnodeThemanagementnodeperformstheimportantroleofpublishingrelevantclusterinformationacrossthenodesinitscluster,alongwithnodemanagement.NodesformanagementworkatstartupwhenallnodeswanttojointheMySQLclusterandalsowhenreconfigurationofthesystemisrequired.Themanagementnodecan be stopped and restart all services without damaging or impacting anongoingoperation,execution,orprocessingofdataandtheapplicationnode.

DatanodeThedatanodesstores thedata.Tablesget sharedacross thedatanodes,whichalsohelpstohandleloadbalancing,replication,andhighavailabilityfailover.

Data nodes are the main nodes of aMySQL cluster solution. It provides thefollowingfunctionalityandbenefits:

Datastorageandmanagementofdisk-basedandin-memorydata

Inashared-nothingscenario,dataisstoredinatleastonereplicawithouttheuseof shared-disk space.MySQLcreate one replica of the databasewhich does asynchronous replication process. If any data node fails due to any specificreason,thereplicateddatawilltakecareofitandprovidetherespectiveoutput.It does a synchronous copy of the node so it consists of the same data as themainnodedata.

We can store the data either in memory or partially on disk based on therequirement.

Data that frequently change are suggested tobe stored in-memory. In-memorydataisroutinelycheckedwiththelocaldiskandcoordinatestoupdatethedatatotherestofthedatanodes.

Automatic and user-definedpartitioningof tablesor shardingoftablesMySQL cluster provides low latency, high throughput, scalability, and highavailability.

Thisadoptshorizontalscalingandautoshardingtoserveheavyloadread/writeoperations

throughthedifferentNoSQLqueries.AnNDBclusterisasetofdifferentnodeswhere

eachtaskisrunningonitsownprocessor.

Synchronous data replicationbetweendatanodesWhen we have data replication for the data node it follows synchronousreplication,soatanytimeallnodedatawillbein-sync.Ifanynodefailsforanyreason, the other nodes have the samedata and sowill be able to provide thedataforaquery.

So, without any downtime for data response, MySQL provides a perfectsolution.

DataretrievalandtransactionsMySQLsupportseachofthetransactionsthatcanbemapped,asitiscommittedonthemasterserverandappliedontheslaveserver.Thismethodisnotreferringtobinlog filesor therelevantposition in thebinlog file.GTID replication issolelyworking based on transactions; it becomes very easy to identify whether themasterandtheslaveserversareinsyncornot.

AutomaticfailoverIf any data node fails for any reason, the other nodes take responsibility andgivestheresponsetotherequest.Replicationofthedatabaseisveryhelpful incriticalconditionsofdowntimeorafailureinanyofthenodes.

Automatic re-synchronization forself-healingafterfailureIf any node is failed it will start automatically and again perform thesynchronizationofdatatotherestofthenodes,whichareactivenodes,andcopyallrecentdatainthenode.Inthatcaseitdoesself-healingofthefailures.

Scaling using memcached inMySQL8

In MySQL 8, using memcached is one of the ways to achieve scalability.Memcachedisasimpleandhighlyscalablesolutionforstoringdatainkeyandvalue form in cachewhenevermemory is available.Memcached is commonlyusedforquickaccessofdata.

Data stored inmemorydoesn't have I/Ooperationsperformed for fetching thedata.

As all the information is stored inmemory, the access speed for data ismuchfaster than compared to loading every time from disk and results in a betterqueryexecutiontimeonthedatabaseserver.Thispluginalsohasthefeatureofserialization,which converts binary files, code blocks, or any other objects tostringsthatcanbestored,andprovidesasimplemeanstoretrievesuchobjects.Whilespecifyingamemoryallocationitshouldnotbelargerthantheavailablephysicalmemoryoftheserver.

If you specify too large of a value then some of the memory allocated formemcached will use swap space and not physical memory. This may lead to

delays when storing and retrieving values because data is swapped to diskinsteadofstoringthedatadirectlyinmemory:

Thepreceding imagedepictsmemcachedarchitecture,whichdisplays the flowofthedatafrommemcachedtoaclientoranenduser,orarequestofthedatafromanapplication.

Thedatainmemcachednevergetsstoredinthedatabase.It'salwaysavailableinmemoryitself.Ifeitherofthememcachedserversfail,datawillbefetchedfromthedatabase, so itwillnot impact end-users fordata retrievalorhaveamajorperformance impact on the application. The only thing need to keep in mindwhile we use a memcached server is that data related to any importantinformation, for instance a financial transaction, should not be placed inmemcached.Inthatcaseifthereisafailureinmemcached,thedatamightnotberetrieved. In a memcached server data integrity is not healthy as it stores inmemory, soduring failure itwouldbegood tohavedata that is important notsaved inmemcached.When configuring amemcached server,memory size isthe key factor. If there is improper configuration, then you can expect a badsituation.

Thiswaywecanusememcached to scale theMySQLserver for an increaseddataresponsetime,andtoprovidefasterperformance.ItwillreducetheloadonMySQLserverandmultipleserversasapartofcachegroupandalsoprovidesaninterfaceformultiplelanguages.Itissuggestedtobeusedideallywhenthereareheavyreadoperations.

NoSQLAPIsMySQLcluster provides numerousways to help access thedata store.Oneofmostgeneric

wayisleveragingSQL;however,inreal-worldusecaseswecanalsodependonnative

APIs, which allow the fetching of the data from within the database withoutaffecting

performance or adding further complexity by developing an application toconvertSQL.

Scalingusingreplication

Replication is thecopyingofaMySQLdatabase.MySQLprovides replicationwith a different approach. MySQL has a feature of replication that providesscale-outsolutions,datasecurity,longdistancedatadistributionandmanymorebenefits.WehavediscussedthisatlengthinChapter8,Replication inMySQL8.The following imageexplains thebasicarchitectureof replication inMySQL:

ReplicationisoneofthebestfeaturesofMySQL.Itsimplycreatesacopyofthedatatothenewserveroranotherphysicalmachine,whichwill importthedatafrom the master server. Whenever the data is required it will populate theaccurateresults.

Itfollowsthemasterandslaveapproachforthereplication.Themasterdatabaseis the actual database of the application and the slave database is created byMySQL

inthedatabaseserverofanotherphysicalserver,whichcontainsreplicateddatafrom the master server. We can configure the slave database for the specificoperation,suchaswhenthequeryrelatestoreadingthedatafromthedatabase;wecanexecutethisontheslaveserver.Inthiscasethemasterdatawillhavelessloadthanearlier.

Supposewehavearatioofthe40%writedataqueryand60%readdataquery;inthiscaseifwehaveasingleserveritwillhandlealloperationsrelatedtotheread and write operation. But, as defined in the preceding image we havereplicated the database in two different servers and read operations are

performedontheslaveservers,sowecanmakeuseofoneoftheslaveservertoperform complex read queries. This makes it possible to generate reports fordoingdataanalysisonMySQL8,asperformingcomplexreadquerieswillnotimpacttheoverallapplicationperformance.

In standardMySQL replication the master server creates binary log files andmaintains the indexof the log files tomaintainandkeep trackof log rotation.Thebinary log files serve the recordsupdatesandare sent to the slave server.Whentheslaveserverconnectstothemasterdatabaseserveritconsidersthelastpositionithasreadinthelogfiles,afterwhichtheslaveserverthenreceivesanyupdatesthathavetakenplacesincethattime.Theslavesubsequentlyblocksandwaitsforthemastertonotifyitforfurtherupdates.

Thequestioninmindiswhydoweneedreplication?Or,whatisthepurposeofthereplication?If replicationrequiresanotherdatabaseserver,complexity,andadditionalconfiguration,itincreasesthemaintenanceandmonitoringtime.Still,wehavemanyadditionalbenefitsforbusinessanddatabaseadmin.

SingleserverdependancyInanycase,ifthemasterdatabaseserverfailswecaneasilyswitchourdatabaseconnectiontothereplicatedslaveservertoprovidestabilityincriticalsituations.This includes if there is network failure, server failure, hardware issues, andmanymorereasonsforfailure.

PerformancePerformance is the main part in the database. When we have a distributeddatabaseovermultipleserverswecanconnectdifferentapplicationstodifferentdatabaseserverstoincreaseperformance.Thisfeaturereducestheresponsetimeofthequery.

BackupandrecoveryReplicationhelpsbackup themasterdatabase. It ismoreefficient thanstoringthe database on disk. Users can store the database in the master using thereplicated database as a backup instead of digging up the backup files.Whenrequiredtorestorethedataofthemasterserverausercaneasilygetitfromtheslaveserver,withnoneedtoworkonthebackupfilesandgoaboutfindingthelastupdatesandotheroperations.

Loaddistribution

Byusing the replication loadof thedatabase,queryexecutioncanbe reduced;wecan split read andwriteoperationsover thedatabases. Ifweexecutewriteoperationsinthemasterdatabaseandreadoperationsintheslavedatabasethatwillimprovetheresponsetimeoftheapplication.Wecancreateloadbalancedenvironments inMySQL,which share the load of all requests to the databaseserver. The load balancer then further sends requests to the database that canhandleeachtransactionwithmuchbetterthroughput.

AsynchronousdatareplicationAsynchronousdata replicationmeans that data is copied fromonemachine toanother,with a resultant delay. This delay is based on networking bandwidth,resourceavailability,oratimeintervalsetbytheadministratorinconfiguration.The correct configuration and time setting provides an accurate result inresponse. It's basedon thenetwork administrator's configuration.Synchronousdata replication implies thatdata iscommitted tooneormoremachinesat thesametime.

GeographicaldatadistributionGroupreplicationmakesitpossibletocopythemaster'sdatatotheslaveserverthatresidesataremotelocationandperformthereadoperationsforaseparategroupofclientwithoutimpactingthemaster'soperations.

GTIDreplication

Global transaction identifiers (GTID) uses transaction based replication ofdatainsteadofbinarylogfilebasedreplication.

Until and unless transactions that have been operated and committed on themaster servers are present on all the slave servers, GTID will not considerreplicationinaconsistentstate.

InMySQL8 replicationcanbedoneeither in asynchronousmodeor in semi-synchronousmode. In asynchronousmode,write operations are performed onthemasterserverimmediately,whilereplicationinslavesisdoneperiodicallyaspertheconfiguration.

Inthesemi-synchronousmodeofreplication,ifsemi-synchronousconfigurationisenabledonthemasterandatleastoneslaveserver,atransactiononthemasternode waits before getting a transaction time out until the semi-synchronousenablednodeconfirms that requireddataorupdatehasbeen received.Andontime-out, the master node again looks for the semi-synchronous slave andperformsthereplication.

MySQL8providesanewreplicationmethod,GTID,whereauniqueidentifieriscreatedandconnectedwitheach transactionsavedorcommittedon themaster

server.Theuniquenessoftheseidentifiersisinallserversthatareintheserverwhere it's created, and also in the replicated servers. GTID have one to onemappingbetweenalltransactions.

The concept of the log file referencing a position within files is not requiredwhenstartinganewslavecreationorfailovertoanewMySQLmaster.Youcanuseeitherrow-basedorstatement-basedreplicationwithGTIDs.

UsingaglobaltransactionIDprimarilyprovidestwomajorbenefits:

It's easy to change a master server to connect with a slave serverduringfailover:GTIDisuniqueoutofallserversinthereplicationgroup.TheslaveserverrememberstheglobaltransactionIDofthelasteventfromtheoldmaster.Thismeansitbecomeseasytoidentifywheretoreinitializereplication on the newMySQLmaster, as the global transaction IDs areknownthroughouttheentirereplicationhierarchy.

The status of the slave provides a crash-safemethod: The slave holdscurrent position information in the mysql.gtid_slave_pos system table. If thistable is using a transactional storage engine (such as InnoDB, which is thedefault),thenfurtherupdatesaredoneinthesametransaction.

AGTIDsisauniquekeycreatedandassociatedwitheachtransaction(insertandupdateoperations)committedonthemasterserver.Thekeyisnotonlyuniquetothemasterserver,butit'suniqueacrossallserversinreplication.

ZFSreplication

TheZFSfile systemhas theability toprovisiona snapshotof theserver files,such as file system contents, transfer the snapshot to another machine, andextract thesnapshot to recreate the filesystemonadifferentserver.Userscancreateasnapshotatanytimeandcancreateasmanysnapshotsasrequired.Bycontinually creating, transferring, and restoring snapshots, it can providesynchronizationbetweenoneormoremachinesinafashionsimilartoDRBD.

We have seen all the possible ways to scale a database in MySQL, usingdifferenttechniques.

As per business need and flexibility we can perform scaling with databasebackup.

Scaling is not an easy task but its possible in MySQL 8, with the properknowledge and understanding of the requirements of the business and aconfigurationprovidedbyMySQL8.Fordatabasescaling,wemusthaveproperunderstanding of the entire workflow of the database and communicationapproach.

ChallengesinscalingMySQL8

Wehaveseenhowscalingworksandtheadvantagesandpurposeofscalingintheprevioustopic.WhenwestartworkingwithscalinginMySQL8,whattypeof challengeswillwe face, andwhat steps need to be kept inmindwhilewework towardsscaling?Wehave toaccount for ifwearedoingscalingand themasterserverfails,limitsarereached,readandwriteoperationsarenotabletohandle the requests of the application, or while re-platforming the database.Scaling is not an easy task; it needs to ensure it is able to handle increasingtransactions without any difficulty. At the time of performing the scaling weneedtokeepinmindmanypoints,suchasthewriteandreadoperationlimitsinthemasterandslaveservers.Databaseloadbalancingisoneof theapproachesthathelpreducethetrafficofthetransaction,butagainitneedsperfection,andneedstounderstandtheloadbalancingconfigurationproperly.

Thefollowingarethemajorchallengesfacedwhenweperformscaling.

Businesstypeandflexibility

This is thefirstpoint thatneeds tokept inmindwhileweperformthescaling.Business type or business behavior is the core part; if the business is an e-commerce, we already know e-commerce businesses have a number offunctionalities and very critical data about clients, such as product details,monopolyofthebusinessforoffersanddiscounts.

Themainthingiscustomerdetailsandpaymentinformationsuchascreditcarddetails,debitcarddetails,andcustomerfeedback.

Inthiscase,whilewearedoingscalinginMySQL8,allparametersneedtokeptinmind,suchasdatabasebackup,security,roles/privilegesofthedatabaseandbackwardcompatibilityofthescaling.Whiledoingscalingbyclusteringalldatanodesneedtobeonthesamepage.Iftheapplicationisdevelopedusingmultipletechnologies and we perform the scaling for each of the stack, we can havedifferentdatanodesavailable;inthiscasethedatabasesyncisoneofthemostimportantthingsthatneedtobesureinconfigurationwhilescaling.Whichtypeofdatashouldresideincachedmemoryinmemcachedandindiskshouldalsobeclearbeforewedesignscaling.

Thebehavioroftheapplicationaccessesdatafromtheshareddatanodes.Ifwe

have an e-commerce site and we perform the sharding for the same and at acertain level the data are not available for the clientwho uses the data of theothershardserverforanyreason,atthattimethecross-nodetransactionwouldberequired.It'scompletelybasedonthebusinessbehavior,anddependsonhowflexible thebusiness iswhenitcomestoacceptingchangesregardingdatabasescaling.

Understandserverworkload

For thesetupof flexibility, scale,andperformance improvementmanyoptionsand

actions are available inMySQL 8.Many people face issueswhile performingsuchactivities

becausetheydonothaveenoughunderstandingorknowledgetohandlevarioustechnology

stacks and configuration option selections that can improve scalability,performance,

security, and the flexibility of the application and deployment activity. Theseconfiguration

options includingclustering, replication, sharding,memcached, storageengine,and

manymore,which can bewell designed to handle thewholeworkload of theapplication.

The database workload and business behavior helps to decide the MySQLconfiguration.

Read-writeoperationlimit

Whathappenifthereadandwritelimitisreachedandthetransactionincreaseson the master database server. MySQL has limitations to the capacity; forinstance if a number of customers are visiting the site at the same time that aread-write operation is running and the server or node are not synced, then atthattimeitwillcreateconfusionormisunderstandingfortheenduser.Or,inane-commerce site, ifonecustomer ispurchasing theproduct,which is last itemleft instock,andatsametimeanothercustomersearchesfor thesameproductandit'sstillavailable,inthiscasebothoperationsarenotinsyncintermsofthereadandwriteoperationofthedatabase.

Intheend,theothercustomermightpurchasethesameproduct,whichwedon'thaveinthewarehouse.Thisimpactsinventorycalculation,andcustomershavedoubtsabouttheprocessofthepurchasecycle.Inthatcasewewouldloosethefaithofthecustomerinthebusinessandthecreditofthebusinesswouldalsobeimpacted.

Anotherapproachistohavedatabasesharding.Shardingcanbesimplystatedaspartitioning thedatabase inmultipleservers.Shardinghelps to reduce the loadon a single database or the master database. If we have databases shardinggeographically,andfordifferentcountryorregionwehavedifferentserversfor

thedatabase,wecansolvetheissueofthelimitofreadandwriteoperationsontheMySQLserver.Butagain,thetechniquewhichweusefortheshardingalsodeterminestheperformanceofthedatabase.WehavealreadylearnedaboutthisindetailinChapter9,PartitioninginMySQL8.

Maintenance

WhilewehaveperformedscalinginMySQL8,wemustknowhowtomanagemasterandslaveservers,andwhichconfigurationisrequiredwhileperformingscaling.What are the steps thatneed tobe takencareof at the timewhen theserver is in a critical stage? What steps needs to be performed at time ofsharding,clustering,orreplicationofthedatabaseserver?

Scalingispossiblebutitsnotaneasyoperation.Ifwewanttoperformscalingwe should know that the database can handle more transactions without anyissue.Weshouldknowtheappropriateconfigurationtobedonetoovercomethedefault limits on themaster server for thewrite and read operation. Once it'scompleted,weneed to perform similarly steps to configure the slave databaseserver,which should only have read operations available for the end user andshouldalwaysbeinsyncwiththemasterdatabase.

Ifwehavemultipleservers,thenthemaintenanceoftheserveralsobecomesacostlyoverhead.Alltheserverneedstobeonsamepage,configurationshouldbeinpropermanner,andthecostoftheserverwillalsoaffectthebusiness.Ifthenumberofdataconstantly increasesat that time, server spacealsoneeds tobemanagedinanappropriatemanner.

MasterserverfailureIf themasterserverfailsanddata isnotavailable to thecustomerat that time,the enduserwill get frustrated and thebusinesswill behampered in termsofcreditinmarketandinlosingthecustomeralso.Thebusinesswillhavetosufferfromtheloss.

SynchronizationWhetherweperformthescalingwithclusteringorreplication,inbothcasesweneedtosecuresynchronization.Allslaveserversshouldhavethesamedatabaseasthemasterserver.Ifwriteoperationsareperformedonthemasterserverandreadoperationsareperformedontheslaveserver,atthattimealldataneedstobesyncedup.Allresultsshouldbesame,andifanyservergoesdownatatimewhendatawasnotsynced,itwillcreateissuesregardingthelossofdata.

Databasesecurity

How do we secure the database if we have different servers and sharding isperformed?

If we have different database servers at different locations and access to thedatabaseisnotrestrictedtotheuserspecificatthattimethentheissueofadataleak isastrongpossibility.Wehave tocompletelyunderstandaccesspointsofthe data in terms of IP configuration of the database server, with appropriaterolesandprivilegesforthedatabaseuserswhoperformvariousactivities.WhichIPs have access and which IPs need to restrict the data transaction from theserver?While we are performing the cross node transaction on the database,accurate data should be available; it should not give the permission to accessrestricteddatafromtheserver.

CrossnodetransactionCross node transaction is required when we have multiple nodes after doingscalingandonenoderequirestheothernodedataasapartofinput.Forinstance,ifwehavedifferentnodesatdifferentlocationsandwehaveasingleinventoryfor all the locations at that time, one user request for any product that is notavailableinthatdatanodeatthattimewillhavetocommunicatewithotherdatanodesfortheinformationoftheproduct,basedontheuser'srequest.

Growingteamfordevelopment

While theapplicationmayhaveapositive responseand itscontinuoussuccessincreasethebusinessteam,theexpansionofthedatabaseadministratorswillalsoberequired.

WhenweperformingsharingandscalingorreplicationinMySQL8,werequireappropriateteammemberswiththeproperknowledgeandexperiencetohandlecontinuous expansion and the management of database servers. It's not onlylimited to the setup of database servers; we also need to keep an eye onmaintenanceoftheserverandkeepwatchingtheserveractivityalso.

Managechangerequest

When we have a change in any database structure and we have alreadyperformedthescalingorreplicationthanafewthingsneedtobetakencareofaspartofachangerequest,or ifweaddanewfeatureoranenhancementof thefunctionality.Thisincludesthingssuchasupdatingsharingkeys,modifyingthedata distribution with replication of the nodes, updating the queries to takereplication latency into account to avoid stale data with on-going managingshards,databalancing,andensuringthatdataisavailablewithnewupdates.

Scale-upandscale-out

Scale-updescribestheprocessofmaximizingthecapacitythatasingleMySQLnode canhandle.Theprocessof scaling-up can involveoptimally tuningyourdatabasesoftwareandchoosingtherightstorageengine,aspreviouslydiscussedin Chapter 6,MySQL 8 Storage Engines, and selecting appropriate hardware.Therearelimitsonhowfaryoucanscale-upanodeandthesearedeterminedbysome combination of data size, schema complexity, CPU cycles, systemmemory, and disk IO operations. While scale-out has been garnering muchattentionbecauseoftheneedtohandleincreasinglymassivedatasets,itisveryimportanttorememberthatthebetterwescale-up,thefewerscale-outnodesthatwewillrequireandsothelessweneedtospendonhardware.

Scale-outcanbeusedtodeliversolutionsthatcoverseveraldifferentusecases.

Someofmostcommononesaretoincreasereadcapacitybyusingreplicationortousedatabase sharding to increase totaldatabase size andoverall transactionthroughput.

All of these are the key challenges faced while scaling MySQL 8. Thesechallenges need to be considered while we are performing scaling of thedatabaseinMySQL8.

Asinglemistakecanputabusinessintoansituationwhichnoneofuswouldliketo be in. Scaling is the the better way to improve the performance of thedatabase.

Achievinghighavailability

Highavailability refers tosystems thataredurableandcanperformoperationswithoutanyhindranceonthedatathatisrequiredfortheresponseoranyrequestfrom any mobile, web portals, websites, social, e-commerce, enterprise, andcloud applications. Data availability is considered an utmost concern for anybusinessororganization.Anyissueswithdowntimemayhaveanimpactonthebusinesscredit,andinsomecasesbusinesseshavetosufferfinancialloss.

For instance, if we have an e-commerce application with a single databaseserver, if that server goes down for reasons such as hardware failure, networkissue,virus,oroperatingsystemissues,itimpactsthedataalso.Ane-commerceapplicationmay have a large number of customer hits at same time, and anyserver failures toserve theresponse touser requestswill impact theuser; theywillsearchforotheroptionsforthepurchasecommodity.

MySQL8hascapabilitiestoprovidebackendfortheapplicationtohelpachievehighavailabilityandprepareafullyscalablesystem.Anabilityofthesystemtokeeptheconnectionpersistent, incaseapartoftheinfrastructurefails,andtheabilityofthesystemtorecoverfromsuchfailuresisconsideredhighavailability.A failureof the systemcanbecausedbyeither amaintenanceactivityononepartofthesystem,suchasahardwareorsoftwareupgrade,orbythefailureof

theinstalledsoftware.

Purposeofhighavailability

The requirements for achieving HA and scalability may vary from system tosystem.

Eachsystemrequiresadifferentconfigurationinordertoachievetheseabilities.

MySQL 8 also supports different approaches, such as the replication of dataacross multiple MySQL servers, or preparing multiple data centers based ongeographical locations and serving the client requests from the data centersclosest to the locationof the client.Such solutions canbeused to achieve thehighestuptimeofMySQL.

Today, in theworldofcompetitivemarketing,anorganization'skeypoint is tohave their system up and running. Any failure or downtime directly impactsbusiness and revenue. Hence, high availability is a factor that cannot beoverlooked.MySQLisquitereliableandhasconstantavailabilityusingclusterandreplicationconfigurations.

Cluster servers instantly handle failures andmanage the failover part to keepyoursystemavailablealmostallthetime.Ifoneservergetsdown,itwillredirecttheuser'srequesttoanothernodeandperformtherequestedoperation.

DataavailabilityDataisalwaysavailableinanysituation.Inanyapplication,dataisthecorepart,which isactually thewealthof theapplicationowner. Ifwehaveahealthcaresystem and at the time of medical check up of any patient their data is notavailable, due to server downtime or any other reason, it might block furtherprocessofthedoctorandinthiscaseimpactsthelifeofthepatient.

Securityofdata

Thefirstthingthatcomestomindissecuringdata,becausenowadaysdatahasbecomeprecious and it can impact business continuity if legal obligations arenotmet;infact,itcanbesobadthatitcanclosedownyourbusinessinnotime.MySQL is themost secureand reliabledatabasemanagement system,usedbymanywell-knownenterprisessuchasFacebook,Twitter,andWikipedia.Itreallyprovidesagoodsecuritylayerthatprotectssensitiveinformationfromintruders.MySQLgivesaccesscontrolmanagementsothatgrantingandrevokingrequiredaccessontheuser iseasy.Rolescanalsobedefinedwitha listofpermissionsthatcanbegrantedorrevokedfortheuser.

All user passwords are stored in an encrypted format using plugin-specificalgorithms.

SynchronizationofdataWhilewehaveasingledatabaseserver,ifitgoesdownforanyreasonwewouldlose the whole database, and if we have database backup available up to thecurrentday,wecanrestorethedatabasetillthatday,butallcurrenttransactionswouldalsobelostinthiscase.Thelasttransactiondatawouldnotbeavailableatthattime.

BackupofthedataDatabasebackuptill the last transactionshouldbe in theplanwhenabusinesshasanyserverbaseapplicationwhereasingledatabaseserverisperformingallthetasks.Whendoinghighavailability,includeallscenariosofthebackupandrestoreoperationinthearchitecture.

CompetitivemarketInthemarketmanycompetitorsareavailablewiththesamenatureofbusiness.In this case, if a business is having issueswith data availability to end users,customersmight not continuewith that business and insteadmove to anotherprovider.Itsanintegralpartofbusinesscontinuity.

Performance

High availability is also important in terms of the performance of the dataoperation.

Ifwehaveasingleserverandalloperationsareperformedonthatserveronly,itwillreachitslimitatsomestage,wheretheservercapacityisexhausted.So,inthatcase,ifwehavehighavailabilityarchitectureimplementeditwouldprovideameans to load the balance of a transaction and the performance of the datamanipulation operation. Replication and clustering enables for betterconcurrencyandmanagestheworkload.

UpdatesinthesystemWhile any online site or application requires updates or any new productionrelease

is planned it directly impacts the end users. If an application has only limitedusers

at that time, we canmanage the update regarding all end-users via emails ormessages

within the application before the release.But in caseswhere there are a largenumber

ofuserinasingleapplication,atthattimeitwillimpactthebusiness.Itwill

stop all users at the same time, anddue to this running transactionswould beimpacted.

Choosingthesolution

Again,we have to think about selecting the right solution for the availability.ManythingsneedtobekeptinmindwhileweplanhighavailabilityinMySQL.The requirements for achieving HA and scalability may vary from system tosystem.Eachsystemrequiresadifferentconfigurationinordertoachievetheseabilities.

SuchsolutionscanbeusedtoachievethehighestuptimeofMySQLwithregardtothefollowing:

Thelevelofavailabilityrequired

Thetypeofapplicationbeingdeployed

Acceptedbestpracticeswithinyourownenvironment

InMySQL, replication and clustering are the best options for achieving highavailability.

Allapplicationshavetheirownarchitecture,andnatureoftheirbusinessneeds

to be considered when we are selecting any technique to achieve highavailabilityofMySQL8.

Advantagesofhighavailability

The following are the advantages that we have when we perform highavailabilityinMySQL:

MySQL is quite reliable and has constant availability using cluster andreplicationconfigurations.

Cluster servers instantly handle failures and manage the failover part tokeepyoursystemavailablealmostallthetime.Ifoneservergoesdown,itwill redirect theuser's request to anothernode andperform the requestedoperation.

Anabilityofthesystemtokeeptheconnectionpersistent,incaseapartofinfrastructure fails, and the ability of the system to recover from suchfailureisconsideredashighavailability.

MySQL 8 also supports different approaches such as replication of dataacrossmultipleMySQLserversorpreparingmultipledatacentersbasedongeographicallocationsandservingtheclientrequestsfromthedatacentersclosesttothelocationoftheclient.

MySQLgiveshighspeedtransactionprocessingwithoptimalspeed.Itcancachetheresults,whichboostsreadperformance.

Replication and clustering enables better concurrency and manages theworkload.Groupreplicationbasicallytakescareofcommittingtransactionsonce most of the members in group replication have acknowledged thetransactionhasbeenreceivedconcurrently.

Thishelpscreatebetterthroughputiftheoverallnumberofwritesdoesnotexceedthecapacityofthemembersinthegroupreplication.

Clustering enables data to be replicated tomultipleMySQL serverswithautomatedsharing.Itisdesignedforbetteravailabilityandthroughput.

Memcached removes the SQL layer and directly accesses the InnoDBdatabasetables.

Hence,overheadoperations likeSQLparsingwill no longerbe executed,whichreallyimpactstheperformance.

MemcachedwithMySQLalsoprovidesyouwithawaytomakein-memorydatapersistentsothatwecanuseitforvariousdatatypeswithoutlosingit.

MemcachedAPIsareavailableindifferentprogramminglanguagessuchasPerl,Python,Java,PHP,C,andRuby.WiththehelpofaMemcachedAPI,an application can interact with the Memcached interface to store andretrieveinformation.

Summary

Inthischapter,westartedwithanoverviewofscalabilityandhighavailabilityinMySQL 8, which covered the various scalability needs, advantages, methods,andkeypointstobenotedwhenwemakescalabledesignsofMySQL8.Wealsodiscussed the shortcomings that we generally come across when we performscalabilityandhowtoovercomechallengeswithappropriatesolutions.Wehavelearned about scaling in MySQL 8 and troubleshooting challenges in scalingMySQL8.WealsolearnedaboutmanydiversewaystoachievehighavailabilityinMySQL8.

In thefollowingchapter,wewill learnhowto takecareofMySQL8security.We will learn about general factors that affect security, the security of coreMySQL8files,accesscontrol,andsecuringthedatabasesystemitself.Wewillalso learn thedetailsofsecuritypluginsandgainan in-depthunderstandingofdatabasesecurityingeneralforrelationaldatabases.

MySQL8–Security

Inpreviouschapters,welearnedabout thescalabilityofMySQL8andhowtotroubleshoot challenges when scaling MySQL 8. Apart from that, we alsolearnedhowtomakeMySQL

8highlyavailableforuse.Nowadays,securityisimportantforanyapplication,right?

Whenwetalkaboutsecurity,itincludesaccountmanagement,roles,privileges,andmore. Considering these aspects,wewill cover all of these topics in thischapter.

This chapter mainly focuses on MySQL 8 database security and its relatedfeatures.

Thefollowingtopicsarecoveredinthischapter:

OverviewofsecurityforMySQL8

Commonsecurityissues

AccesscontrolinMySQL8

AccountmanagementinMySQL8

EncryptioninMySQL8

Securityplugins

OverviewofsecurityforMySQL8

The term security is not bound to a specific topic; it covers a wide range oftopics related to MySQL 8. Before starting a detailed discussion on it, let'smentionsomeimportantpointsrelatedtosecurity:

Considersecuritywithinadatabasewhereusersandtheirprivilegesrelatedtovariousdatabaseobjectsneedtomanage.

Passwordsecurityforusers.

Security configuration during the installation process, which includesvarious typesof files, suchas log files,data files, andmanymore.Thesefilesmustbeprotectedfortheirread/writeoperations.

To handle system level failure scenarios, you must have a backup andrecoveryplan.

This includes all the required files, such as database files, configurationfiles,andmanymore.

Manage network security of the system where MySQL 8 was installed,whichpermitsalimitednumberofhostsfortheconnection.

Nowyourridebeginswithanotherimportantandveryinterestingtopic.Herewego.

CommonsecurityissuesBeforegoingintodetailoncomplexissues,youmustfirstunderstandsomebasicpointsthatwillhelpyoupreventmisuseorattacks.

<strong>shell>tcpdump-l-ieth0-w-srcordstport3306|strings</strong>

Guidelines for a securepasswordIn this section,we describe guidelines for securing passwordswith respect todifferentusersandcoverhowtomanageitduringtheloggingprocess.MySQL8provides the validate_password plugin to define the policy for acceptablepasswords.

<strong>cmd>mysql-uroot--password=your_pwd</strong><br/><strong>--OR</strong><br/><strong>cmd></strong>

<strong>cmd>mysql-uroot-p</strong><br/><strong>Enterpassword:*********</strong>

<strong>[client]</strong><br/><strong>password=your_pass</strong>

<strong>shell>chmod600.my.cnf</strong>

Guidelinesforadministrators

For a database administrator, the following guidelines should be followed tosecurepasswords:

Usevalidate_passwordtoapplythepolicyonacceptedpasswords

MySQL8usesthemysql.usertabletostoreuserpasswords,soconfigurethesysteminawaythatonlyadministrativeuserscanaccessthistable

Usersshouldbeallowedtoresetaccountpasswordsinthecaseofexpiredpasswords

Applyprotectiononthelogfileifitcontainspasswords

Manage access to the plugin directory and the my.cnf file, because it canmodifycapabilitiesprovidedbytheplugins

PasswordandloggingMySQL8allowsyoutowritepasswordsasplaintextinSQLstatements,suchasCREATEUSER,SETPASSWORD,andGRANT.Ifweexecutethesestatements,MySQL8willwritepasswordsastextinlogfiles,andtheyarevisibletoalltheusersthathaveaccesstothelogfiles.Toovercomethisproblem,avoiddirectupdatesongranttablesusingthementionedSQLstatements.

Secure MYSQL 8 againstattackers

TosecureMySQL8againstattackers,stronglyconsiderthefollowingpoints:

SetapasswordforallMySQLaccounts.Neverdefineanaccountwithnopassword,becausethispermitsaccesstoyouraccountbyanyuser.

TomakeaconnectionwithMySQL8,usesecureprotocols/channels,suchascompressedprotocols,MySQL8 internalSSLconnections,orSSHforencryptedTCP/IPconnections.

ForaUnix-basedsystem,setread/writeprivilegesonthedatadirectoryfortheUnixaccountwhichisusedforrunningmysqld.Don'tusetherootusertostarttheMySQL8server.

Use thesecure_file_privvariable tospecify thedirectoryfor readandwritepermission. Using this variable, you can restrict non-administrative usersfrom accessing important directories.Use this variable to set permissionson plugin_dir; it is very important. In the same way, do not provide FILE

privilegestoalltheusers,becauseitpermitsuserstowritefilesanywhereinthesystem.

Usethemax_user_connectionsvariabletorestrictthenumberofconnectionsperaccount.

At the time of creating grant table entries, use wildcards properly. It ispreferabletouseIPsinsteadofDNS.

Followsecurityguidelinesduringstoredprogramandviewcreation.

Security options and variablesprovidedbyMySQL8

The following mysqld options and variables are provided by MySQL 8 forsecurity:

Name

Cmd-Line

OptionFile

SystemVar

StatusVar

VarScope

allow-suspicious-udfs Yes Yesautomatic_sp_privileges Yes Globalchroot Yes Yesdes-key-file Yes Yeslocal_infile Yes Global

old_passwords Yes Bothsafe-user-create Yes Yessecure-auth Yes Yes Global-Variable:secure_auth Yes Globalsecure-file-priv Yes Yes Global-Variable:

secure_file_priv Yes Global

skip-grant-tables Yes Yesskip-name-resolve Yes Yes Global-Variable:

skip_name_resolve Yes Global

skip-networking Yes Yes Global-Variable:

skip_networking Yes Global

skip-show-database Yes Yes Global-Variable:

skip_show_database Yes Global

Reference:https://dev.mysql.com/doc/refman/8.0/en/security-options.html

Security guidelines for clientprogramming

Don't trust any data entered by the application user, because there is thepossibility that the user has entered a drop or delete statement for theMySQLdatabase. So, there is always the risk of security leaks and data loss. As anadministratorofaMySQLdatabase,thefollowingchecklistshouldbefollowed:

ThesizeofthedatamustbecheckedbeforepassingittoMySQL8.

TomakeMySQL8morerestrictive,enablethestrictMySQLmode.

Fornumericfields,entercharacters,specialcharacters,andspaces insteadof numeric itself. Change field values to their original forms by yourapplicationbeforesendingthemtotheMySQL8server.

Use twodifferentusers forapplicationconnection to thedatabaseand fordatabaseadministration.

Modifydatatypesfromnumerictocharactertypesbyaddingquotesinthe

case of dynamicURLs andweb forms.Also add%22 ("),%23 (#), and%27(')indynamicURLs.

Previouslydefinedfunctionalitiesareavailablebuiltintoalloftheprogramminginterfaces. For example, Java JDBC provides prepared statements withplaceholders,andRubyDBIprovidesthequote()method.

AccesscontrolinMySQL8

Privilegesaremainlyusedtoauthenticateusersandwillverifyusercredentialsand check if a user is allowed for the requested operation or not. When weconnectwiththeMySQL8server,itwillfirstchecktheidentityoftheuserbytheprovidedhostandusername.Afterconnection,whenarequestcomesin,thesystem will grant privileges according to the user's identity. Based on thisunderstanding,wecansaythataccesscontrolcontainstwostageswhenwetrytoconnectwiththeMySQL8serverusingtheclientprogram:

Stage 1: The MySQL server will either accept or reject the connection,basedontheprovidedidentity

Stage2:AftergettingaconnectionfromtheMySQLserver,whentheusersendsarequestforperforminganyoperation,theserverwillcheckwhethersufficientprivilegesareavailablefortheuserornot

TherearesomelimitationsoftheMySQL8privilegesystem:

Userisnotallowedtosetapasswordonspecificobjects,suchasatableoraroutine.

MySQL8allowsitgloballyattheaccountlevel.

As an adminuser,we cannot specifyprivileges in away that create/droptableisallowedbutcreate/dropdatabaseofthattableisnotallowed.

Youarenotallowedtorestrictuseraccessexplicitly,whichmeansthatexplicitlymatchingtheuserandrefusingitsconnectionisnotpossible.MySQL8managesthecontentofgranttablesinmemory,sointhecaseofINSERT,UPDATE,andDELETEstatements,executionongranttablesrequirestheservertorestartforeffect.

To avoid server restarts, MySQL has provided a command for flushingprivileges.Wecanexecutethiscommandinthreedifferentways:

1. ByissuingFLUSHPRIVILEGES.

2. Usingmysqladminreload.

3. Usingmysqladminflush-privileges.

Whenwereloadgranttables,itwillworkasperthefollowingmentionedpoints:

Tableandcolumnprivileges:Changesoftheseprivilegeswillbeavailablefromthenextclient'srequest

Databaseprivileges:ChangesoftheseprivilegeswillbeavailablethenexttimetheclientexecutesaUSEdbnamestatement

Global privileges and passwords: Changes of these privileges areunaffectedforaconnectedclient;itwillbeapplicablefromthesubsequentconnections

PrivilegesprovidedbyMySQL8

Privilegesdefinewhichoperations arepermissible to theuser accounts.Basedonthelevelofoperationandthecontextinwhichitisapplied,itwillwork.Itismainlyclassifiedasfollows:

Database privileges: Applied on the database, and all objects of thedatabasewithinit.Itcanbegrantedtoasingledatabaseordefinedgloballytoapplyonalldatabases.

Administrativeprivileges:Itisdefinedatthegloballevel,sonotrestrictedtoasingledatabase.ItenablesuserstomanageoperationoftheMySQL8server.

Databaseobject'sprivileges:Itisusedtodefineprivilegesonthedatabaseobjects,suchastables,views,indexes,andstoredroutines.Itcanbeappliedonaspecificobjectofthedatabase,canbeappliedonallobjectsofagiventypeinadatabase,orcanbeappliedgloballyforalltheobjectsofagiventypeinalldatabases.

MySQL8willstoreaccountprivilegerelatedinformationintogrant tablesand

store the contents of these tables intomemory upon server start-up for betterperformance.

Privilegesarefurtherclassifiedintermsofstaticanddynamicprivileges:

Staticprivileges:Theseareavailablebuiltinwiththeserverandcannotbeunregistered. These privileges are always available for the user to begranted.

Dynamicprivileges:Theseprivilegescanberegisteredorunregisteredatruntime. If privileges are not registered, then they are not available to begrantedforuseraccounts.

Granttables

Granttablescontaininformationrelatedtouseraccountsandgrantedprivilegesfor that user. MySQL 8 automatically inserts data into these tables when weexecuteanyaccountmanagementstatementsinthedatabase,suchasCREATEUSER,GRANT,andREVOKE.MySQLallowsinsert,update,ordeleteoptionsongranttablestotheadminuser,butit'snotapreferableapproach.ThefollowingtablesoftheMySQL8databasecontaingrantinformation:

user:Itcontainsdetailsrelatedtouseraccounts,globalprivileges,andothernon-privilegecolumns

password_history:Itcontainshistoryofpasswordchanges

columns_priv:Itcontainscolumnlevelprivileges

procs_priv:Itcontainsprivilegesrelatedtostoredproceduresandfunctions

proxies_priv:Itcontainsprivilegesforproxyusers

tables_priv:Itcontainstablelevelprivileges

global_grants: It contains details related to dynamic global privilegesassignments

role_edges:Itcontainsedgesforrolesubgraphs

db:Itcontainsprivilegesatthedatabaselevel

default_roles:Itcontainsdetailsrelatedtodefaultuserroles

Granttablescontainscopeandprivilegecolumns:

Scopecolumn:Thiscolumndefinesthescopeofrowsinthetables,whichmeansthecontextunderwhichtherowapplies.

Privilegecolumn: This column indicateswhich operation is permitted totheuser.TheMySQLservercombines informationfromthevariousgranttablestobuildacompletedetailofauser'sprivileges.

FromMySQL8.0onward,granttablesusetheInnoDBstorageenginebymanagingtransactionalstates,butbeforethat,MySQLusedtheMyISAMenginebymanagingnontransactional states. This change enables users to manage all accountmanagement statements in the transactional mode, so in the case of multiplestatements, either all of them are successfully executed or none of them areexecuted.

Verification of access controlstagesMySQL8performsaccesscontrolchecksintwodifferentstages.

Stage1-Connectionverification

This is the connection verification stage, so after verification,MySQL 8 willeither accept or reject your connection request.Verificationwill be performedwiththefollowingconditions:

1. Basedonauser'sidentity,withitspassword.

2. Whetherauser'saccountislockedornot.

The server will deny access if either of these cases fails. Here, the identitycontainstheusernameandhostnamefromwhichtherequestiscoming.MySQLperforms a locking check on the account_locked column of the user table and acredential check on the three columns of the user table scope: Host, User, andauthentication_string.

Stage2-RequestverificationOnceaconnectionisestablishedwiththeMySQLserver,stage2comesintothepicture,wheretheMySQLservercheckswhichoperationyouwant toperformandwhetherthatispermissibletoyouornot.Forthisverification,MySQLusesthe privilege columns of the grant tables; it might be coming from user, db,tables_priv,columns_priv,orprocs_privtables.

AccountmanagementinMySQL8As the name implies, this topic describes how to manage user accounts inMySQL8.Wewilldescribehowtoaddnewaccounts,howtoremoveaccounts,howtodefineusernamesandpasswordsfortheaccounts,andmore.

<strong>#1mysql>CREATEUSER'user1'@'localhost'IDENTIFIEDBY'user1_password';</strong><br/><strong>#2mysql>GRANTALLPRIVILEGESON*.*TO'user1'@'localhost'WITHGRANTOPTION;</strong><br/><br/><strong>#3mysql>CREATEUSER'user2'@'%'IDENTIFIEDBY'user2_password';</strong><br/><strong>#4mysql>GRANTALLPRIVILEGESON*.*TO'user2'@'%'WITHGRANTOPTION;</strong><br/><br/><strong>#5mysql>CREATEUSER'adminuser'@'localhost'IDENTIFIEDBY'password';</strong><br/><strong>#6mysql>GRANTRELOAD,PROCESSON*.*TO'adminuser'@'localhost';</strong><br/><br/><strong>#7mysql>CREATEUSER'tempuser'@'localhost';</strong><br/><br/><strong>#8mysql>CREATEUSER'user4'@'host4.mycompany.com'IDENTIFIEDBY'password';</strong><br/><strong>#9mysql>GRANTSELECT,INSERT,UPDATE,DELETE,CREATE,DROPONdb1.*TO'user4'@'host4.mycompany.com';</strong>

<strong>mysql>DROPUSER'user1'@'localhost';</strong>

Thiscommandwilldropthe'user1'accountfromthesystem.

SecurityusingrolesThesameasauseraccountrolehavingprivileges,wecanalsosaythataroleisacollectionofprivileges.Asanadminuser,wecangrantand revokeprivilegesfrom the roles. MySQL 8 provides the following commands, functions, andvariablesrelatedtoroleconfiguration.

SETROLESETROLEchangestheactiveroleswithinthecurrentsession.Refertothefollowingcommands related to SET ROLE: mysql> SET ROLE NONE; SELECTCURRENT_ROLE();+----------------+|CURRENT_ROLE()|+----------------+|NONE|+----------------+mysql>SETROLE'developer_read';SELECTCURRENT_ROLE();+----------------+|CURRENT_ROLE()|+----------------+|`developer_read`@`%`|+----------------+

The first commandwilldeactivate all roles for theuser in the current session.YoucanseetheeffectwiththeCURRENT_ROLE();function.Inthesecondcommand,wearesetting the'developer_read' roleasdefault,and thenchecking thecurrentroleusingthepredefinedfunctionagain.

CREATEROLECREATEROLE isused tocreatea role; refer to thefollowingcommand,whichwillcreatearolewiththename'developer_role':

CREATEROLE'developer_role';

DROPROLEDROPROLE isusedtoremovearole.Refertothefollowingcommand,whichwillremovethe'developer_role'role:

DROPROLE'developer_role';

<strong>GRANTALLONmy_db.*TO'developer_role';</strong>

<strong>GRANT'developer_role'TO'developer1'@'localhost';</strong>

<strong>CREATEUSER'user1';</strong><br/><strong>CREATEROLE'role1';</strong><br/><strong>GRANTSELECTONmydb.*TO'user1';</strong><br/><strong>GRANTSELECTONmydb.*TO'role1';</strong><br/><strong>CREATEUSER'user2';</strong><br/><strong>CREATEROLE'role2';</strong><br/><strong>GRANT'user1','role1'TO'user2';</strong><br/><strong>GRANT'user1','role1'TO'role2';</strong>

Inthisexample,user1androle1iscreatedandGRANTisappliedontheminasimplewaybyusingtheGRANTcommand.Now,foruser2androle2,wehaveappliedGRANTfromtheuser1androle1,respectively.

<strong>REVOKEdeveloper_roleFROMuser1;</strong><br/><strong>REVOKEINSERT,UPDATEONapp_db.*FROM'role1';</strong>

Thefirstcommandisusedtoremove'developer_role'foruser1,andthesecondcommandisusedtoremoveinsertandupdateprivilegesfromthe'role1'on'app_db'.

<strong>mysql>SETDEFAULTROLEapp_developerTOroot@localhost;</strong><br/><br/><strong>mysql>SELECTCURRENT_ROLE();</strong><br/><strong>+---------------------+</strong><br/><strong>|CURRENT_ROLE()|</strong><br/><strong>+---------------------+</strong><br/><strong>|`app_developer`@`%`|</strong><br/><strong>+---------------------+</strong><br/><strong>1rowinset(0.04sec)</strong>

Aftersettingthedefaultrole,restarttheserverandexecutethecurrent_role()functiontocheckwhetheraroleisassignedornot.

<strong>mysql>showgrantsforapp_developer;</strong><br/><strong>+-------------------------------------------+</strong><br/><strong>|Grantsforapp_developer@%|</strong><br/><strong>+-------------------------------------------+</strong><br/><strong>|GRANTUSAGEON*.*TO`app_developer`@`%`|</strong><br/><strong>+-------------------------------------------+</strong><br/><strong>1rowinset(0.05sec)</strong>

<strong>mysql>showgrantsforroot@localhost;</strong>

<strong>[mysqld]</strong><br/><strong>mandatory_roles='app_developer'</strong>

<strong>SETPERSISTmandatory_roles='app_developer';</strong>

ThisstatementapplieschangesontherunningMySQL8instanceandalsosavesitforsubsequentrestarts.Ifyouwanttoapplychangesfortherunninginstanceandnotforotherrestarts,thenusethekeywordGLOBALinsteadofPERSIST.

<strong>ALTERUSER'testuser'@'localhost'PASSWORDEXPIRE;</strong>

Thiswillmarkapasswordasexpiredforthementioneduser.Forpasswordpolicies,youhavetodefinethedurationintermsofthenumberofdays.MySQLusesthesystemvariabledefault_password_lifetime,whichcontainsapositiveintegernumbertodefinethenumberofdays.Wecandefineitinthemy.cnffileorcandefineitatruntimeusingthePERSISToption:

Passwordreuserestrictions:Usedtopreventtheuseofoldpasswordsagain.MySQL8definesthisrestrictionbasedontwoparameters-thenumberofchangesandtimeelapsed;theycanbeusedseparatelyorincombination.MySQL8definespassword_historyandpassword_reuse_intervalsystemvariables,respectively,toapplyrestrictions.Wecandefinethesevariablesinthemy.cnffile,orcanpersistthem.password_history:Thisvariableindicatesthatnewpasswordscannotbeset/duplicatedfromtheoldpasswords.Here,considerthemostrecentoldpasswordsasperthespecifiednumber.password_reuse_interval:Thisvariableindicatesthatthepasswordcannotbeset/duplicatedfromtheoldpassword.Here,intervaldefinesthespecificperiodandMySQL8willchecknewpasswordwithallthepasswordswhichwerefallsunderthatperiodforauser.Forexample,iftheintervalissetas20days,thenthenewpasswordshouldnothaveexistedinthelast20daysofchangeddata.Passwordstrengthassessment:Usedtodefinestrongpasswords.Itisimplementedusingthevalidate_passwordplugin.

EncryptioninMySQL8

When there is a need to transfer data over the network, it is a must to useencryptionfortheconnection.Ifyouareusingunencrypteddata,thensomeonewhohasaccess to thenetworkcaneasilywatchallofyour trafficandcanseewhatdataistransferredbetweentheclientandserver.Toprotectyourdataoverthenetwork,useencryption.

Make sure the encryption algorithmusedcontains security elements toprotectyour connection fromknown attacks, like changing the order of amessage orreplay twiceondata.Basedonyour application requirements, youcanchooseeither an encrypted or unencrypted type connection. MySQL 8 performsencryptionperconnectionusingTransportLayerSecurity(TLS)protocol.

Configuring MySQL 8 to useencryptedconnectionsThissectiondescribeshowtoconfigure theserverandclientfor theencryptedconnection.

<strong>[mysqld]</strong><br/><strong>ssl-ca=ca.pem</strong><br/><strong>ssl-cert=server-cert.pem</strong><br/><strong>ssl-key=server-key.pem</strong>

The--ssloptionisenabledbydefault,soonserverstartup,MySQL8willtrytofindthecertificateandkeyfileunderthedatadirectory,eventhoughyouhavenotdefineditinthemy.cnffile.Ifthosefilesarefound,thenMySQL8willprovideanencryptedconnection,orelsecontinuewithoutanencryptedconnection.

Client-side configuration forencryptedconnections

Attheclientside,MySQLusesthesame–ssloptionsusedattheserversidetospecify thecertificateandkey file,butapart fromthat, ithas–ssl-modeoptions.By default, the client is allowed to set up an encrypted connection with theserver if the server permits it. For further control, the client programuses thefollowing–ssl-modeoptions:

--ssl-mode=REQUIRED:Thisoptionindicates thatanencryptedconnectionmustbeestablished,andfailsifitisnotestablished

--ssl-mode=PREFFERED:Thisoptionindicatestheclientprogramcanestablishanencrypted connection if the server permits it, or else establish anunencryptedconnectionwithoutafail

--ssl-mode=DISABLED:Thisoptionindicatestheclientprogramisunabletouseanencryptedconnection,andonlyanunencryptedconnectionisallowed

--ssl-mode=VERIFY_CA: This option is the same as REQUIRED, but in addition to

that,itverifiestheCAcertificateagainsttheconfiguredCAcertificateandreturnsafailifnomatchesarefound

--ssl-mode=VERIFY_IDENTITY:ItisthesameastheVERIFY_CAoption,butinadditiontothat,itwillperformthehostnameidentity

Command options for encryptedconnections

The followingoptions are available inMySQL8 for anencryptedconnection.Youcanuse theseoptionson thecommand line,oryoucandefine theminanoptionfile:

Format Description--skip-ssl Donotuseencryptedconnection--ssl Enableencryptedconnection--ssl-ca FilethatcontainsalistoftrustedSSLCertificateAuthorities

--ssl-

capath

DirectorythatcontainstrustedSSLCertificateAuthoritycertificatefiles

--ssl-cert FilethatcontainsX509certificate--ssl-

cipher Listofpermittedciphersforconnectionencryption

--ssl-crl Filethatcontainscertificaterevocationlists

--ssl-

crlpath Directorythatcontainscertificaterevocationlistfiles

--ssl-key FilethatcontainsX509key--ssl-mode Securitystateofconnectiontoserver--tls-

version Protocolspermittedforencryptedconnections

Reference:https://dev.mysql.com/doc/refman/8.0/en/encrypted-connection-options.html

Connect with MySQL 8 remotelyfromWindowswithSSH

ToconnectremotelywithMYSQL8byusingSSHfromtheMicrosoftWindowssystem,performthefollowingsteps:

1. InstalltheSSHclientonyourlocalsystem.

2. Afterstarting theSSHclient, set thehostnameanduser IDbywhichyouwanttoconnectwiththeserver.

3. Configureportforwardingasfollowsandsavetheinformation:

For remote forwarding configure: local_port:3306,remote_host:mysqlservername_or_ip,remote_port:3306

For local forwarding configure: local_port:3306, remote_host:localhost,remote_port:3306

4. LogintotheserverwiththecreatedSSHsession.

5. In your localMicrosoftWindows machine, start any ODBC application,suchasMicrosoftAccess.

6. In your local system, create new file and try to linkwithMySQL serverusing the ODBC driver. Make sure you have defined localhost in theconnectioninsteadofmysqlservername.

SecuritypluginsMySQL8providesseveralpluginstoimplementsecurity.Thesepluginsprovidevariousfeaturesrelatedtoauthenticationprotocols,passwordvalidation,securestorage,andmuchmore.Let'sdiscussthevarioustypesofpluginsindetail.

<strong>CREATEUSER'testsha256user'@'localhost'</strong><br/><strong>IDENTIFIEDWITHsha256_passwordBY'userpassword';</strong>

SHA-2pluggableauthentication

SHA-2pluggableauthentication is thesameas theSHA-256pluggableplugin,except its plugin name is caching_sha2_password.When compare to sha256_password,thispluginhasthefollowingadvantages:

1. If you are using Unix socket-file and shared-memory protocols, thensupportisprovidedforclientconnections.

2. In-memorycachingisavailableinSHA-2plugins,whichprovidesfasterre-authenticationforuserswhohaveconnectedpreviously.

3. This plugin provides RSA-based password exchange, which worksregardlessoftheSSL

libraryprovidedbyMySQL8.

Client-side cleartext pluggableauthenticationThis plugin is used to send passwords to the server without hashing orencryption. It is available at the client side with the name mysql_clear_password.MySQL8providesitbuilt-inwithintheclientlibrary.

<strong>[mysqld]</strong><br/><strong>plugin-load-add=mysql_no_login.so</strong>

<strong>INSTALLPLUGINmysql_no_loginSONAME'mysql_no_login.so';</strong>

<strong>UNINSTALLPLUGINmysql_no_login;</strong>

<strong>[mysqld]</strong><br/><strong>plugin-load-add=auth_socket.so</strong>

<strong>INSTALLPLUGINauth_socketSONAME'auth_socket.so';</strong>

<strong>UNINSTALLPLUGINauth_socket;</strong>

TestpluggableauthenticationA test plugin is provided byMySQL 8 to check account credentials and logsuccessesorfailuresontheserverlogs.Itisnotabuilt-inpluginandneedstobeinstalledbeforeuse.Thisisavailableforboththeserverandclientside,andisnamed test_plugin_server and auth_test_plugin, respectively. MySQL 8 uses theauth_test_plugin.so library for this plugin. To install and uninstall this plugin,performthesamestepsmentionedintheprecedingplugin.

Theconnection-controlpluginsMySQL 8 uses these plugins to introduce an increasing delay in the serverresponsetotheclientaftersomespecificnumberoffailedconnectionattempts.MySQLhasprovidedtwopluginsforconnectioncontrol.

<strong>CONNECTION_CONTROL_FAILED_LOGIN_ATTEMPTS</strong>

ThispluginimplementsusesoftheINFORMATION_SCHEMAtabletoprovidedetailsonmonitoringofthefailedconnections.

<strong>[mysqld]</strong><br/><strong>plugin-load-add=connection_control.so</strong>

<strong>INSTALLPLUGINCONNECTION_CONTROLSONAME<br/>'connection_control.so';</strong><br/><strong>INSTALLPLUGINCONNECTION_CONTROL_FAILED_LOGIN_ATTEMPTSSONAME<br/>'connection_control.so';</strong>

Variables related toCONNECTION-CONTROL

ThefollowingvariablesareprovidedbyCONNECTION-CONTROLplugins:

Connection_control_delay_generated: This is a status variable, mainly used tomanagethecounter.Itindicateshowmanytimestheserveraddedadelayinits response on failed connection attempts. It also depends on theconnection_control_failed_connections_threshold system variable, because thisstatusvariabledoesnotincrementthecountunlessthenumberofattemptsreachesthelimitdefinedbythethresholdvariable.

connection_control_failed_connections_threshold:Thisisasystemvariablewhichindicates how many consecutive failed attempts are allowed to clientsbeforetheserveraddsadelayoneachattempt.

connection_control_max_connection_delay:Thisisasystemvariablewhichdefinesthemaximumdelay time inmillisecondsfor theserver responseonfailedconnections attempts. MySQL 8 will consider this variable once thethresholdvariablecontainsavaluehigherthanzero.

connection_control_min_connection_delay: This system variable defines theminimum delay time in milliseconds for the server to failed connectionattempt.MySQL8will consider this variable once the threshold variablecontainsavaluehigherthanzero.

Thepasswordvalidationplugin

Forpasswordvalidation,MySQLprovidesapluginnamedvalidate_password.Itismainlyusedtotestpasswordsandimprovesecurity.Thefollowingarethetwomajorcapabilitiesofthisplugin:

VALIDATE_PASSWORD_STRENGTH():AnSQL function used to find the strength of apassword.Ittakesapasswordasanargumentandreturnsanintegervaluebetween0and100.Here,0indicatesaweakpasswordand100indicatesastrongpassword.

Check password as per policy in SQL statements: For all the SQLstatementswhichuseapasswordasacleartextvalue,thepluginwillcheckthe provided password against the policy of the password and, based onthat, return a response. In the case of a weak password, the plugin willreturn an ER_NOT_VALID_PASSWORD error. ALTERUSER,CREATE USER, GRANT, SET PASSWORDstatements,andthePASSWORD()functionarealwayscheckedbythispluginifthepasswordisdefinedascleartextinanargument.

<strong>[mysqld]</strong><br/><strong>plugin-load-add=validate_password.so</strong>

<strong>INSTALLPLUGINvalidate_passwordSONAME'validate_password.so';</strong>

Variables and options related tothepasswordvalidationplugin

MySQL 8 provides following system variables, status variables and optionsrelatedtopasswordvalidationplugin.

validate_password_check_user_name:Thisisasystemvariable,andisenabledbydefaultinMySQL8.Asthenameimplies,itisusedtocompareapasswordwiththeusernameofthecurrentlyeffectiveuser.

If thepasswordmatcheswith theusernameor its reverse,MySQL8willreject the password, irrespective of the VALIDATE_PASSWORD_STRENGTH() functionvalue.

validate_password_dictionary_file:This systemvariablecontains thepathnameofthedirectorywhichisusedbythevalidate_passwordplugin.Youcansetitat runtimewithout a server restart, and it is available once the plugin isinstalled. Set the password policy value as 2(strong), if you define thedirectory for the password check. Possible values for password policy isdescribeundervalidate_password_policysystemvariable.

validate_password_length: This system variable is available once a plugin isinstalled and is used to define the minimum number of characters for apasswordtocheckwiththevalidate_passwordplugin.

validate_password_mixed_case_count: This system variable is available once apluginisinstalledandisusedtodefinetheminimumnumberoflowercaseanduppercasecharactersforapasswordcheck.

validate_password_number_count: This system variable is available once thepluginisinstalledandisusedtodefinetheminimumnumberofdigitsforthepasswordcheck.

validate_password_special_char_count: This system variable is available once aplugin is installed and is used to define the minimum number of non-alphanumericcharactersinapasswordcheck.

validate_password_policy: This system variable is available once a plugin isinstalled,andit indicateshowapluginshouldbehavein thecaseofothersystem variables. The following values of this variable describe thebehaviorofthevalidate_passwordplugin:

Policy TestsPerformed

0orLOW Length

1orMEDIUM Length;numeric,lowercase/uppercase,andspecialcharacters

2orSTRONG

Length;numeric,lowercase/uppercase,andspecialcharacters;dictionaryfile

Reference:https://dev.mysql.com/doc/refman/8.0/en/validate-password-options-variables.html

validate_password_dictionary_file_last_parsed: This is a status variable used toindicatethetimewhenadirectoryfilewaslastparsed.

validate_password_dictionary_file_words_count: This is a status variable used toindicatethenumberofwordsreadfromthedirectoryfile.

--validate-password[=value]:Thisoption isused todefinehowa server loadsthe validate_password plugin on startup. This option is available only if thepluginwasregisteredwithINSTALLPLUGINor if it is loadedwith the--plugin-load-addfeature.

MySQL8keyringMySQL8providesakeyringservice,whichallowstheMySQLserver'sinternalcomponentsandpluginstostoretheirsensitiveinformationforlateruse.Forthisfeature,MySQL8usesthekeyring_fileplugin,whichwillstoredataintothefilelocatedontheserverhost.ThispluginisavailableinalldistributionsofMySQL,suchastheCommunityEditionandEnterpriseEdition.

<strong>mysqld]</strong><br/><strong>plugin-load-add=keyring_file.so</strong>

<strong>INSTALLPLUGINkeyring_fileSONAME'keyring_file.so';</strong>

System variables related tokeyringplugin

MySQL8providesbelowsystemvariablerelatedtokeyringplugin:

keyring_file_data:Thissystemvariableisavailableonceapluginisinstalledand is used to define a pathname of the data file which is used by thekeyring_fileplugintostoresecuredata.Keyringoperationsaretransactional,sothispluginusesabackupfileduringwriteoperationtohandlearollbackscenario. In this case, the backfile is also named with the same namingconventionasdefinedinthekeyring_file_datasystemvariable,withthesuffixas.backup.

Summary

Inthischapter,westartedwithanoverviewofsecurity,andthentheridebeganwith theMySQL8 security related features.Firstwediscussed somecommonsecurity issues, thenwe showed how to assign privileges and how tomanageaccess control in MySQL 8. Encryption was also covered in this chapter, tosecureyoursensitivedata.

And finally,we covered some important security plugins,which are useful toimplementsecurityinMySQL8.

It's now time to move on to our next chapter, where we will be configuringMySQL8

foroptimization.Foroptimization,wewillcoverdifferentareasofthedatabase,suchasoptimizingqueries,optimizingtables,optimizingbufferingandcaching,andmuchmore.Apartfromserverconfiguration,italsocovershowtoconfigureaclientforoptimization.

OptimizingMySQL8

Inthepreviouschapter,welearnedaboutsecurity,whichisanimportantaspectofanyproduction-gradeapplication.Thechapterstartedwithanintroductiontosecurity and identifying common security issues. Moving on, the chaptercovered access control mechanisms, account management, and encryption inMySQL8.WelearnedvariousMySQL

8 security plugins in the later part of the chapter. Security is an importantbenchmark for every production-grade application. That's why the previouschapterisanimportantone.

Movingalongasimilarline,withtheobjectiveofdevelopinghighlyoptimizeddatabases, this chapter focuses on optimization methods. It starts with anoverviewofwhatoptimizationmeansinMySQL8. It takes thereader throughMySQL 8 server and client optimization, optimizing database structure, andoptimizingcommonqueriesanddatabasetables.Laterinthechapter,emphasisisgiventobufferingandcachingtechniques.

Thefollowingisalistoftopicstobecovered:

OverviewofMySQL8optimization

OptimizingMySQL8serversandclients

Optimizingdatabasestructure

Optimizingqueries

Optimizingtables

Leveragingbufferingandcaching

Overview of MySQL 8optimization

Let's start with understanding MySQL 8 optimization. Optimization is theprocessof

identifying performance bottlenecks and implementing optimized solutions toovercome

these issues. Optimization in MySQL 8 involves performance measurement,configuration,

andtuningatseveraldifferentlevels.Itisanimportanttaskforanadministrator

to optimize the performance at different levels, like individual SQL queries,entire

database applications, database servers, or distributed database servers.Performance

optimizationattheCPUandmemorylevelsimprovesscalability.Italsoallowsthe

database to handlemore complex queries without slowing down the databaseserver.

The performance of a database depends on multiple factors. At the databaselevel,

thesefactorscanbetables,queries,andconfigurations.Databaseserverstartups

anddatabasequeryexecutionsareacoupleoftheeventswhentheseconstructsimpact

theCPUorperformI/O(Input/Output)operationsatthehardwarelevel.Thisisa

responsibility of the MySQL 8 database administrator: to make sure that thehardware

performancestandsatanoptimumlevel.Itisrequiredthatthehardwareisusedwith

the maximum efficiency possible. At the software level, performanceoptimizationstarts

bylearninggenericrulesandguidelinesandmeasuringperformancewithclocktime.

Gradually,weunderstand the internals of variousdatabaseoperations.Wecanmeasure

theperformance in termsofCPUcyclesand I/Ooperations.Toattain thebestdatabase

performance, we can optimize the software and hardware configurations at abasiclevel.

At an advanced level, we can improve MySQL itself by developing customstorageengines

andhardwareappliances,whichexpandtheMySQLecosystem.

Optimizingthedatabase

What is the most important factor in making the database perform at theoptimumspeed?

Theansweris,basicdatabasedesign.Thefollowingisachecklisttokeepaneye

onfordatabasedesign:

Thedatabasecolumnshavetobeoftherightdatatypes.Tablesmusthaveappropriate

columns for the purposes to be served. Applications that have frequentoperations

to be performed on the database have many tables with fewer columns,whereasapplications

thatanalyzelargeamountsofdatahavelimitedtableswithmanycolumns.

Aswelearnedinoneofthepreviouschapters,databaseindexingplaysan

important

role in enhancing query performance. So, it is important to have correctindexesin

placeforqueryexecutionefficiency.

We discussed database storage engines, such as MyISAM or InnoDB, in earlierchapters.Useofanappropriatestorageengineforeachindividualtable

is important.InnoDB ispreferable for transactionaldatabase tables,whereasMyISAMispreferablefordefiningnon-trasactionaldatabasetables.Thechoiceofstorage

engineplaysavitalroleindefiningtheperformanceandscalabilityofthedatabase.

In the chapter onMySQL8 data types,we learned about row formats indetail.Itis

againimportantforeachtohaveanappropriaterowformat.Thechoiceofrowformat

dependsonthestorageenginechosen.Compressedtablesoccupylessdiskspaceand

requirefewerdiskI/Ooperations.ForInnoDBtables,compressionisavailableforallreadandwriteoperations.Onthecontrary,

compressionisavailableforread-onlyMyISAMtables.

TheMySQLdatabasesupportsmultiplelockingstrategies.Thelockingcanbeatatable-level

orarow-level.Theapplicationmustuseanappropriatelockingstrategy.Bygranting

shared access wherever appropriate, it becomes possible to run databaseoperations

concurrently.Also,itshouldbepossibletorequestexclusiveaccess,sothatcritical

databaseoperationscanbeexecutedwithdataintegrityissuesandprioritycanbe

maintained. In this case, thechoiceof storageengine is again significant.The InnoDB storage engine handles most locking issues without userinvolvement.Itallowsfor

betterconcurrencyand reduces theamountofexperimentationand tuningforthecode.

The memory areas must use the correct caching size. It should be largeenoughtohold

frequently accessed data, and at the same time, not so large that theyoverloadphysical

memoryandcausepaging.TheInnoDBbufferpoolandMyISAMkeycachearethemainmemoryareastobeconfigured.

For newly created tables, MyISAM is the default storage engine. Inpractical use, InnoDB advanced performance features mean thattableswith InnoDB storage enginesoutperform the MyISAM tables foranoperations-heavydatabase.

Optimizingthehardware

Growthisthenatureofeverysoftwareapplication.Astheapplicationgrows,sodoes the database. The database becomesmore andmore busy in performingoperations. At a certain point, the database application eventually hits thehardware limits. An administrator must evaluate the possibility of tuning theapplicationor re-configuring theserver toavoid these issues. It shouldalsobeevaluated whether deploying more hardware resources would help. Systembottlenecksusuallyarisefromthefollowingsources:

Diskseeks:Aspartofthediskreadoperation,findingapieceofdatatakestimeforthedisk.

The mean time for finding a piece of data is usually lower than 10millisecondswithmoderndisks.So, in theory, it shouldbe100seekspersecond.Withtechnologicalevolution,thenewdiskshaveimprovementsonthedisktime,butitisveryhardtooptimizeforsingletables.Tooptimizetheseektime,itisnecessarytodistributedataacrossmorethanonedisk.

Diskreadingandwriting:Toreadorwritedatafromadisk,itisrequiredforthedisktobeatthecorrectposition.Onediskdeliversatleast10to20

MB of throughput per second (throughput is the amount of data read orwritten per second). So, the read and write throughput is more easilyoptimizedthantheseektime,aswecanreadinparallelfrommultipledisks.

CPUcycles:Wemustprocessthedatawhenitisinthemainmemorytogetthedesiredresult.

With large tables, the amount of memory is the most common limitingfactor.Withsmalltables,however,speedisusuallynotanissue.

Memory bandwidth: In an uncommon scenario, the main memorybandwidthbecomesabottleneckwhentheCPUneedsmoredatathancanbefitintheCPUcachememory.

OptimizingMySQL 8 servers andclientsThissectionfocusesonoptimizationforMySQL8databaseserversandclients,starting

with optimizing the server and followed by optimizing MySQL 8 client-sideentities.This

section ismore relevant todatabaseadministrators, to ensureperformanceandscalability

acrossmultiple servers. Itwouldalsohelpdeveloperspreparingscripts (whichincludes

settingupthedatabase)andusersrunningMySQLfordevelopmentandtestingtomaximize

theproductivity.

OptimizingdiskI/O

Inthissection,wewill learnwaystoconfigurestoragedevicestodevotemoreand

fasterstoragehardwaretothedatabaseserver.Amajorperformancebottleneckis

disk seeking (finding the correct place on the disk to read or write content).When

theamountofdatagrowslargeenoughtomakecachingimpossible,theproblemwith

diskseedsbecomesapparent.Weneedatleastonediskseekoperationtoread,and

several disk seek operations towrite things in large databaseswhere the dataaccess

is donemore or less randomly.We should regulate orminimize the disk seektimes

usingappropriatedisks.

In order to resolve the disk seek performance issue, increasing the number ofavailable

diskspindles,symlinkingthefilestodifferentdisks,orstrippingdiskscanbe

done.Thefollowingarethedetails:

Using symbolic links:Whenusing symbolic links,we can create aUnixsymboliclinksforindexanddata

files. The symlink points from default locations in the data directory toanother

disk in the case of MyISAM tables. These links may also be striped. Thisimprovestheseekandreadtimes.The

assumption is that the disk is not used concurrently for other purposes.Symbolic

linksarenotsupportedforInnoDBtables.However,wecanplaceInnoDBdataandlogfilesondifferentphysicaldisks.

Striping:Instriping,wehavemanydisks.Weputthefirstblockonthefirstdisk,thesecond

blockontheseconddisk,andsoon.TheNblockonthe(N%number-of-disks)disk.Ifthestripesizeisperfectlyaligned,

the normal data size will be less than the stripe size. This will help toimprove

theperformance.Stripingisdependentonthestripesizeandtheoperatingsystem.

Inanidealcase,wewouldbenchmarktheapplicationwithdifferentstripe

sizes.

The speed difference while striping depends on the parameters we haveused,likestripe

size.Thedifference inperformancealsodependson thenumberofdisks.Wehaveto

choose ifwewant tooptimizefor randomaccessorsequentialaccess.Togainreliability,

wemaydecide to set upwith striping andmirroring (RAID0+1).RAIDstandsforRedundantArrayofIndependentDrives.Thisapproachneeds2xN drives to holdN drives of data.With a good volumemanagementsoftware,wecanmanagethissetup

efficiently.

Thereisanotherapproachtoit,aswell.Dependingonhowcriticalthetypeofdata

is,wemayvarytheRAIDlevel.Forexample,wecanstorereallyimportantdata,such

ashostinformationandlogs,onaRAID0+1orRAIDNdisk,whereaswecanstoresemi-important

dataonaRAID0disk.InthecaseofRAID,paritybitsareusedtoensuretheintegrity

ofthedatastoredoneachdrive.So,RAIDNbecomesaproblemifwehavetoomany

writeoperationstobeperformed.Thetimerequiredtoupdatetheparitybitsinthis

caseishigh.

If it is not important tomaintainwhen the filewas last accessed,we can

mountthe

filesystemwiththe-onoatimeoption.Thisoptionskips theupdatesonthefilesystem,whichreducesthedisk

seek time. We can also make the file system update asynchronously.Dependingupon

whetherthefilesystemsupportsit,wecansetthe-oasyncoption.

UsingNFSwithMySQL

While using a Network File System (NFS), varying issues may occur,dependingontheoperatingsystemandtheNFSversion.

Thefollowingarethedetails:

DatainconsistencyisoneissuewithanNFSsystem.Itmayoccurbecauseofmessagesreceivedoutoforderorlostnetworktraffic.WecanuseTCPwithhardandintrmountoptionstoavoidtheseissues.

MySQLdataandlogfilesmaygetlockedandbecomeunavailableforuseifplacedonNFSdrives.IfmultipleinstancesofMySQLaccessthesamedatadirectory,itmayresultinlockingissues.ImpropershutdownofMySQLorpower outage are other reasons for filesystem locking issues. The latestversionofNFSsupportsadvisoryandlease-basedlocking,whichhelpsinaddressing the locking issues.Still, it isnot recommended toshareadatadirectoryamongmultipleMySQLinstances.

Maximumfilesizelimitationsmustbeunderstoodtoavoidanyissues.WithNFS2,onlythelower2GBofafileisaccessiblebyclients.NFS3clients

supportlargerfiles.ThemaximumfilesizedependsonthelocalfilesystemoftheNFSserver.

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>performance_schema<spanclass="tokenpunctuation">.</span>setup_instruments<spanclass="tokenkeyword">WHERE</span><spanclass="tokenkeyword">NAME</span><spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'%memory%'</span><spanclass="tokenpunctuation">;</span></strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>performance_schema<spanclass="tokenpunctuation">.</span>setup_instruments<spanclass="tokenkeyword">WHERE</span><spanclass="tokenkeyword">NAME</span><spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'%memory/innodb%'</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span>

<spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>NAME<spanclass="tokenpunctuation">|</span>ENABLED<spanclass="tokenpunctuation">|</span>

TIMED<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span>

<spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/adaptivehashindex<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/buf_buf_pool<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/dict_stats_bg_recalc_pool_t<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/dict_stats_index_map_t<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/dict_stats_n_diff_on_level<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/other<spanclass="token

punctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/row_log_buf<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/row_merge_sort<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/std<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/trx_sys_t::rw_trx_ids<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|<br/></span></span><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="token

punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span></strong>

<strong><spanclass="tokenconstant">performance-schema-instrument</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>'memory/%=COUNTED'</span></strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>performance_schema<spanclass="tokenpunctuation">.</span>memory_summary_global_by_event_name<spanclass="tokenkeyword">WHERE</span>EVENT_NAME<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'memory/innodb/buf_buf_pool'</span>\G;<br/><br/>EVENT_NAME:<spanclass="tokenkeyword">memory</span><spanclass="tokenoperator">/</span>innodb<spanclass="tokenoperator">/</span>buf_buf_pool<br/>COUNT_ALLOC:<spanclass="tokennumber">1<br/></span>COUNT_FREE:<spanclass="tokennumber">0<br/></span>SUM_NUMBER_OF_BYTES_ALLOC:<spanclass="tokennumber">137428992<br/></span>SUM_NUMBER_OF_BYTES_FREE:<spanclass="tokennumber">0<br/></span>LOW_COUNT_USED:<spanclass="tokennumber">0<br/></span>CURRENT_COUNT_USED:<spanclass="tokennumber">1<br/></span>HIGH_COUNT_USED:<spanclass="tokennumber">1<br/></span>LOW_NUMBER_OF_BYTES_USED:<spanclass="tokennumber">0<br/></span>CURRENT_NUMBER_OF_BYTES_USED:<spanclass="tokennumber">137428992<br/></span>HIGH_NUMBER_OF_BYTES_USED:<spanclass="tokennumber">137428992</span></strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenfunction">SUBSTRING_INDEX</span><spanclass="tokenpunctuation">(</span>event_name<spanclass="tokenpunctuation">,</span><spanclass="tokenstring">'/'</span><spanclass="tokenpunctuation">,</span><spanclass="tokennumber">2</span><span

class="tokenpunctuation">)</span><spanclass="tokenkeyword">AS</span><br/>code_area<spanclass="tokenpunctuation">,</span>sys<spanclass="tokenpunctuation">.</span>format_bytes<spanclass="tokenpunctuation">(</span><spanclass="tokenfunction">SUM</span><spanclass="tokenpunctuation">(</span>current_alloc<spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">)</span><br/><spanclass="tokenkeyword">AS</span>current_alloc<br/><spanclass="tokenkeyword">FROM</span>sys<spanclass="tokenpunctuation">.</span>x$memory_global_by_current_bytes<br/><spanclass="tokenkeyword">GROUP</span><spanclass="tokenkeyword">BY</span><spanclass="tokenfunction">SUBSTRING_INDEX</span><spanclass="tokenpunctuation">(</span>event_name<spanclass="tokenpunctuation">,</span><spanclass="tokenstring">'/'</span><spanclass="tokenpunctuation">,</span><spanclass="tokennumber">2</span><spanclass="tokenpunctuation">)</span><br/><spanclass="tokenkeyword">ORDER</span><spanclass="tokenkeyword">BY</span><spanclass="tokenfunction">SUM</span><spanclass="tokenpunctuation">(</span>current_alloc<spanclass="tokenpunctuation">)</span><spanclass="tokenkeyword">DESC</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="token

punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>code_area<spanclass="tokenpunctuation">|</span>current_alloc<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="token

punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb<spanclass="tokenpunctuation">|</span>843.24MiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/performance_schema<spanclass="tokenpunctuation">|</span>81.29MiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/mysys<spanclass="tokenpunctuation">|</span>8.20MiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/sql<spanclass="tokenpunctuation">|</span>2.47MiB<spanclass="tokenpunctuation">|

</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/memory<spanclass="tokenpunctuation">|</span>174.01KiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/myisam<spanclass="tokenpunctuation">|</span>46.53KiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/blackhole<spanclass="tokenpunctuation">|</span>512bytes<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/federated<spanclass="tokenpunctuation">|</span>512bytes<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/csv<spanclass="tokenpunctuation">|</span>512bytes<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/vio<spanclass="tokenpunctuation">|</span>496bytes<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span>

<spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span></strong>

<strong>mysql>showglobalstatus;</strong><br/><strong>+-----------------------------+--------+</strong><br/><strong>|Variable_name|Value|</strong><br/><strong>+-----------------------------+--------+</strong><br/><strong>|Aborted_clients|0|</strong><br/><strong>|Aborted_connects|1|</strong><br/><strong>|Acl_cache_items_count|0|</strong><br/><strong>|Binlog_cache_disk_use|0|</strong><br/><strong>|Binlog_cache_use|0|</strong><br/><strong>|Binlog_stmt_cache_disk_use|0|</strong><br/><strong>|Binlog_stmt_cache_use|0|</strong><br/><strong>|Bytes_received|443|</strong><br/><strong>|Bytes_sent|346|</strong><br/><strong>|Threads_cached|0|</strong><br/><strong>|Threads_connected|1|</strong><br/><strong>|Threads_created|1|</strong><br/><strong>|Threads_running|2|</strong><br/><strong>+-----------------------------+--------+</strong>

<strong>mysql>showstatuslike'%Thread%';</strong><br/><strong>+------------------------------------------+-------+</strong><br/><strong>|Variable_name|Value|</strong><br/><strong>+------------------------------------------+-------+</strong><br/><strong>|Delayed_insert_threads|0|</strong><br/><strong>|Performance_schema_thread_classes_lost|0|</strong><br/><strong>|Performance_schema_thread_instances_lost|0|</strong><br/><strong>|Slow_launch_threads|0|</strong><br/><strong>|Threads_cached|0|</strong><br/><strong>|Threads_connected|1|</strong><br/><strong>|Threads_created|1|</strong><br/><strong>|Threads_running|2|</strong><br/><strong>+------------------------------------------+-------+</strong>

OptimizinglockingoperationsAsdiscussedinoneoftheearlierchapters,MySQL8useslockingmechanismstomanagecontention.Contentionoccurswhenconcurrentlyexecutingqueriesinmultiplethreadstry togetaholdofonetableat thesametime.If thesequeriesareperformedonthetableconcurrently,thetabledataisleft inaninconsistentstate. MySQL 8 supports two types of locking: internal locking and externallocking.

Internal locking is performedbymultiple threadswithin theMySQLserver tomanagecontentionfortablecontents.ThistypeoflockingisperformedentirelybytheMySQLserver,withoutinvolvinganyotherprograms.So,whyitiscalledinternal locking? In the caseof external locking, theMySQL server andotherprograms lock table files to decide which programs can access the table at atime.

Thefollowingarethetwomethodsforinternallocking:

Row-levellocking.Table-levellocking.

Row-level locking in MySQL supports simultaneous write access to multiplesessions. This enables multi-user and highly concurrent applications. Whileperformingmultiple concurrentwrite operations on a single table, it is highlypossiblethatadeadlockmayoccur.

Inordertoavoidsuchadeadlocksituation,alockingmechanismacquireslocksat the beginning of the transaction using the SELECT ... FOR UPDATE statement foreachsetofrowstobemodified.MySQLappliesthestatementsinthesameorderwithin each transaction if transactions lock more than one table. The InnoDBdatabase engine automatically detects deadlock conditions and rolls back theaffectedtransactions.Consideringthis,deadlocksaffectperformance.

Thedeadlockdetectionmaycauseslowdownifmanythreadswaitforthesamelock in highly concurrent systems. In such cases, it becomesmore efficient todisabledeadlockdetection.Wecanrelyontheinnodb_lock_wait_timeoutsettingfor

transaction rollback when deadlock occurs. Using the innodb_deadlock_detect

configurationoption,wecandisablethedeadlockdetection.

Thefollowingaretheadvantagesofrow-levellocking:

Whendifferentsessionsaccessdifferentrowsinatable,thenumberoflockconflictsisfewerThenumberofchangestoberolledbackisfewerItbecomespossibletolockasingletablerowforalongtime

Table-levellockingisusedbyMySQLforMyISAM,MEMORY,andMERGE tables. In thecase of table-level locking,MySQL permits only one session to update thesetablesatatime.Withtable-levellocking,thesestorageenginesbecomesuitablefor read-onlyor single-user applications.These storage engines request all therequiredlocksatonce,whenthequerybegins,toavoidanydeadlocks.Italwayslocksthetablesinthesameorder.Themajordrawbackwithtable-levellockingis that it affects concurrency. If other sessions need tomodify the table, theymustwaituntiltheconcurrentdatachangestatementfinishes.

Thefollowingaretheadvantagesoftable-levellocking:

Itrequireslessmemorycomparedtorow-levellockingWhenusedonalargepartofthetable,it isfast,becauseonlyonelockisrequiredIfGROUPBYoperationsareperformedfrequently,itisfast

ThefollowingisthestrategyforMySQLtograntwritelocksontables:

1. Putawritelockonthetableiftherearenowritelocksonthetable2. Puta lock request in thewrite lockqueue if the tablealreadyhasawrite

lock

ThefollowingisthestrategyforMySQLtograntreadlocksontables:

1. Putareadlockonthetableiftherearenoreadlocksonthetable2. Putalockrequestinthereadlockqueueifthetablealreadyhasareadlock

Morepriorityisgiventotableupdatesthantableretrievals.Thelockisavailabletothewritelockrequestsfirst,andthentothereadlockrequestswhenalockis

released.

Thefollowingisanexampletoanalyzetablelockcontention:

mysql>SHOWSTATUSLIKE'Table_locks%';

+-----------------------+-------+

|Variable_name|Value|

+-----------------------+-------+

|Table_locks_immediate|5|

|Table_locks_waited|0|

+-----------------------+-------+

TheMyISAMstorageengineinherentlysupportsmultipleconcurrentinsertsinordertoreducecontentionbetweenreadersandwritersforatable.ItallowstheMyISAMtable to insert rows in themiddleofadata file. If the tabledoesnothaveanyfreeblocksinthemiddleofthedatafile,therowsareinsertedattheendofthefile.ThisenablesMySQLtoexecuteINSERTandSELECTqueriesonthesametable,concurrently. concurrent_insert is the global system variable which controls thebehavioroftheMyISAMstorageenginetoallowexecutionofconcurrentINSERTandSELECTstatements.IfthissystemvariableissettoAUTO,concurrentINSERTandSELECTareallowed.

IfconcurrentinsertsarenotpossibleandwewanttoperformmultipleINSERTandSELECToperationsonatabletab1,wecanusethetemporarytabletemp_tab1toholdthetab1tabledataandupdatethetab1tablewiththerowsfromthetemp_tab1table.Thefollowingisanexamplewhichdemonstratesthisscenario:

mysql>LOCKTABLEStab1WRITE,temp_tab1WRITE;

mysql>INSERTINTOtab1SELECT*FROMtemp_tab1;

mysql>DELETEFROMtemp_tab1;

mysql>UNLOCKTABLES;

<strong>BENCHMARK(loop_count,expression)</strong>

<strong>mysql>selectbenchmark(1000000,1+1);</strong><br/><strong>+-------------------------+</strong><br/><strong>|benchmark(1000000,1+1)|</strong><br/><strong>+-------------------------+</strong><br/><strong>|0|</strong><br/><strong>+-------------------------+</strong><br/><strong>1rowinset(0.15sec)</strong>

Fromtheprecedingexample,wecanfindthatthetimetakentocalculate1+1for1000000timesis0.15seconds.

<strong>mysql>showprocesslist;</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|Id|User|Host|db|Command|Time|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>+------------------------+-----------------------+</strong><br/><strong>|State|Info|</strong><br/><strong>+------------------------+-----------------------+</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|4|event_scheduler|localhost|NULL|Daemon|214901|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|8|root|localhost:58629|NULL|Query|0|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>+------------------------+-----------------------+</strong><br/><strong>|Waitingonemptyqueue|NULL|</strong><br/><strong>+------------------------+-----------------------+</strong><br/><strong>|starting|showfullprocesslist|</strong><br/><strong>+------------------------+-----------------------+</strong>

<strong>mysql>select*frominformation_schema.processlist;</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|ID|USER|HOST|DB|COMMAND|TIME|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>+------------------------+----------------------------------------------+</strong><br/><strong>|STATE|INFO|</strong><br/><strong>+------------------------+----------------------------------------------+</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|8|root|localhost:58629|NULL|Query|0|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|4|event_scheduler|localhost|NULL|Daemon|215640|</strong><br/><strong>+----+-

----------------+-----------------+------+---------+--------+</strong><br/><strong>+------------------------+----------------------------------------------+</strong><br/><strong>|executing|select*frominformation_schema.processlist|</strong><br/><strong>+------------------------+----------------------------------------------+</strong><br/><strong>|Waitingonemptyqueue|NULL|</strong><br/><strong>+------------------------+----------------------------------------------+</strong>

Themysqladminprocesslistcommand.Theperformanceschemathreadstable,stagetables,andlocktables.

Wemustbeabletoviewtheinformationofuserthreads.ThePROCESSprivilegeisrequiredtoviewtheinformationaboutthreadsbeingexecuted.Toaccessthreads,aMutexaccessisnotrequired.IthaslessimpactontheMySQLserverperformance.AccessingINFORMATION_SCHEMA.PROCESSLISTandSHOWPROCESSLISTrequiresaMutexandhasanimpactonperformance.Threadsalsoprovidedetailsofbackgroundthreads.INFORMATION_SCHEMA.PROCESSLISTandSHOWPROCESSLISTdonotprovideinformationaboutbackgroundthreads.

Thefollowingtableshowstheinformationcontainedineachprocesslistentry:

Information Details

Id Clientconnectionidentifierfortheclientthatthethreadisassociatedwith.

User,Host Accountassociatedwiththethread.db DefaultdatabaseforthethreadorNULL.Command,State Itindicateswhatthethreadiscurrentlydoing.

Time Itindicateshowlongthethreadhasbeeninthecurrentstate.

Info Itcontainstheinformationofthestatementbeingexecutedbythethread.

Thefollowingisthethreadstatevaluesassociatedwithgeneralqueryprocessing:

Aftercreate:Itoccurswhenthethreadcreatesatable,includinginternaltemporarytablesAnalyzing:ItoccurswhenthethreadiscalculatingMyISAMkeydistributionCheckingpermissions:ItoccurswhencheckingiftheserverhastherequiredprivilegestoexecutetheSQLstatementCheckingtable:ItoccurswhenthethreadisperformingatablecheckoperationCleaningup:ItoccurswhenthethreadhasprocessedonecommandandfreesthememoryClosingtables:ItoccurswhenthethreadisflushingthechangedtabledatatodiskandclosingtheusedtablesAlteringtable:ItoccurswhentheserverisprocessingtheALTERTABLEstatementCreatingindex:ItoccurswhenthethreadisprocessingALTERTABLE...ENABLEKEYSfortheMyISAMtableCreatingtable:Itoccurswhenthethreadiscreatingatableend:Itoccursattheend,butbeforethecleanupofALTERTABLE,CREATEVIEW,DELETE,INSERT,SELECT,orUPDATEstatementsexecuting:Itoccurswhenthethreadhasbegunexecutingastatementinit:ItoccursbeforetheinitializationofALTERTABLE,DELETE,INSERT,SELECT,andUPDATEstatements

Thefollowingisthelistofcommonstatesinthemaster'sbinlogdumpthreadforreplicationmasterthreads:

Finishedreadingonebinlog;switchingtonextbinlogMasterhassentallbinlogtoslave;waitingformoreupdatesSendingbinlogeventtoslaveWaitingtofinalizetermination

ThefollowingisalistofcommonstatesforaslaveserverI/Othread:

CheckingmasterversionConnectingtomasterQueueingmastereventtotherelaylogReconnectingafterafailedbinlogdumprequestReconnectingafterafailedmastereventreadRegisteringslaveonmasterRequestingbinlogdumpWaitingforitsturntocommitWaitingformastertosendeventWaitingformasterupdateWaitingforslaveMutexonexitWaitingfortheslaveSQLthreadtofreeenoughrelaylogspaceWaitingtoreconnectafterafailedbinlogdumprequestWaitingtoreconnectafterafailedmastereventread

ThefollowingisalistofcommonstatesforaslaveserverSQLthread:

KillingslaveMakingtemporaryfile(append)beforereplayingLOADDATAINFILE

Makingtemporaryfile(create)beforereplayingLOADDATAINFILE

ReadingeventfromtherelaylogSlavehasreadallrelaylog;waitingformoreupdatesWaitingforaneventfromcoordinatorWaitingforslaveMutexonexit

WaitingforslaveworkerstofreependingeventsWaitingforthenexteventinrelaylogWaitinguntilMASTER_DELAYsecondsaftermasterexecutedevent

OptimizingdatabasestructureAsadatabaseadministrator,wemust look for efficientways toorganize tableschema, tables, and columns.Weminimize I/O, plan ahead, and keep relateditems together to tune the application code in order to keep performance highwithanincreaseindatavolume.Itusuallystartswithefficientdatabasedesign,whichmakesiteasierfor teammemberstowritehigh-performanceapplicationcode.Italsomakesthedatabaselikelytosustainitselfasapplicationsevolveorarerewritten.

Optimizingdatasize

In order to minimize the space on the disk, we should start designing thedatabase tables. This results in huge performance improvements, as it reducestheamountofdatatobewrittentoandreadfromthedisk.Smallertablesusuallyneedlessmainmemory,whilethecontentsareactivelyprocessedduringqueryexecution.Anyreductionintabledataspaceresultsinaneedforsmallerindexesthatcanbeprocessedfaster.

As discussed in the chapter onMySQL 8 data types,MySQL supports manydifferent storage engines and row formats. We can decide the storage andindexingmethodtobeusedforeachtable.Itisabigperformancegaintochoosethepropertableformat.

Tablecolumns

Weshouldusethesmallestfeasibledatatypeforatablecolumn.Thisresultsinthe most efficient approach. MySQL supports specialized data types to savememory and disk space. For example, we should use integer types whereverpossible to get smaller tables.Comparing MEDIUMINT and INT,MEDIUMINT is a betterchoice,asituses25%lessspacecomparedtoINT.

Wemustdeclarecolumns tobe NOTNULLwhereverpossible.ThisenablesbetteruseofindexesandeliminatestheoverheadoftestingwhethereachvalueisNULLornot.ItresultsinfasterSQLoperations.Wecanalsosaveonebitpercolumnofstoragespace.WeshoulduseNULLifwereallyrequireit.NULLvaluesshouldnotbeallowedasaresultofdefaultsettingsforeverycolumn.

We can attain huge performance gain for a table andminimize storage spacerequirementbyusingfollowingtechniques:

RowformatAsadefault,theDYNAMICrowformatisusedwhencreatingInnoDB tables.Wecanconfigureinnodb_default_row_format touse row formatsother than DYNAMIC.We canalso specify the ROW_FORMAT option explicitly in a CREATE TABLE or ALTER TABLE

statement.

The row formats include COMPACT, DYNAMIC, and COMPRESSED. They decrease rowstoragespaceatthecostofincreasedCPUuseforsomeoperations.

Fortheaverageworkload,whichislimitedbythecachehitratesanddiskspeed,itwillbefaster.IfitislimitedbytheCPUspeed,itwillbeslower.

TherowformatsalsooptimizetheCHARdatatypecolumnstoragewhenitusesavariable length character set.With the REDUNDANT row format, the CHAR(N)columnvalueoccupiesNtimesthemaximumbytelengthinthecharacterset.TheInnoDBstorageengineallocatesvariableamountsofstoragewithintherangeofNtoNtimesthemaximumbytelengthinthecharacterset.

Afixed-sizerowformatisusedifwedonothavevariable-lengthcolumns,suchasVARCHAR,TEXT,orBLOB,inthecaseofMyISAMtables.

Indexes

A table's primary index must be as short as feasible. This enables easyidentification of each row. It is efficient, too. In the case of InnoDB tables, theprimarykeycolumn isduplicated ineach secondary indexentry. Ifwehaveashortprimarykey,itsavesspaceinthecaseofmanysecondaryindexes.

We should create only those indexes which improve query performance. Theindexesimproveinformationretrieval,buttheyslowdowntheinsertandupdateoperations. Indexes must be created with proper attention to the performanceimpact. If it is required to access a table by searching on a combination ofcolumns, it is preferred to have a composite index on the combination ofcolumns, rather than a separate index on each of the columns.Themost usedcolumnshouldbethefirstpartoftheindex.Ifitisacommonrequirementtousemany columns in selected operations on the table, it is advisable to have thecolumn with the most duplicates as the first column in the index. This givesbettercompressionoftheindex.

If a long string column is supposed to have a unique prefix as the first fewcharacters, it is advisable to indexonly theprefix,usingMySQL's support forindexingon the leftmostpartof thecolumn.Shorter indexesarepreferred,notonlyforthelessspacetheyrequire,butalsobecausetheyprovidemorehitsin

theindexcacheandrequirefewerdiskseeks.

Joins

Ifatableisscannedveryoften,itisbeneficialtosplitthetableintotwotables,iffeasible. This holds true especially if it is a dynamic-format table. It is alsopossible to use smaller static format tables, which can be used to search forrelevantrowswhilescanningthetables.

The columnswith identical information should be declared in different tableswithidenticaldatatypes.Thisspeedsupjoinsbasedonmatchingcolumns.

Columnnamesmustbekeptsimple,soastousethesamenameacrosstables.Itsimplifies join queries. For example, in a customer table, we should use thecolumnnameofname,ratherthanusingcustomer_name.InordertomakethenamesportabletootherSQLservers,weshouldkeepthecolumnnamesshorterthan18characters.

Normalization

Thedatainthetablecolumnsmustbekeptnon-redundant,consideringthethirdnormalforminthenormalizationtheory.Ifthecolumnholdsrepeatinglengthyvalues, such as names or addresses, it is preferable to assign unique IDs andrepeat these IDs acrossmultiple smaller tables. In the event of searching, joinqueriesshouldbeusedbyreferencingIDsinthejoinclauses.

In an application, if the preference is speed and not disk space or themaintenancecostsofusingmultiplecopiesofdata, it is advisable toduplicatethe information or create summary tables to gain more speed. An examplescenario could be a business intelligence system,where data is analyzed fromlargetables.Inthiscase,normalizationrulesarenotstrictlyfollowed.

OptimizingMySQLdatatypes

Thefollowingaretheguidelinesforoptimizingnumericdatatypes:

Numeric columnsmust be preferred over string columns to store uniqueIDsorother

valuesthatcanberepresentedaseitherstringsornumbers.Itisfasterandoccupies

less memory to transfer and compare, because large numeric values arestoredinfewer

bytescomparedtostrings.

Itisfastertoaccessinformationfromadatabasethanfromatextfile.Thisis

especially truewhen numeric data is used. Information in the database isstoredin

a more compact format than in the text file. So, it requires fewer diskaccesses.

Thefollowingaretheguidelinesforoptimizingcharacterandstringdatatypes:

The binary collation order (logical sequence) should be used for fastercomparisons

andsortoperations.Thebinaryoperatorcanalsobeusedwithinaquerytousebinary

collationorder.

WithanInnoDBtable,whenweusearandomlygeneratedvalueasaprimarykey,itshouldbeprefixed

withanascendingvalue,suchasthedateandtime,iffeasible.Inthiscase,primary

keyvalues are stored nearer to eachother, physically. InnoDB can insert orretrievesuchvaluesfaster.

The binary VARCHAR data type should be used instead of BLOB for columnvaluesthatareexpectedtoholdlessthan8KBofdata.Iftheoriginal

table does not have any BLOB columns, the GROUP BY and ORDER BY clausesgeneratetemporarytables.ThesetemporarytablescanusetheMEMORYstorageengine.

In order to avoid string conversions while running a query, the columnsshouldbe

declared with the same character set and order wherever possible whencomparingthe

valuesfromdifferentcolumns.

Ifthetableholdsstringcolumnswhicharenotfrequentlyusedinretrievaloperations,

splitting the string columns into a separate table shouldbe considered. Intheretrieval

operations, join queries should be used with a foreign key wherevernecessary.MySQL

readsadatablockcontainingallthecolumnsofarowwhenitretrievesanyvalue

fromarow.Itallowsmorerowstofitwithineachdatablockwhenwekeeptherows

small, with only frequently used columns. These compact tables reducememoryusage

anddiskI/O.

ThefollowingaretheguidelinesforoptimizingBLOBdatatypes:

The performance requirements for a BLOB column may be different whenretrievinganddisplayinginformation.So,storingthe

BLOB specific table in a different storage device or a separate databaseinstanceshould

be considered. For example, it is required to retrieve a BLOB in a largesequentialdiskread.So,atraditionalharddriveoranSSDdevicemight

bettersuitneeds.

Inordertoreducethememoryrequirementsforaquerywhichdoesnotusea BLOB column, for a table with several columns, splitting the BLOB intoseparate tablesand referencingwith joinqueries shouldbeconsidered, asneeded.

Ifa tablecolumnisa largeblobwith textualdata,compressingshouldbeconsidered

first.Iftheentiretableiscompressedbythestorageengine,suchasInnoDBorMyISAM,thistechniqueshouldnotbeused.

<strong>>mysqladminstatus</strong><br/><strong>Uptime:262200Threads:2Questions:16Slowqueries:0Opens:111Flushtables:2Opentables:87Queriespersecondavg:0.000</strong>

<strong>mysql>SHOWGLOBALSTATUSLIKE'%Opened_Tables%';</strong><br/><strong>+---------------+-------+</strong><br/><strong>|Variable_name|Value|</strong><br/><strong>+---------------+-------+</strong><br/><strong>|Opened_tables|112|</strong><br/><strong>+---------------+-------+</strong>

UseofaninternaltemporarytableinMySQL

TheMySQL 8 server creates temporary internal tables while processing SQLstatements, in some cases. The following are the conditions under which theservercreatestemporarytables:

UNIONstatements

ViewswhichusestheTEMPTABLEalgorithm,UNION,oraggregation

Derivedtables

Commontableexpressions

Tablescreatedforsubqueryorsemijoinmaterialization

StatementsthatcontainORDERBYandGROUPBYclauses

StatementswithDISTINCTcombinedwithORDERBY

QueriesthatusetheSQL_SMALL_RESULTmodifier

INSERT...SELECTstatementsthatselectfromandinsertintothesametable

MultipletableUPDATEstatements

GROUP_CONCAT()orCOUNT(DISTINCT)expressions

TheEXPLAINstatementcanbeusedtodeterminewhetherthestatementrequiresatemporarytable.

TheEXPLAINstatementhaslimitations.Itwillnotindicateifthestatementrequiresatemporarytableforderivedormaterializedtemporarytables.

The Created_tmp_tables status variable keeps track of the number of temporarytablescreatedininternalmemory.WhentheMySQLservercreatesatemporarytable, it increments the value in the Created_tmp_tables status variable.Created_tmp_disk_tablesisanotherstatusvariablethatkeepstrackofthenumberoftablescreatedonthedisk.

Basedonthequeryconditions,theserverpreventstheuseoftemporarytablesinmemory.Insuchcases,theservercreatesatableonthedisk.Thefollowingaresomeinstances:

IfthetablehasaBLOBorTEXTcolumn

Ifthestatementhasastringcolumnwithamaximumlengthlargerthan512bytesintheSELECTlist,ifUNIONorUNIONALLisused

IftheSHOWCOLUMNSandDESCRIBEstatementsuseBLOBasthetypeofthecolumn

Thefollowingare theconditions inwhichaUNION isevaluatedwithoutcreatingtemporarytables:

TheunionisUNIONALLandnotUNIONorUNIONDISTINCT

ThereisnoglobalORDERBYclause

InaSELECTquery,theunionisnotatthetop-levelqueryblock

OptimizingqueriesSimilartotables,databasequeriesarethemostcrucialelementofanydatabase.Applications interactwith the databases using queries.Queries are also calledexecutableSQLstatements.This section focuseson techniques to improve theperformanceofqueryexecution.

<strong><spanclass="tokenpunctuation">(</span><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator">AND</span>b<spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">AND</span>c<spanclass="tokenoperator">OR</span><spanclass="tokenpunctuation">(</span><spanclass="tokenpunctuation">(</span><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator">AND</span>b<spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">AND</span><spanclass="tokenpunctuation">(</span>c<spanclass="tokenoperator">AND</span>d<spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">)</span><br/><spanclass="tokenprompt">-></span><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator">AND</span>b<spanclass="tokenoperator">AND</span>c<spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">OR</span><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator">AND</span>b<spanclass="tokenoperator">AND</span>c<spanclass="tokenoperator">AND</span>d<spanclass="tokenpunctuation">)</span></strong>

<strong><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator"><</span>b<spanclass="tokenoperator">AND</span>b<spanclass="tokenoperator">=</span>c<spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">AND</span>a<spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span><br/><spanclass="tokenprompt">-></span>b<spanclass="tokenoperator">></span><spanclass="tokennumber">5</span><spanclass="tokenoperator">AND</span>b<spanclass="tokenoperator">=</span>c<spanclass="tokenoperator">AND</span>

a<spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span></strong>

<strong><spanclass="tokenpunctuation">(</span>B<spanclass="tokenoperator">>=</span><spanclass="tokennumber">5</span><spanclass="tokenoperator">AND</span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span><spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">OR</span><spanclass="tokenpunctuation">(</span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">6</span><spanclass="tokenoperator">AND</span><spanclass="tokennumber">5</span><spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span><spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">OR</span><spanclass="tokenpunctuation">(</span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">7</span><spanclass="tokenoperator">AND</span><spanclass="tokennumber">5</span><spanclass="tokenoperator">=</span><spanclass="tokennumber">6</span><spanclass="tokenpunctuation">)</span><br/><spanclass="tokenprompt">-></span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span><spanclass="tokenoperator">OR</span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">6</span></strong>

Optimizingindexes

Thebasicuseofindexesistoquicklyfindtherowswithspecificcolumnvalues.

Iftheindexisnotpresent,MySQLbeginswiththefirstrowandreadsthroughtheentiretabletofindallthematchingrows.Ittakesmoretime,dependingonhowlargeatableis.Iftheindexispresentfortheappropriatecolumns,MySQLisabletoquicklydeterminethepositiontoseektointhemiddleofthedatafile,withoutlookingatthewholetabledata.

ThefollowingisalistofoperationsforwhichMySQLusesindexes:

Tofindmatchingrows,basedonaWHEREclause,quickly.

MySQLuses the indexwith the smallest number of rows (most selectiveindex)inthecaseofchoosingfrommultipleindexestoeliminaterowsfromconsideration.

Theoptimizeruses the leftmostprefixof the index to lookuprows if thetable has a composite index. For example, in a tablewith three columnsindexed(oncol1,col2,col3),theoptimizercanlookforrowswithindexed

searchcapabilitieson(col1),(col1,col2),and(col1,col2,col3).

MySQLusesindexeswhileitfetchesrowsfromothertablesusingjoins.Ifthe indexesaredeclaredas thesame typeandsize,MySQLcanuse themefficientlyonthecolumn.

TheVARCHARandCHARareconsideredthesamewhendeclaredasthesamesize.

MySQLalsousesindexestofindtheminimum(MIN())ormaximum(MAX())valueforanindexedcolumnkey_col.ThepreprocessorcheckswhetheritisusingWHEREkey_part_N=constantonallkeypartstooptimizeit.

It is also possible to optimize the query to retrieve values withoutconsultingthedatarows.(Acoveringindexisanindexthatprovidesalltheresultsforaquery.)Ifthequeryusesonlythosecolumnsfromatablewhichare included in some index, the selected valueswill be fetched from theindextree.Thiswillhaveahigherspeedinretrievingvalues.

Queryexecutionplan

TheMySQLoptimizerconsidersoptimizationtechniquestoefficientlyperformthe lookups involved in the query, depending on the details of the tables,columns,andindexes,andtheconditionsintheWHEREclause.Aquerycanalsobeperformedwithoutreadingalltherowsonahugetable.

An SQL join can also be performedwithout comparing every combination ofrows.Aqueryexecutionplan is a setofoperations that theMySQLoptimizerchoosestoperformthemostefficientquery.ItisalsoknownastheEXPLAINplan.Asanadministrator,thegoalistorecognizetheaspectsofthequeryexecutionplanwhichindicateifaqueryisoptimized.

The EXPLAIN statement is used to determine the query execution plan. ThefollowingisthesetofinformationprovidedbytheEXPLAINstatement:

The EXPLAIN statement works with SELECT, DELETE, INSERT, UPDATE, and REPLACEstatements.

MySQLdisplays information from theMySQLoptimizer about thequeryexecution planwhen EXPLAIN is usedwith the SQL statement. Thismeans

MySQL explains the process with which the statement is executed. Itincludesinformationabouthowtablesarejoined,andinwhichorder.

If EXPLAIN displays the execution plan for the statement execution in thenamedconnection if it isusedwithFORCONNECTION connection_id insteadofexplainableSQLstatement.

EXPLAINdisplaysadditionalexecutionplaninformationforSELECTstatements.

EXPLAINisalsousefulinexaminingquerieswhichinvolvepartitionedtables.

EXPLAIN supports a FORMAT option, which can be used to select the outputformat.TheTRADITIONALformatdisplaystheoutputinatabularformat.Thisisthedefaultformatoption.

The JavaScript Object Notation (JSON) format option producesinformationintheJSONformat.

Based on the output from the EXPLAIN statement, it can be figured out whereindexescanbeaddedto the tables,so that thestatementexecutesfaster. Itcanalso be found whether the optimizer joins the tables in the optimized order.BeginthestatementwithSELECTSTRAIGHT_JOIN,insteadofjustSELECT,togiveahintto theoptimizer touse the joinordercorresponding to theorder the tablesarenamedintheSELECTstatement.AsSTRAIGHT_JOINdisablessemi-jointransformations,itmaypreventtheuseofindexes.

The optimizer trace is another tool to find the information on the queryexecution.

ItispossiblethattheoptimizertracemayprovideinformationdifferingfromthatofEXPLAIN.Theformatandcontentoftheoptimizertracearesubjecttovariation,basedontheversions.

ThefollowingtableshowstheoutputformatoftheEXPLAINstatement:

Column JSONName Details

id select_id TheSELECTidentifier

select_type NoneTheSELECTtype

table table_name Thetablefortheoutputrowpartitions partitions Thematchingpartitionstype access_type Thejointypepossible_keys possible_keys Thepossibleindexestochoosekey key Theindexactuallychosenkey_len key_length Thelengthofthechosenkeyref ref Thecolumnscomparedtotheindexrows rows Estimateofrowstobeexaminedfiltered filtered PercentageofrowsfilteredbytableconditionExtra None AdditionalInformation

Reference:https://dev.mysql.com/doc/refman/8.0/en/explain-output.html#explain-output-column-table

OptimizingtablesDatabase tables are the most basic building blocks for any database. In thissectionofthechapter,wewillfocusonoptimizingtables.Thesectionprovidesdetailed guidelines for improving performance through table optimizationtechniques.

OptimizationforInnoDBtables

TheInnoDB storage engine is preferred in production environments in situationswherereliabilityandconcurrencyareimportant.ItisthedefaultstorageengineforMySQLtables.

ThissectionfocusesonoptimizingdatabaseoperationsforInnoDBtables.

ThefollowingaretheguidelinestooptimizeInnoDBtables:

Useof the OPTIMIZETABLE statement should be considered to reorganize thetableandcompact thewastedspaceonce thedata reachesastablesizeorthetablehasincreasedbytensofmegabytes.

ItrequireslessdiskI/Otoperformfulltablescansforreorganizedtables.

The OPTIMIZE TABLE statement copies the data in the table and rebuilds theindexes. It is beneficial because of improved packing of the data withinindexes, and fragmentation reductionwithin the table spaces on the disk.The benefits may vary, depending on the data in each table. It may benoticeable that thegainsaresignificant insomecases,andnot forothers.

Thegainsmayalsodecreaseovertimeuntil thenexttableoptimizationisdone.Theoperationcanbeslow if the table is largeor the indexesbeingrebuiltdonotfitinthebufferpool.

AlongprimarykeyinaInnoDBtablewastesalotofdiskspace.Itshouldbeavoided.

InInnoDBtables,preferenceshouldbegiventotheVARCHARdatatypeinsteadoftheCHARdatatypetostorevariablelengthstrings,orforcolumnswhichareexpected to contain NULL values. A CHAR(N) column always occupies Ncharacters to storedata, even if thevalue is NULL. Smaller tables aremoresuitabletofitinthebufferpoolandreduceddiskI/O.

Considerusinga COMPRESSED row format forbig tables, or tables containinglotsofrepetitivetextornumericdata.

OptimizationforMyISAMtables

Forread-onlyorread-mostlydata,orforlowconcurrencyoperations,theMyISAMstorage engine fits the best. This is because table locks limit the ability toperform simultaneousupdates. In this section, the focuswill beonoptimizingqueriestobeexecutedonMyISAMtables.

ThefollowingaretheguidelinesforspeedingupqueriesonMyISAMtables:

AvoidexecutingcomplexSELECTqueriesonfrequentlyupdatedMyISAMtables.It prevents problemswith table locking that occur because of contentionbetweenwritersandreaders.

TheMyISAM storageenginesupportsconcurrent inserts. If the tabledata filedoesnothavefreeblocksinthemiddle,wecanINSERTnewrowsinitatthesametimethatotherthreadsarereadingfromthetable.Considerusingthetable to avoiddeleting rows if it is important tobe able todo concurrentread-write operations. Another option is to execute OPTIMIZE TABLE todefragment the table after deletion of the rows. This behavior can becontrolledormodifiedbysettingtheconcurrent_insertsystemvariable.

Avoidallvariable-lengthcolumnsforfrequentlychangingMyISAMtables.Thedynamicrowformatisusedbythetableifitincludesevenasinglevariablelengthcolumn.

The myisamchk --sort-index --sort-records=1 command can be used to sort anindex.Italsosortsdataaccordingtotheindex.

This makes the queries run faster if we have unique indexes, based onwhichwewanttoreadallrowsintheorderaccordingtotheindex.Ittakesalongtimewhenwesortalargetablethiswayforthefirsttime.

Ifweusuallyretrieverowsintheorderofexpression1,expression2,andsoon,use ALTER TABLE ... ORDER BY expression1, expression2,.., and so on. Thiswillgive higher performance, if this option is used after extensive changes tothetable.

OptimizationforMEMORYtables

MySQLMEMORYtablesshouldbeconsideredforuseonlyfornoncriticaldatathatisaccessed often and is read-only and rarely updated.The application should bebenchmarkedagainstequivalentInnoDBorMyISAM tablesunderrealisticworkloadstoconfirmthatadditionalperformanceisworththeriskoflosingdata.

Weshouldexaminethekindsofqueriesagainsteachtableforbestperformancewith MEMORY tables.We should also specify the type of use for each associatedindex.ItcanbeaB-treeindexorahashindex.UsetheUSINGBTREEorUSINGHASHclauseontheCREATEINDEXstatement.

LeveragingbufferingandcachingThissection focusesonusingbufferingandcaching techniques to increase theperformance.

InnoDBbufferpooloptimization

TheInnoDBstorageenginemaintainsastorageareaknownasthebufferpool.Itisusedfor

cachingdataandindexesinthememory.ItisimportanttoknowhowtheInnoDBbufferpoolworks,soastotakeadvantageofittokeepfrequentlyaccesseddata

inmemory.ItisanimportantaspectofMySQLtuning.

The following are the general guidelines for improving performance with theInnoDBbufferpool:

In an ideal case, the size of the buffer pool should be set large enough,whileleaving

enoughmemoryforotherprocessesontheservertorunwithoutexcessivepaging.With

larger buffer pools,more InnoDB functions, like an in-memorydatabase. Inthiscase,itreadsdatafromthedisk

once,andthenaccessesthedatafrommemoryinsubsequentreads.

Wecanconsidersplittingthebufferpoolintomanypartsfor64-bitsystemswith

large memory sizes. This minimizes contention for memory duringconcurrentoperations.

Thefrequentlyaccesseddatashouldbekeptinmemory.

ItispossibletocontrolwhenandhowInnoDBperformsread-aheadrequeststoprefetchpagesintothebufferpoolasynchronously.

InnoDBuses tworead-aheadalgorithmsto improveI/Operformance.Linearreadaheadpredicts

whatpagesmightbeneededsoon,basedonthepagesbeingaccessedinthebufferpool

sequentially. Random read ahead predicts when pages might be neededbasedonthepages

in the buffer pool, regardless of the order in which pages are read. Theinnodb_read_ahead_thresholdconfigurationparametercontrolsthesensitivityoflinearreadahead.Wecanenable

randomreadaheadsbysettinginnodb_random_read_aheadtoON.

innodb_buffer_pool_read_ahead determines the number of pages read into theInnoDBbufferpool.innodb_buffer_pool_read_ahead_evicteddeterminesthenumberofpagesreadintothebufferpoolbytheread-aheadbackground

thread that was subsequently evicted without having been accessed byqueries.The

innodb_buffer_pool_read_ahead_rnd determines the number of random readaheadsinitiatedbyInnoDB.

MyISAMkeycache

The MyISAM storage engine incorporates a strategy that is supported by manydatabasemanagementsystemstominimizethediskI/O.Thecachemechanismis employed by MyISAM to keep the most frequently accessed table blocks inmemoryasfollows:

A special structureknownas akey cache ismaintained for indexblocks.Themostusedindexblocksareplacedinthestructurecontaininganumberofblockbuffers.

MySQL relies on the native operating system filesystem cache for datablocks.

Thekey_buffer_sizesystemvariabledetermines thesizeof thekeycache. If it isset to zero, no key cache is used. The key cache is also not used if thekey_buffer_sizevalueistoosmalltoallocatetheminimumorderofblockbuffers.All theblockbuffers in thekeycachestructureareof thesamesize.Thissizecanbeequal to,greater than,or less than the sizeof the table indexblock. Inusualcases,oneofthesetwovaluesisamultipleoftheother.

When it is required to access data fromany table indexblock, the server firstchecks if it is available in some block buffer of the key cache. If the data isavailable,theserveraccessesdatafromthekeycacheratherthanonthedisk.Ifthedata isnotavailable, theserverselectsacacheblockbuffer thatcontainsadifferent table index block and replaces the data in it by copying the requiredtableindexblock.

Theindexdatacanbeaccessedassoonasthenewindexblockisavailableinthecache.

TheMySQLserverfollowstheLeastRecentlyUsed(LRU)strategy.Accordingto it, it selects the least recently used index blockwhile choosing a block forreplacement.ThekeycachemodulecontainsallusedblocksintheLRUchain(aspecial list).Thelist isorderedbythe timeofuse.It is themostrecentlyusedwhentheblockisaccessed.Theblockisplacedattheendofthelist.Blocksatthebeginningofthelistaretheleastrecentlyusedwhentheblocksneedtobereplaced.So,theblockatthetopbecomesthefirstcandidateforeviction.

The block is considered dirty if the block selected for replacement has beenmodified.

Theblockcontentsareflushedtothetableindexfromwhichtheycamepriortoreplacement.

Based on the following conditions, the threads can access key cache bufferssimultaneously:

Thebufferwhichisnotbeingupdatedcanbeaccessedbymultiplesessions

Thebufferwhichisbeingupdatedcausessessionsthatrequirewaitinguntiltheupdateiscompletetouseit

As long as the sessions are independent and do not interfere with eachother, multiple sessions can initiate requests resulting in cache blockreplacements

Inthisway,sharedaccesstothekeycacheimprovesperformancesignificantly.

Summary

In this chapter, we learned, in detail, the techniques to optimize MySQL 8components.

The chapter started with the basics of optimization, including hardware andsoftwareoptimizationguidelines.WealsodiscussedoptimizationguidelinesfortheMySQL8

server and client, database structure, queries, and tables. We also coveredoptimization for tables belonging to different storage engines, such as MyISAM,InnoDB,andMEMORY.Welearnedthetools,suchasEXPLAINandEXPLAINANALYZE,neededto understand the query execution plan. In the later part of the chapter, welearnedbufferingandcachingtechniquestoimproveperformance.

It's time to move on to the next chapter now. The next chapter focuses ontechniques to extend MySQL 8. The chapter will cover in-depth details ofMySQL8plugins,whichhelp toextend thedefaultMySQL8 features. Itwillalsoexplain the services tocall theseplugins.Thechapterwilldiscussaddingnewfunctions,debugging,andportingmethods. It isgoing tobean importantchapterfordatabaseadministrators.

ExtendingMySQL8

Inthepreviouschapter,welearnedhowtooptimizeMySQL8.Wealsolearnedwhat configurationsneed to bedone to achieveoptimization, and alsohow toleveragecachingandbufferingforoptimization.Wewent throughtheusecasestudystepbystepforachievingoptimizationinthefollowingcomponents:

OptimizingMySQL8serverandclient

Optimizingdatastructures

Optimizingqueries

Optimizingtables

In this chapter,wewill learn about extendingMySQL8.Wewill checkwhatMySQL8

components are allowed to extend, and we will look at how to customizeMySQL 8 for specific business needs. You will learn about the fundamentalcomponentspriortoextendingMySQL8andthefeaturesoftheMySQLplugin

API thatwill be used to extendMySQL8. The following is the list of topicscoveredinthischapter:

AnoverviewofextendingMySQL8

Extendingpluginsandusingservicestocallthem

Addingnewfunctions

Debuggingandporting

AnoverviewofextendingMySQL8

In thissection,youwill learnaboutoneof themostexciting topicsonhowtoextendMySQL8asperyourneeds.ThereareseveralcomponentsofMySQL8thatyoushouldunderstandwellpriorattemptingtoextendMySQL8.HereisalistofthecomponentsthatareimportantforextendingMySQL8:

MySQL8internals

MySQL8pluginAPI

MySQL8servicesforcomponentsandplugins

AddingnewfunctionstoMySQL8

DebuggingandportingMySQL8

MySQL8internals

TherearefewthingsyoushouldknowbeforeyoustartworkingontheMySQLcode. To contribute or track MySQL development you should follow theinstructionsfortheinstallationofsourcecodedistributionasperyoursystemoroperating system platform. The source code includes internal documentation,whichisveryimportanttounderstandhowMySQL

internally works from developer's perspective. You can also subscribe to theinternalsmailinglistfromhttps://lists.mysql.com/internals,whichincludespeoplewhoworkonMySQLcode,andyoucanalsodiscusstopicsrelatedtoMySQLdevelopmentorpostingpatches:

MySQL 8 threads: MySQL server creates threads such as connectionmanager threads, signal threads, read and write threads if using InnoDBstorageengine,schedulerthreadstohandleconnection,andreplicationandeventprocessing.

MySQL 8 test suite: MySQL 8 provides the test systems included withUnixsourcedistributiontohelpusersanddevelopersperformingregressiontestingwithMySQLcode.Youcanalsowriteyourowntestcasesusingthe

testframework.

MySQL8pluginAPI

MySQL 8 provides support for plugin API by which server componentsthemselvescanbecreated.Thepluginscanbeloadedduringserverstartupandcanalsobeloadedandunloadedduringruntime;thereisnoneedtorestart theserver.TheAPIisverygenericinthatitdoesnotspecifywhatpluginscandointerms of limitation but instead they are allowed to do more than build-incomponents. The API supports interfaces for components such as storageenginesplugins,full-textparserplugins,serverextensionsandsoon.

Theplugin interfacemakesuseof theplugin table in theMySQL8database tostore the information about installed plugins permanently by using the INSTALLPLUGIN statement. During the MySQL 8 installation process the plugin table iscreated.Forsingleserverinvocationthepluginscanalsobeinstalledusingthe--plugin--loadoption,butusing thisoptiondoesnot record the installedplugin totheplugintable.

MySQL8 also provides supportAPI for client plugins to be used for specificpurposes such as enabling the server connection by client through differentauthenticationmethods.

MySQL 8 services forcomponentsandplugins

The MySQL 8 server plugins can access and initiate server plugin services;similarly, the server components can also access and request componentservices.TheMySQL8

plugin Services interface complements the API plugin by exposing serverfunctionality, which can be called by plugins. The following are the pluginservicecharacteristics:

The services enable plugins to access the server code using ordinaryfunctioncallsandcanalsocalluser-definedfunctions

Theservicesareportableandcanworkonmultipleplatforms

The services provide versioning support that protects againstincompatibilitiesbetweenpluginsandservices

Theservicesalsoprovidesupportfortestingpluginservices

MySQL provides two services types for plugins and components, listed asfollows:

1. Thelockingservice:Thelockingserviceinterfaceisprovidedattwolevels—that is, at C level and at SQL level. The interface works on locknamespace,lockname,andlockmodeattributes.

2. Thekeyringservice:Thekeyringserviceprovidesaninterfaceforsecurelystoringsensitiveinformationforinternalservercomponentsandpluginstoretrievelater.

AddingnewfunctionstoMySQL8

YoucanaddyourownfunctionstoMySQL8,andthiscanbedonewithanyoneof the three supported types of function. The new function can be called thesame way we invoke the built-in functions such as ABS(), and that is trueirrespectiveofwhichfunctiontypeyouhavenewlyadded.ThefollowinglistisofthesupportedthreetypesofnewfunctioninMySQL8:

1. Addingafunctionthroughtheuser-definedfunction(UDF)interface.

2. Addingafunctionasnative(built-in)MySQLfunction.

3. Addingafunctionbycreatingastoredfunction.

DebuggingandportingMySQL8

PortingMySQL 8 to other operating systems is currently supported bymanyoperatingsystems;thelistofsupportedoperatingsystemsisprovidedathttp://www.mysql.com/support/supportedplatforms/database.html. In case you have added a newportandarerunningintoproblemswiththenewport,youmightusedebuggingofMySQL8.

There are different possible ways to start debugging based on where you arerunningintotheproblems—theycouldbeinMySQLserverorinMySQLclient.Dependingontheproblem'slocation,youcanstartdebugginginMySQLserveror client respectively and also get help from the DBUG package to trace theprogram'sactivities.

Extending plugins and usingservicestocallthem

In this section, you will gain an understanding of how the plugin API, itsinterface, and the MySQL services interact with one another and provideextensionsinMySQL8.

ThepluginsarealsoconsideredascomponentsintheMySQL8architecture,andthereforeyoucanuse them toprovidepluggable features.ThepluginAPIandthepluginservicesinterfaceshavethefollowingdifferences:

ThepluginAPIenablespluginsthatwillbeusedbytheserver.Thecallingandinvokingofpluginsisinitiatedbytheserver,sothepluginscanextendthe server's functionality or can register themselves in order to receiveserverprocessingnotifications.

The plugin services interface allows plugins to call the server code. Thecallingandinvokingofservicefunctionsisinitiatedbythepluginssothatthe same server functionality can be leveraged bymany plugins withoutrequiringindividualimplementationforthefunctionality.

Writingplugins

To create a plugin library, providing the required descriptor information is amust,asitspecifieswhichpluginsthelibraryfilecontains.Writingtheinterfacefunctionforeachofthepluginsspecifiedisalsonecessary.

Everyserverpluginmusthaveageneraldescriptorprovidinginformationtotheplugin APIs, and a type specific descriptor providing information about theinterface for specified plugin types. The structure for specifying a generaldescriptor is the same for all theplugin types, and the type specificdescriptorcan vary based on the requirements of the plugin's behavior or function. Theserverplugininterfaceallowspluginstoexposesystemvariablesandstatus.

Client-sidepluginshaveaslightlydifferentarchitecturethanthatofserversideplugins.Forexample,eachpluginmusthavedescriptorinformation,butthereisnoseparatedivisionbetweengeneralandtypespecificdescriptors.

PluginscanbewritteninCorC++oranyotherlanguagethatcanuseCcallingconventions.

Pluginsareloadedandunloadeddynamically,hencetheoperatingsystemmustdynamically support where you have dynamically compiled the callingapplication.Specifically,forserverpluginsthismeansthatmysqldmustbelinked

dynamically.

Aswecannotbesureofwhatapplicationwilluse theplugin, thedependencieson the symbolsof the callingapplication shouldbeavoidedbytheclientpluginwriters.

The following are the types of supported plugin creations that can implementseveralcapabilities:

Authentication

Passwordvalidationandstrengthchecking

Protocoltracing

Queryrewriting

Securekeyringstorageandretrieval

Storageengines

Full-textparsers

Daemons

INFORMATION_SCHEMAtables

Semisynchronousreplication

Auditing

Componentandpluginservices

YoucanidentifythecomponentservicesandfunctionsprovidedbyMySQLbylooking into the include/mysql/components and respective services directories of theMySQL8sourcedistribution.

Similarly, you can identify the plugin services and functions provided byMySQL by looking into the include/mysql directory of the MySQL 8 sourcedistributionandtherelevantfilesasfollows:

Theplugin.hfileincludestheservices.hfile,whichservices.hfilecontainsalltheavailableservice-specificheaderfileswithinit

Service-specificheaderfileswillhavenamesintheformofservice_xxx.h

ThefollowingisalistofavailablecomponentservicesinMySQL8:

component_sys_variable_register, component_sys_variable_unregister: For registeringandunregisteringsystemvariables

log_builtins,log_builtins_string:Forlogcomponentsservices

mysql_service_udf_registration, mysql_service_udf_registration_aggregate: Forenablingregistrationandunregistrationofscalarandaggregateuser-definedfunctionsincomponentsandplugins

mysql_string:ForstringserviceAPIs

pfs_plugin_table:FordynamicPerformanceSchematablemanipulation

ThefollowingislistofavailablepluginsservicesinMySQL8:

get_sysvar_source:Forretrievingsystemvariablesettings

locking_service: For lock implementation with C language and SQL levelinterfaces,havingtheattributesnamespace,name,andmode

my_plugin_log_service:Forwritingerrorsmessagestologs

my_snprintf: For string formatting to keep the output consistent acrossplatforms

status_variable_registration:Forregisteringthestatusvariable

my_thd_scheduler:Forthreadschedulerselection

mysql_keyring:Forkeyringstorageservice

mysql_password_policy:Forpasswordstrengthandvalidationchecking

plugin_registry_service: For accessing the component registry and relatedservices

security_context:Formanagingthreadsecuritycontexts

thd_alloc:Formemoryallocation

thd_wait:Forreportingtosleeporstall

Now,youhaveaclearunderstandingofpluginservicesandcomponentservices.MySQL

8 provides the following types of services to support plugins and componentsservices:

1. Thelockingservice

2. Thekeyringservice

Thefollowingsectionsgivedetailedinformationonbothtypesofservices.

Thelockingservice

Thelockingserviceinterfaceisprovidedattwolevels:ClevelandatSQLlevel.

The interface works on the lock namespace, lock name, and lock modeattributes. The C language interface is callable as a plugin service from user-definedfunctionsorserverplugins,andtheSQLlevelinterfaceisusedassetofuser-definedfunctions,beingmappedtocalltheserviceroutines.

Thefollowingarethecharacteristicsofthelockinginterface:

Lock namespace, lock name, and lockmode are three three attributes oflocks.

Locks are identified by forming a lock namespace and lock namecombination.

Lockmodecanbeeitherreadorwrite.Readlocksaresharedwhereaswritelocksareexclusive.

Lock names and namespaces can have amaximum of 64 characters and

mustbenon-NULL

andnon-emptystrings.

Locknamesandnamespacearetreatedasbinarystringssocomparisonwillbecase-sensitive.

Functionsareprovidedtoacquireandreleaselocksanddonotrequireanyspecialprivilegestocallthefunctions.

Detectsdeadlockduringlockacquisitioncallsindifferentsessions;acallerischosenandterminatedforitslockacquisitionrequestandcallersessionsholdingreadlocksarepreferredoverthesessionsholdingwritelocks.

A typical session can request formultiple locks acquisitionwith a singlelock acquisition call. It provides atomic behavior for the request andsucceedsifalllocksareacquiredorfailsifanyofthelockacquisitionsfail.

Multiple locksfor thesame lock identifiercanbeacquiredby thesessionwhere the lock instances canbewrite locks, read locks,or amixofbothreadandwritelocks.

Acquired locks are released from the session by explicitly calling therelease-lockfunction,orimplicitlyifthesessiongetsterminated.

All locks in the given namespace when released are released togetherwithinthesession.

<strong>boolmy_key_generate(constchar*key_id,constchar*key_type,<br/>constchar*user_id,size_tkey_len)</strong>

<strong>boolmy_key_fetch(constchar*key_id,constchar**key_type,<br/>constchar*user_id,void**key,size_t*key_len)</strong>

<strong>boolmy_key_remove(constchar*key_id,constchar*user_id)</strong>

<strong>boolmy_key_store(constchar*key_id,constchar*key_type,<br/>constchar*user_id,void*key,size_tkey_len)</strong>

Addingnewfunctions

NewfunctionscanbeaddedwithanyofthethreesupportedtypesinMySQL8.Each of the types have their own advantages and disadvantages. Where andwhich type of function should be added or implemented depends on therequirementsofthefunction.

ThefollowingisthelistofthesupportedthreetypesofnewfunctioninMySQL8,whichwewilllookatinthefollowingsection:

1. Addingafunctionthroughtheuser-definedfunctioninterface.

2. Addingafunctionasanative(built-in)MySQLfunction.

3. Addingafunctionbycreatingastoredfunction.

Features of a user-definedfunctioninterface

A user-defined function interface provides independent capabilities to a userpurposefunction.

ThefollowingfeaturesandcapabilitiesareprovidedbytheMySQLinterfaceforuser-definedfunctions:

Functions can accept arguments of integer, string, or real values and canreturnvaluesforthesametypes

Simplefunctionscanbedefinedtooperateonasinglerowata timeor itcanbeaggregatefunctionstooperateongroupsofrows

Functionsaregiveninformationtoenablethemsothattheycancheckthetypes,names,andnumbersofargumentspassed

Beforepassingargumentstothegivenfunction,youcanalsoaskMySQLtocoercearguments

IndicationscanbemadeifthefunctionresultsinanyerrororreturnsNULL

Adding a new user-definedfunction

TheUDFfunctionsmustbewritten inCorC++and theunderlyingoperatingsystemmustsupportdynamicloadingbehavior.Thereisafile,sql/udf_example.cc,that defines five UDF functions and it's included in the MySQL sourcedistributions.

Analyzing the filewill letyouknowhowcallingconventionswork forUDFs.User-defined function related symbols and data structures are defined in theinclude/mysql_com.hfileandthefileisincludedinthemysql.hheaderfile.

Typicalcodecontained in theUDFsgetsexecuted in therunningserver,soallconstraints are applicable when writing UDF code—server code. Currentlyapplicableconstraintsmayget revisedwhenaserver isupgraded,and thiscanpossiblyresultintotheneedtorewriteUDFcode,soitisessentialtobecarefulwhenwritingcodefortheUDF.

InordertouseUDF,linkingmysqlddynamicallyisamust.Foranyfunctiontobeused in SQL statements there must be underlying C or C++ functions. Theconvention for separating SQL and C/C++ code is followed where xxx() in

uppercaseindicatesanSQLfunctioncallwhereasxxx()withlowercaseindicatesaC/C++functioncall.

EncapsulateyourCfunctionasshowninfollowingsentencewhenyouareusingC++: extern"C"{ ... }This way it is ensured thatyour C++ function names are readable in the completed user-definedfunction.

TowriteandimplementtheinterfacefunctionnameXXX(),themainfunctionxxx()isamustandadditionallyrequiresoneormorefunctiontobeimplementedfromthefollowing:

xxx():Themainfunctionwherethefunctionresultisbeingproduced

xxx_init():The initialization function for themain function xxx(), it canbeusedforanyofthefollowingpurposes:

CheckingnumberofargumentstobepassedontoXXX()

Verifying argument types with a declaration when calling the mainfunction

Allocatingmemorytothemainfunctionwheneverrequired

Result'smaximumlengthverification

Settingadecimalnumberlimitformaximumintheresult

SpecifyingwhethertheresultcanbeNULLornot

xxx_deinit(): Represents deinitialization for the main function anddeallocatesmemoryifanyisallocatedbytheinitializationfunctionforthemainfunction

AggregateUDFsarehandledasinthefollowingsequenceinMySQL8:

1. Call xxx_init() so that it allocates the required memory to store resultinformation.

2. Sortthetable/resultasspecifiedbytheGROUPBYfunction.

3. Callxxx_clear()sothatitresetthecurrentaggregatevalueforthefirstrowineachnewgroup.

4. Callxxx_add()thataddstheargumenttothecurrentaggregatevalue.

5. Callxxx() toget the resultofaggregatedataongroupbychangesorafterprocessingthelastrow.

6. Repeatsteps3-5untilallspecified/resultedrowsareprocessed.

7. Callxxx_deinit()tofreeanyallocatedmemoryfortheUDF.

All the functionsmust be thread-safe, including themain function as well asother additional functions as required, along with the initialization anddeinitializationfunctions.

Similartotheabovesequence,thefollowingareimportantaspectsthatneedtobetakencareofwhileaddingnewuser-definedfunctions:

UDFargumentprocessing

UDFreturnvaluesanderrorhandling

UDFcompilingandinstalling

UDFsecurityprecautions

<strong>doubleItem_func_newname::val()</strong><br/><strong><span>longlongItem_func_newname::val_int()<br/></span>String*Item_func_newname::Str(String*str)</strong>

Ifyourobjectisinheritedfromanyofthestandarditemsthenyouprobablyneedtodefineonlyoneoftheprecedingfunctions,astheparentobjectwilltakecareotherofthefunction.YoucanrefertotheItem_str_funcclassthathasdefinedtheval()functionthatexecutestheatof()functiononthereturnedvalueofthe::str()function.

5. Ifthefunctionisnondeterministic—thatis,ifthereturnedresultvariesatdifferentinvocationsforfixedgivenarguments-thenthefollowingstatementneedstobeincludedintheitemconstructor,indicatingthatthefunctionresultsshouldnotbecached:current_thd->lex->safe_to_cache_query=0;.

6. Youprobablyalsoneedtodefinethefollowingobjectfunctionforyournativefunction:

voidItem_func_newname::fix_length_and_dec()

Thefunctionshouldatleastincludethemax_lengthcalculationonthegivenargumentsYoushouldalsosetmaybenull=0ifyourmainfunctioncannotreturnanyNULLvaluesYoucanrefertoItem_func_mod::fix_length_and_decforthesame

Threadsafetyisamustforallfunctions.Youshouldnotbeusinganystaticorglobalvariablesinthefunctionswithoutbeingprotectedbymutexes.

Debuggingandporting

PortingMySQL 8 to other operating systems is currently supported bymanyoperatingsystems.Thelistofthelatestsupportedoperatingsystemsisprovidedathttp://www.mysql.com/support/supportedplatforms/database.html. If you have added orattemptedtoaddnewports(supportedplatforms)andarerunningintoproblems,youmightusedebuggingofMySQL8tofindandfixtheproblems.

First,youshouldgetthetestprogrammysys/thr_locktoworkbeforedebuggingmysqld.Thismakessurethatyourthreadinstallationcanhavearemotechancetowork!

Therearedifferentpossibilitiesforstartingdebugging,basedonwhereyouarerunning into theproblems - it couldbe inMySQLserveror inMySQLclient.Dependingontheproblem'slocationyoucanstartdebugginginMySQLserverorMySQLclient respectively,and for tracing theprogram'sactivitiesyouwillgethelpfromtheDEBUGpackage.

TheMySQL source code includes internal documentationwrittenusingDoxygen,whichisveryhelpfulinunderstandingthedeveloperperspectiveonhowMySQLworks.

Inthissection,youwillseedetailedinformationonthefollowingtopics:

DebuggingMySQLserver

DebuggingMySQLclient

TheDBUGpackage

DebuggingMySQLserver

If you are using some of very new functionality inMySQL and facing someissues—let's say the server is crashing—youcan try running mysqldwith the --skip-new option. This option tells the MySQL server to disable all new andpotentiallyunsafefunctionality.

In caseswhere mysqld is not getting started, verify the my.cnf files, as they caninterferewiththesetup!Youcanchecktheargumentsinmy.cnfwiththemysqld--print-defaults option and then start mysqld with the --no-defaults option to avoidusingthem.

Incaseswheremysqldstarts toeatupmemoryorCPUorhangs,youcancheckmysqladminprocessliststatusandfindoutifaqueryexecutedbysomeoneistakingalongtime.Incaseswhereyouarefacingperformanceissuesorproblemsandnewclientsarenotabletoconnect,youcanusemysqladmin-i10processliststatus.

Youcanalsousethedebugcommandmysqladmin,whichdumpsinformationaboutquery usage,memory usage, and locks in use to theMySQL log file and cansolve someproblems foryou.Thiscommandalsoworks incaseyouhavenotcompiledMySQLfordebugging,providingsomeusefulinformation.

Incaseswhereyouarefacinganyissuewiththetablegettingslower,youshould

try to optimize the table using myisamchk or OPTIMIZE_TABLE. You should probablychecktheslowqueries,ifthereareany,usingEXPLAINtofindandfixtheproblemwithqueries.

ThefollowingaretheimportantareastoconsiderwhendebugginginMySQL8:

CompilingMySQLfordebugging:Incaseofveryspecificproblemsyoucanalways try todebugMySQL.Todo thatyoumustconfigureMySQLwith the -DWITH_DEBUG=1 option. The debugging configuration automaticallyenableslotsofextrasafetycheckfunctionsthatmonitorthehealthofmysqld.

Creating trace files: You can attempt to find the problem by creating atrace file. To do that you must have mysqld compiled with debuggingsupport.Youcan thenuse the --debug option,whichwill add trace logs in/tmp/mysqld.traceonUnixand\mysqld.traceonWindows.

Using WER with PDB to create a Windows crashdump: ProgramdatabasefilesareincludedintheZIParchivedebugbinariesandtestsuiteas a separate distribution ofMySQL. These files provide information ondebugging for a MySQL installation problem. They can be used withWinDbgorVisualStudiotodebugmysqld.

Debugging mysqld under gdb: You can use this option when you arefacingissueswiththreadsorwhenthemysqldserverhangspriortoreadyforconnections.

Using a stack trace: You can also use this option when mysqld diesunexpectedlyandfindouttheproblem.

Using server logs to find causes of errors in mysqld: You can use thisoptionbyenablingthegeneralquerylog-prior to that,youshouldcheckallyourtablesusingthemyisamchkutilityandverifyifthereareanyproblemsfromthelogs.

Making a test case if you experience table corruption: This option isusedwhenyouare facingan issuewith tablecorruptionand isapplicableonlytoMyISAMtables.

DebuggingMySQLclientIn cases where you are facing an issue inMySQL client you can also debugwithinMySQLclientaswell,butinordertodosoyoumusthavetheintegrateddebug package. You need to configure MySQL with -DWITH_DEBUG=1 to enabledebugginginMySQLclient.

Prior to running MySQL client, you should set the environment variableMYSQL_DEBUGasfollows:shell>MYSQL_DEBUG=d:t:O,/tmp/client.traceshell>exportMYSQL_DEBUG

ThismakesMySQL client generate a trace file in /tmp/client.trace forUnix or\client.traceforWindows.

Incaseswhereyouhaveproblemswithyourownclientcode,youcanattempttoconnect to the server by running your query using the client that is known towork.Fordoingthisyoushouldrunmysqldindebuggingmode:shell>mysql --debug=d:t:O,/tmp/client.trace

This tracewillprovideuseful information ifyouwant tomailabugreport fortheproblem.

Incaseswhereyourclientcrashesatsomelegallookingcode,youcancheckthatyourmysql.hheaderfileincludesfilematcheswithyourMySQLlibraryfile.Thisis one of the very common mistakes, using an older mysql.h file from an oldMySQLinstallationwithanewMySQLlibrary,resultinginthisissue.

<strong><spanclass="tokenoperator">--</span>debug<spanclass="tokenoperator">=</span>d<spanclass="tokenoperator">:</span>t<br/><spanclass="tokenoperator">--</span>debug<spanclass="tokenoperator">=</span>d<spanclass="tokenoperator">:</span>f<spanclass="tokenpunctuation">,</span>main<spanclass="tokenpunctuation">,</span>subr1<spanclass="tokenoperator">:</span>F<spanclass="tokenoperator">:</span>L<spanclass="tokenoperator">:</span>t<spanclass="tokenpunctuation">,</span>20<br/><spanclass="tokenoperator">--</span>debug<spanclass="tokenoperator">=</span>d<spanclass="tokenpunctuation">,</span>input<spanclass="tokenpunctuation">,</span>output<spanclass="tokenpunctuation">,</span>files<spanclass="tokenoperator">:</span>n<br/><spanclass="tokenoperator">--</span>debug<spanclass="tokenoperator">=</span>d<spanclass="tokenoperator">:</span>t<spanclass="tokenoperator">:</span>i<spanclass="tokenoperator">:</span>O<spanclass="tokenpunctuation">,</span>\\mysqld<spanclass="tokenpunctuation">.</span>trace</strong>

Summary

Inthischapter,youlearnedhowtoextendMySQL8throughcustomfunctionsandAPIs.

Youalsogottoknowaboutwritingfunctionsandtheassociatedcharacteristicsof the plugin services and APIs. You can now create your own function orplugin,catertospecificbusinessrequirements,andalsodebugifafunctiondoesnotworkasperexpectations,andtestwhetheritdoes.

Innextchapter,youwilllearnaboutMySQL8bestpracticesandbenchmarkinginMySQL

8.Youwilllearnaboutbenchmarkingandtoolsusedforbenchmarking.YouwillalsolearnbestpracticesforsomeofveryimportantfeaturesofMySQL8,suchasmemcached,replication,datapartitioning,andindexing.

MySQL 8 Best Practices andBenchmarking

Inthepreviouschapter,youlearnedhowtoextendMySQL8.Itcoveredalotofinterestingaspects,suchasextendingpluginsandcallingthembyusingservicesinMySQL8,addinganddebuggingnewfunctionstoMySQL8,andsoon.Inthischapter,wewillgothroughthebestpracticesofMySQL8,whichisamuch-awaited version that promises to address many of the shortfalls of the priorversionsandhasexcitingnewfeatures.

MySQL8promisesnot tobejustastandalonedatabase,but itwillalsoplayasignificantroleinvariousareas,includingbigdatasolutions.Wewilllearnhowbest practices can be implemented for optimal use of features in MySQL 8.Benchmarkingwillenhanceourunderstandingfurther.

Wewillcoverthefollowingtopicsinthischapter:

MySQLbenchmarkingandtools

Bestpracticesforthememcached

Bestpracticesforreplication

Bestpracticesfordatapartitioning

Bestpracticesforqueriesandindexing

Due to prominent optimizations and changes,MySQL 8 advanced its versiondirectlyfromthereleaseofMySQL5.7.MySQL8willnothavethelimitationoffiles,whichwaspreviously restricting thenumberof databases that you couldhave.Therearemanymoreexcitingfeatures,whichwehavecoveredinChapter1, Introduction toMySQL 8. MySQL 8 can now store millions of tables in adatabase.Itwillalsomakemodificationstotablesswiftly.

I am excited to go through this chapter, asMySQL 8 best practices not onlyimpactyourdatabaseperformance,scalability,security,andavailability,butwillalso,on thewhole,exposehowyoursystemperformsfor theenduser.This isourendgoal, isn't it?Let's lookatsomebenchmarksthathavebeenderivedinour test lab, which will raise your eyebrows for sure:

MySQLbenchmarkingandtools

WehavegonethroughvariousnewfeaturesandimprovementsinMySQL8.Itmakesusmoreexcited,asperformanceisalwayswhatwecrave.WithMySQL8notbeinggenerallyavailableyet,Oraclehasn'tpublisheditsbenchmarkresults.Wedidn'twaitforittodosoandcarriedoutourownanalysisinafewareas.

Configuration best practices ofMySQL is the cherry on the cake;without thecherry,thecakeseemsincomplete.Inadditiontoconfigurations,benchmarkinghelps us validate and find bottlenecks and address them. Let's look at a fewspecific areas thatwill help us understand the best practices for configurationandperformancebenchmarking.

ResourceutilizationIOactivity,CPU,andmemoryusageissomethingthatyoushouldnotmissout.These metrics help us know how the system is performing while doingbenchmarking and at the time of scaling. It also helps us derive impacts pertransaction.

Stretching your benchmarkingtimelines

We may often like to have a quick glance at performance metrics; however,ensuringthatMySQLbehavesinthesamewayforalongerdurationoftestingisalsoakeyelement.Thereissomebasicstuffthatmightimpactonperformancewhen you stretch your benchmark timelines, such as memory fragmentation,degradation of IO, impact after data accumulation, cachemanagement, and soon.

Wedon'twantourdatabasetogetrestartedjusttocleanupjunkitems,correct?

Therefore, it issuggested to runbenchmarkingfora longdurationforstabilityandperformancevalidation.

Replicatingproductionsettings

Let's benchmark in a production-replicated environment. Wait! Let's disabledatabase replication in a replica environment until we are done withbenchmarking.Gotcha!

Wehavegotsomegoodnumbers!

Itoftenhappensthatwedon'tsimulateeverythingcompletelythatwearegoingto configure in theproduction environment. It couldprove tobe costly, aswemightunintentionallybebenchmarkingsomethinginanenvironmentthatmighthave an adverse impactwhen it's in production.Replicate production settings,data, workload, and so on in your replicated environment while you dobenchmarking.

Consistency of throughput andlatencyThroughput and latency go hand in hand. It is important to keep your eyesprimarily focused on throughput; however, latency over time might besomethingtolookoutfor.

Performancedips,slowness,orstallswerenoticedinInnoDBinitsearlierdays.Ithas improveda lot since then,but as theremightbeothercasesdependingonyour workload, it is always good to keep an eye on throughput along withlatency.

SysbenchcandomoreSysbench is a wonderful tool to simulate your workloads, whether it bethousands of tables, transaction intensive, data in-memory, and so on. It is asplendidtooltosimulateandgivesyounicerepresentation.

VirtualizationworldIwould like tokeep this simple;baremetalascompared tovirtualization isn'tthesame.Hence,whiledoingbenchmarking,measureyourresourcesaccordingtoyourenvironment.Youmightbe surprised to see thedifference in results ifyoucompareboth.

ConcurrencyBig data is seated on heavy data workload; high concurrency is important.MySQL 8 is extending itsmaximumCPU core support in every new release,optimizing concurrency based on your requirements and hardware resourcesshouldbetakencareof.

HiddenworkloadsDonotmissoutfactorsthatruninthebackground,suchasreportingforbigdataanalytics,backups,andon-the-flyoperationswhileyouarebenchmarking.Theimpactofsuchhiddenworkloadsorobsoletebenchmarkingworkloadscanmakeyourdays(andnights)miserable.

NervesofyourqueryOops!Didwemisstheoptimizer?Notyet.Anoptimizerisapowerfultoolthatwillreadthenervesofyourqueryandproviderecommendations.It'satoolthatIusebeforemakingchangestoaqueryinproduction.It'sasaviorwhenyouhavecomplexqueriestobeoptimized.

These are a few areas that we should look out for. Let's now look at a fewbenchmarks that we did on MySQL 8 and compare them with the ones onMySQL5.7.

BenchmarksTo start with, let's fetch all the column names from all the InnoDB tables. Thefollowingisthequerythatweexecuted:

SELECTt.table_schema,t.table_name,c.column_name

FROMinformation_schema.tablest,

information_schema.columnsc

WHEREt.table_schema=c.table_schema

ANDt.table_name=c.table_name

ANDt.engine='InnoDB';

The following figure showshowMySQL8performeda thousand times fasterwhenhavingfourinstances:

Followingthis,wealsoperformedabenchmarktofindstatictablemetadata.Thefollowingisthequerythatweexecuted:

SELECTTABLE_SCHEMA,TABLE_NAME,TABLE_TYPE,ENGINE,ROW_FORMAT

FROMINFORMATION_SCHEMA.TABLES

WHERETABLE_SCHEMALIKE'chintan%';

The following figure shows howMySQL 8 performed around 30 times fasterthanMySQL5.7:

Itmadeuseager togointoabitmoredetail.So,wethoughtofdoingonelasttesttofinddynamictablemetadata.

Thefollowingisthequerythatweexecuted:

SELECTTABLE_ROWS

FROMINFORMATION_SCHEMA.TABLES

WHERETABLE_SCHEMALIKE'chintan%';

The following figure shows howMySQL 8 performed around 30 times fasterthanMySQL5.7:

MySQL8.0bringsenormousperformanceimprovementstothetable.Scalingtoone million tables, which is a need for many big data requirements, is nowachievable.WelookforwardtomanymorebenchmarksbeingofficiallyreleasedonceMySQL8isavailableforgeneralpurposes.

Let'snowlookatournexttopic,whichwillmakeyourlifeeasier.It'sallabouttakingthingsintoconsiderationforbestpracticesofmemcached.

BestpracticesformemcachedMultiple get operations are now possible with the InnoDB memcached plugin,whichwill really help in improving the read performance.Now,multiple keyvalue pairs can be fetched in a single memcached query. Frequentcommunicationtraffichasalsobeenminimized,aswecangetmultipledatainasingleshot.

Thekeytakeawaysthatyoushouldconsiderformemcachedconfigurationbestpracticesarewhatwewillbegoingthroughnow.

ResourceallocationMemory allocation for memcached shouldn't be allocated over the availablephysicalmemoryorwithoutconsideringotherresourcesthatwouldbeutilizingmemory. If we over-allocatememory, there is a high chance thatmemcachedwould havememory allocated from the swap space. Thismay lead to delayswhileinsertingorfetchingvaluesbecausetheswapspaceisstoredonthedisk,whichisslowerthanin-memory.

OperatingsystemarchitectureAs theoperating systemarchitecturehas32-bits,oneneeds tobecautious.Aswe know, there are limitations to provision resources in a 32-bit operatingsystemarchitecture.

Similarly, memcached with 4 GB RAM with a 32-bit operating systemarchitectureshouldn'tbesetmorethan3.5GBRAM,asitcanbehavestrangelyinperformanceandcanalsoresultincrashes.

Defaultconfigurations

Somekey default configuration parameters should always be fine-tuned basedonyourneeds:

Memory allocation: By default, this is 64 MB; instead it should bereconfiguredbasedonyourrequirementsandtesting

Connections: By default, this is 1,024 concurrent connections; instead itshouldbereconfiguredbasedonyourrequirementsandtesting

Port:Bydefault,thislistensonport11211;insteaditshouldlistentoanotherportforsecuritypurposes

Network interface:Bydefault, thisacceptsconnectionsfromallnetworkinterfaces;insteaditshouldbelimitedforsecuritypurposes

MaxobjectsizeYoushouldlookatconfiguringthemaximumobjectsize,whichbydefaultis1MB.

However, it canbebumpedup to128MB. It ispurelybasedonwhat typeofdatayouaregoingtostoreand,accordingly,itsmaximumobjectsizeshouldbeallowed. Allowing overhead data to be stored in memcached can have anadverse impact, as theremaybemuchmoredata to retrieve,which can causefailures.

Backlogqueuelimit

Thebacklogqueue limit isallabout thenumberofconnections tomemcachedthat should be kept in queue if it reaches the limit of allowed connections.Ideally,yournumberofconnectionsallowedshouldbeconfiguredinawaythatshouldsuffice formostofyourneeds.Thebacklogqueue limitcanbehelpfulwhenthereisanunexpectedpeakloadonmemcached.Ideally,itshouldnotgobeyond20%ofthetotalconnectionsoritcouldimpacttheexperienceofsystemfetchinginformationfrommemcachedbecauseofheavydelays.

LargepagessupportOnsystemsthatsupportlargememorypages,youshouldenablememcachedtoleverage them. Large pages support helps allocate a large data chunk to storedataandalsoreducesthenumberofcachesmissedcallsusingthis.

SensitivedataStoring sensitive data in memcached could be a security threat, as somebodywith access to memcached could view the sensitive information. You shouldobviously take precautions to limit the exposure ofmemcached.You can alsohavesensitiveinformationencryptedbeforestoringitonmemcached.

RestrictingexposureMemcacheddoesn'thavemanysecurityfeaturesbuiltin.Onemeasureinvolvesexposingmemcachedaccesswithintherequiredboundaries.Ifyourapplicationserver needs to talk tomemcached, it only allowsmemcached to be accessedfrom that server with the help of system firewall rules, such as IP Tables orsimilartechniques.

FailoverMemcacheddoesn'thavegoodfailovertechniques.Itissuggestedthatyouhaveyour application configured in a way to failover to an unavailable node andregeneratedataintoanotherinstance.Itisgoodtohaveatleasttwomemcachedconfiguredtoavoidfailureowingtotheunavailabilityoftheinstance.

NamespacesYou can leverage namespaces provided by memcached, which basically addsprefixestothedatabeforestoringit inmemcached.Itcanhelpwhenyouhavemultiple applications talking to memcached. This is helpful and, using somebasicprinciplesofnamingconventions,youcanderiveasolution.Ifthereisdatathatisstoringfirstnamesandlastnames,youcanuseprefixes,suchasFNandLN,respectively.Thiswouldhelpyoueasilyidentifyandretrievedatafromtheapplication.

Cachingmechanism

One of the easiestways to start leveraging caching inmemcached is to use atwo-columntable;youcanleveragenamespacesprovidedbymemcached,whichbasicallyaddsprefixes.

Thefirstcolumnswouldbeaprimarykey,anddatabaseschemashouldbe theaddressrequirementofauniqueidentifierwiththehelpofprimarykeymappingalongwithuniqueconstraints.Incaseyouwanttohaveasingleitemvaluebycombining multiple column values, you should make sure you chooseappropriatedatatypes.

QuerieswithasingleWHEREclausecanbemappedeasilyintomemcachedlookupswhileusing=orINoperatorsinthequeriesthemselves.IncaseswheremultipleWHERE clauses areusedor complexoperations areparsed, such as <,>,LIKE, andBETWEEN,memcachedwouldgetyou throughchallenges. It is suggested thatyouhave such complex operations using traditional SQL queries added to yourdatabase.

ItwouldbebeneficialtocacheentireobjectsinmemcachedinsteadofoptingtocacheindividualrowsfromMySQL8.Forinstance,forabloggingwebsite,youshouldcachetheentireobjectoftheblogportinmemcached.

MemcachedgeneralstatisticsTohelpyouunderstand the statisticsofmemcachedbetter,wewillprovideanoverviewofhealthandperformance.Statisticsreturnedbymemcachedandtheirmeaning are shown in the following table:Terms used to define the value foreachofthestatisticsare:

32u:32-bitunsignedinteger64u:64-bitunsignedinteger32u:32u:Two32-bitunsignedintegersseparatedbyacolonString:Characterstring

Statistic Datatype Description

pid 32u ProcessIDofthememcachedinstance.

uptime 32u Uptime(inseconds)forthismemcachedinstance.

time 32u Currenttime(asepoch).

version string Versionstringofthisinstance.

pointer_size string Sizeofpointers for thishost specified inbits (32or64).

rusage_user 32u:32u Total user time for this instance(seconds:microseconds).

Total system time for this instance

rusage_system 32u:32u (seconds:microseconds).

curr_items 32u Currentnumberofitemsstoredbythisinstance.

total_items 32u Totalnumberofitemsstoredduringthelifeofthisinstance.

bytes 64u Current number of bytes used by this server tostoreitems.

curr_connections 32u Currentnumberofopenconnections.

total_connections 32u Total number of connections opened since theserverstartedrunning.

connection_structures 32u Number of connection structures allocated by theserver.

cmd_get 64u Totalnumberofretrievalrequests(getoperations).

cmd_set 64u Totalnumberofstoragerequests(setoperations).

get_hits 64u Number of keys that have been requested andfoundpresent.

get_misses 64u Numberofitemsthathavebeenrequestedandnotfound.

delete_hits 64u Numberofkeysthathavebeendeletedandfoundpresent.

delete_misses 64u Number of items that have been delete and notfound.

incr_hits 64u Number of keys that have been incremented andfoundpresent.

incr_misses 64u Numberof items thathavebeen incrementedandnotfound.

decr_hits 64u Number of keys that have been decremented andfoundpresent.

decr_misses 64u Numberofitemsthathavebeendecrementedandnotfound.

cas_hits 64u Number of keys that have been compared andswappedandfoundpresent.

cas_misses 64u Number of items that have been compared andswappedandnotfound.

cas_badvalue 64uNumber of keys that have been compared andswapped, but the comparison (original) valuedidnotmatchthesuppliedvalue.

evictions 64u Numberofvaliditemsremovedfromcachetofreememoryfornewitems.

bytes_read 64uTotal number of bytes read by this server fromnetwork.

bytes_written 64u Total number of bytes sent by this server tonetwork.

limit_maxbytes 32u Numberofbytesthisserverispermittedtouseforstorage.

threads 32u Numberofworkerthreadsrequested.

conn_yields 64u Numberofyieldsforconnections(relatedtothe-Roption).

Reference:https://dev.mysql.com/doc/refman/8.0/en/ha-memcached-stats-general.html

These are a few useful items that should be kept handy for best practices ofmemcached. It'snow time forus tomoveaheadand lookatbestpractices forreplication.

BestpracticesforreplicationMySQL8hasmadesomegreatimprovementsonthereplicationside.MySQL8is all about scalability, performance, and securitywith the utmost integrity ofdata,whichisexpectedtobeagame-changerinbigdatatoo.

ThroughputingroupreplicationGroupreplicationbasically takescareofcommitting transactionsoncemostofthemembers in group replication have acknowledged the transaction receivedconcurrently.

This results in a better throughput if the overall number of writes doesn'texceeding the capacity of themembers in group replication. If there is a casewherecapacity isnotplannedappropriately,youwouldnotice lagsonaffectedmembersascomparedtoothermembersinthegroup.

InfrastructuresizingInfrastructure sizing is a common success factor for performance and the bestpractices checklist. If infrastructure sizing is not proper or uneven across thenodes in group replication, it could adversely impact the replicationfundamentalstopology.Eachcomponentshouldbeconsideredwhileconsideringthethroughputrequiredfromthecomponents.

Constantthroughput

To achieve constant throughput is a good success factor. What if you startexperiencing aworkload that starts affecting the rest of themembers ingroupreplication?Itmightbeacasewhereyourmasterkeepsonacceptingadditionalworkload and is lagging behind, after which it might return to an acceptablelevel before burning out all the resources. Additionally, you can implement aqueuingmethodology that can prevent you from burning down resources andonlyallowsyoutopassonworkloadstothemembersthatarepredefinedbasedoncapacity.

While considering a queuing methodology, you mustn't allow queues growexponentially.

This would impact the end user, as there would be a lag in the data beingupdated.

However,youneedstodecidebasedonyourneedsandthebusinessrequirementtoachieveconstantthroughputacrossthesystem.

Contradictoryworkloads

Fundamentally,group replication isdesigned toallowupdates fromanyof themembers in the group. Rollback of transactions based on overlap of rows ischecked for each of the transactions; the rest are committed and sent to beupdated to other members in the group. If several updates on the same rowhappen frequently, it can result inmultiple rollbacks. Youmight come acrosscyclic situations where one server updates, requests others to update, and, inparallel, another has already updated for the same row. This would result inrollback.

Topreventsuchascenario,youcanhavethelastmemberofthegroupapplytheupdate, afterwhichyouproceed to anotherone.Youcanhave similar updatesrouted only from the same node where the earlier one had been executed topreventthechancesofcyclicrollbackconditions.

Writescalability

Distribute your write workload by sharing out write operations, which mightresultinbetterthroughputandbetterscalabilityonwriteperformance.Itwouldbe dependent on contradictory workloads that you would be expecting in thesystem.This ishelpfulwhenyourpeakworkloadisbeingexecutedisonethatcansharetheload.Incommoncases, ifyouhavegoodcapacityplanningdonewithwritescalability,youwouldseetrivialimprovement.

Refer to the following diagram that depicts this:

Youwillnotice thatwith thehelpofmulti-masters todistribute,your loadhasbetterthroughput.Italsoconsidersthegroupsizeinmulti-masterconfiguration.

Best practices for datapartitioning

In general terms, partitioning is logically dividing anything into multiplesubgroups so that each subgroup can be identified independently and can becombinedintoasinglepartition.

Let's now learn different partitioning methods and how partitioning can helpwheretherearelargedatatables.

For any organization, it is very important to store data in such away that thedatabase provides scalability, performance, availability, and security. Forinstance,inahighlyaccessede-commercestore,therearethousands,ormore,ofordersplacedfrequently.

Sotomaintainday-to-dayorderdeliveryshowingadashboardofcurrentorders,whatisrequiredistoqueryatableshowingordersfromthepastfiveyears;theprocesswill take a long time to executewith the currentdata.Here, historicalorderdataisneededfortheanalyticalpurposeoffindinguserbehaviorortrends,butthiswillberequiredtobeperformedonlimiteddatasets.

Therearevariouswaystoachievethebestsuitablesolutionforhighavailability,

scalability,andhighlyperformingarchitecture;thekeyingredientispartitioning.

Inadatabase,dataineachtableisstoredinphysicalfilegroups.Sobydividingthisdatatablefromasinglefilegrouptoamultiplefilegroupcanreducethefilesizeandhelpuscreateascalableandhigh-performingdatabase.

Thefollowingarethekeybenefitsofusingpartitioninginadatabase:

Scalability:Asdatawillbesharedamongmorethanonepartition,serverscanbe configured to usemultiple nodes andpartitions canbe configuredamong multiple nodes. Doing so will eliminate any hardware limits andallow the database to scale up to a large extent to accommodate highvolumedata.

Highperformance:Asdataisstoredamongmultiplepartitions,eachquerywill be executed on a small portion of the data. For example, in an e-commercestorewithanorderhistoryofmorethantwoyears,togetalistoforders placed in the current month will require checking only a singlepartitionandnottheentireorderhistory,thusreducingthequeryexecutiontime.Tofetchthequeryonmorethanonepartition,wecanalsorunthisinparallel,thusreducingtheoveralltimetofetchdatafromthedatabase.

High availability: In partitioning, data is divided across multiple filegroups. Each file group is logically connected but can be accessed andworked on independently. So if one of the file groups or partitions getscorrupted or one of the nodes in the server fails, then we will not loseaccess to the entire table, but only a section of the database will not beavailable, thuseliminating thechancesofsystemfailureandmakingyoursystemhighlyavailable.

Security: Itmaybe that someof thedata in tables requireshigh securitymeasurements to avoid data theft or data leaks. By partitioning, you canprovideadditionalsecurity tooneormorepartitionstoavoidanysecurityissues,thusimprovingdataaccessibilitywithdatasecurity.

In general terms, partitioning is logically dividing anything into multiple

subgroups so that each subgroup can be identified independently and can becombined into a single partition. Let's understand what partitioning means intermsofRDBMS.

Partitioningisgenerallyusedtodividedataintomultiplelogicalfilegroupsforthepurposeofperformance,availability,andmanageability.Whendealingwithbigdata,thenormaltendencyofdataistobeintermsofbillionsofrecords.Soto improve performance of the database, it is better to divide data amongmultiple file groups. These file groups can be on a singlemachine or sharedacrossmultiplemachinesandidentifiedbyakey.Thesefilegroupsareknownaspartitioneddata.

Datainthetablecanbepartitionedintwoways:

Horizontalpartitioning

Verticalpartitioning

HorizontalpartitioningWhenthenumberofrowsinthetableisverylarge,thetablecanbedividedintomultiplepartitions; this isknownashorizontalpartitioning.Whenhorizontalpartitioning is used, each partition of the table contains the same number ofcolumns. It is possible to access all partitions at once, oryou can access eachpartitionindividually.

Verticalpartitioning

In vertical partitioning, the columns of the tables are partitioned to achieveperformanceandbettermanagementofthedatabase.Verticalpartitioningcanbeachievedintwoways.Thefirstoneisbynormalizingtables.Insteadofhavingtoomanycolumnsin the table,columnscanbedividedintomultiple tablesbydividingthedata.Thesecondoneisbycreatingseparatephysicalfilegroupsfordefinedcolumnsinthetable.

VerticalpartitioningiscurrentlynotsupportedinMySQL8.

Let'slookatafewofthebenefitsassociatedwithpartitioning:

Ifatablecontainshistoricaldata,suchaslogsofanapplication,dataolderthansixmonthsdoesnotprovideanysignificancetotheapplicationtobeactive. If partitioning is created based onmonths, you can easily removeoneofthepartitions.

In thesameprecedingcaseof logs, ifwewant to filterdatabetween twodates, theMySQL optimizer can identify the specific partitions, where itcanfindthefilteredrecords,whichcanresultinmuchfasterqueryresults,

asthenumberofrowstocheckisreduceddrastically.

MySQL8alsosupportsqueryingdataonparticularpartitions.Itcanreducethenumberofrecordstocheckwhenyouknowthepartitionthatneedstobequeriedforthedatarequired.

PruningpartitionsinMySQL

Pruningistheselectiveextractionofdata.Aswehavemultiplepartitions,itwillgothrougheachpartitionduringretrieval,whichistimeconsumingandimpactsonperformance.Someofthepartitionswillalsobeincludedinsearchingwhilethe requested data is not available inside that partition, which is an overheadprocess. Pruning helps here to search for only those partitions that have therelevant data, which will avoid the unnecessary inclusion of those partitionsduringretrieval.

Thisoptimization that avoids the scanningofpartitionswhere there canbenomatchingvaluesisknownasthepruningofpartitions.Inpartitionpruning,theoptimizer analyzes FROM and WHERE clauses in SQL statements to eliminateunneededpartitions,andscansthosedatabasepartitionsthatarerelevant to theSQLstatement.

Best practices for queries andindexingIt would be difficult to write the best queries for reference and reuse. It willalwaysvarybasedon thenatureofyourapplication,architecture,design, tablestructure,andsoon.However,precautionscanbe takenwhilewritingMySQLqueriesforbetterperformance,scalability,andintegrity.

Let'sgothroughafewofthebestpracticesthatweshouldkeepinmindwhiledesigningorwritingMySQLqueries.

Datatypes

A database table could consist of multiple columns with data types, such asnumerics or strings. MySQL 8 provides various data types rather than justlimitingtonumericsorstrings:

Small isgood.AsMySQL loadsdata inmemory,a largedata sizewouldhaveanadverseimpactonitsperformance.Smallersetscanaccommodatemoredatainmemoryandreduceoverheadsofresourceutilization.

Fixyour length. Ifyoudon't fix thedata type length, itwouldhave togoandfetch therequired informationeach timeitneeds to.So,wherever it'spossible,youcanlimitthedatalengthbyusingthechardatatype.

NotnullNotnulldataissomethingthatMySQLdoesn'tlikemuch.Notnullcolumnsusemorestorage,impacttheperformance,andrequireadditionalprocessingwithinMySQL.

Optimizingsuchqueries referring tonulldata isdifficultaswell.Whenanulldatacolumnisindexed,itusesadditionalbytesforeachentry.

IndexingIndexingisimportant,asitcanimprovetheperformanceofyourbadlydesignedqueryand table structure or it can even turn awell-designedquery into a badone,whichcanimpactperformancetoo.

SearchfieldsindexGenerally,wedoindexingonfieldsthatareusedasfiltersinMySQLqueries.Itobviously helps reading faster but can adversely impact writes/updates soindexingonlywhatyouneedwouldbeasmartdecision.

DatatypesandjoinsMySQLcandojoinsfordatatypesthataredifferentbuttheperformancecanbeimpacted if MySQL is asked to use different data types for join fields, as itwouldhavetoconvertfromonetoanotherforeachrow.

CompoundindexIfaqueryissupposedtorefertomultiplecolumnsofatable,acompositeindexforsuchcolumnsmightbehelpful.Acompoundindexrefersthecolumnsfromtheresultssetbythefirstcolumn,secondcolumn,andsoon.

Theorderofcolumnsplaysasignificantroleintheperformanceofthequery,sowhiledesigningthetablestructureandindex,youneedtouseiteffectively.

Shorteningupprimarykeys

Small isgoodforprimarykeys too.Shorteningupprimarykeyswouldbenefitanalogously to howwediscusseddatatypes.Becauseof smaller primarykeys,yourindexsizewouldbesmallerandhencetheusageofcachewouldbeless,soitcanaccommodatemoredatainmemory.

It is preferred to use numeric types, as these would be much smaller thancharacters toachieve thegoalofshorteningupprimarykeys. Itcanbehelpfulwhiledoingjoins,asgenerally,primarykeysarereferredforthejoining.

IndexingeverythingIndexing everything is a good idea; however,MySQL won't do this. Do youknow thatMySQLwill do a full table scan if it is supposed to scan an indexhigherthan30%?

Donotindexvaluesthatdon'tneedtobeindexed.

We need to keep inmind that indexing helps—if done correctly—in fetchingdata;however,whilewriting/updatingdata,itisanoverhead.

Fetchingalldataselect *... - Arrghh! Do not use this unless it is really needed. So far, myexperience hasn't needed this. Fetching all datawill slow down the executiontimeandimpactheavilyonresourceutilizationoftheMySQLserver.Youneedtoprovideaspecificcolumnnameorappropriateconditions.

LettingtheapplicationdothejobLet the application alsodo the job forMySQL.Youcan avoidhaving clausessuch as order by by letting applications do the ordering. Doing ordering inMySQL is much slower than in applications. You can identify queries thatshouldbeplannedtobetakencareofbytheapplication.

ExistenceofdataCheckingtheexistenceofdatawiththehelpoftheEXISTSclauseismuchfaster.TheEXISTSclausewillreturntheoutputassoonasitfetchesthefirstrowfromthefetcheddata.

LimitingyourselfLimit yourself to the data that you need to fetch.Always ensure that you useappropriate limits while fetching the data, as unwanted data being fetchedwouldn'tbeusefulandwouldimpactperformance.UsetheLIMITclauseinyourSQLqueries.

AnalyzingslowqueriesThisisagoodpracticetofollow.Wemightmissoutqueriestoeitheroptimizeorrealize having adverse impact as data grows. Youmight have changes in therequirementofthedatatobefetchedwherewemightmissseeingtheimpactofthe queries. It is good to always keep a watch on slow queries that can beconfiguredinMySQLandoptimizethem.

Querycost

What is the cost of your query? Explain is the right answer to this. Use theexplainqueryparameter toknowwhat is impactingyourquery—whether it isafull table scan, index scans, range access, and so on. Use the informationprovided by explain wisely, to optimize the query further. It is a wonderful,quickhandytoolofMySQL.Ifyouknowthatyouhavedoneyourbest,indexingcomesasasaviortooptimizeitfurtherbasedonyourneeds.

Best practices while writing a query start with requirements, designs,implementations, and ongoingmaintenance. It's a complete life cycle that wecan'tdiversify.Understandingschemas,indexes,andanalysesplaysasignificantrole.Whatmatterstousistheresponsetimeandoptimumresourceutilization.

Ipersonallylovetodeepdiveintothismuchmorethanwecanmentionhere—it'saworldofrelations!Yourquerywillmeetaroworcolumnofatableorgetjoinedwith another table.On topof this, if youhaven't done it right, you aretrying to find a relation from a subset that is not required.Howdowe forgetindexesthataresaviorsifusedappropriately?Allthesetogetherwouldshowourrelationsandwouldpromptlyrespondtoarequestedquery.

Summary

Iamsurethatwhilereadingthechapter,youhavekeptinmindthethingstobetakencareoforrecollectingthem,ifthere'sanythingmissinginyourMySQL8implementation.

In these chapter, we discussed best practices for MySQL 8 which would behelpful at various stages, such as implementation, usage, management, andtroubleshootingandwouldactaspointersforbestpracticesofMySQL8;thesemight varybasedondifferent use cases.Proper testing andverificationwouldhelpaffirmthebenefitsofhavingbestpracticesimplemented.

WehavebroadlycoveredsomeexcitingtopicsaboutMySQL8benchmarksanda few configuration parameters along with best practices of memcached. WediscussedMySQL replication best practices, inwhichwewent through a fewcritical pointers. Lastly, MySQL queries and indexing pointers were alsodiscussedwithbestpracticesfordatapartitioning.

Anything written in this chapter would be less, but the pointers provided arenecessary.

By now,we should have a good understanding ofMySQL8; it's now time tosolveproblems.

Let'snowmoveon to thenextchapterand lookathowwecouldcomeacrossmany common issues, identifying error codes alongwith real-world scenariosfortroubleshootingMySQL8.

TroubleshootingMySQL8

In the previous chapter, we learned an important aspect of the MySQL 8database,benchmarking, andbestpractices.Benchmarkinghelps incomparingthecurrentdatabaseperformanceagainsttheexpectedperformancematrices.Welearned what benchmarking is and the tools that can be used to find thebenchmarkperformanceofaMySQL8server.Inalaterpartofthechapter,welearned about the best practices to be followed for memcached, replication,partitioning,andindexing.BestpracticeshelpensuretheoptimumconfigurationoftheMySQL8database.

In this chapter, the focuswill beonunderstanding the commonerrors thatwemayencounterwhileworkingwith theMySQL8database.Theerrorsmaybeservererrorsorclienterrors.Wewilllookatawaytodeterminethattheproblemhasoccurred.

Wewillalsolearntroubleshootingandresolutiontechniquesforerrors.Inalaterpartofthechapter,wewilllookintoreal-worldscenarioswherethesetechniquesareapplicable.Thefollowingisthelistoftopicstobecovered:

MySQL8commonproblems

MySQL8servererrors

MySQL8clienterrors

MySQL8troubleshootingapproach

Real-worldscenario

MySQL8commonproblems

When troubleshooting is an issue, the first thing to be done is to find out theprogramorpieceofequipmentthatiscausingitwhenwerunintoaproblem.

Thefollowingaresymptomsthatindicateahardwareorkernelproblem:

The keyboard is not functioning. It can be checked by pressing theCapsLockkey.IfthelightontheCapsLockkeydoesnotlightup,itisanissuewiththekeyboard.Similarly,themousenotmovingindicatesanissuewiththemouse.

ping is an operating system command to check the accessibility of onemachine from another machine. The machine from which the pingcommand is executed is called the localmachine, whereas the machinepinged is called the remote machine. If the remote machine does notrespond to the local machine's pings, it indicates a hardware or networkrelatedissue.

Itmay indicate an issuewith the operating system kernel program if theprogramsotherthanMySQLarenotworkingcorrectly.

Itmayindicateanissuewiththeoperatingsystemorhardwareifthesystemrestartsunexpectedly. Ina typicalcase,auser-levelprogramshouldneverbeabletotakethesystemdown.

Totroubleshoottheissue,oneormoreofthefollowingcanbedone:

Runadiagnostictooltocheckhardware

Ensuretherelevantlibraryfilesareuptodate

Check for the availability of updates, patches, or service packs for theoperatingsystem

Checkallconnections

ECC memory is an error correcting code memory. It can detect and correct mostcommon internaldatacorruption issues. It isadvisable touseECCmemory inordertodetectthememoryissuesatanearlystage.

Thefollowinginstructionsmayhelpfurtheridentifytheissue:

Examining the system log files may help to discover the reason for theproblem.MySQL

log filesmust also be checked in case there appears to be an issuewithMySQL.

Operating system specific commands can be used to check issues withmemory,filedescriptors,diskspace,orothercriticalresources.

A bug can be identified in the operating system kernel if a problematicrunawayprocessdoesnotdieeventhoughwehaveexecutedacommandtokillit.

Ifthereappearsnottobeaproblemwiththehardware,attemptsshouldbe

made to identify the program that may be causing the problem. Usingoperatingsystemspecificcommands,suchasTaskmanageronWindows,psandtoponLinux,orsimilarprograms,wecanidentifyprogramsthateatupCPUorblocksystemprocesses.

Itispossibletorecovertheaccesstothemachineeventhoughthekeyboardis lockedup.Thiscanbedoneby loggingon to the systemfromanothermachine.Executethekbd_mode-acommanduponsuccessfullogin.

MySQLuserscanreportissuesbyusingoneofthemultiplechannelsprovidedbyMySQL.

After having examined all the possible alternatives, if it can be decided thateither theMySQL server or theMySQLclient causes the problem, a user caneither create a bug report for the mailing list or contact the MySQL supportteam.Thebugreportermustprovidedetailedinformationaboutthebug,systeminformation,andbehaviorandtheexpectedbehavior.ThereportermustdescribethereasonbasedonwhyitseemstobeaMySQLbug.Itisusefultoknowthefollowinginformationiftheprogramfails:

With the help of the top command, check if the program in question hastakenupalltheCPUtime.Insuchcases,weshouldallowaprogramtorunfor a while because it is possible that the program may be executingintensivecomputationalinstructions.

ObservetheresponsefromtheMySQLserverwhenaclientprogramtriesto connect to it. Has it stopped responding? Did the server provide anyoutput?

If it is found that the MySQL server is causing problems in the mysqldprogram,trytoconnectusingthemysqladminprogramtocheckwhethermysqldresponds. The mysqladmin -u root ping or mysqladmin -u root processlist

commandscanbeused.

Didthefailedprogrammakeasegmentationfault?

MostcommonMySQLerrorsThis section provides a list of the most common MySQL errors that usersencounterveryfrequently.

Accessdenied

MySQL provides a privilege system that authenticates the user who connectsfroma host, and associates the userwith access privileges on a database.Theprivileges include SELECT, INSERT, UPDATE, and DELETE and are able to identifyanonymoususersandgrantprivilegesforMySQLspecificfunctions,suchasLOADDATAINFILEandadministrativeoperations.

Theaccessdeniederrormayoccurbecauseofmanycauses.Inmanycases,theproblemiscausedbecauseofMySQLaccounts that theclientprogramsuse toconnectwiththeMySQLserverwithpermissionfromtheserver.

<strong><spanclass="tokencommand">>mysqladmin</span>version<br/><spanclass="tokencommand">>mysqladmin</span>variables<br/><spanclass="tokencommand">>mysqladmin</span><spanclass="tokenproperty">-h</span>`hostname`versionvariables<br/><spanclass="tokencommand">>mysqladmin</span><spanclass="tokenproperty">-h</span>`hostname`<spanclass="tokenconstant">--port</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>3306</span>version<br/><spanclass="tokencommand">>mysqladmin</span><spanclass="tokenproperty">-h</span>host_ipversion<spanclass="tokencommand"><br/>>mysqladmin</span><spanclass="tokenconstant">--protocol</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>SOCKET</span><spanclass="tokenconstant">--socket</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>/tmp/mysql.sock</span>version</strong>

Intheprecedingcommands,hostnameisthehostnameofthemachineonwhichtheMySQLserverisrunning.host_ipistheIPaddressoftheservermachine.

LostconnectiontoMySQLserver

ThelostconnectiontoMySQLservererrorcanoccurbecauseofoneofthethreelikely

causesexplainedinthissection.

One of the potential reasons for the error is that the network connectivity istroublesome.

Networkconditionsshouldbecheckedif this isafrequenterror. If theduringquery message is part of the error message, it is certain that the error hasoccurredbecause

ofnetworkconnectionissues.

The connection_timeout system variable defines the number of seconds that themysqldserverwaitsforaconnectionpacketbeforeconnectiontimeoutresponse.Infrequently,

this error may occur when a client is trying for the initial connection to theserver

andtheconnection_timeoutvalueissettoafewseconds.Inthiscase,theproblemcanberesolvedbyincreasing

theconnection_timeoutvaluebasedon the thedistanceandconnectionspeed.SHOWGLOBAL STATUS LIKE and Aborted_connects can be used to determine if we areexperiencingthismorefrequently.Itcanbecertainly

saidthatincreasingtheconnection_timeoutvalueisthesolutioniftheerrormessagecontainsreadingauthorizationpacket.

It is possible that the problemmay be faced because of largerBinaryLargeOBject(BLOB)valuesthanmax_allowed_packet.Thiscancausea lostconnectionto the MySQL server error with clients. If the ER_NET_PACKET_TOO_LARGE error isobserved,itconfirmsthatthemax_allowed_packetvalueshouldbeincreased.

<strong>>mysql-uuser_name-p</strong><br/><strong>Enterpassword:</strong>

Onafewsystems,itmayhappenthatthepasswordworksfinewhenspecifiedinanoptionfileoronthecommandline.ButitdoesnotworkwhenenteredinteractivelyontheCommandPromptattheEnterpassword:prompt.Itoccursbecausethesystem-providedlibrarytoreadthepasswordslimitsthepasswordvaluestoasmallnumberofcharacters(usuallyeight).ItisanissuewiththesystemlibraryandnotwithMySQL.Asaworkaroundtothis,changetheMySQLpasswordtoavaluethatiseightorfewercharactersorstorethepasswordintheoptionfile.

<strong>Host'host_name'isblockedbecauseofmanyconnectionerrors.</strong><br/><strong>Unblockwith'mysqladminflush-hosts'</strong>

<strong>>mysqld_safe--max_connect_errors=10000</strong>

<strong>mysql>SETGLOBALmax_connect_errors=10000;</strong>

ItshouldbecheckedfirstthatthereisnothingwrongwithTCP/IPconnectionsfromthehostifthehost_nameisblockederrorisreceivedforaparticularhost.Increasingthevalueofthemax_connect_errorsvariabledoesnothelpifthenetworkhasproblems.

Toomanyconnections

This error indicates that all available connection are in use for other clientconnections.

Themax_connectionsisthesystemvariablethatcontrolsthenumberofconnectionstotheserver.The

default value for the maximum number of connections is 151. We can set alargervalue

than151forthemax_connectionssystemvariabletosupportmoreconnectionsthan151.

The mysqld server process actually allows one more than max_connections

(max_connections + 1) value clients to connect. The additional one connection iskeptreservedforaccounts

with CONNECTION_ADMIN or the SUPER privilege. The privilege can be granted to theadministrators with access to the PROCESS privilege. With this access, theadministratorcanconnecttotheserverusingthe

reservedconnection.TheycanexecutetheSHOWPROCESSLISTcommandtodiagnose

theproblemseventhoughthemaximumnumberofclientconnections

isexhausted.

OutofmemoryIfthemysqldoesnothaveenoughmemorytostoretheentirerequestofthequeryissued by the MySQL client program, the server throws the following error:mysql:Outofmemoryatline42,'malloc.c'mysql:needed8136byte(8k),memoryinuse:12481367bytes(12189k)ERROR2008:MySQLclientranoutofmemory

Inorder to fix theproblem,wemust firstcheck if thequery iscorrect.Doweexpectthequerytoreturnsomanyrows?Ifnot,weshouldcorrectthequeryandexecuteitagain.Ifthequeryiscorrectandneedsnocorrection,wecanconnectmysql with the --quick option. Using the --quick option results in themysql_use_result() CAPI function for fetching the result set. The function addsmoreloadontheserverandlessloadontheclient.

<strong><spanclass="tokencommand">>mysql</span><spanclass="tokenconstant">--max_allowed_packet</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>32M</span></strong>

ThedefaultvaluefortheMySQLserveris64MB.Itshouldbenotedthatthereisnoharminsettingalargervalueforthissystemvariable,astheadditionalmemoryisallocatedasneeded.

ThetableisfullThetable-fullerroroccursinoneofthefollowingconditions:

ThediskisfullThetablehasreachedthemaximumsize

TheactualmaximumtablesizeintheMySQLdatabasecanbedeterminedbytheconstraintsimposedbytheoperatingsystemonthefilesizes.

<strong>Can'tcreate<spanclass="tokenoperator">/</span>writetofile'\\sqla3fe_0<spanclass="tokenpunctuation">.</span>ism'<spanclass="tokenpunctuation">.</span></strong>

<strong>>mysqld--tmpdirC:/temp</strong>

<strong>[mysqld]</strong><br/><strong>tmpdir=C:/temp</strong>

CommandsoutofsyncIftheclientfunctionsarecalledinthewrongorder,thecommandsoutofsyncerror is received. Itmeans that the command cannot be executed in the clientcode.Asanexample,ifweexecutemysql_use_result()andtrytoexecuteanotherquery before executing mysql_free_result(), this error may occur. It may alsohappen if we execute two queries that return a result set without calling themysql_use_result()ormysql_store_result()functionsinbetween.

<strong>Foundwrongpasswordforuser'some_user'@'some_host';ignoringuser</strong>

TheaccountisignoredbytheMySQLpermissionsystemasaresult.Tofixtheproblem,weshouldassignanewvalidpasswordfortheaccount.

<strong>Table'tbl_name'doesn'texist</strong><br/><strong>Can'tfindfile:'tbl_name'(errno:2)</strong>

Insomecases,theusermaybereferringtothetableincorrectly.ItispossiblebecausetheMySQLserverusesdirectoriesandfilesforstoringdatabasetables.Dependingupontheoperatingsystemfilemanagement,thedatabaseandtablenamescanbecasesensitive.

Fornoncase-sensitivefilesystems,suchasWindows,thereferencestoaspecifiedtableusedwithinaquerymustusethesamelettercase.

MySQL8servererrorsThissectionfocusesonMySQL8servererrors.Thesectiondescribestheerrorsrelated toMySQLserveradministration, tabledefinitions,andknownissues intheMySQL8server.

<strong>ERROR:Can'tfindfile:'path/with/file_name'(Errcode:13)</strong>

<strong><spanclass="tokenprompt">></span><spanclass="tokencommand">UMASK</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>384</span><spanclass="tokencomment">#=600inoctal</span><br/><spanclass="tokenprompt">></span><spanclass="tokencommand">export</span>UMASK<br/><spanclass="tokenprompt">></span><spanclass="tokencommand">mysqld_safe</span></strong>

<strong><spanclass="tokenprompt">></span><spanclass="tokencommand">UMASK_DIR</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>504</span><spanclass="tokencomment">#=770inoctal</span><br/><spanclass="tokenprompt">></span><spanclass="tokencommand">export</span>UMASK_DIR<br/><spanclass="tokenprompt">></span><spanclass="tokencommand">mysqld_safe</span>&</strong>

<strong><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">USER</span><spanclass="tokenstring">'root'</span>@<spanclass="tokenstring">'localhost'</span><spanclass="tokenkeyword">IDENTIFIED</span><spanclass="tokenkeyword">BY</span><spanclass="tokenstring">'NewPassword'</span><spanclass="tokenpunctuation">;</span></strong>

<strong><spanclass="tokenprompt">C:\></span><spanclass="tokencommand">cd</span><spanclass="tokenatrule">"C:\ProgramFiles\MySQL\MySQLServer8.0\bin"</span><br/><spanclass="tokenprompt">C:\></span><spanclass="tokencommand">mysqld</span><spanclass="tokenconstant">--init-file</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>C:\\mysql-root-reset.txt</span></strong>

<strong>>kill'cat/mysql-data-directory/host_name.pid'</strong>

<strong><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">USER</span><spanclass="tokenstring">'root'</span>@<spanclass="tokenstring">'localhost'</span><spanclass="tokenkeyword">IDENTIFIED</span><spanclass="tokenkeyword">BY</span><spanclass="tokenstring">'NewPassword'</span><spanclass="tokenpunctuation">;</span></strong>

<strong><spanclass="tokenprompt">></span><spanclass="tokencommand">mysqld</span><spanclass="tokenconstant">--init-file</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>/home/me/mysql-reset-root</span>&</strong>

<strong>>mysql</strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">FLUSH</span><spanclass="tokenkeyword">PRIVILEGES</span><spanclass="tokenpunctuation">;</span></strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">USER</span><spanclass="tokenstring">'root'</span>@<spanclass="tokenstring">'localhost'</span><spanclass="tokenkeyword">IDENTIFIED</span><spanclass="tokenkeyword">BY</span><br/><spanclass="tokenstring">'NewPassword'</span><spanclass="tokenpunctuation">;</span></strong>

5. Restarttheserverandloginwiththerootuserandnewlysetpassword.

<strong>C:\ProgramFiles\MySQL\MySQLServer8.0\bin>mysqladminversion-uroot-p</strong><br/><strong>Enterpassword:*****</strong><br/><strong>mysqladminVer8.0.3-rcforWin64onx86_64(MySQLCommunityServer(GPL))</strong><br/><strong>Copyright(c)2000,2017,Oracleand/oritsaffiliates.Allrightsreserved.</strong><br/><br/><strong>OracleisaregisteredtrademarkofOracleCorporationand/orits</strong><br/><strong>affiliates.Othernamesmaybetrademarksoftheirrespective</strong><br/><strong>owners.</strong><br/><br/><strong>Serverversion8.0.3-rc-log</strong><br/><strong>Protocolversion10</strong><br/><strong>ConnectionlocalhostviaTCP/IP</strong><br/><strong>TCPport3306</strong><br/><strong>Uptime:9days4hours4min52sec</strong><br/><br/><strong>Threads:2Questions:4Slowqueries:0Opens:93Flushtables:2Opentables:69Queriespersecondavg:0.000</strong>

Theresolve_stack_dumpisautilityprogramthatresolvesanumericstackdumptosymbols.Toanalyzetherootcauseofwherethemysqldserverprocessdied,wefindinthestacktraceerrorlogs.Itcanberesolvedwiththeresolve_stack_dumpprogram.Itmustbenotedthatitispossiblethatvariablevaluesfoundintheerrorlogsmaynotbeaccurate.

CorrupteddataorindexfilescancausetheMySQLservertocrash.Thesefilesareupdatedonthediskusingthewrite()systemcalledupontheexecutionofeachSQLstatementandbeforetheclientisnotifiedabouttheresult.Itmeansthatthecontentsinthedatafilesaresafeevenintheeventofamysqldcrash.Thewritingoftheunflusheddataonthediskistakencareofbytheoperatingsystem.The--flushoptioncanbeusedwithmysqldtoforceMySQLtoflusheverythingtodiskaftereverySQLstatementexecution.

OneofthefollowingcanbethereasonforMySQLcorruptedtables:

Ifthedatafileorindexfilecrashesthenitcontainscorrupteddata.AbugintheMySQLserverprocesscausedtheservertodieinthemiddleofanupdate.Anexternalprogrammanipulatedthedataandindexfilesatthesametimeasmysqldwithouttablelocking.Inthemiddleofanupdate,theMySQLserverprocesswaskilled.Manymysqldserversarerunningonthesystem.Theserversusethesamedatadirectory.Thesystemdoesnothavegoodfilesystemlocksortheexternallockingisdisabled.Itispossiblethatabugisfoundinthedatastoragecode.WecantrytochangethestorageenginebyusingALTERTABLEontherepairedcopyofthetable.

HandlingMySQLfulldisk

ThissectionfocusesontheresponsefromMySQLtodisk-fullerrorsandquota-exceedederrors.ItismorerelevanttowritesinMyISAMtables.Itcanbeappliedtowritesinbinarylogfilesandanindexfile.Itexcludesthereferencestorowsandrecordsthatshouldbeconsideredanevent.

MySQLperformsthefollowingwhenadisk-fullconditionoccurs:

MySQL ensures that there is enough space available towrite the currentrow.

TheMySQLserverwritesanentryinthelogfileevery10minutes.Itwarnsaboutthedisk-fullcondition.

Thefollowingactionsshouldbetakentoremedytheproblem:

Thediskspaceshouldbefreed tomakesureenoughspace isavailable toinsertallrecords.

Wecanexecute the mysqladminkill command toabort the thread.Thenexttimeitchecksthedisk,thethreadisaborted.

Itmayhappen thata few threadsarewaiting for the table thatcaused thedisk-full situation.Outof several locked threads,killing the thread that iswaitingonthedisk-fullconditionwillenableotherthreadstocontinue.

TheREPAIRTABLEorOPTIMIZETABLEstatementsareexceptionsto theprecedingcondition.OtherexceptionsincludeindexescreatedinabatchaftertheLOADDATA INFILE or ALTER TABLE statements. These SQL statements can createtemporary fileswith largevolumes.Thismaycreatebigproblems for therestofthesystem.

MySQLtemporaryfilesstorageThevalueoftheTMPDIRenvironmentvariableisusedbyMySQLontheUnixasthepathnameof thedirectory to store temporary files.MySQLuses a systemdefault,suchas/tmp,/var/tmp,or/usr/tmpiftheTMPDIRisnotset.

The values of the TMPDIR, TEMP, and TMP environment variables are checked byMySQLonWindows. IfMySQLfindsoneset, ituses thatvalueanddoesnotcheckforremainingvalues.Ifnoneofthesethreevariablesareset,MySQLusesthesystemdefault,whichisC:\windows\temp\.

Ifthetemporaryfiledirectoryinthefilesystemistoosmall,wecanusethemysqld--tmpdir option to specifyadirectoryon the filesystemwithenough space.Forthereplication,onslavemachines,wecanuse--slave-load-tmpdirandspecifythedirectory for holding temporary files during the replication of LOAD DATA INFILEstatements. It is possible to set a list of several paths used in a round-robinfashionwiththe--tmpdiroption.OntheUnixsystem,thepathscanbeseparatedby thecoloncharacter(:),whereasonWindows, thesemicoloncharacter(;)canbeusedtoseparatethepaths.

To effectively distribute the load, multiple temporary directorypathsshouldbelongtodifferentphysicaldisksandnotthedifferentpartitionsofthesamedisk.

For the MySQL server working as a replication slave, we must take care ofsetting the --slave-load-tmpdir option so as not to point to a directory in thememory-basedfilesystemortoadirectorythatiscleareduponserverorserverhostrestarts.ToreplicatethetemporarytablesorLOADDATAINFILEoperations,thereplication slave requires its temporary files on the machine restart. Thereplicationfailsifthefilesinthetemporaryfiledirectoryarelost.

MySQL takes care of removing the temporary files when the mysqld serverprocessisterminated.OnUnix-likeplatforms,itcanbedonebyunlinkingafileafteropeningit.Oneofthemajordisadvantageofthisisthatthenamedoesnotappear in directory listings. It also happens thatwe cannot see a big file thatoccupiesthefilesystem.

ibtmp1isthenameofthetablespacefilethattheInnoDBstorageengineusestostoretemporarytables.ThefileislocatedinthedatadirectoryofMySQL.Ifwewantto specify a different filename and location, the innodb_temp_data_file_pathoptioncanbeusedontheserverstartup.

If the ALGORITHM=COPY technique is used by the ALTER TABLE operation on the InnoDBtable, the storage engine creates a temporary copy of the original table in thesamedirectory.The temporary table filenamesstartwith the#sql- prefix.TheyonlyappearbrieflywhiletheALTERTABLEoperationisbeingperformed.

If the InnoDB table is rebuilt by the ALTER TABLE SQL statement using theALGORITHM=INPLACEmethod,theInnoDBstorageenginecreatesanintermediatecopyofthe original table in the same directory as that of the original table. Theintermediate table filenames start with the #sql-ib prefix. They only appearbrieflywhiletheALTERTABLEoperationisbeingperformed.

The innodb_tmpdir option cannot be applied to intermediate table files. Theseintermediatefilesarealwayscreatedandstoredinthesamedirectoryasthatoftheoriginaltable.

The ALTER TABLE SQL statements that rebuild the InnoDB table with theALGORITHM=INPLACE method create temporary sort files in the default MySQLtemporary directory. The default temporary directory is denoted by $TMPDIR onUnix,%TEMP%onWindows,orthedirectorymentionedbythe--tmpdiroption.tmpdirmayneed tobe reconfigured if the temporarydirectory is not large enough tostoresuchfiles.Asanalternative,wecandefineanothertemporarydirectoryforonlineInnoDBALTERTABLEstatementsusingtheinnodb_tmpdiroption.Theinnodb_tmpdiroptioncanbeconfiguredatruntime,usingtheSETGLOBALorSETSESSIONstatements.

Replicating an innodb_tmpdir configuration should be considered in replicationenvironmentsifalltheservershavethesameoperatingsystemenvironment.Inother cases, an innodb_tmpdir setting replication can result in a failed replicationwhile executing online ALTER TABLE operations. It is recommended to configureinnodb_tmpdir for each server separately if the operating environments aredifferent.

<strong>chmod+t/tmp</strong>

<strong>[mysqld]</strong><br/><strong>socket=/path/to/socket</strong><br/><br/><strong>[client]</strong><br/><strong>socket=/path/to/socket</strong>

<strong>mysqladmin--socket=/path/to/socketversion</strong>

TimezoneproblemsTheMySQLservermustbetoldtheuser'scurrenttimezoneifwehaveproblemwith

SELECTNOW()returningavalueinUTCinsteadoftheuser'scurrenttimezone.Itisalsoapplicable

ifUNIX_TIMESTAMP() returnsawrongvalue. It shouldbedone for theenvironmentrunningtheserver;

forexample,mysqld_safeormysql.server.

Wecanalsoset theserver timezonebyusing the--timezone=timezone_nameoptionwithmysqld_safe. Itcanalsobesetbyassigning thevalue to theTZenvironmentvariablebeforethemysqldisstarted.

Theallowedlistofvaluesfor--timezoneorTZdependsonthesystem.

MySQL8clienterrorsThissection focusesonerrors thatoccuron theMySQL8client.The jobofaMySQL

clientis toconnecttotheMySQLserversoastoexecutetheSQLqueriesandget the results from theMySQL8database.This section lists errors related toexecutionofthequeries.

<strong>col_name<spanclass="tokenkeyword">COLLATE</span>utf8mb4_0900_as_cs<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'a%'</span></strong><br/><strong>col_name<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'a%'</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_0900_as_cs</strong><br/><strong>col_name<spanclass="tokenkeyword">COLLATE</span>utf8mb4_bin<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'a%'</span></strong><br/><strong>col_name<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'a%'</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_bin</strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SET</span><spanclass="tokenkeyword">NAMES</span><spanclass="tokenstring">'utf8mb4'</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SET</span><spanclass="tokenvariable">@s1</span><spanclass="tokenoperator">=</span><spanclass="tokenstring">'MySQL'</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_bin<spanclass="tokenpunctuation">,</span><spanclass="tokenvariable">@s2</span><spanclass="tokenoperator">=</span><spanclass="tokenstring">'mysql'</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_bin<spanclass="tokenpunctuation">;</span><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenvariable">@s1</span><spanclass="tokenoperator">=</span><spanclass="tokenvariable">@s2</span><spanclass="tokenpunctuation">;<br/></span><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="token

punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>@s1=@s2<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>0<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="token

variable">@s1</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_0900_ai_ci<spanclass="tokenoperator">=</span><spanclass="tokenvariable">@s2</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>@s1COLLATE

utf8mb4_0900_ai_ci=@s2<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>1<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="token

punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span></strong>

<strong><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>table_name<spanclass="tokenkeyword">WHERE</span><spanclass="tokendatatype">date_col</span><spanclass="tokenoperator">>=</span><spanclass="tokenstring">'2011-06-02'</span><spanclass="tokenpunctuation">;</span></strong>

WhenaconstantstringiscomparedtoDATE,TIME,DATETIME,orTIMESTAMPusing<,<=,=,>=,>,orBETWEENoperators,MySQLconvertsthestringintoaninternallongintegervalue.MySQLdoesthissoastoachieveafastercomparison.However,thefollowingexceptionsareapplicabletothisconversion:

ComparingtwocolumnsComparingaDATE,TIME,DATETIME,orTIMESTAMPcolumntoanexpressionUseofacomparisonmethodotherthanthosejustlisted,suchasINorSTRCMP()

Thecomparisonisdonebyconvertingtheobjectsintostringvaluesandperformingastringcomparisonincaseoftheseexceptions.

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">INSERT</span><spanclass="tokenkeyword">INTO</span>my_table<spanclass="tokenpunctuation">(</span>phone<spanclass="tokenpunctuation">)</span><spanclass="tokenkeyword">VALUES</span><spanclass="tokenpunctuation">(</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">INSERT</span><spanclass="tokenkeyword">INTO</span>my_table<spanclass="tokenpunctuation">(</span>phone<spanclass="tokenpunctuation">)</span><spanclass="tokenkeyword">VALUES</span><spanclass="tokenpunctuation">(</span><spanclass="tokenstring">''</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">;</span></strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">,</span><spanclass="tokennumber">1</span><spanclass="tokenoperator">+</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">,</span><spanclass="tokenfunction">CONCAT</span><spanclass="tokenpunctuation">(</span><spanclass="tokenstring">'Invisible'</span><spanclass="tokenpunctuation">,</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">;</span></strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>my_table<spanclass="tokenkeyword">WHERE</span>phone

<spanclass="tokenoperator">=</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">;</span></strong>

<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>my_table<spanclass="tokenkeyword">WHERE</span>phone<spanclass="tokenoperator">IS</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>my_table<spanclass="tokenkeyword">WHERE</span>phone<spanclass="tokenoperator">=</span><spanclass="tokenstring">''</span><spanclass="tokenpunctuation">;</span></strong>

MySQL 8 troubleshootingapproach

In this section of the chapter,wewill focus on theMySQL8 troubleshootingapproach.

WhydoweneedtotroubleshootMySQL8?Thereasonsfortroubleshootingareasfollows:

FasterexecutionofSQLqueries

Performanceenhancement

Efficientuseofresources

TheprimarysetofresourcesincludeCPU,diskIO,memory,andnetwork.TherearetwoapproachestomeasureMySQLperformance:

Inaqueryfocusedapproach,itisimportanttomeasurehowfastthequeriesgetexecuted

In resource focused approach, it is important that the queries use fewerresources.

LetustakeadeeperlookatwaystotroubleshootMySQLproblems.

<strong>mysql>SELECT*FROMtbl_nameWHEREprimary_key=1;</strong><br/><strong>mysql>SELECT*FROMtbl_nameWHEREprimary_key_part1=1ANDprimary_key_part2=2;</strong>

ref:Foreachcombinationofrowsfromtheearliertables,allrowswithmatchingindexvaluesarereadfromthereftable.Ifthejoinusesonlytheleftmostprefixofthekey,therefisused.

Real-worldscenario

MySQL query optimization is referred to as improving the time of queryexecution.

Forexample,whenaqueryisnotperformingwellmeansthatthequeryistakinga longer time than expected for execution.The timeof the query execution isimportant but there are other matrices as well that are used to measureperformance.Thissectionexplainswhatshouldbemeasuredandhowitshouldbedoneaspreciselyaspossible.

Thefollowingquestionarises:whyshouldweoptimizethequery?Doesitreallyrequireoptimizationifitonlytakesahundredthofasecond?Yes,itdoesrequireoptimizationunlessthequeryisexecutedrarely.Weshouldoptimizethequeriesthataremostexpensive.

Let'sdiscussareal-timeexample.Inoneoftheapplications,wehadareportthatwas generated based on a complex query andwas taking toomuch time.Theexecution time was in minutes. To optimize such a complex query, weconsideredthefollowingapproach:

1. Analyze the query plan using EXPLAIN: MySQL provides two ways toanalyze theperformanceof a query.One is the EXPLAINmethod,whichwehavealreadylearnedaboutintheprecedingsectionofthischapter.

Another tool is SHOW STATUS. Usually, we should prefer to use EXPLAIN tounderstandthequeryplanofaSELECTquery.Inthecaseofthereportquery,weconvertafewofthenon-SELECTqueriestoSELECTqueries.Thishelpsusinunderstandingthequeryexecutionplanfornon-SELECTqueriesaswell.Forexample,rewritinganUPDATEquerytoSELECTcanbedonebyusingtheWHEREclauseintheUPDATEquery,whichispassedontotheSELECTquery.Wecouldalsofindfewmissingindexesonthetables.

2. SHOW STATUS: The SHOW STATUS statement outputs the internal counters forMySQL. The counters are incremented by MySQL upon every queryexecution.Withthehelpofthesecounters,wecouldunderstandthetypesofoperationsperformedbytheserverinaggregate.Italsohelpsinindicatingtheworkdonebyeachindividualquery.

ThefollowingarethemeasurementsperformedforMySQLservervariables:

Select_: This counter is incrementedwhenever a SELECT query is executed.Thiscountercanalsobeusedtoidentifyifatablescanisperformed.

Key_read:Thisvariableprovidesadditional informationon theusageof thekeyindex.

Last_query_cost:Thisvalue indicateshowexpensive the lastexecutedquerywas.

Thefollowingarethestepstoperformqueryoptimization:

1. Querytheexecutionafewtimestoensureitreturnsthesameresult.

2. ExecuteSHOWSTATUS.Theoutputshouldbesaved.

3. Executethequery.

4. ExecuteSHOWSTATUStoobservethedifferencesfromthepreviousexecution.

5. ExecuteEXPLAINifrequired.

The following parameters should be analyzed for query performanceoptimization:

Tableindex

Sorting

Overallperformance

Rowleveloperations

DiskI/Ooperations

Summary

Inthislastchapterofthebook,welearnedanimportantaspectofanydatabase:troubleshootingerrorsthatwemayencounterusingtheMySQLserverorclient.We started the discussion by understanding what troubleshooting is. Wediscussed different ways for initial diagnostics of the error. We understoodcommonMySQL errors and what the error messages mean.We also learnedaboutthewaystofixtheseerrors.WealsolearnedabouttheMySQLserverandclienterrorsandfixesfortheseerrors.Inthelaterpartofthechapter,welearnedabouttheMySQLtroubleshootingapproachandlookedatareal-worldscenario.Prettyimportantstuffforthelastchapter,huh?

That'sitforthebook.

OtherBooksYouMayEnjoy

Ifyouenjoyedthisbook,youmaybeinterestedintheseotherbooksbyPackt:

MySQL8forBigDataShabbirChallawala, JaydipLakhatariya,ChintanMehta,KandarpPatel ISBN:978-1-78839-718-6

ExplorethefeaturesofMySQL8andhowtheycanbeleveragedtohandleBigData

Unlock the new features of MySQL 8 for managing structured andunstructuredBigData

IntegrateMySQL8andHadoopforefficientdataprocessing

PerformaggregationusingMySQL8foroptimumdatautilization

ExploredifferentkindsofjoinandunioninMySQL8toprocessBigDataefficiently

AccelerateBigDataprocessingwithMemcached

IntegrateMySQLwiththeNoSQLAPI

ImplementreplicationtobuildhighlyavailablesolutionsforBigData

MySQL8CookbookKarthikAppigatla

ISBN:978-1-78839-580-9

InstallandconfigureyourMySQL8instancewithoutanyhassle

Get togripswithnewfeaturesofMySQL8 likeCTE,Windowfunctionsandmanymore

Perform backup tasks, recover data and set up various replicationtopologiesforyourdatabase

MaximizeperformancebyusingnewfeaturesofMySQL8likedescendingindexes,controllingqueryoptimizerandresourcegroups

Learn how to use general table space to suit the SaaS or multi-tenant

applications

Analyze slow queries using performance schema, sys schema and thirdpartytools

Manage and monitor your MySQL instance and implement efficientperformance-tuningtasks

Leaveareview-letotherreadersknowwhatyouthink

Pleaseshareyourthoughtsonthisbookwithothersbyleavingareviewonthesite that you bought it from. If you purchased the book fromAmazon, pleaseleaveusanhonestreviewonthisbook'sAmazonpage.Thisisvitalsothatotherpotential readers can see and use your unbiased opinion to make purchasingdecisions,wecanunderstandwhatourcustomersthinkaboutourproducts,andourauthorscanseeyourfeedbackonthetitlethattheyhaveworkedwithPackttocreate. Itwillonly takeafewminutesofyour time,but isvaluable tootherpotentialcustomers,ourauthors,andPackt.Thankyou!