greenplum dca administration guide - dell emc virtual ip for master servers ... synopsis ......

0

EMC® Greenplum® Data Computing ApplianceAdministration Guide

Appliance Version 1.2.0.

P/N: 300-013-847Rev: A01

EMC Confidential - for EMC use only

Copyright © 2012 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com

All other trademarks used herein are the property of their respective owners.

EMC Greenplum DCA Administration Guide – Contents

EMC Greenplum DCA Administration Guide - ContentsPreface ............................................................................................... 6

About This Guide.............................................................................. 6Document Conventions .................................................................... 6

Text Conventions........................................................................ 6Command Syntax Conventions ................................................... 7

Getting Support ............................................................................... 8Product information .................................................................... 8Technical support ....................................................................... 8

Chapter 1: System information and configurations ............... 9Greenplum DCA configurations......................................................... 9

DCA configuration rules .............................................................. 9System rack components...........................................................11Aggregation Rack Components ..................................................12Expansion Rack Components .....................................................13

Power Supply Reference..................................................................14Network and Cabling Configurations................................................15

Interconnect Switch Reference ..................................................15Administration Switch Reference................................................19Network Hostname and IP Configuration ...................................21Multiple Rack - Aggregation Cabling...........................................24Multiple Rack Cabling Information .............................................25

Configuration Files ..........................................................................27Default Passwords...........................................................................28

Chapter 2: DCA Networking.........................................................29Create a VLAN overlay on the DCA..................................................29

Configure Interconnect switches with custom VLAN...................29Configure custom VLAN on DCA hosts .......................................31

Set custom hostnames on DCA hosts ..............................................32Restrictions................................................................................32Set custom hostnames using DCA Setup interactive mode ........32Set custom hostnames using a DCA Setup config file.................33Unset custom hostnames...........................................................34View hostname mapping............................................................35

Setup Virtual IP for Master servers..................................................36Setup network interface card bonding for Master servers ................37Configure DNS settings ...................................................................38

Chapter 3: DCA Storage ................................................................39Storage overview ............................................................................39

Storage layout ...........................................................................39Common storage commands .....................................................40

Control PERC learn cycle .................................................................41Condition PERC battery immediately..........................................41Schedule conditioning for PERC battery .....................................41

PERC call home behavior.................................................................42Upgrade PERC firmware, server BIOS and OMSA ............................42

Table of Contents 3EMC Confidential - for EMC use only


Chapter 4: Hadoop on the DCA ...................................................44Hadoop overview.............................................................................44

Hadoop modules........................................................................44Hadoop platforms ......................................................................45Common commands ..................................................................46

Hadoop configuration ......................................................................46Configuration files......................................................................46Default ports .............................................................................47

Hadoop examples............................................................................47Wordcount .................................................................................47Create table and insert rows in Hive ..........................................48

Chapter 5: Master Server Failover .............................................49Orchestrated Failover ......................................................................49

What Happens during an Orchestrated Failover .........................50Syntax for the dca_failover Utility..............................................50DCA Failover Configuration ........................................................51Orchestrated Failover Examples.................................................52

Automated Failover .........................................................................52Enable, Disable and Status of Automated Failover .....................52Triggers for Automatic Failover ..................................................53Monitor a Failover in Progress....................................................54Failback After an Automated Failover.........................................54

Chapter 6: Secured Mode .............................................................55Enable Secured Mode on the DCA ...................................................55Revert to Unsecured Mode on the DCA............................................56Verify Secured Mode State ..............................................................56

Chapter 7: SNMP on the DCA.......................................................58DCA MIB information.......................................................................58

MIB Locations ............................................................................58MIB Contents.............................................................................58View MIB ...................................................................................59

Integrate DCA MIB with environment ..............................................60Change the SNMP community string ..........................................60Set a SNMP Trap Sink................................................................61

Chapter 8: DCA Utility Reference ...............................................62dca_setup .......................................................................................62dca_shutdown.................................................................................66

Description ................................................................................67Options......................................................................................67

dca_hadoop ....................................................................................68Synopsis ....................................................................................68Description ................................................................................68

dcacheck.........................................................................................69dca_healthmon_ctl..........................................................................71dcaperfcheck...................................................................................72dca_blinker .....................................................................................73gppkg..............................................................................................74



Chapter 9: Troubleshooting.........................................................77Call home information .....................................................................77

ConnectEMC Error Codes ...........................................................77Verifying SNMP Data Collection..................................................80Starting Stopping and checking Status of ConnectEMC ..............81Server Configuration Parameter for ConnectEMC .......................81

Log Files..........................................................................................82Common Issues ..............................................................................83

Host does not boot ....................................................................83Host iDRAC IP address not correct .............................................83Host interfaces eth2, eth3, eth4 or eth5 do not come up ...........83Greenplum Database errors during initialization in DCA Setup...84Errors connecting to Interconnect switches for download, cluster

map, upload ............................................................................84Slow disk performance with dcaperfcheck..................................84Unexpected return code from dca_hadoop.................................85


EMC Greenplum DCA Administration Guide – Preface

Preface

This guide is intended for EMC field personnel responsible for configuring or managing an EMC Greenplum Data Computing Appliance (DCA). This guide provides information on DCA specific management tools and features. The following sections are included in this preface.

• About This Guide

• Document Conventions

• Getting Support

About This GuideThis guide provides procedural and reference information to help technical field personnel manage and configure an EMC Greenplum Data Computing Appliance. This guide should be used as a supplement to the DCA documentation set, which references the management tools in this guide.

This guide assumes knowledge of Linux/UNIX system administration, network administration, and database administration.

Document ConventionsThe following conventions are used throughout the Greenplum Database documentation to help you identify certain types of information.

• Text Conventions

• Command Syntax Conventions

Text Conventions

Table 0.1 Text Conventions

Text Convention Usage Examples

bold Button, menu, tab, page, and field names in GUI applications

Click Cancel to exit the page without saving your changes.

italics New terms where they are defined

Database objects, such as schema, table, or columns names

The master instance is the postgres process that accepts client connections.

Catalog information for Greenplum Database resides in the pg_catalog schema.

Running H/F 1 6



Command Syntax Conventions

monospace File names and path names

Programs and executables

Command names and syntax

Parameter names

Edit the postgresql.conf file.

Use gpstart to start Greenplum Database.

monospace italics Variable information within file paths and file names

Variable information within command syntax

/home/gpadmin/config_file

COPY tablename FROM 'filename'

monospace bold Used to call attention to a particular part of a command, parameter, or code snippet.

Change the host name, port, and database name in the JDBC connection URL:

jdbc:postgresql://host:5432/mydb

UPPERCASE Environment variables

SQL commands

Keyboard keys

Make sure that the Java /bin directory is in your $PATH.

SELECT * FROM my_table;

Press CTRL+C to escape.

Table 0.1 Text Conventions


Table 0.2 Command Syntax Conventions


{ } Within command syntax, curly braces group related command options. Do not type the curly braces.

FROM { 'filename' | STDIN }

[ ] Within command syntax, square brackets denote optional arguments. Do not type the brackets.

TRUNCATE [ TABLE ] name

... Within command syntax, an ellipsis denotes repetition of a command, variable, or option. Do not type the ellipsis.

DROP TABLE name [, ...]

Document Conventions 7



Getting SupportEMC support, product, and licensing information can be obtained as follows.

Product information

For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Powerlink website (registration required) at:

http://Powerlink.EMC.com

Technical support

For technical support, go to Powerlink and choose Support. On the Support page, you will see several options, including one for making a service request. Note that to open a service request, you must have a valid support agreement. Please contact your EMC sales representative for details about obtaining a valid support agreement or with questions about your account.

| Within command syntax, the pipe symbol denotes an “OR” relationship. Do not type the pipe symbol.

VACUUM [ FULL | FREEZE ]

$ system_command

# root_system_command

=> gpdb_command

=# su_gpdb_command

Denotes a command prompt - do not type the prompt symbol. $ and # denote terminal command prompts. => and =# denote Greenplum Database interactive program command prompts (psql or gpssh, for example).

$ createdb mydatabase

# chown gpadmin -R /datadir

=> SELECT * FROM mytable;

=# SELECT * FROM pg_database;

Table 0.2 Command Syntax Conventions


Getting Support 8




EMC Greenplum DCA Administration Guide – Chapter 1:

1. System information and configurations

This section contains reference information on the DCA hardware and software configurations and settings. The following information is included:

• Greenplum DCA configurations

• Power Supply Reference

• Network and Cabling Configurations

• Configuration Files

• Default Passwords

Greenplum DCA configurationsThis section includes information on the available DCA configurations, and components included. The following information is included:

• DCA configuration rules

• System rack components.

• Aggregation Rack Components

• Expansion Rack Components

DCA configuration rules

There are three basic types of racks that can be shipped from Manufacturing: System - DCA1-SYSRACK, Aggregation - DCA1-AGGREG and Expansion - DCA1-EXPAND. The following rules apply for population of racks:

• A configuration of two racks or smaller will use one System and one Aggregation rack.

• A configuration of one to six racks will use a System and Aggregation for the first two racks and Expansion for each remaining rack.

• A configuration of one to twelve racks will use a System as the first rack, aggregation as the second rack, and expansion for the remaining racks.

• The largest configuration allowed is 12 racks.

• The smallest configuration allowed is 1/4 rack

• Each rack can contain 4, 8, 12 or 16 servers.

• Racks must be fully populated with 16 servers before an additional rack is added.

Greenplum DCA configurations 9



Figure 1.1 1 to 6 Rack Configuration

Figure 1.2 1 to 12 Rack Configuration

Figure 1.3 Brocade VDX-6720 Aggregation Switch Locations




System rack components.

Table 1.1 System Rack Components

Component Quantity

Master Host 2 (one primary and one standby)

Hosts 4-16

Interconnect Switch 2

Administration Switch 1




Aggregation Rack Components

Component Quantity

Master Host 0

Hosts 4-16



Aggregation Switch 2 - 1 to 6 Rack Cluster

1 - 1 to 12 Rack Cluster. Second aggregation switch located in rack 7.

In a 1 to 12 rack configuration, the Brocade 8000 aggregation switches, shown as aggr-sw-1 and aggr-sw-2 are replaced by a single 2U Brocade VDX switch. The Brocade VDX switch is placed in rack 2 and 7.




Expansion Rack Components

Component Quantity

Master Host 0

Hosts 4-16






Power Supply ReferenceFigure 1.4 shows the external power cables connected to their respective PDPs (Power Distribution Panel), while each PDP is connected to an internal PDU (Power Distribution Unit). Viewed from the rear of the DCA, Power Zone A is located on the right hand side, while Power Zone B is located on the left hand side of the system. The figure shows a full system rack.

Figure 1.4 DCA Power Cable Configuration

Power Supply Reference 14



Network and Cabling ConfigurationsThis section describes the network cabling configurations for the administration and interconnect networks. The following information is included:

• Interconnect Switch Reference

• Administration Switch Reference

• Network Hostname and IP Configuration

• Multiple Rack - Aggregation Cabling

Interconnect Switch Reference

The DCA contains two Brocade 8000 switches in each rack for the Greenplum Interconnect. Ports are numbered starting at 0. Ports 0-23 are used for segment server and rack connectivity - not including ports 18 and 19. Ports 18 and 19 are reserved for external connectivity. Ports 0-7 on the far right of the switch are fibre channel and unused by default.

Figure 1.5 Brocade 8000 Portmap

Network and Cabling Configurations 15



Interconnect 1 - Bottom Brocade Switch References

The bottom Brocade Interconnect switch connects servers to the first interconnect. The bottom Brocade switch will always use odd-numbered hostnames, i-sw-1, i-sw-3, i-sw-5, etc.

Figure 1.6 Bottom Interconnect Switch Cabling Reference




Interconnect 2 - Top Brocade Switch References

The top Brocade Interconnect switch connects servers to the second interconnect. The top Brocade switch will always use even-numbered hostnames, i-sw-2, i-sw-4, i-sw-6, etc.

Figure 1.7 Top Interconnect Switch Cabling Reference




Interconnect Switch Port References

Table 1.2 Interconnect Switch Port Reference

Switch Port System RackAggregation and Expansion Racks

Server Port

0 sdw1 Server 1

Interconnect 1 Switches (bottom) connect to RIGHT server port:

Interconnect 2 switches (top) connect to LEFT server port:

1 sdw2 Server 2

2 sdw3 Server 3

3 sdw4 Server 4

4 Server 5 Server 5

5 Server 6 Server 6

6 Server 7 Server 7

7 Server 8 Server 8

8 Server 9 Server 9

9 Server 10 Server 10







16 mdw ---

17 smdw ---

18 Reserved for external Connectivity

Reserved for external Connectivity

19 Reserved for external Connectivity

Reserved for external Connectivity




Administration Switch Reference

The DCA contains one Allied Telesis switch per rack, except for the Aggregation rack which contains two. The Allied Telesis Administration switch is used for management traffic. This switch connects all of the hosts and switches in a DCA, and allows service connectivity through a red service cable..

Figure 1.8

Figure 1.9 Rack 1 Admin Switch Cabling Reference

Allied Telesis AT9924TL Portmap




Table 1.3 Admin Switch Port Reference

Switch Port System RackAggregation and Expansion Racks

server port

1 sdw1 Server 1

Top NIC (GB1):

Left NIC (GB1):

2 sdw2 Server 2

3 sdw3 Server 3

4 sdw4 Server 4

5 Server 5 Server 5

6 Server 6 Server 6

7 Server 7 Server 7

8 Server 8 Server 8

9 Server 9 Server 9








17 smdw ---

18 mdw ---

19 Bottom Brocade Interconnect Switch

Bottom Brocade Interconnect Switch

20 Top Brocade Interconnect Switch

Top Brocade Interconnect Switch

21 aggr-a-sw-1 aggr-a-sw-1




Network Hostname and IP ConfigurationTable 1.4 DCA Network Configuration

Component hostnameiDRAC IPhost-sp

NIC 1 IPhost-cm

IC 1host-1

IC 2host-2

Administration switch in rack 1

a-sw-1 172.28.0.190


a-sw-2 172.28.0.191


a-sw-3 172.28.0.192


a-sw-4 172.28.0.193


a-sw-5 172.28.0.194

Administtation switch in rack 6 a-sw-6 172.28.0.195


a-sw-7 172.28.0.196


a-sw-8 172.28.0.197


a-sw-9 172.28.0.198


a-sw-10 172.28.0.199


a-sw-11 172.28.1.190


a-sw-12 172.28.1.191

Aggregation Administration Switch in rack 2 (first aggregation rack)

aggr-a-sw-1 172.28.1.248

Lower Interconnect switch in rack 1

i-sw-1 172.28.0.170

Upper Interconnect switch in rack 1

i-sw-2 172.28.0.180


i-sw-3 172.28.0.171


i-sw-4 172.28.0.181


i-sw-5 172.28.0.172


i-sw-6 172.28.0.182





i-sw-7 172.28.0.173


i-sw-8 172.28.0.183


i-sw-9 172.28.0.174


i-sw-10 172.28.0.184


i-sw-11 172.28.0.175


i-sw-12 172.28.0.185


i-sw-13 172.28.0.176


i-sw-14 172.28.0.186


i-sw-15 172.28.0.177


i-sw-16 172.28.0.187


i-sw-17 172.28.0.178


i-sw-18 172.28.0.188


i-sw-19 172.28.0.179


i-sw-20 172.28.0.189


i-sw-21 172.28.1.170


i-sw-22 172.28.1.180


i-sw-23 172.28.1.171


i-sw-24 172.28.1.181

Aggregation Interconnect Switch 1 in AGGREG

aggr-sw-1 172.28.0.248

Aggregation Interconnect Switch 2 in AGGREG

aggr-sw-2 172.28.0.249

Primary Master Server mdw 172.28.0.250 172.28.4.250 172.28.8.250 172.28.12.250

Standby Master Server smdw 172.28.0.251 172.28.4.251 172.28.8.251 172.28.12.251


NIC 1 IPhost-cm

IC 1host-1

IC 2host-2




**Hadoop Worker nodes are numbered 1-320. In order to accomodate the required number of hosts, the third IP address octet is incremented by 1 and the fourth octet restarts at 1 when the node number reaches 161. For example, the host hdw160-sp uses a third octet of 2 and a fourth octet of 160 - host hdw-161-sp uses a third octet of 3 and a fourth octet of 1.

To see a complete list of IP addresses and hostnames, view the /etc/hosts file.

GPDB Segment Server sdw# 172.28.0.# 172.28.4.# 172.28.8.# 172.28.12.#

DIA Server etl# 172.28.0.20# 172.28.4.20# 172.28.8.20# 172.28.12.20#

Hadoop Master Node 1-8 hdm1

hdm2

hdm3

hdm4

172.28.1.250

172.28.1.251

172.28.1.252

172.28.1.253

172.28.5.250

172.28.5.251

172.28.5.252

172.28.5.253

172.28.9.250

172.28.9.251

172.28.9.252

172.28.9.253

172.28.13.250

172.28.13.251

172.28.13.252

172.28.13.253

Hadoop Worker Node 1-160 hdw1-160 172.28.2.# 172.28.6.# 172.28.10.# 172.28.14.#

Hadoop Worker Node 161-320

hdw161-320 172.28.3.#

# = node number minus 160. Example: hdw162-sp = 172.28.3.2**

172.28.7.#

# = node number minus 160. Example:

hdw162 -cm= 172.28.7.2**

172.28.11.#

# = node number minus 160. Example: hdw162-1 = 172.28.11.2**

172.28.15.#

# = node number minus 160. Example: hdw162-2 = 172.28.15.2**

Hadoop Hbase Node 1-9 hbw1-9 172.28.0.161 - 172.28.0.169

172.28.4.161 - 172.28.4.169

172.28.8.161 - 172.28.8.169

172.28.12.161 - 172.28.12.169

Hadoop Hbase Node 10 hbw10 172.28.0.200 172.28.4.200 172.28.8.200 172.28.12.200


172.28.4.217 - 172.28.4.229

172.28.8.217 - 172.28.8.229

172.28.12.217 - 172.28.12.229


172.28.5.161 - 172.28.5.169

172.28.9.161 - 172.28.9.169

172.28.13.161 - 172.28.13.169



172.28.5.217 - 172.28.5.229

172.28.9.217 - 172.28.9.229

172.28.13.217 - 172.28.13.229


172.28.6.161 - 172.28.6.169

172.28.10.161 - 172.28.10.169

172.28.14.161 - 172.28.14-169



172.28.6.217 - 172.28.6.229

172.28.10.217 - 172.28.10.229

172.28.14.217 - 172.28.14.229


172.28.7.161 - 172.28.7.169

172.28.11.161 - 172.28.11.169

172.28.15.161 - 172.28.15.169



172.28.7.217 - 172.28.7.229

172.28.11.217 - 172.28.11.229

172.28.15.217 - 172.28.15.229


NIC 1 IPhost-cm

IC 1host-1

IC 2host-2




Multiple Rack - Aggregation Cabling

Servers in a multi-rack configuration communicate through the aggregation switches located in the AGGREG rack. The following diagram and table show the proper ports to cable:

FROM (ports 20-23)TO - Aggregation Switch Ports6 Rack or Smaller Config

TO - Aggregation Switch Ports7 Rack or Larger Config

Rack 1 0 - 3 1-4

Rack 2 8 - 11 6-9

Rack 3 16 - 19 11-14

Rack 4 4 - 7 16-19

Rack 5 12 - 15 21-24

Rack 6 20 - 23 26-29

Rack 7 n/a 31-34

Rack 8 n/a 36-39

Rack 9 n/a 41-44

Rack 10 n/a 46-49

Rack 11 n/a 51-54

Rack 12 n/a 56-59

Table 1.5 Interconnect Switch to Aggregation Switch Cabling

Figure 1.10 1 to 6 Rack Interconnect Switch to Aggregation Switch Cabling




Figure 1.11 7+ Rack Aggregation Switch Cabling

Table 1.6 DCA Multi-Rack Cable Kits

Kit NamePart Number

Quantity

Component Description

DCA1-CBL5 100-652-574 1 BRCD 10GBE TWINAX COPPER 5M 8-PACK

038-003-476 1 25’ CAT6 Ethernet Cable

DCA1-CBL10

100-652-576 2 BRCD 10GBE SFP+ SHORT WAVE 8-PACK

038-003-733 8 10m LC to LC Optical 50 Micron MM Cable Assemblies


DCA1-CBL30

100-652-576 2 BRCD 10GBE SFP+ SHORT WAVE 8-PACK

038-003-740 8 30m LC to LC Optical 50 Micron MM Cable Assemblies


Multiple Rack Cabling Information

Table 1.7 Cabling Kits for a 6 Rack or smaller cluster

FROM AGGREG (Rack 2) TO Cable Kit

Rack 1 - SYSRACK DCA1-CBL5

Rack 2 - AGGREG DCA1-CBL5

Rack 3 - 1st EXPAND DCA1-CBL5

Rack 4 - 2nd EXPAND DCA1-CBL10




Rack 5 - 3rd EXPAND DCA1-CBL10

Rack 6 - 4th EXPAND DCA1-CBL10

Table 1.8 Cabling Kits for a 12 Rack of smaller cluster

FROM AGGREG TO Cable Kit

Rack 2


Rack 2 - AGGREG 1 DCA1-CBL5











Rack 7













Table 1.7 Cabling Kits for a 6 Rack or smaller cluster

FROM AGGREG (Rack 2) TO Cable Kit




Configuration FilesConfiguration files are text files that contain the hostnames of servers for available configurations such as quarter, half or full rack. The file used will be based on the desired function. Refer to the table below for a description of each configuration and host file:

File Description

gpexpand_map Expansion MAP file used during GPDB upgrades to reallocate primary and mirror instances on the new hardware.

gpinitsystem_map MAP file used during installation of GPDB blocks to assign primary and mirror segments to each server.

hostfile Contains one hostname per server for ALL servers in the system. Includes GPDB, DIA and HD (if present)

hostfile_segments Contains the hostnames of the segment servers of all GPDB blocks

hostfile_gpdb_ic1 Contains the hostnames of the subnet 1 interface for GPDB servers.

hostfile_gpdb_ic2 Contains the hostnames of the subnet 2 interface for GPDB servers.

hostfile_dia Contains the hostnames of the DIA servers.

hostfile_dia_ic1 Contains the hostnames of the subnet 1 interface for DIA servers.

hostfile_dia_ic2 Contains the hostnames of the subnet 2 interface for DIA servers.

hostfile_hadoop Contains the hostnames of the Hadoop servers.

hostfile_hdm Contains the hostnames of all Hadoop Master servers.

hostfile_hdw Contains the hostnames of all Hadoop Worker servers.

hostfile_hbw Contains the hostnames of all Hadoop HBase servers.

hostfile_hadoop_ic1 Contains hostnames of the subnet 1 interface for Hadoop servers.

hostfile_hadoop_ic2 Contains hostnames of the subnet 2 interface for Hadoop servers.

hostfile_exkeys Contains the hostnames of all interfaces of all server blocks in the system.

Table 1.9 Hostfiles created by the DCA Setup utility

Configuration Files 27



Default Passwords

Component User Password

Dell R510 and R610 Master Servers

iDRAC root user calvin

root changeme

gpadmin changeme

Brocade 8000 / VDX Interconnect / Aggregation Switch admin changeme

Allied Telesis Administration Switch

manager friend

The following table lists default passwords for all the components in a DCA

Default Passwords 28


EMC Greenplum DCA Administration Guide – Chapter 2: DCA Networking

2. DCA Networking

This chapter contains information related to networking on a DCA. The following topics are included:

• Create a VLAN overlay on the DCA

• Set custom hostnames on DCA hosts

• Setup Virtual IP for Master servers

• Setup network interface card bonding for Master servers

• Configure DNS settings

Create a VLAN overlay on the DCAA VLAN overlay is used to separate network traffic from the DCA internal network. A common use for a VLAN overlay is Hadoop external connectivity.

Figure 2.1 VLAN overlay through Interconnect switch to Hadoop hosts

Creating a VLAN overlay involves two tasks:

• Configure Interconnect switches with custom VLAN

• Configure custom VLAN on DCA hosts

Configure Interconnect switches with custom VLAN

Interconnect switch configurations to use a VLAN overlay are implemented with the DCA Setup utility. The DCA Setup utility generates and uploads switch configurations based on the custom VLAN ID specified. Brocade 8000 and VDX switches are supported.

1. Log in to the Primary Master server as the user gpadmin.

2. Stop the Greenplum Database:

$ gpstop -af

Create a VLAN overlay on the DCA 29



3. Switch to the user root:

su -

4. Launch the DCA Setup utility:

# dca_setup

5. Select option 2: Modify DCA Settings:

6. Select option 15 for Networking: Switches:

7. Select option 3 to Generate and upload switch configurations

8. Select option 1 for All Switches.

9. Press ENTER to confirm the base VLAN ID for Interconnect 1 and 2. These values should not change from 199 and 299.

Generate & Upload Switch Configurations

************************************************************

DCA Switch Configuration

Enter Base Vlan ID for Interconnect 1 [default=199]: ENTER

Enter Base Vlan ID for Interconnect 2 [default=299]: ENTER

10. Press y, then enter the customer VLAN ID for Interconnect 1 and 2. The VLAN IDs 100 is used as an example:

Will there be a customer VLAN configured on Interconnect 1? Yy|Nn (default=y):

> y

Enter Customer Vlan ID for Interconnect 1 [default=100]: 100

Will there be a customer VLAN configured on Interconnect 2? Yy|Nn (default=y):

> n

11. Press ENTER to select the default values for the following questions:

Configure ports 18/19 in 'trunk' Mode for tagged traffic? Yy|Nn (default=y):

> ENTER

Configure ports 18/19 as a port LAG? Yy|Nn (default=y):

> ENTER

Block all Spanning Tree Protocol from leaving ports 18/19? Yy|Nn (default=n):

> ENTER

12. Enter the Interconnect switch and local user password:

Enter interconnect switch password:




Password: changeme

Enter localhost root password:

Password: changeme

13. Press ENTER to accept the default options, then press y to confirm a switch reboot:

Utilize verbose mode for switch communications? Yy|Nn (default=n):

> ENTER

Ignore errors from downloading backup configs? Yy|Nn (default=n):

> ENTER

Reboot switches after uploading new config? Yy|Nn (default=y):

> ENTER

WARNING! Network switches are rebooting. Ensure you are ready for this operation. Type yes to continue? Yy|Nn (default=n):

> y

14. Enter y when prompted. The new VLAN ID will be added to all Interconnect switches in the cluster:

Enter yes to continue or no to exit this application: (Yy|Nn): y

Configure custom VLAN on DCA hosts

1. Log in to the Primary Master server as the user root.

2. Launch the dca_setup utility:

# dca_setup

3. Select option 2: Modify DCA Settings.

4. Select option 14 for Networking: Segments.

5. Selection option 4 to Edit Custom VLAN Configuration.

6. Select option 4 to change Interconnect 1 VLAN IP Addresses.

7. A list of all hosts in the cluster will be displayed. Starting with the first host, enter the corresponding number to modify it’s Interconnect 1 VLAN IP Address.

8. Enter the customer provided IP Address for the host on Interconnect 1 VLAN. For example, hdm1:

Enter IP for host 'hdm1':

9. Select the next host. Repeat this until all desired hosts have been provided IP addresses for the Interconnect 1 VLAN.




10. Once all hosts have been provided Interconnect 1 VLAN IP Addresses, select option 1 to Apply Changes.

11. From the DCA Customer VLAN Configuration menu, select option 5, Interconnect 1 vlan netmask.

12. Enter the subnet mask of the Interconnect 1 VLAN.

13. From the DCA Customer VLAN Configuration menu, select option 6 to set the Interconnect 1 VLAN Gateway.

14. Enter the gateway of the Interconnect 1 VLAN.

15. From the DCA Customer VLAN Configuration menu, select option 1 to Apply Changes.

16. From the Network Configuration: Hosts menu, select option 1 again to Apply Changes.

17. Enter y to continue with updates to the VLAN configuration:


Set custom hostnames on DCA hostsCustom hostnames may be assigned to DCA hosts in conjunction with a VLAN overlay. Follow the steps in this section to assign custom hostnames.

The following sections are included:

• Restrictions

• Set custom hostnames using DCA Setup interactive mode

• Unset custom hostnames

• View hostname mapping

Restrictions

• Customized hostnames are not reflected in the Greenplum Database if they are set after the database has been initialized.

• If hostnames are customized, and the DCA software is reverted to 1.1.0.x, these changes will be lost.

Set custom hostnames using DCA Setup interactive mode

Follow the steps in this section to set custom hostnames using the interactive mode of DCA Setup.



$ gpstop -af

Set custom hostnames on DCA hosts 32




$ su -


# dca_setup



7. Select option 2 for Edit DCA Hostnames.

8. A list of all hosts in the cluster will be displayed. Starting with the first host, enter the corresponding number to add a custom hostname.

9. Enter the custom hostname for the host. For example, hdm1:

Enter new custom hostname [default=hdm1]: hd_master_1

10. Select the next host. Repeat this until all desired hosts have been provided custom hostnames.

11. Once all hosts have been assigned custom hostnames, select option 1 to Apply Changes.


13. Enter y to continue:


14. Start the Greenplum Database as the user gpadmin:

# su - gpadmin

$ gpstart -a

Set custom hostnames using a DCA Setup config file

Follow the steps in this section to configure custom hostnames using a configuration file. This method can quickly update multiple hosts at one time.



$ gpstop -af


$ su -

4. Navigate to the /home/gpadmin directory:

# cd /home/gpadmin




5. Copy the DCA Setup networking example file to the /home/gpadmin directory and rename it as custom_hostnames.cnf:

# cp /opt/dca/var/dca_setup/networking_sample.cnf ./custom_hostnames.cnf

6. Open the custom_hostnames.cnf file for editing:

# vi custom_hostnames.cnf

7. Locate the section starting with text custom_hostname.sdw1. A list of 18 hostnames are provided as examples.

8. Edit each line to reflect the ORIGINAL hostname (example sdw1). Add the custom hostname after the = symbol. If the ORIGINAL hostname for a server you wish to modify does not appear, add it to the list.

For example, to set a custom hostname on host hdm1, add the text in BOLD:

custom_hostname.hdm1 = acme_hd_master1

9. Delete each custom_hostname line that you do not wish to use.

10. Scroll through the rest of the file and delete the remaining lines in each section. For example, if you need to set custom hostnames on servers hdm1 to hdm4, the file should only contain the following text:

[global]

generate_etchostfile = 1





11. Save and close the custom_hostnames.cnf file.

12. Launch the DCA Setup utility with the config file parameter:

# dca_setup -c /home/gpadmin/custom_hostnames.cnf




# su - gpadmin

$ gpstart -a

Unset custom hostnames






$ gpstop -af


$ su -


# dca_setup



7. Select option 2 for Edit DCA Hostnames.

8. Select option 2 to Unset All Hostnames.

9. Select option 1 to Apply Changes.





# su - gpadmin

$ gpstart -a

View hostname mapping

Issue the following command to view mapping of the server type, default hostname and custom hostname:

$ cat /opt/dca/etc/dca_setup/hostmapping.state




Setup Virtual IP for Master serversA virtual ip address (VIP) is used for system availability during master server failure - if the primary master server fails, the same IP address can be used to access the Greenplum Database on the Standby Master server.

Figure 2.2 Master servers configured with VIP 10.10.10.3

1. Launch the DCA Setup utility.

# dca_setup


3. Select option 13: Networking: Masters.

4. Select option 3: Configure Virtual IP on Master.

5. The system will be polled to determine the external network configuration. Enter the virtual IP, gateway, and netmask when prompted:

Enter ip address of failover vip: : 10.10.10.5

Enter ip address of failover gateway: : 10.10.10.10

Enter ip address of failover netmask: : 255.255.255.0

6. Edit the pg_hba.conf file to allow client connections over the VIP. Refer to the Greenplum Database Administrator Guide - Allowing Connections to the Greenplum Database for instructions.

7. Restart the Greenplum Database:

# su - gpadmin

$ gpstop -r

Setup Virtual IP for Master servers 36



Setup network interface card bonding for Master serversNetwork interface card (NIC) bonding is used for network redundancy over the external connections on a Primary or Standby Master server. NIC bonding requires a second physical connection for each Master server.

Figure 2.3 Network bonding between eth1 and eth2 on a Master server

Considerations:

• NIC bonding is configured through the DCA Setup utility.

• The Greenplum Database should be stopped before setting up NIC bonding. The DCA Setup utility will warn if Greenplum Database is detected to be running.

• If you are connected to the DCA through the external connection, you will be disconnected and my not be able to log back in. All changes should be performed locally, through the administration network.

• A new network interface bond0 is created - eth1 and eth2 are set as a SLAVE device with bond0 as the MASTER.

Customer networking requirements:

• (2) 1GB/s cat6 ethernet connections to EACH Master server.

• External IP address.

• Gateway.

• Subnet mask.

Configure NIC bonding on the Primary or Standby Master:

1. Log in to the Primary Master server as the user root.


# dca_setup


4. Select option 13 for Networking: Masters.

5. Select option 1 to modify the Primary Master or option 2 to modify the Standby Master server.

6. Enter y to enable external interface networking bonding.

Setup network interface card bonding for Master servers 37



7. Enter the customer provided IP address, gateway and subnet mask.

8. Review the New Network Configuration Settings. Enter y to continue.

Configure DNS settingsCustomer DNS settings can be applied to the DCA through the DCA Setup utility. These settings will apply globally - across all servers in the DCA.

1. Launch the dca_setup utility as the user root:

# dca_setup

2. Select option 2: Modify DCA Settings:

3. Select option 14: Networking: Segments

4. Select option 5: Edit Name Resolution Settings

5. Select option 3: Nameserver 1. Enter the customer provided domain name server address.

6. Select option 4 or 5 if you wish to specify a second or third domain name server.

7. Select option 6: Resolve search path. Enter the customer provided search path.

8. Select option 7: Resolve domain. Enter the customer provided resolve domain.

9. Select option 1: Apply Changes.

10. Select option 1: Apply Changes.

Configure DNS settings 38


EMC Greenplum DCA Administration Guide – Chapter 3: DCA Storage

3. DCA Storage

This chapter contains information on the DCA storage system, including Dell PowerEdge Raid Controller (PERC) information.

The following sections are included:

• Storage overview

• Control PERC learn cycle

• PERC call home behavior

• Upgrade PERC firmware, server BIOS and OMSA

Storage overviewThe DCA uses direct attached storage to house application and operating environment data. Each server in a DCA contains a Dell PERC with 512MB of cache protected by a lithium-ion battery.

The following information is included in this section:

• Storage layout

• Common storage commands

Storage layout

There are different physical and virtual disk configurations based on the type and role of a server in the DCA:

Table 3.1 Storage layout by server role

Server Role Server Model Drives RAIDVirtual Disks

Disk Mount

Master Servers

Dell R610 6 Raid 5 (4+1) with Hot Spare

3 /dev/sda1 /boot

/dev/sda2 /

/dev/sdb /swap

/dev/sdc /data

GPDB Segment Servers

Dell R510 (Standard and Capacity)

12 2 x Raid 5(5+1) 4 /dev/sda1 /boot

/dev/sda2 /

/dev/sdb /data1

/dev/sdc /swap

/dev/sdd /data2

Storage overview 39


Common storage commands

The following commands can be used to view information about the storage system on a DCA server. All commands should be run as the user root.

View PERC cache policy:# omreport storage vdisk | grep -e ID -e 'Write Policy'

View PERC battery learn cycle state:# omreport storage battery | grep 'State'

View PERC battery charge capacity:# omconfig storage controller action=exportlog controller=0

# cat /var/log/lsi_mmdd.log | grep 'Full Charge Capacity'

View PERC battery manufacturer:# omconfig storage controller action=exportlog controller=0

# cat /var/log/lsi_mmdd.log | grep 'Manufacturer Name'

View disk free space by OS mount:# df -h

DIA Servers Dell R510 (Capacity)

12 2 x Raid 5(5+1) 4 /dev/sda1 /boot

/dev/sda2 /

/dev/sdb /data1

/dev/sdc /swap

/dev/sdd /data2

GP HD Master Servers

Dell R510 (Capacity)

12 Raid 6 (9+2) with Hot Spare

6 /dev/sda1 /boot

/dev/sda2 /

/dev/sdb /swap

/dev/sdc UNUSED

/dev/sdd UNUSED

/dev/sde UNUSED

/dev/sdf /data

GP HD Worker and HBase Servers

Dell R510 (Capacity)

12 12 x Raid 0 16 /dev/sda1 /boot

/dev/sda2 /

/dev/sdb UNUSED

/dev/sdc /data1

/dev/sdd /swap

/dev/sde /UNUSED

/dev/sdf - /dev/sdp /data2 - /data12

Table 3.1 Storage layout by server role

Server Role Server Model Drives RAIDVirtual Disks

Disk Mount

Storage overview 40


Control PERC learn cycleThe PERC learn cycle is a process that occurs on each server every 90 days. During a learn cycle, server performance will suffer due to storage cache being unavailable. The learn cycle cannot be disabled, but can be synchronized across all servers in the DCA in order to ensure a predictable time for execution.

There are two options to control the PERC learn cycle:

• Condition PERC battery immediately

• Schedule conditioning for PERC battery

Condition PERC battery immediately

1. Open an SSH connection to the Primary Master server as the user root.

2. Start the DCA Setup utility:

# dca_setup


4. Select option 19: RAID controller battery.

5. Select option 1 or 2:

a. Select option 1: Condition battery on all hosts to condition the PERC batteries on all servers in the DCA.

b. Select option 2: Condition battery on selected hosts to condition the PERC batteries on selected server(s). Multiple servers can be entered.

6. Enter y to continue.

Schedule conditioning for PERC battery

1. Open an SSH connection to the Primary Master server as the user root.


# dca_setup


4. Select option 19: RAID controller battery.

5. Select option 3: Adjust scheduling for battery conditioning.

6. Select option 1 or 2:

Option 1: Every other Month by Day of Week:

• Select the day to execute the learn cycle.

• Select the time to execute the learn cycle.

• Select the week of the month to execute the learn cycle.

Control PERC learn cycle 41


• Specify if you would like to condition batteries immediately. This should be done to ensure all learn cycles are synchronized.

Option2: Every other Month by Day of Month:

• Select the date to execute the learn cycle.

• Select the time to execute the learn cycle.

• Specify if you would like to condition the batteries immediately. This should be done to ensure all learn cycles are synchronized.


PERC call home behaviorThe PERC call home logic is explained in this section. When the PERC battery is unavailable, due to a degraded state (low charge capacity) or learn cycle, there will be a performance impact from a write-through cache policy being enabled. Based on the state of the PERC battery and re-occurring battery learn cycle, the following will occur:

Learn cycle has NOT been scheduled through DCA Setup

• If the PERC battery is in a degraded state, and the PERC controller is NOT in a learn cycle, a warning call home is generated.

• If the PERC battery is in a degraded state, and the PERC controller is in a learn cycle, no call home is generated.

• If the PERC cache policy changes to write-through mode, and the PERC is in a learn cycle, no call home is generated.

• If the PERC cache policy changes to write-through mode, and the PERC is NOT in a learn cycle, a warning call home will be generated.

Learn cycle has been scheduled through DCA Setup

• If the PERC battery is in a degraded state due to a scheduled learn cycle, no call home is generated.

• If the DCA software initiates a scheduled learn cycle, battery and cache policy call homes are suppressed for 1 hour. After 1 hour, if the battery is in a degraded state, and the learn cycle is NOT running, a warning call home is generated.

Upgrade PERC firmware, server BIOS and OMSAThis procedure contains information on how to upgrade the PERC firmware, server BIOS and OMSA.

1. Open an SSH connection to the Primary Master server and log in as the user root.

2. Start the DCA Setup utility

# dca_setup

3. Select option 2 to Modify DCA Settings

PERC call home behavior 42


4. Select option 18 to Upgrade Bios/Firmware/OMSA

5. The DCA Setup utility will poll each host to determine the software revision of BIOS, Firmware and OMSA. Enter y to continue:

6. Alist of hosts requiring upgrades will be reported. If any host is identified as having down-rev Bios, Firmware or OMSA, the utility will report ALL hosts requiring the upgrade for that item. Only hosts that are down-rev will be upgraded.

Note: If the following warning is displayed, the Primary Master server needs to be upgraded. The Primary Master server can only be upgraded if the DCA Setup utility is run from the Standby Master server.

[WARNING]:-mdw host needs a BIOS upgrade WHICH WILL BE SKIPPED

7. Confirm the upgrade, enter y to continue:


8. The upgrade will begin - the following will occur:

• Based on the number of hosts that require upgrades, time will vary - a 2 module cluster will take about an hour.

• Any host that does not requiring an upgrade will be verified, but not upgraded again or rebooted. For example, if only one host in a 2 module cluster requires an upgrade, the upgrade will take about 15 minutes.

• The upgrade is divided into 4 groups - one host form each module belongs to a group. For example, a 2 module configuration will have sdw1 and sdw5 in group 1, sdw2 and sdw6 in group 2, etc.

• The upgrade will occur in three stages, first BIOS will be upgraded, then Firmware then OMSA. A reboot will occur after each stage.

• The Primary Master server cannot be upgraded if the DCA Setup utility is run from itself - it must be upgraded from the Standby Master server.

Upgrade PERC firmware, server BIOS and OMSA 43

EMC Greenplum DCA Administration Guide – Chapter 4: Hadoop on the DCA

4. Hadoop on the DCA

This chapter contains information on the Hadoop implementation on the DCA.

The following information is included:

• Hadoop overview

• Hadoop configuration

• Hadoop examples

Hadoop overviewThis section contains the following information:

• Hadoop modules

• Hadoop platforms

• Common commands

Hadoop modules

The DCA implementation of Hadoop uses three module types to build a Hadoop cluster: Master, Worker and HBase.

Hadoop Master Module

Master modules are configured to assume the role of namenode, secondary namenode and jobtracker. A Master module is required as the foundation to a Hadoop installation. Up to one module of Hadoop Masters is allowed in a DCA, future provisions allow for a second. Each of the four servers in the Master module servers a role:

Table 4.1 Hadoop Master Module server roles

Hostname Role

hdm1 Namenode

hdm2 Standby Namenode

hdm3 Jobtracker

hdm4 Secondary Namenode

The DCA’s modular architecture allows the Master module to be installed in any location, but the Master module will typically be the first Hadoop module installed.

Hadoop overview 44


Hadoop Worker Module

Worker modules serve as the data storage for a Hadoop installation. A minimum of one Worker module is required to use Hadoop. a maximum of 46 Worker modules is allowed - this will populate a 12 rack DCA with 1 module of Greenplum Database and 1 module of Master modules.

Hadoop Worker modules run the tasktracker and datanode services.

Hadoop HBase Module

Hadoop HBase modules are configured to run Hadoop HBase software. No HBase modules are required for a Hadoop implementation, however, a minimum configuration of 1 Master and 1 Worker module are required to use HBase.

Hadoop platforms

The following platforms are installed with Hadoop modules and configured for use.

Hive

Apache Hive is data warehousing software that can access files stored in the HDFS or HBase storage system. Hive on the DCA has an initial configuration and is ready for use by running the following command from any Hadoop Master module server:

$ hive

Hive binaries, examples and documentation are located in:

/opt/dca/libexec/hive

The Hive configuration is located in:

/opt/dca/etc/hive_conf

Mahout

Apache Mahout is a collection of machine learning libraries. Mahout binaries and examples are located in:

/opt/dca/libexec/mahout

Pig

Apache Pig is a platform for analyzing large data sets. Pig on the DCA has an initial configuration and is ready for use by running the following command from any Hadoop Master module server:

$ pig

Pig binaries, examples and documentation are located in:

/opt/dca/libexec/pig

The Pig configuration is located in:

/opt/dca/etc/pig_conf

ZooKeeper

Apache ZooKeeper is a coordination service for distributed applications in Hadoop. The ZooKeeper binaries are located in:

Hadoop overview 45


/opt/dca/libexec/zookeeper

The ZooKeeper configuration is located in:

/opt/dca/etc/zookeeper_conf

Common commands

The following are common commands for interacting with the Hadoop file system on a DCA. These commands are not specific to the DCA, for a full list, view the Apache Hadoop documentation.

View Hadoop FS commands:$ hadoop dfs -help

List Directories in HDFS:$ hadoop dfs -ls /

Run the Hadoop filesystem check tool:$ hadoop fsck /

Remove files in a directory:$ hadoop dfs -rmr TEST

Copy files from a local directory into HDFS:$ hadoop dfs -copyFromLocal /home/gpadmin/gpAdminLogs/* TEST

Hadoop configurationThis section contains information on the configuration of Hadoop in the DCA.

Configuration files

This follow table lists the location of important configuration files used by Hadoop.

Table 4.2 File location of Hadoop configuration files

Filename Location

core-site.xml /opt/dca/etc/hadoop_conf

on Hadoop Mastershdfs-site.xml

dca_hdm_settings.cnf

mapred-site.xml

Hadoop configuration 46


Default ports

The following table lists ports configured on the DCA for various Hadoop services.

Table 4.3 Hadoop ports on the DCA

Service Port

Hadoop IPC 8020

Datanode 1604

Datanode HTTP 1606

Namenode HTTP 50070

Namenode HTTPS 50470

Secondary Namenode HTTP 50090

Secondary Namenode HTTPS 50490

Hadoop examplesThe following examples can be run on a DCA with Hadoop installed:

• Wordcount

• Create table and insert rows in Hive

Wordcount

The following example uses a built-in function to count the number of words in files present in the specified directory:

1. Log in to hdm1 as the user gpadmin.

1. Create a test directory named TEST:

$ hadoop dfs -mkdir TEST

2. View the TEST directory:

$ hadoop dfs -ls

3. Copy a sample file from the OS file system into the Hadoop filesystem:

$ hadoop dfs -copyFromLocal /opt/dca/etc/hadoop_samples/imagine.txt TEST

4. Run the example wordcount job:

$ hadoop jar /opt/dca/libexec/hadoop/hadoop-examples-1.0.0-gphd-1.1.0.0.jar wordcount TEST OUTPUT

5. View the OUTPUT directory, this contains files created from the Hadoop job:

$ hadoop dfs -ls OUTPUT

Hadoop examples 47


6. View the contents of the file created in the OUTPUT directory:

$ hadoop dfs -cat OUTPUT/part-r-00000

Create table and insert rows in Hive

1. Log in to hdm1 as the user gpadmin.

2. Create a text file /home/gpadmin/test_hive and populate it with integer values, one on each line:

$ vi /home/gpadmin/test_hive

1

2

3

...

3. Start Hive:

$ hive

4. Create a table named test:

hive> create table test (foo int);

5. Show the table:

hive> show tables;

6. Load data from the test_hive file into the table test:

hive> load data local inpath '/home/gpadmin/test_hive' overwrite into table test;

7. View the contents of the table test:

hive> select * from test;

8. Exit the Hive shell:

hive> quit;

9. View the contents of the table test from the HDFS:

$ hadoop fs -cat /user/hive/warehouse/test/test_hive

Hadoop examples 48

EMC Greenplum DCA Administration Guide – Chapter 5: Master Server Failover

5. Master Server Failover

The following chapter describes the Master Server Failover feature of the DCA. In the Greenplum DCA, a backup master server called the Standby is available to resume operations if the Primary master has a failure. A failover to the Standby Master - hostname smdw - is performed when the Primary Master - hostname mdw - can no longer accept connections to the Greeplum Database.

There are two types of failover, orchestrated and automatic. An orchestrated failover is done by invoking the failover manually - this may be due to marginally failing hardware or scheduled maintenance of the Primary Master. Automatic failover occurs when the DCA detects the Primary Master has failed and performs an unattended failover to the Standby Master.

The failover scenarios listed in this chapter only pertain to the case where the Primary Master fails. If a Standby Master fails, there is no impact to availability of the Greenplum Database. For information about the replacement process of a Primary or Standby Master, refer to the Greenplum DCA Field Replaceable Units Field Procedures Guide, available on Powerlink.

The following sections are included in this chapter:

• Orchestrated Failover

• Automated Failover

Figure 5.1 Overview of Master Server Failover Process

Orchestrated FailoverOrchestrated failover refers to the manual involving of a failover from Primary to Standby Master. The Primary Master may still be operational, but does not need to be.

An orchestrated failover can be done for several reasons, including:

• Marginally failing hardware

• Scheduled maintenance

Orchestrated Failover 49


• Control of when failover occurs - user does not want to activate Automated Failover

Orchestrated failover does not require any special setup beforehand. The utility used to invoke an orchestrated failover is an automation of steps that were required to be run manually in previous versions of the DCA software.

The following topics are included in this section:

• What Happens during an Orchestrated Failover

• Syntax for the dca_failover Utility

• DCA Failover Configuration

• Orchestrated Failover Examples

What Happens during an Orchestrated Failover

An orchestrated failover is an aggregation of functionality which moves operations from the Primary to Standby Master. The following operations are performed in order during an orchestrated failover:

1. The Greenplum Database is stopped - if the --stopmasterdb parameter is given.

2. The virtual IP address of the failed Primary Master is deleted - if the --deletevip parameter is given.

3. The gpactivatestandby command is run on the Standby Master, it becomes the new Primary Master.

4. A query is run against the new Primary Master to validate the Greenplum Database is running.

5. The failed master server is shutdown - if the --shutdown parameter is given.

6. Add the virtual IP address to the new Primary Master - if the --shutdown and --vip parameters are given.

7. The new Primary Master sends an ARP request to the gateway with the virtual IP address it was assigned - if the --shutdown and --vip parameters are given.

8. Performance Monitor is started on the new Primary Master - if the --shutdown parameter is given.

Syntax for the dca_failover Utility

The dca_failover utility is used to initiate an orchestrated failover from Primary to Standby Master. The utility parameters are listed below. Parameters highlighted in BOLD are required.

dca_failover --master_data_dir --port --user --vip --gateway --netmask --shutdown --deletevip --noninteractive --stopmasterdb --help



--master_data_dir: Master Data Directory of Greenplum Database. This value is read from the /opt/dca/etc/env/default/master_data_directory file by default. This parameter is not required.

--port: The port which the Greenplum Database is running on. If the parameter is not given, the value is read from the /opt/dca/etc/dca_setup/dca_setup.cnf file. This parameter is not required.

--user: The user to perform Greenplum Database operations - gpactivatestandby, test query with. This is gpadmin by default. This parameter is not required.

--vip: The virtual IP address to move. If the parameter is not given, the value is read from the /opt/dca/etc/dca_setup/dca_setup.cnf file. This parameter is not required.

--gateway: The gateway of the virtual IP address. The --vip parameter must be given in addition to this parameter. If the parameter is not given, the value is read from the /opt/dca/etc/dca_setup/dca_setup.cnf file. This parameter is not required.

--netmask: The netmask of the virtual IP address. The --vip parameter must be given in addition to this parameter. If the parameter is not given, the value is read from the /opt/dca/etc/dca_setup/dca_setup.cnf file. This parameter is not required.

--shutdown: Shutdown the failed Primary Master after the Standby has been initialized as the new Primary Master. This parameter is required if the failed Primary Master is still powered on.

--noninteractive: Don’t prompt user for confirmation during failover.

--stopmasterdb: Stop the Greenplum Database on the failed Primary Master server before failover. This parameter is required if the failed Primary Master is powered on and still accepting connections to the Greenplum Database.

--help: Display syntax of dca_failover command.

DCA Failover Configuration

The DCA Failover utility uses values from configuration files if certain command-line parameters are not specified. These values are populated by the DCA Setup utility during installation and configuration.

/opt/dca/etc/dca_setup/dca_setup.cnf

dbwatcher.failover_vip = 192.168.0.3

dbwatcher.failover_gateway = 192.168.0.1

dbwatcher.failover_netmask = 255.255.255.0

gpdb_port = 5432

/opt/dca/etc/env/default/master_data_directory

/data/master/gpseg-1



Orchestrated Failover Examples

To perform an orchestrated failover where the Primary Master is powered on and the Greenplum Database is online, enter the following command:

# dca_failover --shutdown --stopmasterdb --noninteractive

To perform an orchestrated failover where the Primary Master server has completely failed - powered off - and the Greenplum Database is offline, enter the following command:

# dca_failover --noninteractive

Automated FailoverThe automated failover feature monitors the Primary Master, and automatically initiates a failover to the Standby Master when certain conditions are met. Automated failover only refers to a failover from the Primary to Standby Master. If a Standby Master fails, there is no affect to Greenplum Database availability, and no operations are moved.

The algorithm that determines if the Primary Master has failed uses pessimistic logic, queries and ping requests across network interfaces from multiple hosts must all fail in order for a determination to be made. By default, the automated failover feature is not enabled. The feature is enabled by the DCA Setup utility.

The following topics are included in this section:

• Enable, Disable and Status of Automated Failover

• Triggers for Automatic Failover

• Monitor a Failover in Progress

• Failback After an Automated Failover

Enable, Disable and Status of Automated Failover

Automated failover is controller by the DCA Setup utility. In order to run the DCA Setup utility, the user must have root permissions.

Enable Automated Failover

1. As the user root, log into the Primary Master Server - mdw.

2. Launch the DCA Setup Utility:

# dca_setup

3. Select option 2 to Modify DCA Settings.

4. Selection option 12 for DB Watcher.

5. Select option 2 to Enable dbwatcherd.


Automated Failover 52


Disable Automated Failover

1. As the user root, log into the Primary Master Server - mdw.

2. Launch the DCA Setup Utility:

# dca_setup


4. Selection option 12 for DB Watcher.

5. Select option 3 to Disable dbwatcherd.


Check Status of Automated Failvoer

The status of automated failover can be verified by checking if the dbwatcherd service is running. If the service is running, the feature is enabled and monitoring the Primary Master for a failure. If the service is not running, the feature is not enabled. To check if the dbwatcherd service is running:

# ps -ef | grep dbwatcherd

root 24934 1 0 17:01 ? 00:00:00 /opt/dca/bin/dbwatcherd

Triggers for Automatic Failover

The following triggers are evaluated by the failover logic. The triggers are checked in order - on a 15 second poll interval. If the first trigger fails, the next is tried, and all must fail in order for the failover to the Standby Master to initiate. If any trigger is successful, the failover does not occur, and checks start from the beginning on the next poll interval.

The failover monitoring and execution is carried out by a service called dbwatcherd. The dbwatcherd service must

Note: A partial failure such as a failed NIC or Greenplum Database down may not trigger a failover to the Standby. These failures will be reported to EMC Support by the ConnectEMC software.

running in order for these triggers to be monitored.

• The Standby Master connects to the Primary Master and runs a query.

• Standby Master issues ping request to Primary Master.

• Standby Master asks segment servers to validate Primary Master availability:

• 5 different segments will query the Primary Master.

• 5 segments (used above) - will issue ping requests to the Primary Master.

• All 5 segments must report the Primary Master as unreachable.



Monitor a Failover in Progress

When an automated failover is in progress, events are written to the OS log files. By monitoring the /var/log/messages file, you can see what phase of the failover is occurring. Use the following command from the Standby Master:

# tail -f /var/log/messages

Failback After an Automated Failover

After an automated failover occurs, the original Standby Master - smdw - will be the new Primary Master - smdw. Once the original Primary Master - mdw - is replaced, it can be integrated back into the cluster in one of two ways: as the Standby Master or as the Primary Master. The supported process is to always return the replaced master server to its original role - only in certain circumstances, where additional downtime is not feasible, should the replaced master server be left in a reversed role.

This section contains guidelines and high level information about restoring normal operation after a failover. For the replacement procedure, follow the Greenplum DCA Field Replaceable Units Field Procedures Guide, available on Powerlink.

Return Master Server to Original Role After Replacement

The following high level steps should be performed to return the replaced Master server to its original role as Primary Master Server. At completion of the activity, both Master servers will be in their original role, and able to perform an automated failover should a new failure of the Primary Master occur.

• Initialize the replaced Master Server - mdw - as the Standby Master

• Perform an Orchestrated failover to move operations from the new Primary Master - smdw - to the original Primary Master - mdw.

• Initialize the original Standby Master - smdw - as the new Standby Master.

Keep the Master Servers in a Reversed Role

After the original Primary Master - mdw - is physically replaced, the DCA will be operating with the original Standby Master - smdw - as the Primary Master and no Standby Master. In order to reduce the downtime that would be induced by a fail-back, the original Primary Master - mdw - can be initialized as a Standby Master.

This configuration is transparent to the Greenplum Database, and in the event of a failure, the failover process would function correctly. This configuration is not supported by EMC, and should only be done under special circumstances.

Note: The Primary Master should always be returned to its original role due to serviceability concerns regarding rack position, cabling, and hostnames.


EMC Greenplum DCA Administration Guide – Chapter 6: Secured Mode

6. Secured Mode

By default, the DCA ships in a non-hardened state. At the time of installation or upgrade, secured mode can be enabled. Secured mode is based on the Department of Defense DSA STIG guidelines. This chapter is OPTIONAL and should only be done if required by the customer for security reasons. The following sections are included:

• Enable Secured Mode on the DCA

• Revert to Unsecured Mode on the DCA

• Verify Secured Mode State

Enable Secured Mode on the DCAPerform the following steps to enable secured mode on the DCA. This procedure will cause temporary unavailability for the Greenplum Database.

1. Log into the DCA master server as the user gpadmin.

2. Stop the Greenplum Database if it is running:

$ gpstop -a

3. Switch to the user root.

$ su -

4. Stop the healthmon daemon

# dca_healthmon_ctl -d

5. Run the secure mode upgrade utility. The host file specified should contain the name of ALL hosts in the system - this file should already be present if the dca_setup utility was run as part of installation.

# cd /home/gpadmin/gpconfigs

# /opt/dca/sbin/dca_upgrade-1.2.0.0s.bin -u -f hostfile

6. The script will upgrade software on each server - the Primary Master server will be rebooted last, requiring you log back in. Wait 15 minutes for the upgrade process to complete before attempting to log in the Primary Master Server.

7. Log back into the Primary Master Server and verify the script completed successfully


# /opt/dca/sbin/dca_upgrade-1.2.0.0s.bin -s -f hostfile

8. Start the healthmon daemon and the Greenplum Database

# dca_healthmon_ctl -e

# su - gpadmin

$ gpstart -a

Enable Secured Mode on the DCA 55


Revert to Unsecured Mode on the DCAPerform the following steps to revert a system that is in secured mode to an unsecured state. This procedure will cause temporary unavailability for the Greenplum Database.

1. Log into the DCA master server as the user gpadmin.

2. Stop the Greenplum Database if it is running:

$ gpstop -a

3. Switch to the user root.

$ su -

4. Stop the healthmon daemon


5. Run the secure mode upgrade utility with revert option. The host file specified should contain the name of ALL hosts in the system - this file should already be present if the dca_setup utility was run as part of installation.


# /opt/dca/sbin/dca_upgrade-1.2.0.0s.bin -r -f hostfile

6. The script will revert to the unsecured settings on each server - the Primary Master server will be rebooted last, requiring you log back in. Wait 15 minutes for the revert process to complete before attempting to log in the Primary Master Server.

7. Log back into the Primary Master Server and verify the script completed successfully


# /opt/dca/sbin/dca_upgrade-1.2.0.0s.bin -s -f hostfile

8. Start the healthmon daemon and the Greenplum Database

# dca_healthmon_ctl -e

# su - gpadmin

$ gpstart -a

Verify Secured Mode StateThe state of secured mode on the DCA can be verified by following the steps below:

1. Login to the DCA primary master server as the user root.

2. Run the following command to print the contents of the appliance version file.

# cat /etc/gpdb-appliance-version

The output will should read 1.2.0.0s if secured mode is ENABLED or 1.2.0.0 if secured mode is DISABLED.

3. Verify FIPS-compliant mode of SSH is also enabled - this is automatically done as part of enabling secured mode, however it should also be verified.

Revert to Unsecured Mode on the DCA 56


# cat /proc/sys.crypto/fips_enabled

The output should read 1 if the FIPS compliant SSH is ENABLED or 0 if the FIPS compliant SSH is DISABLED.

Verify Secured Mode State 57

EMC Greenplum DCA Administration Guide – Chapter 7: SNMP on the DCA

7. SNMP on the DCA

This chapter contains information on the DCA implementation of SNMP. The Greenplum DCA has a SNMP version 2 management information base (MIB). The MIB can be used by enterprise monitoring systems to identify issues with components and services in the DCA.

The following sections are included in this chapter:

• DCA MIB information

• Integrate DCA MIB with environment

DCA MIB informationThis section contains information on the DCA MIB. The following information is included:

• MIB Locations

• MIB Contents

• View MIB

MIB Locations

The DCA MIBs are located in the following locations:

/usr/share/snmp/mibs/GP-DCA-TRAP-MIB.txt

/usr/share/snmp/mibs/GP-DCA-DATA-MIB.txt

MIB Contents

The DCA public MIB is organized in the following way:

Figure 7.1 MIB OID Structure

DCA MIB information 58


Table 7.1 DCA Data MIB - v1 Hardware and DCA Services Components

Data MIB Contents Component OID Description

1 - gpDCAv1Hardware gpMasterNodes 1 GPDB Primary and Standby Master servers.

gpSegmentNodes 2 GPDB Segment servers.

gpAdminSwitches 3 DCA administration switches.

gpInterconnectSwitches 4 DCA Interconnect switches.

gpEtlNodes 5 DIA servers.

gpHadoopMasterNodes 6 Hadoop Master servers.

gpHadoopWorkerNodes 7 Hadoop Worker servers.

gpAggAdminSwitches 8 DCA aggregation administration switches.

gpAggInterconnectSwitches 8 DCA aggregation Interconnect switches.

gpHbaseWorkerNodes 10 Hadoop HBase servers.

3 - gpDCAServices gpDbService 1 Greenplum Database processes.

gpHadoopService 2 Hadoop processes.

Table 7.2

Trap MIB Contents Description

1 - gpDCATrap This OID is used for notifications generated for a hardware or database event.

2 - gpDCATrapSymCode Symptom code for the event.

3 - gpDCATrapDetailedSymCode Detailed symptom code for the event.

4 - gpDCATrapDesc Description of the event.

5 - gpDCATrapSeverity Severity of the event:

0 - unknown

1 - error

2 - warning

3 - info

4 - debug

6 - gpDCATrapHostname Server where the event occurred.

DCA Trap MIB

View MIB

Issue the following commands from a Master server as the user root:

# MIBS+=GP-DCA-DATA-MIB

# export MIBS

DCA MIB information 59


# snmpwalk -v 2c -c public 172.28.4.250 1.3.6.1.4.1.1139.23.1.1.2

Integrate DCA MIB with environmentThis section contains information on how to integrate the DCA MIB with an environment. The following information is included:

• Change the SNMP community string

• Set a SNMP Trap Sink

Change the SNMP community string

The SNMP community string can be modified through the DCA Setup utility. Changing the SNMP community string through DCA Setup will update all hosts in the DCA. Follow the instructions below to modify the community SNMP string.

The following restrictions apply when modifying the community SNMP string:

• The Greenplum Database must be version 4.1.1.3 or later. If the Greenplum Database is a version earlier than 4.1.1.3, the option to modify the SNMP community string will not be available.

• If the SNMP community string is modified while running Greenplum Database 4.1.1.3 or later, and the Greenplum Database is downgraded to a version earlier than 4.1.1.3, the modified SNMP file will not function properly. Also, dial-home and health monitoring will be affected.

• If the DCA cluster is expanded with new hosts, the new hosts will not use the default SNMP configuration by default. The updated SNMP configuration must be copied from an existing host to the new hosts.



# dca_setup


4. Select option 20 for Health Monitoring.

5. Select option 7 for snmp_community.

6. Enter the new SNMP community string:

Enter the SNMP community string [public]:cust-string

7. Select option 1 Apply Changes.

8. Enter y to accept changes.

Integrate DCA MIB with environment 60


Set a SNMP Trap Sink

Up to 6 SNMP Trap Sink servers can be specified through the DCA Setup utility. Follow the instructions below to set Trap Sink servers:



# dca_setup



5. Select option 8 through 13 to set trapsink_hostname1 through trapsink_hostname6.

6. Select option 1 for Change this trapsink.

7. Enter the trap sink server in the format hostname or hostname:port:

Enter snmp trap destination 1: trapserver1

8. Select option 1 Apply Changes.

9. Enter y to accept changes.

Integrate DCA MIB with environment 61

EMC Greenplum DCA Administration Guide – Chapter 8: DCA Utility Reference

8. DCA Utility Reference

This chapter contains reference information on DCA utilities. The following information is included:

• dca_setup

• dca_shutdown

• dca_hadoop

• dcacheck

• dca_healthmon_ctl

• dcaperfcheck

• dca_blinker

• gppkg

dca_setupThe DCA Setup utility is an administration tool for the DCA. Through the DCA Setup utility, a user can install, upgrade, and modify settings on a DCA.

Synopsisdca_setup

dca_setup [ --config configuration_file | -c configuration_file ] [-a]

dca_setup --help

Description

The DCA Setup utility was implemented starting in DCA software version 1.1.0.0 to automate common tasks performed on a DCA. The DCA Setup utility can be run in an interactive mode (default) or as a batch script using a configuration file. Using a configuration file is best done when there are multiple changes to be made, such as configuration of custom IP addresses in a VLAN overlay.

Usage:

• Must be run as the user root

• Must be run from the Primary or Standby Master server

• Navigation is done by entering the numeric option or sub-option and pressing ENTER.

• Use CTRL-C to exit without committing changes

62



Available operations

The following operations are available through DCA Setup:

Table 8.1 DCA Setup Operations

Option Sub Option Description

Install DCA Install a new DCA with interactive options. Configure GPDB, DIA and GP HD modules.

Regenerate DCA Config Files

Generate or regenerate hostfiles. Files are created in /home/gpadmin/gpconfigs. Health monitoring is configured with hosts entered in this option.

Change DCA Locale Change locale of the DCA. Available locales can be displayed by using the locale -a shell command.

Change DCA Timezone

Change timezone of the DCA.

Clocks Configuration 1) Reset ntpd configuration Modify settings related to the time/clocks on DCA servers. Reset configuration to default using.

2) Add ntpd external servers Add customer timeserver.

3) Synchronize Clocks across Cluster

Synchronize time across all hosts in the DCA.

Regenerate SSH Keys Regenerate SSH keys across all servers in a the DCA. This operation should be used any time new hardware is added to the DCA.

Change Machine Passwords

Change the gpadmin, root and iDRAC password across all servers in the DCA.

Initialize GPDB Configure and initialize the Greenplum Database.

Initialize GPDB Standby Master

Initialize a Standby Master server for an existing Greenplum Database instance.

Expand DCA Add GPDB, DIA or GP HD servers to a DCA. Values must be entered in increments of 4 (modules).

Rebalance Mirrors 1) Rebalance Mirrors Into Failure Groups

Create /home/gpadmin/gpconfigs/move_mirrors_failure_group.conf. This option is used to move from spread to group mirroring. Spread mirroring was used pre-GPDB 4.1 / DCA software 1.1.0.0.

2) Rebalance Mirrors onto a SAN

Create /home/gpadmin/gpconfigs/move_mirrors_san.conf. This option is used when a DCA is attached to external storage.

Login Banner 1) Set a new login banner Set a login banner across all servers in the DCA. The login banner is used as part of a secure-mode DCA. You must create a plain-text file with the content before-hand.

2) Delete existing login banner

Delete the login banner across all servers in a DCA.

63



DB Watcher 1) Enable dbwatcherd Enable Master server failover.

2) Disable dbwatcherd Disable Master server failover.

Networking: Masters 1) Modify GPDB Master Network Settings

Set Primary Master NIC bonding, external IP address/gateway/netmask.

2) Modify GPDB Standby Master Network Settings

Set Standby Master NIC bonding, external IP address/gateway/netmask.

3) Configure Virtual IP on Master

Set a Virtual IP on the Primary Master server (used in failover).

4) Deactivate Virtual IP on Master

Remove a Virtual IP on the Primary Master server.

Networking: Segments 1) Apply Changes Apply changes once option 2 - 5 have been run.

2) Edit DCA Hostnames Set custom hostnames of the servers in a DCA.

3) Edit Non-DCA Hostnames Set a hostname in /etc/hosts for a server that is not part of the DCA.

4) Edit VLAN Configuration Set a custom VLAN overlay.

5) Edit Name Resolution Settings

Edit the name resolution settings (DNS).

Networking: Switches 1) Generate new switch configurations

Generate a new Interconnect (IC) switch configuration based on an interview. Configuration can be created for a single (FRU) or all switches. Configurations are created in /opt/dca/var/switch_config/upload.

2) Upload switch configurations

Upload IC switch configurations in /opt/dca/var/switch_config/upload.

3) Generate and upload switch configurations

Generate new IC switch configurations, then upload the files immediately to the switch(es).

4) Download switch configurations

Download the live IC switch configurations.

5) Generate cluster map Generate a cluster map. A cluster map shows which host MAC addresses are mapped to a switch. This is used to identify what rack a host is located.

6) Clear ssh known_hosts on switches

Clear the known_hosts file on switches. This is useful if the switch was replaced or for communication issues. This must be run if a Master server is imaged.

Hadoop 1) Enable NFS Server on Hadoop Master

Enable NFS on a Hadoop Master server, hdm2 (default) or hdm1.

2) Disable NFS Server on Hadoop Master

Disable NFS on a Hadoop Master server.

3) Configure namenode failover

Configure namenode failover.



64



Automate DCA Setup configuration

DCA Setup operations can be run in an automated batch mode using a configuration file. A sample file is provided containing all options:

/opt/dca/var/dca_setup/dca_setup.cnf

/opt/dca/var/dca_setup/networking_sample.cnf

Light Bars Turn on the DCA door light bar and a server LED. In order for this operation to function, a cluster map must have been generated during install or using Networking: Switches, option 5.

Bios/Firmware/OMSA Upgrade the server BIOS, PERC Firmware and Dell Open Manage Server Administration software.

RAID Controller Battery

1) Condition battery on all hosts

Enter a learn cycle for the RAID batteries on all servers.

2) Condition battery on selected hosts

Enter a learn cycle for the RAID batteries on selected servers.

3) Adjust scheduling for battery conditioning

Synchronize a schedule for the RAID batteries on all servers.

Health Monitoring 1) Apply Changes Use option to commit changes once option 2 - 13 have been run.

2) emcconnect: on Change ConnectEMC operation.

3) warning_disk_space_percentage: 80

Change the threshold a warning notification will be generated.

4) error_disk_space_percentage: 90

Change the threshold an error notification will be generated.

5) do_periodic_licensing_data_report: 1

Turn ON or OFF periodic licensing report call home.

6) do_periodic_report: 1 Perform an immediate licensing report.

7) snmp_community: NOT_SET

Set the SNMP Community String.

8) trapsink_hostname1: NOT_SET






Set 1 to 6 trapsink server hostnames.



65



/opt/dca/var/dca_setup/custom_gpdb_port.cnf

Perform the following to use a configuration file:

• Always edit a copy of the file:# cp /opt/dca/var/dca_setup/dca_setup.cnf my_config.cnf

# vi my_config.cnf

• Remove comment markers ‘#’ to make a line active

• Select parameters (top of file), then the associated action to perform in the # ACTIONS section

• Execute DCA Setup with the config file:# dca_setup -c myconfig.cnf

Options

--config configuration_file | -c configuration_file

Specify a configuration file for a batch mode. This option is best used when making multiple changes, such as assigning custom IP addresses to multiple hosts for a VLAN overlay.

-a

Use this parameter with the --config option to perform unattended changes. Using this option will not prompt for confirmation before performing an action.

--help

Display the online help.

Examples

Run DCA Setup in interactive mode:

# dca_setup

Run DCA Setup with a configuration file, in unattended mode:

# dca_setup -c my_config.cnf -a

dca_shutdownThe DCA shutdown utility will safely power off all servers in a DCA.

Synopsisdca_shutdown { -f hostfile | -h hostname } [ --ignoredb ] [ --password= password ] [ --passfile= password_file ] [--statusonly]

dca_shutdown

dca_shutdown --help

66



Description

The dca_shutdown utility will safely power down all servers in a DCA. The utility can be run with no parameters, and will use the system inventory generated by DCA Setup during an installation or Regenerate DCA Config Files operation. If the utility is run with a hostfile or hostname specified, only those hosts will be shutdown. This utility will not shut down the administration, Interconnect or aggregation switches.

The utility should be run as the user root. Prior to running the dca_shutdown, the following steps should be performed to ensure a clean shutdown:

1. Stop Greenplum Database:$ gpstop -af

2. Stop Command Center:$ gpcmdr --stop

3. Stop health monitoring as the user root:$ su -


Options

-?, --help

Print usage and help information

-i, --ignoredb

Do not check if Greenplum Database, health monitoring or Command Center are running. Shut down all servers immediately.

-h, --host hostname

Perform a shutdown on the host specified.

-f, --hostfile hostfile

Perform a shutdown on the hosts listed in the hostfile. This option can not be used with the --host option.

-p, --password password

Specify a password to connect to the server’s IPMI (iDRAC) to perform the shutdown. The password is originally set during installation with DCA Setup - if an installation through DCA Setup has never been run, the user will be prompted for a password.

-s, --passfile password_file

Specify a file containing the password to use to connect to the server’s IPMI (iDRAC) to perform the shutdown. This file is generated during installation with DCA Setup, and is located in /opt/dca/etc/ipmipasswd.

67



-o, --statusonly

Print the power status (ON | OFF) of all servers. This will not power off any servers.

Examples

Shut down all servers in a DCA:

dca_shutdown

Shut down servers listed in the file hostfile:

dca_shutdown -f /home/gpadmin/gpconfigs/hostfile

dca_hadoopControl Hadoop services across all Hadoop servers. Also used to control Hadoop namenode failover.

Synopsisdca_hadoop { --start | --stop | --status }

dca_hadoop --nnfailover hostname --verbose

dca_hadoop --help

Description

The dca_hadoop utility is used to start, stop and monitor Hadoop servers across all Hadoop servers. The utility will only control Hadoop servers based on the configuration specified in DCA Setup during an installation or expansion. In order for the namenode Virtual IP address to be active, dca_hadoop --start must be run.

This utility can only be run as the user gpadmin from a Hadoop master server.

The dca_hadoop utility also controls namenode failover. During namenode failover, the following actions are performed:

• Backup of hdfs-site.xml is made - hdfs-site.xml.bkp - on all Hadoop masters.

• Edit hdfs-site.xml to remove /data/remote/hdfs/name from dfs.name.dir property.

• Stop Hadoop services

• Virtual IP (VIP) removed from old namenode (if running).

• Check that VIP is not present on any other servers in the DCA.

• VIP added to new namenode

• Specify new namenode hostname in dca_hdm_settings.cnf

• Start Hadoop services

68



Options

--start

Starts Hadoop services on all Hadoop servers. If services are already running on some hosts, they will be skipped. Virtual IP address will be enabled on namenode.

--stop

Stop Hadoop services on all Hadoop servers. If services are already stopped on some hosts, they will be skipped. Virtual IP address will be removed from namenode.

--status

Print status of all Hadoop servers. This option will display services running on each server and if any services are stopped or missing.

--nnfailover host

Perform a namenode failover to the host specified. The available options for host are hdm1 or hdm2. Specify host hdm2 for an initial failover and host hdm1 to fail back.

-v, --verbose

Print detailed information during a namenode failover. This option can only be used in addition to --nnfailover.

--help

Print the online help.

Examples

Start Hadoop services on all Hadoop servers:

dca_hadoop --start

Perform a namenode failover from hdm1 to hdm2 (standby namenode):

dca_hadoop --nnfailover hdm2

dcacheckValidate hardware and operating system settings.

Synopsisdcacheck { -f hostfile | -h hostname } { --stdout | --zipout } [ --config config_file ]

dcacheck --zipin dcacheck_zipfile

dcacheck -?

69



Description

The dcacheck utility validates DCA operating system and hardware configuration settings. The dcacheck utility can use a host file or a file previously created with the --zipout option to validate settings. At the end of a successful validation process, DCACHECK_NORMAL message displays. If DCACHECK_ERROR displays, one or more validation checks failed. You can use also dcacheck to gather and view platform settings on hosts without running validation checks.

Greenplum recommends that you run dcacheck as the user root. If you do not run dcacheck as root, the utility displays a warning message and will not be able to validate all configuration settings; Only some of these settings will be validated.

If dcacheck is run with no parameters, it will validate settings in the following file:

/opt/dca/etc/dcacheck/dcacheck_config

Different configuration parameters are validated by DCA software release.

Options

--config config_file

The name of a configuration file to use instead of the default file /opt/dca/etc/dcacheck/dcacheck_config.

-f hostfile

The name of a file that contains a list of hosts dcahceck uses validate settings. This file should contain a single host name for all hosts in the DCA.

-h hostname

The name of a host that dcacheck will validate platform-specific settings.

--stdout

Display collected host information from dcacheck. No checks or validations are performed.

--zipout

Save all collected data to a .zip file in the current working directory. dcacheck automatically creates the .zip file and names it dcacheck_timestamp.tar.gz. No checks or valications are performed.

--zipin file

Use this option to decompress and check a .zip file created with the --zipout option. dcacheck performs validation tasks against the file you specify in this option.

-?


70



Examples

Verify and validate the DCA settings on specific servers:

# dcacheck -f /home/gpadmin/gpconfigs/hostfile

Verify custom settings on all DCA servers:

# dcacheck --config my_config_file

dca_healthmon_ctlControl the healthmond service on a DCA.

Synopsisdca_healthmon_ctl { -e | -d | -s } [ -n ]

dca_healthmon_ctl --help

Description

The dca_healthmon_ctl utility controls the DCA health monitoring service. This utility is used to manually enable, disable, or check the status of health monitoring. The health monitoring service must be stopped during most service activities to avoid false call home messages. This utility must be run as the user root.

Options

-e, --enable

Enable health monitoring services on the DCA.

-d, --disable

Disable health monitoring services on the DCA.

-s, --status

Query the status of health monitoring services on the DCA.

-n, --node mdw|smdw

Run command on a specific server, mdw or smdw. Normal operation runs the command on both mdw and smdw. This option must be used with the --enable or --disable parameters.

-h, -?, --help


Examples

Stop heath monitoring services on a DCA:


71



Start health monitoring services on the Primary Master server, mdw:

# dca_healthmon_ctl -e -n mdw

dcaperfcheckRun performance tests on disk, memory and network.

Synopsisdcaperfcheck { -f hostfile | -h hostname } { -r [d | s | n | N | M ] } [-B size ] [ -S size ] {-d test_dir | --device } {-v | -V } [ -D ] [ --duration seconds ] [ --netperf ]

dcaperfcheck -?

Description

The dcaperfcheck utility is used to test performance of the hardware in a DCA. This test is run to validate network, disk and memory are performing as expected. This is a useful tool to determine hardware failures, or mis-cabling. This utility can be run as the user gpadmin or root. If the utility is run as the user gpadmin, this user must have permissions to write and read from the test directory.

Options

-d test_directory

Directory where data will be written to and read from. Multiplt -d flags may be specified for multiple directories on each host. During network and memory tests, this can be the /tmp directory. During disk tests, use operating system mount points that will exercise each drive.

-v

Enable verbose output.

-V

Enable very verbose output.

-D

Print statistics for each host. The default output will print only the hosts with lowest and highest values.

-rd, -rs, -rn, -rN, -rM

Specify type of test to run, d - disk, s - stream (memory), n - serial netperf, N - parallel netperf, or M - full matrix netperf. These options can be conbined, for example, -rds. The default is dsn. Typically, disk and network tests are seperated, because disk tests require more test directories specified, where network tests only require a single temporary directory.

72



-B size

Specify the block size for disk performance tests. The default is 32kb. Examples: 1KB, 4MB.

-S size

Specify the file size for disk performance tests. The default is 2x server memory. On a DCA, there is 48GB of memory, so the default is 96GB. Examples: 500MB, 16GB.

-h hostname

Specify a host to run the utility. Multiple hosts can be specified.

-f hostfile

Specify a file with a list of hosts to run the utility. The hostfile will differ based on the test (disk or network) you are running. A network test will typically be run against once interconnect, so hostnames should reflect only interfaces on this interconnect

--duration seconds

Specify a length of time to run the network test. Time specified is in seconds.

--netperf

Use the netperf network test instead of gpnetbenchServer/gpnetbenchClient. This option can only be run if the network test is specified.

--device

Use a raw device instead of a test directory, for example /dev/sda1, /dev/sda2. Multiple devices may be specified. This option requires dcaperfcheck be run as the user root. WARNING THIS WILL CAUASE DATA LOSS FOR SPECIFIED DEVICES.

-?

Print online help.

Examples

Run a parallel network and stream test on Interconnect 1:

# dcaperfcheck -f /home/gpadmin/gpconfigs/hostfile_gpdb_ic1 -rsN -d /tmp

Run a disk test, using all the data directories on a segment server, sdw1:

# dcaperfcheck -h sdw1 -rd -d /data1 -d /data2

dca_blinkerIdentify a DCA server by flashing it’s marker LED.

73



Synopsisdca_blinker { -h hostname | -f hostfile } [ -a ON | OFF ] [ -t interval ]

dca_blinker -?

Description

The dca_blinker utility turns on/off the marker LED on a DCA server. This utility can be used from any server in a DCA. The default flash interval 15 minutes. This utility must be run as the user root.

Options

-h hostname

Specify the host to flash the marker LED. Multiple hosts can be specified.

-f hostfile

Specify a list of hosts to flash the marker LED.

-a ON|OFF

Set the LED flash to ON or OFF.

-t interval

Interval in seconds to flash the server marker LED. Default interval is 15 minutes.

Examples

Flash the marker LED on sdw1 for 20 seconds:

dca_blinker -h sdw1 -a ON -t 20

Turn off the flashing LED on sdw1:

dca_blinker -h sdw1 -a OFF

gppkgInstalls Greenplum Database extensions such as pgcrypto, PL/R, PL/Java, PL/Perl, and PostGIS, along with their dependencies, across an entire cluster.

Synopsisgppkg [-i package | -u package | -r name-version | -c] [-d master_data_directory] [-a] [-v]

gppkg --migrate GPHOME_1 GPHOME_2 [-a] [-v]

gppkg [-q | --query] query_option

gppkg -? | --help | -h

74



gppkg --version

Description

The Greenplum Package Manager (gppkg) utility installs Greenplum Database extensions, along with any dependencies, on all hosts across a cluster. It will also automatically install extensions on new hosts in the case of system expansion and segment recovery.

First, download one or more of the available packages from the EMC Download Center then copy it to the master host. Use the Greenplum Package Manager to install each package using the options described below.

Note: After a major upgrade to Greenplum Database, you must download and install all extensions again.

The following packages are available for download from the EMC Download Center.

• PostGIS

• PL/Java

• PL/R

• PL/Perl

• Pgcrypto

Options

-a (do not prompt)

Do not prompt the user for confirmation.

-c | --clean

Reconciles the package state of the cluster to match the state of the master host. Running this option after a failed or partial install/uninstall ensures that the package installation state is consistent across the cluster.

-d master_data_directory

The master data directory. If not specified, the value set for $MASTER_DATA_DIRECTORY will be used.

-i package | --install=package

Installs the given package. This includes any pre/post installation steps and installation of any dependencies.

--migrate GPHOME_1 GPHOME_2

Migrates packages from a separate $GPHOME. Carries over packages from one version of Greenplum Database to another.

For example: gppkg --migrate /usr/local/greenplum-db-4.2.0.1 /usr/local/greenplum-db-4.2.1.0

75


https://emc.subscribenet.com




This option is automatically invoked by the installer during minor upgrades. This option is given here for cases when the user wants to migrate packages manually.

Migration can only proceed if gppkg is executed from the installation directory to which packages are being migrated. That is, GPHOME_2 must match the $GPHOME from which the currently executing gppkg is being run.

-q | --query query_option

Provides information specified by query_option about the installed packages. Only one query_option can be specified at a time. The following table lists the possible values for query_option. <package_file> is the name of a package.

Table 8.2 Query Options for gppkg

query_option Returns

<package_file> Whether the specified package is installed.

--info <package_file> The name, version, and other information about the specified package.

--list <package_file> The file contents of the specified package.

--all List of all installed packages.

-r name-version | --remove=name-version

Removes the specified package.

-u package | --update=package

Updates the given package.

--version (show utility version)

Displays the version of this utility.

-v | --verbose

Sets the logging level to verbose.

-? | -h | --help

Displays the online help.

76


EMC Greenplum DCA Administration Guide – Chapter 9: Troubleshooting

9. Troubleshooting

This section contains resources for troubleshooting the DCA. The following information included:

• Call home information

• Log Files

• Common Issues

Call home informationThis section contains the following topics:

• ConnectEMC Error Codes

• Verifying SNMP Data Collection

• Starting Stopping and checking Status of ConnectEMC

• Server Configuration Parameter for ConnectEMC

ConnectEMC Error Codes

ConnectEMC collects and sends event data — files indicating system errors and other information — from the DCA to EMC Global Services systems. This event data (or alerts) provide critical information to EMC support personnel and help prevent potential problems on the DCA, as well as diagnose and resolve current issues on the system.

Alerts that arrive at EMC Global Services can have one of the following severity levels:

• WARNING: This indicates a condition that might require immediate attention.

• ERROR: This indicates that an error occurred on the DCA. System operation and/or performance is likely affected. This alert requires immediate attention.

• UNKNOWN: This severity level is associated with hosts and devices on the DCA that are either disabled (due to hardware failure) or unreachable for some other reason. This alert requires immediate attention.

INFO: An event with this severity level indicates that a previously reported error condition is now resolved. Greenplum Database startup also triggers an INFO alerts.

Code Description

1.1 Host not responding to SNMP calls, host may be down.

1.4 Interface status: could not open session to host 1.

2.15 Greenplum Database is ready to accept connections.

2.15005 Greenplum Database panic, insufficient resource queues available.

Call home information 77


3.2000 Status of power supply, if PS fails, will get error with this code.

4.3000 Status of battery on system. Will report error on failure.

5.4001 Status of cooling device, e.g. fan failure.

5.4002 Temperature of system.

6.5001 Status check of a CPU. CPU failure will register here.

7.6001 Status of a CPU Cache device. Cache device failure will register here.

8.1002 Operating System Memory Status.

9.7000 Memory device status. Failed memory devices will get this code.

10.8003 Status of the network device.

10.8005 A configured network bond is unavailable.

10.8006 Network bonding on master servers: The bond interface has no active link/slave.

10.8007 Network bonding on master servers: The bond interface link/slave has changed.

10.8008 Network bonding on master servers: The bond interface links are all down.

10.8009 Network bonding on master servers: One of the bond interface links is down.

11.9001 Status of IO Controller.

11.9002 Status of battery on the IO Controller.

12.10002 Virtual Disk 1 Status: /dev/sda: nonCritical.

12.10004 Virtual disk size (MB).

12.10005 Write cache policy on virtual disk. For example, expected to be write back mode.

12.10006 Read cache policy of virtual disk. For example, expected to be adaptive read ahead.

12.10007 Detects offline, rebuilding raid and other unexpected virtual disk states.

12.10011 Percentage of disk space on virtual disk used.

12.10012 Virtual disk space used (KB).

13.11001 Status of drive. Drive failures use this ID.

14.12002 Interconnect Switch Operational Status.

14.12005 Operational status of Interconnect switch flash memory.

Code Description



14.12006 State of Interconnect switch flash memory.

14.13001 Status errors from switch sensors - Fans, Power Supplies, and Temperature.

14.14 Interface 0 Description: unexpected snmp value: val_len<=0.

14.14001 Interface 0 Status: unexpected status from device.

15.2 An error detected in the SNMP configuration of the host.

15.3 Other SNMP related errors.

15.4 Connection aborted by SNMP.

15.5 Unexpected SNMP errors from the SNMP system libraries.

15.6 Can not find expected OID during SNMP walk.

16.0 Test Dial Home.

18.15000 Sent from inside GPDB when starting up.

18.15001 Sent from inside GPDB when GPDB could not access the status of a transaction.

18.15002 Sent from inside GPDB when interrupted in recovery.

18.15003 Sent from inside GPDB when a 2 phase file is corrupted.

18.15004 A test message sent from inside GPDB.

18.15005 Sent from inside GPDB when hitting a panic.

18.17000 Sent by healthmond when GPDB status is normal.

18.17001 Sent by healthmond when GPDB can not be connected to and was not shutdown cleanly, possible GPDB failure.

18.17002 Sent by healthmond when detecting a failed segment.

18.17003 Sent by healthmond when detecting a segment in change tracking.

18.17004 Sent by healthmond when detecting a segment in resync mode.

18.17005 Sent by healthmond when detecting a segment not in its preferred role, unbalanced cluster.

18.17006 Sent by healthmond when detecting a move of the master segment from mdw to smdw.

18.17007 Sent by healthmond when detecting a move of the master segment from smdw to mdw.

18.17008 Sent by healthmond when a query fails during health checking.

18.17009 Healthmond error querying GPDB State.

Code Description



Table 9.1 DCA Error Codes

Verifying SNMP Data Collection

The SNMP agent collects data from each host in the DCA in one of two locations based on the state of the Greenplum Database:

Greenplum Database is running:$MASTER_DATA_DIRECTORY/gpperfmon/data/snmp/

Greenplum Datbase is stopped or not initialized:/opt/dca/var/healthmond/datafiles/

The following files are created by the SNMP agent:

• hostlistreport.txt: Shows the rollup status of each host in the DCA.

• lastreport.txt: Contains the epoch timestamp of the start and end of the last SNMP report.

• snmp.host.hostname.txt: Each monitored host has its own file. For example, the file for segment host 1 (sdw1) would be named snmp.host.sdw1.txt. Each file contains monitored items and their associated values for that host.

ConnectEMC uses the information in these files to generate the event data files it sends to EMC Global Services. If SNMP is monitoring is enabled and functioning properly, these files should be updating on a regular basis.

19.18000 ID for informational dial homes with general system usage information.

21.20000 Core files were found on the system.

21.20001 Linux kernel core dump files were found on the system - indicates a crash and reboot.

22.21000 Master Node Failover was successful.

22.21001 GPActivatestandby command failed during master node failover.

22.21002 Greenplum Database is not reachable after the failover.

22.21003 Error in bringing the remote(other) master server down during master node failover.

22.21004 Error in taking over the remote(other) master server IP.

22.21005 Unknown error in failover.

23.22002 Host did not complete upgrade within the specified timeout period. Timeout period is 12 hours by default unless set in /opt/dca/etc/healthmond/healthmond.cnf.

Code Description



Starting Stopping and checking Status of ConnectEMC

ConnectEMC starts automatically when you start the start EMC Greenplum Data Computing Appliance and runs as a background Linux daemon.

To stop ConnectEMC

$ su

# /etc/init.d/connectemc stop

To start ConnectEMC

$ su

# /etc/init.d/connectemc start

To Check ConnectEMC Status

$ /etc/init.d/connectemc status

To disable ConnectEMC auto-start

$ su -

# chkconfig --level 012345 off

Server Configuration Parameter for ConnectEMC

ConnectEMC event logging can be controlled through the DCA Setup utility. The server configuration parameter has three values:

• No ConnectEMC logging

• ConnectEMC events are logged locally only

• ConnectEMC events are sent to EMC Global Services Support only

• ConnectEMC events are logged locally and send to EMC Global Services Support

To change ConnectEMC logging:

1. Log into the Primary Master server as the user root.

2. Launch the DCA Setup utility:

# dca_setup

3. Select option 2 for Modify DCA Settings.


5. Select option 2 for emcconnect.

6. Choose one of the following options:

• 1 - ON: (Send to EMC and Store in GPDB

• 2 - LOCAL: Store in GPDB only

• 3 - REMOTE Send to EMC only

• 4 - OFF



7. From the Health Monitoring Configuration menu, select 1 to Apply Changes.

Log FilesThis section contains information on the log files for the DCA.

Table 9.2 Log File Locations

Utility Log File Location

DCA Upgrade /home/gpadmin/gpAdminLogs/dca_upgrade_verify_completion_yyyymmdd.log

/home/gpadmin/gpAdminLogs/dca_upgrade_service_yyyymmdd.log

/home/gpadmin/gpAdminLogs/dca_upgrade_prereq_tester_yyyymmdd.log

/home/gpadmin/gpAdminLogs/dca_upgrade_master_yyyymmdd.log

Hadoop /opt/dca/libexec/hadoop/logs/hadoop-gpadmin-namenode-hdm#.log

(Namenode on hdm1 or hdm2)

/opt/dca/libexec/hadoop/logs/hadoop-gpadmin-jobtracker-hdm3.log

(Jobtracker on hdm3)

/opt/dca/libexec/hadoop/logs/jobtracker-metrics.out

(Jobtracker Metrics on hdm3)

/opt/dca/libexec/hadoop/logs/hadoop-gpadmin-secondarynamenode-hdm4.log

(Secondary Namenode on hdm4)

/opt/dca/libexec/hadoop/logs/hadoop-gpadmin-datanode-hdw#.log

(Datanode on hdw#)

/opt/dca/libexec/hadoop/logs/hadoop-gpadmin-tasktracker-hdw#.log

(Tasktracker on hdw#)

DCA Setup /home/gpadmin/gpAdminLogs/dca_setup_yyyymmdd.log

Command Center

/usr/local/greenplum-cc-web-1.2.0.0/instances/INSTANCE_NAME/logs/lighttpd-access.log

/usr/local/greenplum-cc-web-1.2.0.0/instances/INSTANCE_NAME/logs/lighttpd-error.log

/usr/local/greenplum-cc-web-1.2.0.0/instances/INSTANCE_NAME/logs/gpmonws.log

DCA Check /root/gpAdminLogs/dcacheck_main.py_yyyymmdd.log

DCA Blinker /root/gpAdminLogs/dca_blinker_main.py_yyyymmdd.log

Log Files 82


Common IssuesThis chapter contains common errors and suggested resolutions to be used for troubleshooting purposes. If the steps in this section do not resolve the issue, or there is a risk of system unavailability, you should contact EMC Support.

Host does not boot

Symptoms:

• Host cannot be pinged after a PXE boot / image.

• Host does not load Operating System after a PXE boot / image.

Try This:

• Reboot host by holding down Power button.

• Connect to host iDRAC:# ssh sdw1-sp

admin1>reset /system1

Host iDRAC IP address not correct

Symptoms:

• Host cannot be pinged on iDRAC (hostname-sp) interface, but can be pinged on other interfaces.

• Host iDRAC interface IP did not update after a PXE boot / image.

Try This:

• Set the iDRAC IP address manually (must log in through another interface). For example on host sdw1-sp (172.28.0.1):# ipmitool lan set ipsrc static

# ipmitool lan set ipaddr 172.28.0.1

Host interfaces eth2, eth3, eth4 or eth5 do not come up

Symptoms:

• Host cannot be pinged on sdw1-1 or sdw1-2 but can be pinged on sdw1-cm

• Running ifconfig from host does not list eth2 or eth3

Try This - GPDB Offline:

• Perform a hard shutdown, then power up with power button:# shutdown -h now

• Perform a system stop/start from iDRAC:# ssh sdw1-sp

admin1> stop /system1

admin1> start /system1

Common Issues 83


• Check cables from host to Interconnect switch. Verify a solid green LED is lit on the switch and host CNA. A flashing LED on the CNA indicates a bad connection.

Greenplum Database errors during initialization in DCA Setup

Symptoms:

• DCA Setup quits with an error of Not Enough Segments to Start.

• DCA Setup fails to initialize Greenplum Database.

Try This:

• Check the symbolic link on each GPDB server in the cluster. The symbolic link should indicate the same version of GPDB across the cluster:# gpssh -f /home/gpadmin/gpconfigs/hostfile_gpdb -e 'll /usr/local/greenplum-db'

• Remove and recreate the symbolic link on each GPDB server in the cluster:# gpssh -f /home/gpadmin/gpconfigs/hostfile_gpdb

=> cd /usr/local

=> rm greenplum-db

=> ln -s greenplum-db-4.2.1.0 greenplum-db

=> exit

• Check the Greenplum Database master data directory.

Errors connecting to Interconnect switches for download, cluster map, upload

Symptoms:

• Errors downloading/uploading switch configurations or creating custer map.

Try This:

• Check Cabling to Switch.

• Clear ssh known_hosts with DCA Setup.

Slow disk performance with dcaperfcheck

Symptoms:

• dcaperfcheck -d (disk) shows slow performance

Try This:

• Determine if PERC is in learn cycle:# omreport storage battery|grep 'Learn'

• Determine if PERC is in write-through mode due to a bad battery:# omreport storage vdisk| grep -e ID -e 'Write Through'

Common Issues 84


Unexpected return code from dca_hadoop

Symptoms:

• dca_hadoop start or stop displays the message Unexpected return code 255 controlling service

Try This:

• Exchange SSH keys through the DCA Setup utility.

Common Issues 85

greenplum dca administration guide - dell emc virtual ip for master servers ... synopsis ......

Documents