porting guide (apache) - huawei cloud

Kunpeng BoostKit for Big Data

Porting Guide (Apache)

Issue 07

Date 2021-07-13

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. i

Contents

1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 & openEuler 20.03)..................................... 11.1 Introduction............................................................................................................................................................................... 11.2 Environment Requirements................................................................................................................................................. 21.3 Configuring the Compilation Environment.................................................................................................................... 31.3.1 Installing Basic Libraries.................................................................................................................................................... 31.3.2 Installing OpenJDK.............................................................................................................................................................. 51.3.3 Installing Maven................................................................................................................................................................... 51.3.4 Installing CMake.................................................................................................................................................................. 61.3.5 Installing Protobuf............................................................................................................................................................... 71.4 Performing Porting Analysis................................................................................................................................................ 81.5 Compiling Dependencies...................................................................................................................................................... 81.6 Compiling Hadoop.................................................................................................................................................................. 91.7 Verification.............................................................................................................................................................................. 10

2 HBase 2.1.0 Porting Guide (CentOS 7.6 & openEuler 20.03)...................................... 122.1 Introduction............................................................................................................................................................................ 122.2 Environment Requirements............................................................................................................................................... 132.3 Configuring the Compilation Environment.................................................................................................................. 142.3.1 Installing Basic Libraries.................................................................................................................................................. 142.3.2 Installing OpenJDK............................................................................................................................................................162.3.3 Installing Maven................................................................................................................................................................ 162.3.4 Installing Protobuf.............................................................................................................................................................172.4 Performing Porting Analysis..............................................................................................................................................182.5 Compiling HBase................................................................................................................................................................... 182.6 Verifying HBase..................................................................................................................................................................... 192.7 Troubleshooting..................................................................................................................................................................... 192.7.1 mojo' findbugs' Fails to Be Loaded, Causing the Default findbugs-maven-plugin Fail to Be Executed............................................................................................................................................................................................................ 192.7.2 Failed to Locate protoc During the HBase Compilation Process...................................................................... 20

3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler 20.03)......................................... 213.1 Introduction............................................................................................................................................................................ 213.2 Environment Requirements............................................................................................................................................... 223.3 Configuring the Compilation Environment.................................................................................................................. 233.3.1 Installing Basic Libraries.................................................................................................................................................. 23

Kunpeng BoostKit for Big DataPorting Guide (Apache) Contents

Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. ii

3.3.2 Installing OpenJDK............................................................................................................................................................253.3.3 Installing Maven................................................................................................................................................................ 253.3.4 Installing Protobuf.............................................................................................................................................................263.4 Performing Porting Analysis..............................................................................................................................................273.5 Compiling Hive...................................................................................................................................................................... 273.6 Verifying Hive......................................................................................................................................................................... 293.7 Troubleshooting..................................................................................................................................................................... 29

4 Spark 2.3.0 Porting Guide (CentOS 7.6 & openEuler 20.03).......................................304.1 Introduction............................................................................................................................................................................ 304.2 Environment Requirements............................................................................................................................................... 314.3 Configuring the Compilation Environment.................................................................................................................. 324.3.1 Installing Basic Libraries.................................................................................................................................................. 324.3.2 Installing OpenJDK............................................................................................................................................................344.3.3 Installing Maven................................................................................................................................................................ 344.3.4 Installing the R Language...............................................................................................................................................354.4 Performing Porting Analysis..............................................................................................................................................364.5 Compiling Spark.................................................................................................................................................................... 364.6 Rectifying Faults.................................................................................................................................................................... 384.6.1 "Cannot find 'R_HOME'. Please specify 'R_HOME' or make sure R is properly installed" ReportedDuring the Spark Compilation.................................................................................................................................................384.6.2 "error: cannot compile a simple Fortran program" Reported During the R Language Compilation... 394.6.3 "configure: error: --with-x=yes (default) and X11 headers/libs are not available" Reported Duringthe R Language Compilation................................................................................................................................................... 394.6.4 "/usr/bin/install: cannot stat' NEWS.pdf': No such file or directory" Reported During the R LanguageCompilation.................................................................................................................................................................................... 40

5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 & openEuler 20.03)..............................415.1 Introduction............................................................................................................................................................................ 415.2 Environment Requirements............................................................................................................................................... 415.3 Configuring the Compilation Environment.................................................................................................................. 425.3.1 Installing Basic Libraries.................................................................................................................................................. 425.3.2 Installing OpenJDK............................................................................................................................................................445.3.3 Installing Maven................................................................................................................................................................ 455.4 Analyzing Software Porting...............................................................................................................................................465.5 Compiling ZooKeeper.......................................................................................................................................................... 46

6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler 20.03)........................................ 486.1 Introduction............................................................................................................................................................................ 486.2 Environment Requirements............................................................................................................................................... 486.3 Configuring the Compilation Environment.................................................................................................................. 496.3.1 Installing Basic Libraries.................................................................................................................................................. 496.3.2 Installing OpenJDK............................................................................................................................................................516.3.3 Installing Maven................................................................................................................................................................ 526.4 Analyzing Software Porting...............................................................................................................................................53


Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. iii

6.5 Compiling Flink...................................................................................................................................................................... 536.6 Troubleshooting..................................................................................................................................................................... 556.6.1 Flink Compilation Error................................................................................................................................................... 55

7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler 20.03)........................................ 577.1 Introduction............................................................................................................................................................................ 577.2 Environment Requirements............................................................................................................................................... 577.3 Configuring the Compilation Environment.................................................................................................................. 587.3.1 Installing Basic Libraries.................................................................................................................................................. 587.3.2 Installing Maven................................................................................................................................................................ 607.3.3 Installing OpenJDK............................................................................................................................................................617.4 Performing Porting Analysis..............................................................................................................................................627.5 Compilation.............................................................................................................................................................................62

8 Kafka 2.1.0 Porting Guide (CentOS 7.6 & openEuler 20.03).......................................668.1 Introduction............................................................................................................................................................................ 668.2 Environment Requirements............................................................................................................................................... 668.3 Configuring the Compilation Environment.................................................................................................................. 678.3.1 Installing Basic Libraries.................................................................................................................................................. 688.3.2 Installing OpenJDK............................................................................................................................................................708.3.3 Installing Gradle................................................................................................................................................................. 708.4 Performing Porting Analysis..............................................................................................................................................708.5 Compiling the Source Code............................................................................................................................................... 718.6 Verifying Kafka...................................................................................................................................................................... 72

9 Kafka 2.2.0 Porting Guide (CentOS 7.6 & openEuler 20.03).......................................739.1 Introduction............................................................................................................................................................................ 739.2 Environment Requirements............................................................................................................................................... 739.3 Configuring the Compilation Environment.................................................................................................................. 749.3.1 Installing Basic Libraries.................................................................................................................................................. 759.3.2 Installing OpenJDK............................................................................................................................................................779.3.3 Installing Gradle................................................................................................................................................................. 779.4 Performing Porting Analysis..............................................................................................................................................779.5 Compiling the Source Code............................................................................................................................................... 789.6 Verification.............................................................................................................................................................................. 79

10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 & openEuler 20.03)............................... 8010.1 Introduction.......................................................................................................................................................................... 8010.2 Environment Requirements.............................................................................................................................................8010.3 Configuring the Compilation Environment............................................................................................................... 8110.3.1 Installing Basic Libraries............................................................................................................................................... 8110.3.2 Installing OpenJDK......................................................................................................................................................... 8310.3.3 Installing Gradle.............................................................................................................................................................. 8410.4 Performing Porting Analysis........................................................................................................................................... 8410.5 Compiling the Source Code.............................................................................................................................................85


Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. iv

11 Druid 0.12.1 Porting Guide (CentOS 7.6 & openEuler 20.03)...................................8711.1 Introduction.......................................................................................................................................................................... 8711.2 Environment Requirements.............................................................................................................................................8711.3 Configuring the Compilation Environment............................................................................................................... 8811.3.1 Installing Basic Libraries............................................................................................................................................... 8811.3.2 Installing OpenJDK......................................................................................................................................................... 9011.3.3 Installing Maven.............................................................................................................................................................. 9111.3.4 Installing Protobuf.......................................................................................................................................................... 9211.4 Performing Porting Analysis........................................................................................................................................... 9311.5 Compiling Druid.................................................................................................................................................................. 9411.6 Troubleshooting.................................................................................................................................................................. 9511.6.1 Certificate Error Reported After git clone Is Executed...................................................................................... 9511.6.2 Failed to Verify the github.com Certificate Downloaded by Using wget....................................................9511.6.3 An Error Reported When Downloading the hadoop-lzo Dependency......................................................... 9511.6.4 An Error Reported When Downloading the libthrift Dependency................................................................. 96

12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 & openEuler 20.03).................................9712.1 Introduction.......................................................................................................................................................................... 9712.2 Environment Requirements.............................................................................................................................................9712.3 Configuring the Compilation Environment\.............................................................................................................. 9812.3.1 Installing Basic Libraries............................................................................................................................................... 9812.3.2 Installing OpenJDK....................................................................................................................................................... 10012.3.3 Installing Maven........................................................................................................................................................... 10112.4 Analyzing Software Porting.......................................................................................................................................... 10212.5 Compiling Phoenix........................................................................................................................................................... 103

A Change History....................................................................................................................105


Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. v

1 Hadoop 3.1.1 Porting Guide (CentOS 7.6& openEuler 20.03)

1.1 Introduction

1.2 Environment Requirements

1.3 Configuring the Compilation Environment

1.4 Performing Porting Analysis

1.5 Compiling Dependencies

1.6 Compiling Hadoop

1.7 Verification

1.1 Introduction

DescriptionHadoop is an open-source distributed storage and computing framework that iswidely used for massive data storage and processing. It can process data in areliable, efficient, and scalable manner.

For more information about Hadoop, visit https://hadoop.apache.org/docs/r3.1.1.

Programming language: Java

Brief description: distributed system infrastructure

Kunpeng BoostKit for Big DataPorting Guide (Apache)

1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)

Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 1

https://hadoop.apache.org/docs/r3.1.1

https://hadoop.apache.org/docs/r3.1.1


Hardware RequirementsItem Remarks

Server TaiShan server

CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor

Drive partition No requirement for drive partitions

Network Accessible to the Internet

Software requirementsItem Version

OpenJDK 1.8.0_252

Maven 3.5.4

Hadoop 3.1.1

CMake 3.12.4

Protobuf 2.5.0

CentOSItem Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5

openEulerItem Version

openEuler 20.03 LTS SP1

OS Kernel 4.19.90

GCC 7.3.0





1.3.1 Installing Basic Libraries

Installing GCC

Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop

NO TE

Replace YOUR_OS.iso with the actual iso package name.

Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*

NO TICE

Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.

Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo

Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0

Step 4 Make the Yum source configuration take effect.yum clean allyum makecache

Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64

Step 6 Resolve the -fsigned-char problem (by modifying the GCC).

1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc

2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl

3. Create a new GCC file.vi /usr/bin/gcc

Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"

4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc




5. Check whether the GCC is available.gcc --version

– CentOS: The installation is successful if information similar to thefollowing is displayed:

– openEuler: The installation is successful if information similar to thefollowing is displayed:

Step 7 Resolve the -fsigned-char problem (by modifying G++).

1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++

2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl

3. Create a new G++ file.vi /usr/bin/g++

Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"

4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++

5. Check whether G++ is available.g++ --version



----End

Installing Dependencies

Use the Yum source to install dependencies.




yum install -y wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel

1.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for

example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/

Step 2 Configure Java environment variables.vim /etc/profile

Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH

Step 3 Make the environment variables take effect.source /etc/profile

Step 4 Check whether OpenJDK is successfully installed.java -version

The installation is successful if information similar to the following is displayed:

----End

1.3.3 Installing MavenStep 1 Download the installation package and install Maven to a directory (for

example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/

Step 2 Modify the Maven environment variables.vim /etc/profile

Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH


Step 4 Check whether Maven is successfully installed.mvn -v





Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.

Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.

NO TE

The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.

Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):

<mirror> <id>huaweimaven</id> <name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>

If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:

<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>

----End

1.3.4 Installing CMakeThe CMake version must be 3.12 or later. This document uses CMake 3.12.4 as anexample.

CentOS

The built-in CMake version of CentOS is too early. To manually install CMake,perform the following steps:

Step 1 Download the CMake installation package.wget https://cmake.org/files/v3.12/cmake-3.12.4.tar.gz




Step 2 Decompress the installation package.tar -zxf cmake-3.12.4.tar.gz

Step 3 Compile and install CMake.cd cmake-3.12.4./bootstrapmake -j8make install

----End

openEuleryum install cmake

1.3.5 Installing Protobuf

CentOS

Step 1 Install Protobuf.yum install -y protobuf protobuf-devel

Step 2 Check whether Protobuf is installed successfully.protoc --version


Step 3 Install Maven.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc

----End

openEuler

Step 1 Download and decompress the source code.wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gztar -zxf protobuf-2.5.0.tar.gz

Step 2 Move the decompressed directory to the /opt/tools/installed/ directory.mv protobuf-2.5.0 /opt/tools/installed/

Step 3 Go to the /opt/tools/installed/ directory.cd /opt/tools/installed

Step 4 Download the protoc.zip package and decompress it to obtain the protoc.patchfile whose storage path can be specified, for example, to /opt/tools/installed/.wget https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/bigdata/Patch/protoc.zipunzip protoc.zipcp ./protoc/protoc.patch ./protobuf-2.5.0/src/google/protobuf/stubs/

Step 5 Go to the protobuf-2.5.0/src/google/protobuf/stubs/ directory and install thepatch.




https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/bigdata/Patch/protoc.zip

cd protobuf-2.5.0/src/google/protobuf/stubs/patch -p1 < protoc.patch

Step 6 Go back to the root directory of protobuf-2.5.0, compile the file, and install it inthe default directory.cd /opt/tools/installed/protobuf-2.5.0./autogen.sh && ./configure CFLAGS='-fsigned-char' && make -j8 && make install

Step 7 Deploy Protoc in the local Maven repository.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/local/bin/protoc

----End

1.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Hadoop installation package andobtain information about the third-party dependencies to be ported. For details,see Kunpeng Porting Advisor Case Study. Table 1-1 lists the third-partydependencies to be ported.

Table 1-1 Third-party dependencies to be ported

Original JAR File SO File

lz4-1.2.0.jar liblz4-java.so

netty-all-4.0.52.Final.jar libnetty_transport_native_epoll_x86_64.so

leveldbjni-all-1.8.jar libleveldbjni.so

snappy-java-1.0.5.jar libsnappy.so

NO TE

The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.

1.5 Compiling DependenciesStep 1 Use the Yum source to install the basic library.

yum install openssl-devel zlib-devel automake libtool

Step 2 (Optional) Compile Snappy.

Snappy is an optional compression library. You can add compilation options to theHadoop software package.

1. Download the Snappy dependency package.URL: https://github.com/google/snappy/releases/tag/1.1.7Decompress the package and go to the extracted folder.cd snappy-1.1.7




https://support.huaweicloud.com/intl/en-us/prtg-pa-kunpengbds/kunpengprtgusepa_02_0001.html

https://github.com/google/snappy/releases/tag/1.1.7

2. Modify the CMakeLists.txt file to enable the dynamic link library compilation.vim CMakeLists.txt

Change the value of this parameter from OFF to ON.option(BUILD_SHARED_LIBS "Build shared libraries(DLLs)." ON)

3. Create a compilation directory and perform compilation.mkdir buildcd buildcmake ../make install

----End

1.6 Compiling HadoopStep 1 Download the Hadoop source code package.

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz

Step 2 Decompress the source code package.tar -zxvf hadoop-3.1.1-src.tar.gz

Step 3 Go to the Hadoop directory decompressed.cd hadoop-3.1.1-src

Step 4 Modify the pom.xml file.vim pom.xml

Add the Kunpeng Maven repository in the first line of the repositories tag.

<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url> </repository>

In addition to the dependency repository source, add the plug-in repository source.The node levels of pluginRepositories and repositories are the same.

<pluginRepositories> <pluginRepository> <id>huaweicloud-plugin</id> <url>http://mirrors.huaweicloud.com/repository/maven</url> <snapshots> <enabled>true</enabled> </snapshots> </pluginRepository></pluginRepositories>




Step 5 Run the compilation command.mvn package -DskipTests -Pdist,native -Dtar -Dmaven.javadoc.skip=true

Step 6 (Optional) Add the Snappy repository compilation command.mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib64 -Dbundle.snappy -Dmaven.javadoc.skip=true

NO TE

If you have followed 1.5 Compiling Dependencies to install the Snappy dependencypackage, run the compilation commands above.

After the compilation is successful, the tar.gz package is generated in the hadoop-dist/target/ directory of the source code.

Step 7 Use the Kunpeng Porting Advisor to scan the package generated after thecompilation and ensure that the package does not contain x86 .so or .jarpackages.

NO TE

The compiled hadoop-3.1.1.tar.gz must be scanned by using the Kunpeng Porting Advisorto ensure that no x86 .so or .jar packages are contained. If the compiled directory containsx86 .so or .jar packages, the component functions may be affected.

For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.

----End

1.7 VerificationStep 1 For details about how to install components, see the documents at the official

website. There are two installation modes: Single-node installation and clusterinstallation.● Single-node installation: https://hadoop.apache.org/docs/r3.1.1/hadoop-

project-dist/hadoop-common/SingleCluster.html● Cluster installation: https://hadoop.apache.org/docs/r3.1.1/hadoop-project-

dist/hadoop-common/ClusterSetup.html






https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/SingleCluster.html

https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/SingleCluster.html

https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/ClusterSetup.html

https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/ClusterSetup.html

Step 2 Use the test program of Hadoop to calculate the PI value../hadoop-3.1.1/bin/yarn jar hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar pi 5 10

----End




2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)

2.1 Introduction




2.5 Compiling HBase

2.6 Verifying HBase

2.7 Troubleshooting

2.1 Introduction

HBase OverviewHBase (short for Hadoop Database) is a distributed storage system that iscolumn-based and scalable with high reliability and excellent performance. Withthe HBase technology, a large-scale storage cluster can be set up on low-cost PCservers, allowing random and quick access of massive data.

For more information about HBase, see https://hbase.apache.org.


Brief description: distributed database




https://hbase.apache.org


Hardware Requirements

Item Remarks





Software requirements

Item Version

OpenJDK 1.8.0_252

Maven 3.5.4

HBase 2.1.0

Protobuf 2.5.0

CentOS

Item Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5

openEuler

Item Version


OS Kernel 4.19.90

GCC 7.3.0






Installing GCC


NO TE



NO TICE



























----End






yum install -y wget vim openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel








----End













NO TE






----End

2.3.4 Installing ProtobufStep 1 Install Protobuf using the Yum source.

yum install -y protobuf protobuf-devel

Step 2 Specify the executable files of Protoc.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc

----End




2.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the HBase installation package andobtain information about the third-party dependencies to be ported. For details,see Kunpeng Porting Advisor Case Study. Table 2-1 lists the third-partydependencies to be ported.


Original JAR Package SO File

commons-crypto-1.0.0.jar libcommons-crypto.so

netty-all-4.0.23.Final.jar libnetty-transport-native-epoll.so


snappy-java-1.0.5.jar libsnappyjava.so

hbase-shaded-netty-2.1.0.jar liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so

NO TE


2.5 Compiling HBaseStep 1 Download the HBase installation package.

wget http://archive.apache.org/dist/hbase/2.1.0/hbase-2.1.0-src.tar.gz

Step 2 Decompress the package.tar -zxf hbase-2.1.0-src.tar.gz

Step 3 Go to the directory generated after the decompression and add the HuaweiKunpeng repository to the pom.xml file.cd hbase-2.1.0


Add the Kunpeng Maven repository in the first line of the repositories tag.<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>





Step 5 Perform compilation.mvn clean package -DskipTests assembly:single

After the compilation is successful, a tar.gz package is generated in the hbase-assembly/target/ directory of the source code.


NO TE

The compiled HBase package must be scanned by using the Kunpeng Porting Advisor toensure that no x86 .so or .jar packages are contained. If the compiled directory containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.

----End

2.6 Verifying HBaseDeploy an HBase cluster (see the HBase Cluster Deployment Guide (CentOS7.6) and verify HBase.

2.7 Troubleshooting

2.7.1 mojo' findbugs' Fails to Be Loaded, Causing the Defaultfindbugs-maven-plugin Fail to Be Executed

SymptomExecution of the default findbugs-maven-plugin fails because the mojo 'findbugs'cannot be loaded.

SolutionChange the findbugs-maven-plugin version to 3.0.4.






https://support.huaweicloud.com/intl/en-us/dpmg-apache-kunpengbds/kunpengbds_04_0006.html


2.7.2 Failed to Locate protoc During the HBase CompilationProcess

Symptomprotobuf 2.5.0 has been installed. However, a message is displayed during theHBase compilation process indicating that protoc is missing.

SolutionRun the following command to specify the protoc executable file installed:

mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc




3 Hive 3.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)

3.1 Introduction




3.5 Compiling Hive

3.6 Verifying Hive

3.7 Troubleshooting

3.1 Introduction

Hive OverviewHive is a data warehouse tool running on Hadoop. It maps structured data files toa database table, provides simple SQL search functions, and converts SQLstatements into MapReduce tasks. It features low learning cost. You can use SQL-like statements to quickly implement simple MapReduce statistics and do notneed to develop dedicated MapReduce applications. Therefore, Hive is verysuitable for statistical analysis of data warehouse.

For more information about Hive, visit https://hive.apache.org/.


Brief description: Hadoop-based data warehouse tool


3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)


https://hive.apache.org/







Software RequirementsItem Version

OpenJDK 1.8.0_252

Maven 3.5.4

Hive 3.0.0

CentOSItem Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5



OS Kernel 4.19.90

GCC 7.3.0






Installing GCC


NO TE



NO TICE



























----End








NO TE






----End

3.3.4 Installing ProtobufStep 1 Install Protobuf using the Yum source.

yum install -y protobuf protobuf-devel

Step 2 Specify the executable files of Protoc.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc

----End




3.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Hive installation package. For details,see Kunpeng Porting Advisor Case Study. Table 3-1 lists the third-partydependencies to be ported.





jline-2.12.jar libjansi.so

server-metrics-0.2.8.jar libsigar.so


hbase-shaded-netty-1.0.1.jar liborg-apache-hadoop-hbase-shaded-netty-transport-native-epoll.so

NO TE


3.5 Compiling HiveStep 1 Download the Hive installation package.

wget https://github.com/apache/hive/archive/rel/release-3.0.0.tar.gz

Step 2 Decompress the installation package.mv release-3.0.0.tar.gz hive-rel-release-3.0.0.tar.gztar -zxf hive-rel-release-3.0.0.tar.gz

Step 3 Switch to the directory where the source code package is decompressed.cd hive-rel-release-3.0.0


Add the Kunpeng Maven repository source at the first place under therepositories tag.<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>





vim jdbc/pom.xml

Add the Kunpeng Maven repository source.

<repositories><repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository></repositories>

vim druid-handler/pom.xml

Add the Kunpeng Maven repository source.


Step 5 Perform compilation.mvn clean install -Pdist -DskipTests

After the compilation is successful, the tar.gz package for Hive is generated in thepackaging/target directory of the source code.





NO TE

The compiled Hive package must be scanned by using the Kunpeng Porting Advisor toensure that no x86 .so or .jar packages are contained. If the compiled directory containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.

----End

3.6 Verifying HiveDeploy a Hive cluster (see the Hive Deployment Guide (CentOS 7.6)) and verifyHive.

3.7 Troubleshooting

Symptomprotobuf-2.5.0 has been installed. However, a message is displayed during the Hivecompilation process indicating that protoc is missing.

ProcedureRun the following command to specify the protoc executable file installed:

mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc






4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)

4.1 Introduction




4.5 Compiling Spark

4.6 Rectifying Faults

4.1 Introduction

DescriptionSpark is a unified analysis engine used for large-scale data processing. It featuresscalability and memory-based computing and has become a unified platform forquick processing of lightweight big data. Spark can be used to build the data storeand running system for various applications, such as real-time stream processing,machine learning, and interactive query.

For more information about Spark, visit https://spark.apache.org/docs/latest/building-spark.html.

Programming language: Scala

Brief description: large-scale data computing engine




https://spark.apache.org/docs/latest/building-spark.html

https://spark.apache.org/docs/latest/building-spark.html



Item Remarks





Software Requirements

Item Version

OpenJDK 1.8.0_252

Maven 3.5.4

R 3.1.1

Spark 2.3.0

CentOS

Item Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5

openEuler

Item Version


OS Kernel 4.19.90

GCC 7.3.0






Installing GCC


NO TE



NO TICE



























----End








NO TE






----End

4.3.4 Installing the R LanguageStep 1 Download the R language installation package.

wget http://cran.rstudio.com/src/base/R-3/R-3.1.1.tar.gz

Step 2 Decompress the package.tar -zxf R-3.1.1.tar.gz

Step 3 Install dependencies.yum -y install libXt-devel.aarch64 readline-devel.aarch64

Step 4 Go to the directory generated after the decompression.cd R-3.1.1




Step 5 Perform compilation../configure --enable-R-shlib --enable-R-static-lib --with-libpng --with-jpeglib --prefix=/opt/tools/installed/R-3.1.1make all -j10 && make install

NO TE

For details about how to rectify faults that occur during the compilation of the R language,see 4.6.2 "error: cannot compile a simple Fortran program" Reported During the RLanguage Compilation, 4.6.3 "configure: error: --with-x=yes (default) and X11headers/libs are not available" Reported During the R Language Compilation, and4.6.4 "/usr/bin/install: cannot stat' NEWS.pdf': No such file or directory" ReportedDuring the R Language Compilation.

Step 6 Configure the R language environment variables.export R_HOME=/opt/tools/installed/R-3.1.1export PATH=$R_HOME/bin:$PATH

----End

4.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Spark installation package byreferring to Kunpeng Porting Advisor Case Study. Table 4-1 lists the third-partydependencies to be ported.

Table 4-1 Software porting analysis result



jline-2.12.1.jar libjansi.so



scala-compiler-2.11.12.jar libjansi.so

snappy-java-1.1.2.6.jar libsnappyjava.so

lz4-java-1.4.0.jar liblz4-java.so

NO TE


4.5 Compiling SparkStep 1 Download the Spark installation package.

wget https://github.com/apache/spark/archive/v2.3.0.tar.gz





Step 2 Decompress the package.tar -zxf v2.3.0.tar.gz

Step 3 Go to the directory generated after the decompression.cd spark-2.3.0



Add Huawei Maven source to the first line of the pluginRepository tag.

<pluginRepository> <id>huaweicloud-plugin</id> <url>https://mirrors.huaweicloud.com/repository/maven</url></pluginRepository>

Step 5 Perform compilation.dev/make-distribution.sh --tgz -Pyarn,hive,hive-thriftserver,sparkr

After the compilation is complete, a compiled package is generated in the sourcecode directory.




NO TE

For details about how to rectify faults that occur during the Spark compilation, see 4.6.1"Cannot find 'R_HOME'. Please specify 'R_HOME' or make sure R is properly installed"Reported During the Spark Compilation.


NO TE

The compiled spark-2.3.0-bin-2.6.5.tgz must be scanned by using the Kunpeng PortingAdvisor to ensure that no x86 .so or .jar packages are contained. If the compiled directorycontains x86 .so or .jar packages, the component functions may be affected.


----End

4.6 Rectifying Faults

4.6.1 "Cannot find 'R_HOME'. Please specify 'R_HOME' ormake sure R is properly installed" Reported During the SparkCompilation

Symptom

During the Spark compilation, "Cannot find 'R_HOME'. Please specify 'R_HOME' ormake sure R is properly installed" is displayed.

Cause Analysis

The R language support is enabled during Spark compilation. You need to compileand install the R language in the environment.






Solution

Step 1 Compile and install the R language in the /opt/tools/installed directory, and setthe environment variables.

For details, see 4.3.4 Installing the R Language.

Step 2 Run the Spark compilation command again.

----End

4.6.2 "error: cannot compile a simple Fortran program"Reported During the R Language Compilation

Symptom

During the R language compilation, "error: cannot compile a simple Fortranprogram" is displayed.

Cause Analysis

The gfortran package does not exist in the system.

Solution

Run the yum command to install the gfortran package in the OS image.

yum -y install gcc-gfortran.aarch64

4.6.3 "configure: error: --with-x=yes (default) and X11headers/libs are not available" Reported During the RLanguage Compilation

Symptom

During the R language compilation, "configure: error: --with-x=yes (default) andX11 headers/libs are not available" is displayed.




Cause Analysis--with-x=yes (use the X Window System) is enabled by default. Therefore, youneed to install the libXt-devel module.

SolutionRun the yum command to install the module in the OS image.

yum -y install libXt-devel.aarch64

4.6.4 "/usr/bin/install: cannot stat' NEWS.pdf': No such file ordirectory" Reported During the R Language Compilation

SymptomDuring the R language compilation, "/usr/bin/install: cannot stat' NEWS.pdf': Nosuch file or directory" is displayed.

Cause AnalysisThe NEWS.pdf file is not found in the source code directory R-3.1.1. Therefore, thefile cannot be copied when the make install command is executed.

SolutionThe NEWS text file exists in the doc directory. Therefore, copy the contents of theNEWS file in the doc directory to the NEWS.pdf file.

cat doc/NEWS > doc/NEWS.pdf




5 ZooKeeper 3.4.6 Porting Guide (CentOS7.6 & openEuler 20.03)

5.1 Introduction



5.4 Analyzing Software Porting

5.5 Compiling ZooKeeper

5.1 IntroductionZooKeeper is a distributed and open-source coordination service for distributedapplications, an open-source implementation of Google Chubby, and an importantcomponent of Hadoop and HBase. ZooKeeper provides alignment services fordistributed applications, including configuration maintenance, domain naming,distributed synchronization, and group services.

For more information about ZooKeeper, visit https://zookeeper.apache.org/.








5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 &openEuler 20.03)


https://zookeeper.apache.org/


Item Version

OpenJDK 1.8.0_252

Maven 3.5.4

CMake 3.12.4

CentOS

Item Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5

openEuler

Item Version


OS Kernel 4.19.90

GCC 7.3.0



Installing GCC


NO TE






NO TICE
















1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.




command -v g++








----End


Install dependencies using the Yum source.

yum install -y wget vim openssl-devel zlib-devel autoconf automake libtool ant svn make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel











----End










NO TE



<mirror> <id>huaweimaven</id>




<name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>



----End

5.4 Analyzing Software PortingUse the Kunpeng Porting Advisor to scan the ZooKeeper installation package byreferring to Kunpeng Porting Advisor Case Study. No third-party dependencyneeds to be ported.

5.5 Compiling ZooKeeperStep 1 Download the zookeeper-release-3.4.6 source code package from GitHub and

decompress it.wget https://github.com/apache/zookeeper/archive/release-3.4.6.tar.gztar -zxf release-3.4.6.tar.gz

Step 2 Go to the directory decompressed and install the dependency.cd zookeeper-release-3.4.6yum install cppunit-devel -y

Step 3 The build.xml and ivysettings.xml files need to be modified due to network proxyproblems.

1. Manually download the ant-eclipse-1.0.bin.tar.bz2 file from http://downloads.sourceforge.net/project/ant-eclipse/ant-eclipse/1.0/ant-eclipse-1.0.bin.tar.bz2, copy it to the ${src.dir}/java/ path, and comment outthe content of download and copy operations in the build.xml file.

2. Change the value of repo.maven.org in ivysettings.xml to https://mirrors.huaweicloud.com/repository/maven/.





3. Modify the repo URL in the build.xml file.sed -i 's/http:\/\/repo2/https:\/\/repo1/' build.xml

Step 4 Perform compilation.ant package tar

After the compilation is successful, the zookeeper-3.4.6.tar.gz package isgenerated in the ./build/ directory of the source code directory.


NO TE

The compiled zookeeper-3.4.6.tar.gz must be scanned by using the Kunpeng PortingAdvisor to ensure that no x86 .so or .jar packages are contained. If the compiled directorycontains x86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.

----End






6 Flink 1.8.1 Porting Guide (CentOS 7.6 &openEuler 20.03)

6.1 Introduction




6.5 Compiling Flink

6.6 Troubleshooting

6.1 IntroductionFlink is a distributed, high-performance, and high-availability open-source big dataprocessing framework. It features in-memory computing and stream processingand is used for stateful computing of unbounded and bounded data streams.








6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)



Item Version

OpenJDK 1.8.0_252

Maven 3.5.4

Flink 1.8.1

CentOS

Item Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5

openEuler

Item Version


OS Kernel 4.19.90

GCC 7.3.0



Installing GCC


NO TE






NO TICE




















command -v g++








----End














----End










NO TE



<mirror> <id>huaweimaven</id>




<name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>



----End

6.4 Analyzing Software PortingUse the Kunpeng Porting Advisor to scan the Flink installation package and obtaininformation about the third-party dependencies to be ported. For details, seeKunpeng Porting Advisor Case Study. Table 6-1 lists the third-partydependencies to be ported.



jython-standalone-2.7.1.jar libjansi.so


flink-shaded-netty-4.1.32.Final-6.0.jar

liborg_apache_flink_shaded_netty4_netty_transport_native_epoll_x86_64.so

frocksdbjni-5.17.2-artisans-1.0.jar

librocksdbjni32.so, librocksdbjni64.so,librocksdbjnile.so

NO TE


6.5 Compiling FlinkStep 1 Download the flink-release-1.8.1 source code package from GitHub and

decompress it.wget https://github.com/apache/flink/archive/release-1.8.1.tar.gztar -zxf release-1.8.1.tar.gz





Step 2 Go to the Flink directory decompressed.cd flink-release-1.8.1


Add the Kunpeng Maven repository in the first line of the repositories tag.

<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>

Modify the flink-dist/pom.xml file.

vim flink-dist/pom.xml

Add the following code to line 653 to add the Kunpeng Maven repository source:


Modify the flink-libraries/pom.xml file.

vim flink-libraries/pom.xml

Add the following code to line 69 to add the Kunpeng Maven repository source:





Step 4 Perform compilation.mvn clean install -DskipTests -Dhadoop.version=2.7.0 -Dscala-2.12 -Dpython2.12

After the compilation is complete, the directory flink-1.8.1 is generated in flink-release-1.8.1/flink-dist/target/flink-1.8.1-bin/.

Step 5 Use the Kunpeng Porting Advisor to scan the .tar package generated after thecompilation and ensure that the .tar package does not contain the x86 .so or .jarpackages.

NO TE

The compiled flink package must be scanned by using the Kunpeng Porting Advisor toensure that no x86 .so or .jar packages are contained. If the compiled package containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.

----End

6.6 Troubleshooting

6.6.1 Flink Compilation Error

SymptomThe following error message is reported during Flink compilation: Failed toexecute goal on project flink-avro-confluent-registry: Could not resolvedependencies for project org.apache.flink:flink-avro-confluent-registry:jar:1.8.1:Could not find artifact io.confluent:kafka-schema-registry-client:jar:3.3.1 in big(http://10.93.238.51/maven/)

Cause AnalysisThe dependent third-party .jar package kafka-schema-registry-client-3.3.1.jar ismissing.






SolutionDownload the available kafka-schema-registry-client-3.3.1.jar package to theMaven repository or the */.m2/repository/io/confluent/kafka-schema-registry-client/3.3.1 directory of the local cache library, and compile Flink again.




7 Flink 1.9.1 Porting Guide (CentOS 7.6 &openEuler 20.03)

7.1 Introduction




7.5 Compilation

7.1 IntroductionFlink is a distributed, high-performance, and high-availability open-source big dataprocessing framework. It features in-memory computing and stream processingand is used for stateful computing of unbounded and bounded data streams.











Item Version

OpenJDK 1.8.0_252

Maven 3.5.4

Flink 1.9.1

CentOS

Item Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5

openEuler

Item Version


OS Kernel 4.19.90

GCC 7.3.0



Installing GCC


NO TE






NO TICE




















command -v g++








----End

Installing DependenciesUse the Yum source to install dependencies.














NO TE






----End











----End

7.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Flink installation package and obtaininformation about the third-party dependencies to be ported. For details, see theKunpeng Porting Advisor Case Study. Table 7-1 lists the third-partydependencies to be ported.




flink-shaded-netty-4.1.32.Final-7.0.jar

liborg_apache_flink_shaded_netty4_netty_transport_native_epoll_x86_64.so

frocksdbjni-5.17.2-artisans-1.0.jar

librocksdbjni32.so,librocksdbjni64.so,librocksdbjnile.so

flink-shaded-netty-tcnative-dynamic-2.0.25.Final-7.0. jar

liborg_apache_flink_shaded_netty4_netty_tcnative_linux_x86_64.so,liborg_apache_flink_shaded_netty4_netty_tcnative_linux_x86_64_fedora.so

NO TE


7.5 CompilationStep 1 Download the flink-release-1.9.1 source code package and decompress it.

wget https://github.com/apache/flink/archive/release-1.9.1.tar.gztar -zxf release-1.9.1.tar.gz





Step 2 Go to the directory generated after the decompression.cd flink-release-1.9.1

Step 3 Add the reference to the Kunpeng repository to the flink-release-1.9.1/pom.xmlfile.vim pom.xml

Add the Kunpeng Maven repository in the first line under repositories.<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>

Step 4 Add the reference to the Kunpeng repository to the flink-release-1.9.1/flink-dist/pom.xml file.vim flink-dist/pom.xml

Add the Kunpeng Maven repository in the first line under repositories.

<repositories> <repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url> </repository></repositories>

Step 5 Add the reference of the Kunpeng repository to the flink-release-1.9.1/flink-state-backends/flink-statebackend-rocksdb/pom.xml file.vim flink-state-backends/flink-statebackend-rocksdb/pom.xml






Step 6 Add the reference to the Kunpeng repository to the flink-runtime/pom.xml file.vim flink-runtime/pom.xml



Step 7 Perform compilation in the flink-release-1.9.1 directory.mvn install -DskipTests -Dtar -Dhadoop.version=2.8.3

The following figure shows that the compilation is successful.




After the compilation is complete, the directory flink-1.9.1 is generated in flink-release-1.9.1/flink-dist/target/flink-1.9.1-bin/.


NO TE

The compiled Flink package must be scanned by using the Kunpeng Porting Advisor toensure that no x86 .so or .jar packages are contained. If the compiled directory containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see the Kunpeng PortingAdvisor Case Study.

----End






8 Kafka 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)

8.1 Introduction




8.5 Compiling the Source Code

8.6 Verifying Kafka

8.1 Introduction

Kafka OverviewKafka is an open-source streaming platform developed by Apache SoftwareFoundation in Scala and Java. Kafka is a distributed publish-subscribe messagingsystem with high throughput. It can process all action flow data on the customer-related websites.

For more information about Kafka, visit https://kafka.apache.org/.

Category: application







https://kafka.apache.org/

Item Remarks




Software requirementsItem Version

OpenJDK 1.8.0_252

Maven 3.5.4

Gradle 4.10

Kafka 2.1.0

CentOSItem Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5



OS Kernel 4.19.90

GCC 7.3.0






Installing GCC


NO TE



NO TICE



























----End


Use Yum to install dependencies.

yum install -y wget unzip openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel











----End

8.3.3 Installing GradleStep 1 Download and decompress the Gradle source code package to a directory (for

example, /opt/tools/installed/).wget https://downloads.gradle.org/distributions/gradle-4.10-bin.zip --no-check-certificateunzip gradle-4.10-bin.zipmv gradle-4.10 /opt/tools/installed/

Step 2 Modify the gradle environment variables.vim /etc/profile

Add the following code at the end of the /etc/profile file:export GRADLE_HOME=/opt/tools/installed/gradle-4.10export PATH=$GRADLE_HOME/bin:$PATH

Step 3 Press Esc and run :wq to save the configuration and exit.


----End

8.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Kafka installation package andobtain information about the third-party dependencies to be ported. For details,




see Kunpeng Porting Advisor Case Study. Table 8-1 lists the third-partydependencies to be ported.



rocksdbjni-5.14.2.jar librocksdbjni32.so


rocksdbjni-5.14.2.jar librocksdbjnile.so

NO TE


8.5 Compiling the Source CodeStep 1 Download the kafka-2.1.0 source code package.

Download address: https://archive.apache.org/dist/kafka/2.1.0/kafka-2.1.0-src.tgz

Step 2 Decompress the source code package.tar -zxvf kafka-2.1.0-src.tgz

Step 3 Go to the directory where the source code is located.cd kafka-2.1.0-src

Step 4 Modify the build.gradle file.vim build.gradle

Add multiple Maven repositories under the repositories section of the file.allprojects { repositories { maven { url "https://mirrors.huaweicloud.com/kunpeng/maven" } maven { url "https://mirrors.huaweicloud.com/repository/maven" } maven { url "https://plugins.gradle.org/m2/" } mavenCentral() }

Step 5 Perform compilation.gradle -g /$UserHome/gradleRepository releaseTarGz -info

The compiled package is stored in the following directory:





https://archive.apache.org/dist/kafka/2.1.0/kafka-2.1.0-src.tgz


NO TE

If connection timeout 443 error occurs during compilation, run the compilation commandagain.


NO TE

The compiled kafka_2.11-2.1.0.tgz must be scanned by using the Kunpeng Porting Advisorto ensure that no x86 .so or .jar packages are contained. If the compiled package containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.

----End

8.6 Verifying KafkaDeploy Kafka (see Kafka Deployment Guide (CentOS 7.6)) and performverification.








9.1 Introduction





9.6 Verification

9.1 Introduction

Kafka OverviewKafka is an open-source streaming platform developed by Apache SoftwareFoundation in Scala and Java. Kafka is a distributed publish-subscribe messagingsystem with high throughput. It can process all action flow data on the customer-related websites.










Item Remarks




Software RequirementsItem Version

OpenJDK 1.8.0_252

Maven 3.5.4

Gradle 4.10

Kafka 2.2.0

CentOSItem Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5



OS Kernel 4.19.90

GCC 7.3.0






Installing GCC


NO TE



NO TICE



























----End














----End







----End

9.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Kafka installation package andobtain information about the third-party dependencies to be ported. For details,




see the Kunpeng Porting Advisor Case Study. Table 9-1 lists the third-partydependencies to be ported.






NO TE


9.5 Compiling the Source CodeStep 1 Download the kafka-2.2.0 source code package.

URL: https://archive.apache.org/dist/kafka/2.2.0/kafka-2.2.0-src.tgz

Step 2 Decompress the source code package.tar -zxvf kafka-2.2.0-src.tgz

Step 3 Go to the directory where the source code is located.cd kafka-2.2.0-src


Add multiple Maven repositories under the repositories tag, as shown in thefollowing figure.allprojects { repositories { maven { url "https://mirrors.huaweicloud.com/kunpeng/maven" } maven { url "https://mirrors.huaweicloud.com/repository/maven" } maven { url "https://plugins.gradle.org/m2/" } mavenCentral() }

Step 5 Compile the source code.gradle -g /$UserHome/gradleRepository releaseTarGz -info

NO TE

gradleRepository is the local repository directory of Gradle. You need to manually createthis directory.mkdir -p /$UserHome/gradleRepository







Step 6 Use the Kunpeng Code Scanner to scan the .tar package generated aftercompilation and ensure that the .tar package contains no x86 .so or .jar packages.

NO TE

Use the Kunpeng Code Scanner to scan the compiled kafka_2.12-2.2.0.tgz package andensure that the .so or .jar packages of x86 are not contained. If the compiled directorycontains x86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Code Scanner, see Kunpeng Porting AdvisorCase Study.

----End

9.6 VerificationDeploy Kafka (see the Kafka Deployment Guide (CentOS 7.6)) and verify theporting result.







10 Kafka 0.10.1.1 Porting Guide(CentOS 7.6 & openEuler 20.03)

10.1 Introduction





10.1 Introduction

Kafka Overview

Kafka is an open-source streaming platform developed by Apache SoftwareFoundation in Scala and Java. Kafka is a distributed publish-subscribe messagingsystem with high throughput. It can process all action flow data on the customer-related websites.





Item Remarks




10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)



Item Remarks




Item Version

OpenJDK 1.8.0_252

Maven 3.5.4

Gradle 4.10

Kafka 0.10.1.1

CentOS

Item Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5

openEuler

Item Version


OS Kernel 4.19.90

GCC 7.3.0



Installing GCC





NO TE



NO TICE



























----End


Use Yum to install dependencies.

yum install -y wget unzip openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel











----End







----End

10.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Kafka installation package andobtain information about the third-party dependencies to be ported. For details,see Kunpeng Porting Advisor Case Study. Table 10-1 lists the third-partydependencies to be ported.











NO TE


10.5 Compiling the Source CodeStep 1 Download the kafka-0.10.1.1 source code package.

wget https://archive.apache.org/dist/kafka/0.10.1.1/kafka-0.10.1.1-src.tgz

Step 2 Decompress the source code package.tar -zxvf kafka-0.10.1.1-src.tgz

Step 3 Go to the directory where the source code is located.cd kafka-0.10.1.1-src


Add multiple Maven repositories under the repositories section of the file.allprojects { apply plugin: 'idea' apply plugin: 'eclipse' apply plugin: 'jacoco'

repositories { maven { url "https://mirrors.huaweicloud.com/kunpeng/maven" } maven { url "https://mirrors.huaweicloud.com/repository/maven" } maven { url "https://plugins.gradle.org/m2/" } mavenCentral() }

Step 5 Perform compilation.gradle -g /$UserHome/gradleRepository releaseTarGz -info





NO TE

If connection timeout 443 error occurs during compilation, run the compilation commandagain.


NO TE

The compiled kafka_2.10-0.10.1.1.tgz must be scanned by using the Kunpeng PortingAdvisor to ensure that no x86 .so or .jar packages are contained. If the compiled packagecontains x86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.

----End






11 Druid 0.12.1 Porting Guide (CentOS7.6 & openEuler 20.03)

11.1 Introduction




11.5 Compiling Druid

11.6 Troubleshooting

11.1 IntroductionDruid is an efficient data query system that is used to aggregate and query a largeamount of time-series data. Data can be ingested in real time and can be queriedimmediately after being ingested to the Druid. In addition, data is almostimmutable. After a time-series event occurs and is ingested to Druid, an externalsystem can query the event.

For more information about Druid, visit https://github.com/apache/druid.







11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)


https://github.com/apache/druid

Item Remarks



Item Version

OpenJDK 1.8.0_252

Maven 3.5.4

CMake 3.12.4

CentOS

Item Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5

openEuler

Item Version


OS Kernel 4.19.90

GCC 7.3.0



Installing GCC


NO TE






NO TICE



























----End



yum install -y wget unzip vim











----End










NO TE









----End

11.3.4 Installing Protobuf

CentOS

Step 1 Install Protobuf.yum install -y protobuf protobuf-devel

Step 2 Check whether Protobuf is installed successfully.protoc --version


Step 3 Install Maven.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc

----End

openEuler

Step 1 Download and decompress the source code.wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gztar -zxf protobuf-2.5.0.tar.gz

Step 2 Move the decompressed directory to the /opt/tools/installed/ directory.mv protobuf-2.5.0 /opt/tools/installed/

Step 3 Go to the /opt/tools/installed/ directory.cd /opt/tools/installed




Step 4 Download the protoc.zip package and decompress it to obtain the protoc.patchfile whose storage path can be specified, for example, to /opt/tools/installed/.wget https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/bigdata/Patch/protoc.zipunzip protoc.zipcp ./protoc/protoc.patch ./protobuf-2.5.0/src/google/protobuf/stubs/

Step 5 Go to the protobuf-2.5.0/src/google/protobuf/stubs/ directory and install thepatch.cd protobuf-2.5.0/src/google/protobuf/stubs/patch -p1 < protoc.patch

Step 6 Go back to the root directory of protobuf-2.5.0, compile the file, and install it inthe default directory.cd /opt/tools/installed/protobuf-2.5.0./autogen.sh && ./configure CFLAGS='-fsigned-char' && make -j8 && make install

Step 7 Deploy Protoc in the local Maven repository.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/local/bin/protoc

----End

11.4 Performing Porting AnalysisScan the Druid installation package by referring to Kunpeng Porting AdvisorCase Study and obtain the third-party dependencies to be ported. Table 11-1 liststhe third-party dependencies to be ported.


Original .jar Package .so File

java-util-0.12.1.jar libsigar.so





netty-transport-native-epoll-4.0.52.Final-linux-x86_64.jar

libnetty_transport_native_epoll_x86_64.so




NO TE





https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/bigdata/Patch/protoc.zip



11.5 Compiling DruidStep 1 Download the source code and decompress it.

wget https://github.com/apache/druid/archive/druid-0.12.1.tar.gztar -xvf druid-0.12.1.tar.gzmv druid-druid-0.12.1 druid-0.12.1

Step 2 Switch to the directory where the Druid source code package is decompressed.cd druid-0.12.1


Add the Kunpeng Maven repository source at the beginning of line 1295 in thefile.


Step 4 Modify the java-util/pom.xml file.vim java-util/pom.xml

Change the download path of sigar-dist-1.6.5.132.zip from line 271 in the file.

<url>http://porting-dependence.obs.cn-north-4.myhuaweicloud.com/ARTIFACTS/</url>

Step 5 Perform compilation.mvn package -DskipTests

Obtain the .tar.gz package generated in the distribution/target/ directory.

NO TE

The compiled package druid-0.12.1-bin.tar.gz must be scanned by using the KunpengPorting Advisor to ensure that no x86 .so or .jar packages are contained. If the compileddirectory contains x86 .so or .jar packages, the component functions may be affected.


----End

11.6 Troubleshooting

11.6.1 Certificate Error Reported After git clone Is Executed

Symptom

In the compilation process, "fatal:unable to access 'https://*******': Peer's Certificateissuer is not recognized." is displayed.

Procedure

Before running the git clone command, run the following command:

git config --global http.sslVerify false

11.6.2 Failed to Verify the github.com Certificate Downloadedby Using wget

Symptom

During compilation, an error is reported, indicating that the certificate issued bygithub.com cannot be verified. The permission of the issuer cannot be verifiedlocally.

Procedure

To connect to github.com in an insecure manner, specify the --no-check-certificate parameter in the wget command.

11.6.3 An Error Reported When Downloading the hadoop-lzoDependency

Symptom

The following error is reported during compilation:






[ERROR] Failed to execute goal on project druid-thrift-extensions: Could not resolve dependencies for project io.druid.extensions.contribto collect dependencies at com.twitter.elephantbird:elephant-bird-core:jar:4.8 -> com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19: Failedgplcompression:hadoop-lzo:jar:0.4.19: Could not transfer artifact com.hadoop.gplcompression:hadoop-lzo:pom:0.4.19 from/to twitter (http:/http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/hadoop-lzo-0.4.19.pom. Return code is: 502, ReasonPhrase:badgateway.)

Procedure

Modify the local Maven repository file ./repository/com/twitter/elephantbird/elephant-bird/4.8/elephant-bird-4.8.pom.

The code after modification is as follows:

<repositories><repository><id>twitter</id><url>https://nexus.xebialabs.com/nexus/content/groups/public/</url></repository></repositories><pluginRepositories><pluginRepository><id>twitter</id><url>https://nexus.xebialabs.com/nexus/content/groups/public/</url></pluginRepository></pluginRepositories>

11.6.4 An Error Reported When Downloading the libthriftDependency

Symptom

The following error is reported during compilation:

[ERROR] Failed to execute goal com.twitter:scrooge-maven-plugin:4.11.0:testCompile (thrift-test-sources) on project druid-thrift-extensions: Execution thrift-test-sources of goal com.twitter:scrooge-maven-plugin:4.11.0:testCompile failed: Plugin com.twitter:scrooge-maven-plugin:4.11.0 or one of its dependencies could not be resolved: Failed to collect dependencies at com.twitter:scrooge-maven-plugin:jar:4.11.0 -> com.twitter:scrooge-generator_2.10:jar:4.11.0 -> org.apache.thrift:libthrift:jar:0.5.0-1: Failed to read artifact descriptor for org.apache.thrift:libthrift:jar:0.5.0-1: Could not transfer artifact org.apache.thrift:libthrift:pom:0.5.0-1 from/to maven.twttr.com

Procedure

Download maven-repo and replace the thrift directory of the local Mavenrepository with the downloaded org/apache/thrift directory.




https://github.com/pantsbuild/maven-repo

12 Phoenix 5.0.0 Porting Guide (CentOS7.6 & openEuler 20.03)

12.1 Introduction


12.3 Configuring the Compilation Environment\


12.5 Compiling Phoenix

12.1 IntroductionPhoenix is an intermediate structured query language (SQL) layer built on HBase.It enables you to use the standard JDBC APIs instead of the regular HBase clientAPIs to create tables, insert data, and query your HBase data. Phoenix is written inJava and functions as the JDBC driver embedded in HBase. The Phoenix queryengine converts SQL queries into one or more HBase scans, and orchestrates andexecutes the scans to generate standard JDBC result sets. HBase APIs, collaborativeprocessors, and custom filters are directly used. For simple queries, theperformance is measured in milliseconds. For millions of rows, the performance ismeasured in seconds.

For more information about Phoenix, visit https://phoenix.apache.org/.



Item Remarks




12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)


https://phoenix.apache.org/

Item Remarks




Item Version

OpenJDK 1.8.0_252

Maven 3.5.4

CMake 3.12.4

CentOS

Item Version

CentOS 7.6

OS kernel 4.14.0

GCC 4.8.5

openEuler

Item Version


OS Kernel 4.19.90

GCC 7.3.0

12.3 Configuring the Compilation Environment\


Installing GCC





NO TE



NO TICE



























----End









NO TE






----End

12.4 Analyzing Software PortingUse the Kunpeng Porting Advisor to scan the Phoenix installation package byreferring to Kunpeng Porting Advisor Case Study. Table 12-1 lists the third-partydependencies to be ported.




netty-all-4.0.23.Final.jar libnetty-transport-native-epoll_x86_64.so

hbase-shaded-netty-2.1.0.jar liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so

jline-2.11.1.jar libjansi.so



snappy-java-1.0.5.jar libsnappyjava.so






lz4-java-1.2.0.jar liblz4-java.so


NO TE


12.5 Compiling PhoenixStep 1 Download the Phoenix-5.0.0-HBase-2.0 source code package and decompress it.

wget https://github.com/apache/phoenix/archive/v5.0.0-HBase-2.0.tar.gzmv v5.0.0-HBase-2.0.tar.gz phoenix-5.0.0-HBase-2.0.tar.gztar -zxf phoenix-5.0.0-HBase-2.0.tar.gzcd phoenix-5.0.0-HBase-2.0



Step 3 Perform compilation.mvn clean package -DskipTests -Dmaven.javadoc.skip=true

After the compilation is successful, the phoenix-5.0.0-HBase-2.0.tar.gz package isgenerated in the /phoenix-assembly/target/ directory of the source codedirectory.





NO TE

The compiled phoenix-5.0.0-HBase-2.0.tar.gz must be scanned by using the KunpengPorting Advisor to ensure that no x86 .so or .jar packages are contained. If the compileddirectory contains x86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.

----End






A Change History

Date Description

2021-07-13 This issue is the seventh official release.Added the adaptation to openEuler 20.03 in the porting guides ofApache components.

2021-01-28 This issue is the sixth official release.Added the 9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03).

2020-12-30 This issue is the fifth official release.Changed "Kunpeng Code Scanner" to "Kunpeng Porting Advisor".

2020-10-16 This issue is the fourth official release.Added the 10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03).

2020-09-23 This issue is the third official release.● Deleted "Compiling Dependency Libraries".● Added the 7 Flink 1.9.1 Porting Guide (CentOS 7.6 &

openEuler 20.03).

Kunpeng BoostKit for Big DataPorting Guide (Apache) A Change History


Date Description

2020-05-23 This issue is the second official release.● Modified 1.6 Compiling Hadoop in the 1 Hadoop 3.1.1

Porting Guide (CentOS 7.6 & openEuler 20.03) and addednotes.

● Deleted "Troubleshooting" from the Hadoop 3.1.1 PortingGuide (CentOS 7.6).

● Deleted "Troubleshooting" from the 3 Hive 3.0.0 PortingGuide (CentOS 7.6 & openEuler 20.03).

● Deleted "Troubleshooting" from the 8 Kafka 2.1.0 PortingGuide (CentOS 7.6 & openEuler 20.03).

● Deleted "Reference" from the 11 Druid 0.12.1 Porting Guide(CentOS 7.6 & openEuler 20.03) and added the relatedinformation to 11.1 Introduction.

● Deleted "Reference" from the 12 Phoenix 5.0.0 Porting Guide(CentOS 7.6 & openEuler 20.03) and added the relatedinformation to 12.1 Introduction.

2020-03-20 This issue is the first official release.

Kunpeng BoostKit for Big DataPorting Guide (Apache) A Change History


porting guide (apache) - huawei cloud

Documents