api reference · mapreduce service api reference issue 03 date 2017-07-14 huawei technologies co.,...

MapReduce Service

API Reference

Issue 03

Date 2017-07-14

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2017. All rights reserved.No part of this document may be reproduced or transmitted in any form or by any means without prior writtenconsent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei and thecustomer. All or part of the products, services and features described in this document may not be within thepurchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information,and recommendations in this document are provided "AS IS" without warranties, guarantees orrepresentations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Huawei Technologies Co., Ltd.Address: Huawei Industrial Base

Bantian, LonggangShenzhen 518129People's Republic of China

Website: http://e.huawei.com

Issue 03 (2017-07-14) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.

i

http://e.huawei.com

Contents

1 Overview......................................................................................................................................... 1

2 APIs.................................................................................................................................................. 32.1 Service Usage................................................................................................................................................................. 32.2 API Calling Process........................................................................................................................................................32.3 Obtaining Request Authentication Information............................................................................................................. 42.4 Token Authentication......................................................................................................................................................42.5 AK/SK Authentication................................................................................................................................................... 52.5.1 AK and SK Generation................................................................................................................................................52.5.2 Request Signing Procedure..........................................................................................................................................52.5.3 Sample Code................................................................................................................................................................62.6 Obtaining a Project ID..................................................................................................................................................132.7 Constructing Requests.................................................................................................................................................. 132.8 Initiating Requests........................................................................................................................................................ 142.9 Parsing Responses........................................................................................................................................................ 152.10 Status Codes............................................................................................................................................................... 15

3 Message Header........................................................................................................................... 203.1 Request Message Header..............................................................................................................................................203.2 Response Message Header........................................................................................................................................... 21

4 API Usage......................................................................................................................................224.1 Cluster Management Interface......................................................................................................................................224.1.1 Creating a Cluster and Running a Job....................................................................................................................... 224.1.2 Adding Nodes to a Cluster.........................................................................................................................................354.1.3 Querying a Cluster List..............................................................................................................................................394.1.4 Viewing Cluster Details.............................................................................................................................................444.1.5 Terminating a Cluster................................................................................................................................................ 494.2 Job Object Interface......................................................................................................................................................504.2.1 Adding a Job and Executing the Job......................................................................................................................... 504.2.2 Querying the exe Object List of Jobs........................................................................................................................ 614.2.3 Querying exe Object Details......................................................................................................................................654.3 Job Execution Object Interface.....................................................................................................................................694.3.1 Deleting a Job Execution Object............................................................................................................................... 69

MapReduce ServiceAPI Reference Contents


ii

A Change History........................................................................................................................... 71

MapReduce ServiceAPI Reference Contents


iii

1 Overview

Introduction to MRS

The MapReduce Service (MRS) builds a reliable, secure, and easy-to-use operation andmaintenance (O&M) platform and provides storage and analysis capabilities for massive data,addressing enterprise data storage and processing demands.

MRS enables users to create their own clusters for running MapReduce, Spark, Hive Script,DistCP, Spark Script and Spark SQL jobs.

A cluster supports a maximum of 102 nodes (two Master nodes and 3 to 100 Core nodes).Each node has one Elastic Cloud Server (ECS), and two EVS disks (one 40 GB system datadisk and one user data disk), and one elastic IP address (EIP).

If more than 100 Core nodes are required, contact technical support engineers.

Table 1-1 lists basic concepts in this document.

Table 1-1 Basic concepts

Concept Description

Job object Defines the job template. A job template contains the jobbinary object set for job running, job type (such asMapReduce, Spark, and Spark SQL), and job name. A jobtemplate can generate job execution objects.

Job execution object Generated by a job template and contains information aboutrunning jobs, for example, job status, start time, end time, andID of the cluster that runs the job.

Cluster managementinterface

A cluster is a group of nodes with the same configuration.

Job object interface A job object lists the binaries that a job needs to run. To run ajob, you must specify data sources and job parameters. You canrun a job on an existing or new transient cluster.

Job execution objectinterface

A job execution object represents a Hadoop job that runs on acluster. A job execution polls the status of a running job andreports it to the user. Also a user can cancel a running job.

MapReduce ServiceAPI Reference 1 Overview


1

API ClassificationMRS provides self-developed cluster APIs, and job APIs. To address public cloudauthentication problems and differentiate clusters based on open source, MRS adds specificheaders to request message headers, including X-Sdk-Date, Authorization, Host, Content-type, and Content-Length. For details about headers, see section Request Message Header.

MapReduce ServiceAPI Reference 1 Overview


2

2 APIs

2.1 Service UsagePublic cloud services provide RESTful APIs.

Representational State Transfer (REST) allocates Uniform Resource Identifiers (URIs) todispersed resources so that the resources can be located. Applications on clients use UnifiedResource Locators (URLs) to obtain the resources.

The URL is in the following format: https://Endpoint/uri

Table 2-1 describes the parameters in a URL.

Table 2-1 Parameter description

Parameter Description

Endpoint Specifies the name of the requested ECS. Obtain the value fromRegions and Endpoints.

uri Specifies the API access path for performing a specified operation.Obtain this value from the URI of the API, for example, v3/auth/tokens.

2.2 API Calling ProcessThe process for calling an MRS API is as follows:

1. Obtaining Request Authentication InformationBefore you call an API, obtain request authentication information and add it to themessage header.

2. Obtaining a Project IDBefore you call an API, obtain the project ID from the management console.

3. Constructing RequestsConfigure parameters to construct requests.

MapReduce ServiceAPI Reference 2 APIs


3

http://developer.hwclouds.com/endpoint.html

4. Initiating Requests5. Parsing Responses

2.3 Obtaining Request Authentication InformationThe following authentication methods are available for calling APIs:

l Token authentication: Requests are authenticated using tokens.l AK/SK authentication: Requests are encrypted using the access key (AK) and secret

key (SK). AK/SK authentication provides higher security than token authentication.

2.4 Token Authentication

ScenariosIf you use a token for authentication, you must obtain the user's token and add X-Auth-Tokento the request message header of the service API when making an API call.

This section describes how to make an API call for token authentication.

Making an API Call1. Send POST https://IAM endpoint/v3/auth/tokens to obtain the IAM endpoint and the

region name in the message body. For details, see Regions and Endpoints.An example request message is as follows:

NOTE

Replace the items in italic in the following example with actual ones. For details, see the Identityand Access Management API Reference.

{ "auth": { "identity": { "methods": [ "password" ], "password": { "user": { "name": "username", "password": "password", "domain": { "name": "domainname" } } } }, "scope": { "project": { "name": "cn-north-1"

} } }}

2. Obtain the token. For details, see section Obtaining the User Token in the Identity andAccess Management API Reference. After the request is processed, the value of X-Subject-Token in the message header is the token value.



4


3. Make a call to a service API, add X-Auth-Token to the message header, and set thevalue of X-Auth-Token to the token obtained in step 2.

2.5 AK/SK Authentication

2.5.1 AK and SK GenerationBefore using REST APIs provided by MRS, download the AK/SK certificate to the localcomputer.

NOTE

Each user needs to download the AK/SK certificate only once.

Procedure1. Log in to the management console.2. Click the username and choose My Credential from the drop-down list.

3. Click Access Credentials.4. Click Add Access Key to switch to the Add Access Key page.5. Enter the password used for the current login.6. Enter the authentication code received in the email.

NOTE

l For users created in Identity and Access Management (IAM), if no email address or mobilephone is filled during the user creation, you only need to authenticate the login password.

l For MYWorkplace users, you do not need to authenticate the login password. If the emailaddress or mobile phone information is unavailable, you can create an access passwordwithout the authentication code.

7. Click OK to download the access key.

NOTE

To prevent the access key from being leaked, keep it secure.

2.5.2 Request Signing Procedure

Preparations1. Download the API gateway signing tool from the following website:

Download path: http://esdk.huawei.com/ilink/esdk/download/HW_4567062. Extract the package.3. Create a Java project, and reference the extracted JAR to the dependency path.

Sign a Request1. Create a request com.cloud.sdk.DefaultRequest (JAVA) used for signing.2. Set the target API URL, HTTPS method, and content of request

com.cloud.sdk.DefaultRequest (JAVA).3. Sign request com.cloud.sdk.DefaultRequest (JAVA).



5

http://esdk.huawei.com/ilink/esdk/download/HW_456706

a. Call SignerFactory.getSigner(String serviceName, String regionName) to obtaina signing tool.

b. Call Signer.sign(Request<?> request, Credentials credentials) to sign the requestcreated in step 1.The following code shows the details.//Select an algorithm for request signing.Signer signer = SignerFactory.getSigner(serviceName, region);//Sign the request. The request will change after the signing.signer.sign(request, new BasicCredentials(this.ak, this.sk));

4. Convert the request signed in the previous step to a new request that can be used to makean API call and copy the header of the signed request to the new request.For example, if Apache HttpClient is used, convert DefaultRequest to HttpRequestBaseand copy the header of the signed DefaultRequest to HttpRequestBase.For details, see descriptions of AccessServiceImpl.java in section Sample Code.

2.5.3 Sample CodeThe following code shows how to sign a request and how to use an HTTP client to send anHTTPS request:

Code is classified into three classes for demonstrating request signing and HTTP requestsending.

AccessService: indicates the abstract class that converts the GET, POST, PUT, and DELETEmethods in to the access method.

Demo: indicates the execution entry used to simulate GET, POST, PUT, and DELETE requestsending.

AccessServiceImpl: indicates the implementation of the access method. Code required forAPI gateway communication is in the access method.

For details about the region and serviceName in the following codes, see Regions andEndpoints.

AccessService.java:

package com.cloud.apigateway.sdk.demo; import java.io.InputStream; import java.net.URL; import java.util.Map; import org.apache.http.HttpResponse; import com.cloud.sdk.http.HttpMethodName; public abstract class AccessService { protected String serviceName = null; protected String region = null; protected String ak = null; protected String sk = null; public AccessService(String serviceName, String region, String ak, String sk) { this.region = region;



6



this.serviceName = serviceName; this.ak = ak; this.sk = sk; } public abstract HttpResponse access(URL url, Map<String, String> header, InputStream content, Long contentLength, HttpMethodName httpMethod) throws Exception; public HttpResponse access(URL url, Map<String, String> header, HttpMethodName httpMethod) throws Exception { return this.access(url, header, null, 0l, httpMethod); } public HttpResponse access(URL url, InputStream content, Long contentLength, HttpMethodName httpMethod) throws Exception { return this.access(url, null, content, contentLength, httpMethod); } public HttpResponse access(URL url, HttpMethodName httpMethod) throws Exception { return this.access(url, null, null, 0l, httpMethod); } public abstract void close(); public String getServiceName() { return serviceName; } public void setServiceName(String serviceName) { this.serviceName = serviceName; } public String getRegion() { return region; } public void setRegion(String region) { this.region = region; } public String getAk() { return ak; } public void setAk(String ak) { this.ak = ak; } public String getSk() { return sk; } public void setSk(String sk) { this.sk = sk; } }

AccessServiceImpl.java:

package com.cloud.apigateway.sdk.demo; import java.io.IOException; import java.io.InputStream; import java.net.URISyntaxException; import java.net.URL; import java.util.HashMap; import java.util.Map;



7

import javax.net.ssl.SSLContext; import org.apache.http.Header; import org.apache.http.HttpHeaders; import org.apache.http.HttpResponse; import org.apache.http.client.methods.HttpDelete; import org.apache.http.client.methods.HttpGet; import org.apache.http.client.methods.HttpHead; import org.apache.http.client.methods.HttpPatch; import org.apache.http.client.methods.HttpPost; import org.apache.http.client.methods.HttpPut; import org.apache.http.client.methods.HttpRequestBase; import org.apache.http.conn.ssl.AllowAllHostnameVerifier; import org.apache.http.conn.ssl.SSLConnectionSocketFactory; import org.apache.http.conn.ssl.SSLContexts; import org.apache.http.conn.ssl.TrustSelfSignedStrategy; import org.apache.http.entity.InputStreamEntity; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import com.cloud.sdk.DefaultRequest; import com.cloud.sdk.Request; import com.cloud.sdk.auth.credentials.BasicCredentials; import com.cloud.sdk.auth.signer.Signer; import com.cloud.sdk.auth.signer.SignerFactory; import com.cloud.sdk.http.HttpMethodName; public class AccessServiceImpl extends AccessService { private CloseableHttpClient client = null; public AccessServiceImpl(String serviceName, String region, String ak, String sk) { super(serviceName, region, ak, sk); } /** {@inheritDoc} */ public HttpResponse access(URL url, Map<String, String> headers, InputStream content, Long contentLength, HttpMethodName httpMethod) throws Exception { // Make a request for signing. Request request = new DefaultRequest(this.serviceName); try { // Set the request address. request.setEndpoint(url.toURI()); String urlString = url.toString(); String parameters = null; if (urlString.contains("?")) { parameters = urlString.substring(urlString.indexOf("?") + 1); Map parametersmap = new HashMap<String, String>(); if (null != parameters && !"".equals(parameters)) { String[] parameterarray = parameters.split("&"); for (String p : parameterarray) { String key = p.split("=")[0]; String value = p.split("=")[1]; parametersmap.put(key, value); } request.setParameters(parametersmap); } }



8

} catch (URISyntaxException e) { // It is recommended to add logs in this place. e.printStackTrace(); } // Set the request method. request.setHttpMethod(httpMethod); if (headers != null) { // Add request header information if required. request.setHeaders(headers); } // Configure the request content. request.setContent(content); // Select an algorithm for request signing. Signer signer = SignerFactory.getSigner(serviceName, region); // Sign the request, and the request will change after the signing. signer.sign(request, new BasicCredentials(this.ak, this.sk)); // Make a request that can be sent by the HTTP client. HttpRequestBase httpRequestBase = createRequest(url, null, request.getContent(), contentLength, httpMethod); Map<String, String> requestHeaders = request.getHeaders(); // Put the header of the signed request to the new request. for (String key : requestHeaders.keySet()) { if (key.equalsIgnoreCase(HttpHeaders.CONTENT_LENGTH.toString())) { continue; } httpRequestBase.addHeader(key, requestHeaders.get(key)); } HttpResponse response = null; SSLContext sslContext = SSLContexts.custom() .loadTrustMaterial(null, new TrustSelfSignedStrategy()) .useTLS().build(); SSLConnectionSocketFactory sslSocketFactory = new SSLConnectionSocketFactory( sslContext, new AllowAllHostnameVerifier()); client = HttpClients.custom().setSSLSocketFactory(sslSocketFactory) .build(); // Send the request, and a response will be returned. response = client.execute(httpRequestBase); return response; } /** * Make a request that can be sent by the HTTP client. * * @param url * specifies the API access path. * @param header * specifies the header information to be added. * @param content * specifies the body content to be sent in the API call. * @param contentLength * specifies the length of the content. This parameter is optional. * @param httpMethod * specifies the HTTP method to be used. * @return specifies the request that can be sent by an HTTP client. */ private static HttpRequestBase createRequest(URL url, Header header, InputStream content, Long contentLength, HttpMethodName httpMethod) { HttpRequestBase httpRequest; if (httpMethod == HttpMethodName.POST) { HttpPost postMethod = new HttpPost(url.toString()); if (content != null) { InputStreamEntity entity = new InputStreamEntity(content,



9

contentLength); postMethod.setEntity(entity); } httpRequest = postMethod; } else if (httpMethod == HttpMethodName.PUT) { HttpPut putMethod = new HttpPut(url.toString()); httpRequest = putMethod; if (content != null) { InputStreamEntity entity = new InputStreamEntity(content, contentLength); putMethod.setEntity(entity); } } else if (httpMethod == HttpMethodName.PATCH) { HttpPatch patchMethod = new HttpPatch(url.toString()); httpRequest = patchMethod; if (content != null) { InputStreamEntity entity = new InputStreamEntity(content, contentLength); patchMethod.setEntity(entity); } } else if (httpMethod == HttpMethodName.GET) { httpRequest = new HttpGet(url.toString()); } else if (httpMethod == HttpMethodName.DELETE) { httpRequest = new HttpDelete(url.toString()); } else if (httpMethod == HttpMethodName.HEAD) { httpRequest = new HttpHead(url.toString()); } else { throw new RuntimeException("Unknown HTTP method name: " + httpMethod); } httpRequest.addHeader(header); return httpRequest; } @Override public void close() { try { if (client != null) { client.close(); } } catch (IOException e) { // It is recommended to add logs in this place. e.printStackTrace(); } } }

Demo.java:

package com.cloud.apigateway.sdk.demo; import java.io.BufferedReader; import java.io.ByteArrayInputStream; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; import java.net.MalformedURLException; import java.net.URL; import org.apache.http.HttpResponse; import com.cloud.sdk.http.HttpMethodName; public class Demo {



10

//replace real region private static final String region = "regionName"; //replace real service name private static final String serviceName = "serviceName"; public static void main(String[] args) { //replace real AK String ak = "akString"; //replace real SK String sk = "skString"; // get method //replace real url String url = "urlString"; get(ak, sk, url); // post method //replace real url String postUrl = "urlString"; //replace real body String postbody = "bodyString"; post(ak, sk, postUrl, postbody); // put method //replace real body String putbody = "bodyString"; //replace real url String putUrl = "urlString"; put(ak, sk, putUrl, putbody); // delete method //replace real url String deleteUrl = "urlString"; delete(ak, sk, deleteUrl); } public static void put(String ak, String sk, String requestUrl, String putBody) { AccessService accessService = null; try { accessService = new AccessServiceImpl(serviceName, region, ak, sk); URL url = new URL(requestUrl); HttpMethodName httpMethod = HttpMethodName.PUT; InputStream content = new ByteArrayInputStream(putBody.getBytes()); HttpResponse response = accessService.access(url, content, (long) putBody.getBytes().length, httpMethod); System.out.println(response.getStatusLine().getStatusCode()); } catch (Exception e) { e.printStackTrace(); } finally { accessService.close(); } } public static void patch(String ak, String sk, String requestUrl, String putBody) { AccessService accessService = null; try { accessService = new AccessServiceImpl(serviceName, region, ak, sk); URL url = new URL(requestUrl);



11

HttpMethodName httpMethod = HttpMethodName.PATCH; InputStream content = new ByteArrayInputStream(putBody.getBytes()); HttpResponse response = accessService.access(url, content, (long) putBody.getBytes().length, httpMethod); System.out.println(convertStreamToString(response.getEntity() .getContent())); } catch (Exception e) { e.printStackTrace(); } finally { accessService.close(); } } public static void delete(String ak, String sk, String requestUrl) { AccessService accessService = null; try { accessService = new AccessServiceImpl(serviceName, region, ak, sk); URL url = new URL(requestUrl); HttpMethodName httpMethod = HttpMethodName.DELETE; HttpResponse response = accessService.access(url, httpMethod); System.out.println(convertStreamToString(response.getEntity() .getContent())); } catch (Exception e) { e.printStackTrace(); } finally { accessService.close(); } } public static void get(String ak, String sk, String requestUrl) { AccessService accessService = null; try { accessService = new AccessServiceImpl(serviceName, region, ak, sk); URL url = new URL(requestUrl); HttpMethodName httpMethod = HttpMethodName.GET; HttpResponse response; response = accessService.access(url, httpMethod); System.out.println(convertStreamToString(response.getEntity() .getContent())); } catch (Exception e) { e.printStackTrace(); } finally { accessService.close(); } } public static void post(String ak, String sk, String requestUrl, String postbody) { AccessService accessService = new AccessServiceImpl(serviceName, region, ak, sk); URL url = null; try { url = new URL(requestUrl); } catch (MalformedURLException e) { e.printStackTrace(); } InputStream content = new ByteArrayInputStream(postbody.getBytes()); HttpMethodName httpMethod = HttpMethodName.POST; HttpResponse response;



12

try { response = accessService.access(url, content, (long) postbody.getBytes().length, httpMethod); System.out.println(convertStreamToString(response.getEntity() .getContent())); } catch (Exception e) { e.printStackTrace(); } finally { accessService.close(); } } private static String convertStreamToString(InputStream is) { BufferedReader reader = new BufferedReader(new InputStreamReader(is)); StringBuilder sb = new StringBuilder(); String line = null; try { while ((line = reader.readLine()) != null) { sb.append(line + "\n"); } } catch (IOException e) { e.printStackTrace(); } finally { try { is.close(); } catch (IOException e) { e.printStackTrace(); } } return sb.toString(); } }

NOTE

1. Parameters URI, AK, SK, and HTTP METHOD are mandatory.

2. You can use the request.addHeader() method to add header information.

2.6 Obtaining a Project IDA project ID (project_id or tenant_id) is required in some URIs when APIs are called.project_id and tenant_id have the same meaning in this document. Obtain the project IDfrom the management console as follows:

1. Register an account and log in to the management console.2. Click the username, and select My Credential from the drop-down list.

On the My Credential page, obtain the project ID from the project list. For example,project_id:"5a3314075bfa49b9ae360f4ecd333695".

2.7 Constructing RequestsA request consists of three parts: request line, request header, and a request body (optional).



13

Request Line

A request line starts with a method token, which is followed by a request uniform resourceidentifier (URI) and protocol version. The three parts are separated by spaces. The format ofthe request line is as follows:

Method Request-URI HTTP-Version CRLF

l Method: request method. All methods are capitalized. Their meanings are as follows:– GET: obtains resources identified by a Request-URI.– POST: adds new data to a resource identified by a Request-URI.– PUT: requests a server to save a resource and uses a Request-URI to identify the

resource.– DELETE: requests the server to delete the resource identified by a Request-URI.– PATCH: requests a server to update a resource or create a new resource if the target

resource does not exist.– HEAD: requests only the head of the page.– OPTION: allows the client to view server performance.

l Request-URI: unified resource identifier.

NOTE

Request-URI is in the format [path][?query], where:

l If there are multiple paths, separate them by a forward slash (/).

l If there are multiple queries, separate them by ampersand (&).

l path and query are separated by a question mark (?).

l HTTP-Version: version of the HTTP protocol used by a request.l CRLF: carriage return and new-line characters. CRLF is placed only at the end of a line,

and a separate CR or LF is not allowed.

Request Header

A request header consists of several header fields. Each header field is in the format: fieldname:field value.

For details about request headers of MRS, see section Request Message Header.

Request Body

A request body is a JSON-style nested key:value pair. The mandatory fields and optionalfields in an HTTP request vary with URI objects.

2.8 Initiating RequestsThere are three methods to initiate requests to the server end:

l cURLcURL is a command line tool used to perform URL operations and transmit information.It can serve as an HTTP client to send HTTP requests to the server end and receiveresponse messages. cURL is suitable for use in API tuning scenarios. For moreinformation about cURL, visit https://curl.haxx.se/.



14

https://curl.haxx.se/

l Code

You can call an API through code to assemble, send, and process requests.

l REST clients

Mozilla Firefox and Google provide a graphical browser plug-in for REST clients tosend and process requests. For details, see Firefox REST Client and Chrome RESTClient.

2.9 Parsing ResponsesAfter receiving a request, the server will return an HTTP response.

A response consists of three parts: status line, response header, and response body.

Status Line

The format of a status line is as follows:

HTTP-Version Status-Code Reason-Phrase CRLF

l HTTP-Version: version of the HTTP protocol used by the server.

l Status-Code: status code in the response returned by the server.

A status code consists of three digits. The first digit defines the category of a response.There are five categories:

– 1xx: informational response, indicating that the request has been received and therecipient is continuing to process the request.

– 2xx: success response, indicating that the request has been received, understood,and accepted.

– 3xx: redirection response, indicating that further action needs to be taken by therequester in order to complete the request.

– 4xx: client error response, indicating that the request contains a syntax error orcannot be fulfilled.

– 5xx: server error response, indicating that the server fails to fulfill a valid request.

l Reason-Phrase: text description of the status code.

Response Header

For details about response headers, see Response Message Header.

Response Body

A response body is a JSON text. The content of a response body is UTF-8 coded.

2.10 Status CodesTable 2-2 describes status codes.



15

https://addons.mozilla.org/en-US/firefox/addon/restclient/

https://code.google.com/archive/p/rest-client

https://code.google.com/archive/p/rest-client

Table 2-2 Status codes

StatusCode

Code Description

100 Continue The server has received the initial part of the requestand the client should continue to send the remainingpart.

101 Switching Protocols The requester has asked the server to switch protocolsand the server has agreed to do so.The target protocol must be more advanced than thesource protocol.For example, the current HTTP protocol is switchedto a later version of HTTP.

200 OK The server has successfully processed the request.

201 Created The request has been fulfilled, resulting in thecreation of a new resource.

202 Accepted The request has been accepted for processing, but theprocessing has not been completed.

203 Non-AuthoritativeInformation

The request has been fulfilled.

204 NoContent The server has successfully processed the request, butdoes not return any content.The status code is returned in response to an HTTPOPTIONS request.

205 Reset Content The server has successfully processed the request, butdoes not return any content. Unlike a 204 response,this response requires that the requester reset thecontent.

206 Partial Content The server has successfully processed a part of theGET request.

300 Multiple Choices There are multiple options for the requested resource.For example, this code could be used to present a listof resource characteristics and addresses from whichthe client such as a browser may choose.

301 Moved Permanently This and all future requests should be permanentlydirected to the given URI indicated in this response.

302 Found The requested resource was temporarily moved.

303 See Other The response to the request can be found underanother URI using a GET or POST method.



16

StatusCode

Code Description

304 Not Modified The requested resource has not been modified. Insuch case, there is no need to retransmit the resourcesince the client still has a previously-downloadedcopy.

305 Use Proxy The requested resource is available only through aproxy.

306 Unused This HTTP status code is no longer used.

400 BadRequest The request is invalid.The client should modify the request instead of re-initiating it.

401 Unauthorized The authentication information provided by the clientis incorrect or invalid.

402 Payment Required Reserved for future use.

403 Forbidden The server has received the request and understood it,but the server is refusing to respond to it.The client should modify the request instead of re-initiating it.

404 NotFound The requested resource could not be found.The client should modify the request instead of re-initiating it.

405 MethodNotAllowed A request method is not supported for the requestedresource.The client should modify the request instead of re-initiating it.

406 Not Acceptable The server could not fulfill the request according tothe content characteristics of the request.

407 Proxy AuthenticationRequired

This code is similar to 401, but indicates that theclient must first authenticate itself with the proxy.

408 Request Time-out The server timed out waiting for the request.The client may re-initiate the request withoutmodifications at any later time.

409 Conflict The request could not be processed due to a conflictin the request, such as an edit conflict betweenmultiple simultaneous updates or the resource that theclient attempts to create already exits.

410 Gone The requested resource has been deleted permanentlyand will not be available again.



17

StatusCode

Code Description

411 Length Required The server refused to process the request because therequest does not specify the length of its content.

412 Precondition Failed The server does not meet one of the preconditionsthat the requester puts on the request.

413 Request Entity TooLarge

The request is larger than the server is willing or ableto process.The server may close the connection to prevent theclient from continuing the request. If the servertemporarily cannot process the request, the responsewill contain a Retry-After header field.

414 Request-URI TooLarge

The URI provided was too long for the server toprocess.

415 Unsupported MediaType

The server does not support the media type in therequest.

416 Requested range notsatisfiable

The requested range is invalid.

417 Expectation Failed The server fails to meet the requirements of theExpect request-header field.

422 UnprocessableEntity The request was well-formed but was unable to befollowed due to semantic errors.

429 TooManyRequests The client has sent more requests than its rate limit isallowed within a given amount of time, or the serverhas received more requests than it is able to processwithin a given amount of time. In this case, it isadvisable for the client to re-initiate requests after thetime specified in the Retry-After header of theresponse expires.

500 InternalServerError The server is able to receive the request but it couldnot understand the request.

501 Not Implemented The server does not support the requested function.

502 Bad Gateway The server was acting as a gateway or proxy andreceived an invalid request from a remote server.

503 ServiceUnavailable The requested service is invalid.It is advisable for the client to modify the requestinstead of re-initiating the request.

504 ServerTimeout The server could not return a timely response. Theresponse will reach the client only if the requestcarries a timeout parameter.



18

StatusCode

Code Description

505 HTTP Version notsupported

The server does not support the HTTP protocolversion used in the request.



19

3 Message Header

3.1 Request Message HeaderTable 3-1 shows the request message header.

Table 3-1 Request header

Parameter Description Mandatory orNot

Example

X-Sdk-Date Time to send a request.The time format isYYYYMMDD'T'HHMMSS'Z'.The value is set to thecurrent system GMT.

Mandatorywhen AK/SKauthenticationis used.

20150907T101459Z

Authorization Signature authenticationinformation.The value is the result ofa signature request.For details, see RequestSigning Procedure.


SDK-HMAC-SHA256Credential=QRUP2R3QFNAOVAWMYHZW/20160202/northchina/test/sdk_request,SignedHeaders=host;x-sdk-date,Signature=9d8b56b055c0e1f7a9498d881a7cb726be91b4f0cde1773b0b1557e987a480ce

MapReduce ServiceAPI Reference 3 Message Header


20

Parameter Description Mandatory orNot

Example

Host Requested serverinformation, which isobtained from the URLof a service API. Thevalue ishostname[:port]. If therequest does not containa port, use the defaultport. The default port ofHTTPS is 443.


code.test.comorcode.test.com:443

Content-type MIME type of the entitythat is sent.

Yes application/json

Content-Length Length of the requestbody, expressed by Byte.

Mandatory for aPOST/PUTrequest. A GETrequest cannotcontain theparameter.

3495

X-Project-Id Project ID, used to obtaintokens for differentprojects.

No e9993fc787d94b6c886cbaa340f9c0f4

X-Auth-Token User token. Mandatorywhen tokenauthenticationis used.

-

X-Language Required responselanguage. The validvalues are as follows:l zh-cn: Chinesel en-us: English

No en-us

3.2 Response Message HeaderTable 3-2 shows the response message headers.

Table 3-2 Response header

Parameter Description

Content-Length Length of the response message body, expressed by Byte.

Date System response time.

Content-type MIME type of the entity that is sent.

MapReduce ServiceAPI Reference 3 Message Header


21

4 API Usage

4.1 Cluster Management InterfaceA cluster is a group of nodes with the same configuration.

4.1.1 Creating a Cluster and Running a Job

Function

This API is used to create an MRS cluster and submit a job in the cluster. This API isincompatible with Sahara.

Users can create a maximum of 10 clusters at a time.

Before using the interface, you need to obtain the resources listed in Table 4-1.

Table 4-1 Obtaining resources

Resource Obtain Method

VPC See the operation guide in Virtual Private Cloud > Querying VPCsand VPC > Creating a VPC in the API Reference of the virtualprivate cloud.

Subnet See the operation guide in Subnet > Querying Subnets and Subnet> Creating a Subnet in the API Reference of the virtual privatecloud.

Key pair See the operation guide in ECS SSH Key Management > QueryingSSH Key Pairs (Native OpenStack API) and ECS SSH KeyManagement > Creating and Importing an SSH Key Pair (NativeOpenStack API) in the API Reference of the elastic cloud server.

Region For information about regions and available zones, see Regions andEndpoints.

MapReduce ServiceAPI Reference 4 API Usage


22



Resource Obtain Method

Versioninformation

Currently, MRS 1.3.0 is supported.The latest version of MRS is used by default. Currently, the latestversion is MRS 1.3.0.

Componentinformation

Information about components supported by MRS 1.3.0l Hadoop: MRS 1.3.0_001l Spark: MRS 1.3.0_002l HBase: MRS 1.3.0_003l Hive: MRS 1.3.0_004l Hue: MRS 1.3.0_005l Kafka: MRS 1.3.0_006l Storm: MRS 1.3.0_007

URIl Format:

POST /v1.1/{project_id}/run-job-flowl Parameter description

Table 4-2 URI parameter description

Parameter Mandatory or Not Description

project_id Yes Project ID. For details onhow to obtain the projectID, see Obtaining aProject ID.

Requestl Example:

{ "billing_type": 12, "data_center": "southchina", "master_node_num": 2, "master_node_size": "c2.2xlarge.linux.mrs", "core_node_num": 3, "core_node_size": "c2.2xlarge.linux.mrs", "available_zone_id": "az1.dc1", "cluster_name": "newcluster", "vpc": "vpc1", "vpc_id": "5b7db34d-3534-4a6e-ac94-023cd36aaf74", "subnet_id": "815bece0-fd22-4b65-8a6e-15788c99ee43", "subnet_name": "subnet", "cluster_version": MRS 1.3.0", "cluster_type": 0, "volume_type": "SSD", "volume_size": 100, "node_public_cert_name": "SSHkey-bba1", "safe_mode": 0,



23

"component_list": [ { "component_id": "MRS 1.3.0_001", "component_name": "Hadoop", "component_version": "2.7.2", "component_desc": "" }, { "component_id": MRS 1.3.0_002", "component_name": "Spark", "component_version": "1.5.1", "component_desc": "" }, { "component_id": "MRS 1.3.0_003", "component_name": "HBase", "component_version": "1.0.2", "component_desc": "" }, { "component_id": MRS 1.3.0_004", "component_name": "Hive", "component_version": "1.2.1", "component_desc": "" } ], "add_jobs": [ { "job_type": 1, "job_name": "tenji111", "jar_path": "s3a://bigdata/program/hadoop-mapreduce-examples-2.7.2.jar", "arguments": "wordcount", "input": "s3a://bigdata/input/wd_1k/", "output": "s3a://bigdata/ouput/", "job_log": "s3a://bigdata/log/", "shutdown_cluster": true, "file_action": "", "submit_job_once_cluster_run": true, "hql": "", "hive_script_path": "" } ]}

l Parameter description

Table 4-3 Request parameter description

Parameter Mandatory or Not

Type Description

billing_type Yes Integer The value is 12, indicating on-demand payment.

data_center Yes String Cluster region information. Obtainthe value fromRegions andEndpoints.

master_node_num Yes Integer Number of Master nodesThe value is 2.



24




Type Description

master_node_size Yes Strig Best match based on several years ofcommissioning experience. MRSsupports seven specifications ofhosts, and host specifications aredetermined by CPUs, memory, anddisks. Master nodes supportc2.4xlarge, s1.4xlarge ands1.8xlarge. Core nodes of astreaming cluster support s1.xlarge,c2.2xlarge, c2.4xlarge, s1.4xlarge,s1.8xlarge, and d1.8xlarge. Corenodes of an analysis cluster supportall specifications.l s1.xlarge.linux.bigdata

– CPU: 4-core– Memory: 16 GB– System Disk: 40 GB

l c2.2xlarge.linux.bigdata– CPU: 8-core– Memory: 16 GB– System Disk: 40 GB

l c2.4xlarge.linux.bigdata– CPU: 16-core– Memory: 32 GB– System Disk: 40 GB

l s1.4xlarge.linux.bigdata– CPU: 16-core– Memory: 64 GB– System Disk: 40 GB

l s1.8xlarge.linux.bigdata– CPU: 32-core– Memory: 128 GB– System Disk: 40 GB

l d1.4xlarge.linux.bigdata– CPU: 16-core– Memory: 128 GB– System Disk: 40 GB– Data Disk: 1.8 TB x 12 HDDs

l d1.8xlarge.linux.bigdata– CPU: 36-core



25


Type Description

– Memory: 256 GB– System Disk: 40 GB– Data Disk: 1.8 TB x 24 HDDs

NOTEClusters charged in Monthly/Yearlymode support s1.xlarge, c2.2xlarge,c2.4xlarge, s1.4xlarge, and s1.8xlarge.Clusters charged in Metered modesupport all specifications.

core_node_num Yes Integer Number of Core nodesValue range: 3 to 100A maximum of 100 Core nodes aresupported by default. If more than100 Core nodes are required, contacttechnical support engineers or invokea background interface to modify thedatabase.

core_node_size Yes String Instance specification of a Core nodeConfiguration method of thisparameter is identical to that ofmaster_node_size.

available_zone_id Yes String ID of an available zone. Obtain thevalue from Regions and Endpoints.

cluster_name Yes String Cluster name, which is globallyunique and contains only 1 to 64letters, digits, and underscores (_).

vpc Yes String Name of the VPC where the subnetlocatesObtain VPC name from themanagement console as follows:1. Register an account and log in to

the management console.2. Click Virtual Private Cloud and

select Virtual Private Cloudfrom the left list.

On the Virtual Private Cloud page,obtain the VPC name from the list.



26



Type Description

vpc_id Yes String ID of the VPC where the subnetlocatesObtain the VPC ID from themanagement console as follows:1. Register an account and log in to



On the Virtual Private Cloud page,obtain the VPC ID from the list.

subnet_id Yes String Subnet IDObtain the subnet ID from themanagement console as follows:1. Register an account and log in to



On the Virtual Private Cloud page,obtain the subnet ID from the list.

subnet_name Yes String Name of the subnetObtain the subnet name from themanagement console as follows:1. Register an account and log in to



On the Virtual Private Cloud page,obtain the subnet name from the list.

cluster_version No String Version of the clustersCurrently,, MRS 1.3.0 are supported.The latest version of MRS is used bydefault. Currently, the latest versionis MRS 1.3.0.

cluster_type No Integer Type of clustersl 0: analysis clusterl 1: streaming clusterThe default value is 0.



27


Type Description

volume_type Yes String Type of disksSATA, SAS and SSD are supported.l SATA: common I/Ol SAS: high-speed I/Ol SSD: super high-speed I/O

volume_size Yes Integer Data disk storage space of a CorenodeUsers can add disks to expandstorage capacity when creating acluster. There are the followingscenarios:l Separation of data storage and

computing: Data is stored in theOBS system. Costs of clusters arerelatively low but computingperformance is poor. The clusterscan be deleted at any time. It isrecommended when datacomputing is not frequentlyperformed.

l Integration of data storage andcomputing: Data is stored in theHDFS system. Costs of clustersare relatively high but computingperformance is good. The clusterscannot be deleted in a short term.It is recommended when datacomputing is frequentlyperformed.

Value range: 100 GB to 1000 GB

node_public_cert_name

Yes String Name of a key pairYou can use a key to log in to theMaster node in the cluster.



28


Type Description

safe_mode Yes Integer MRS cluster running model 0: common mode

The value indicates that theKerberos authentication isdisabled. Users can use allfunctions provided by the cluster.

l 1: safe modeThe value indicates that theKerberos authentication isenabled. Common users cannotuse the file management or jobmanagement functions of an MRScluster and cannot view clusterresource usage or the job recordsof Hadoop and Spark. To usethese functions, the users mustobtain the relevant permissionsfrom the MRS Manageradministrator.The request has thecluster_admin_secret parameteronly when safe_mode is set to 1.

cluster_admin_secret

No String Indicates the password of the MRSManager administrator. Thepassword:l Must contain 8 to 64 characters.l Must contain at least four types of

the following:– Lowercase letters– Uppercase letters– Digits– Special characters of `~!@#$

%^&*()-_=+\|[{}];:'",<.>/?– Spaces

l Must be different from theusername.

l Must be different from theusername written in reverse order.

This parameter needs to beconfigured only when safe_mode isset to 1.

component_list Yes Array Service component listFor the parameter description, seeTable 4-4.



29


Type Description

add_jobs Yes Array You can submit a job when youcreate a cluster to save time and useMRS easily. Only one job can beadded. For details about jobparameters, see Table 4-5.

Table 4-4 component_list parameter description


Type Description

component_id Yes String Component IDComponent IDs supported by MRS1.3.0 include:l MRS 1.3.0_001: Hadoopl MRS 1.3.0_002: Sparkl MRS 1.3.0_003: HBasel MRS 1.3.0_004: Hivel MRS 1.3.0_005: Huel MRS 1.3.0_006: Kafkal MRS 1.3.0_007: StormFor example, the component ID ofHadoop is MRS 1.3.0_001.

component_name Yes String Component nameCurrently, Hadoop, Spark, HBase,Hive, Hue, Kafka and Storm aresupported..



30


Type Description

component_version No String Component versionMRS 1.3.0 supports the followingcomponent version:Component version of an analysiscluster:l Hadoop: 2.7.2l Spark: 1.5.1l HBase: 1.0.2l Hive: 1.2.1l Hue: 3.11.0Component version of a streamingcluster:l Kafka: 0.10.0.0l Storm: 1.0.2

component_desc No String Component descriptionContains a maximum of 65535characters.

Table 4-5 add_jobs parameter description


Type Description

job_type Yes Integer Job typel 1: MapReducel 2: Sparkl 3: Hive Scriptl 4: HiveQL (not supported

currently)l 5: DistCp, importing and

exporting data (not supported inthis API currently).

l 6: Spark Scriptl 7: Spark SQL, submitting Spark

SQL statement (not supported inthis API currently).NOTE

Spark and Hive jobs can be added toonly clusters including Spark andHive components.



31


Type Description

job_name Yes String Job nameIt contains only 1 to 64 letters, digits,and underscores (_).NOTE

Identical job names are allowed but notrecommended.

jar_path Yes String Path of the .jar package or .sql filefor program executionThe parameter must meet thefollowing requirements:l Contains a maximum of 255

characters, excluding specialcharacters such as ;|&><'$. Theaddress cannot be empty or full ofspaces.

l Starts with / or s3a://.l Spark Script must end with .sql;

while MapReduce and Spark Jarmust end with .jar. sql and jarare case-insensitive.

arguments No String Key parameter for programexecutionThe parameter is specified by thefunction of the user's program. MRSis only responsible for loading theparameter.The parameter contains a maximumof 255 characters, excluding specialcharacters such as ;|&>'<$, and canbe empty.

input No String Path for inputting data, which muststart with / or s3a://. A correct OBSpath is required.The parameter contains a maximumof 255 characters, excluding specialcharacters such as ;|&>'<$, and canbe empty.



32


Type Description

output No String Path for outputting data, which muststart with / or s3a://. A correct OBSpath is required. If the path does notexist, the system automaticallycreates it.The parameter contains a maximumof 255 characters, excluding specialcharacters such as ;|&>'<$, and canbe empty.

job_log No String Path for storing job logs that recordjob running status. This path muststart with / or s3a://. A correct OBSpath is required.The parameter contains a maximumof 255 characters, excluding specialcharacters such as ;|&>'<$, and canbe empty.

shutdown_cluster No Bool Whether to delete the cluster after thejobs are completel true: Yesl false: No

file_action No String Data import and exportl importl export

submit_job_once_cluster_run

Yes Bool l true: A job is submitted when acluster is created.

l false: A job is submittedseparately.

The parameter is set to true in thisexample.

hql No String HiveQL statement



33


Type Description

hive_script_path Yes String SQL program pathThis parameter is needed by SparkScript jobs only and must meet thefollowing requirements:l Contains a maximum of 255


l Starts with / or s3a://.l Ends with .sql. sql is case-

insensitive.

Responsel Example:

{ "cluster_id": "da1592c2-bb7e-468d-9ac9-83246e95447a", "result": true, "msg": ""}


Table 4-6 Response parameter description


Type Description

cluster_id Yes String Cluster ID, which is returned by thesystem after the cluster is created.

result Yes Bool Operation resultl true: operation succeededl false: operation failed

msg No String System message, which can beempty.

Status Code

Table 4-7 describes the status code of this API.

Table 4-7 Status code

Status Code Description

200 The cluster is successfully created.



34

For the description about error status codes, see section Status Codes.

4.1.2 Adding Nodes to a Cluster

FunctionThis API is used to add Core nodes to a cluster. This API is incompatible with Sahara.

Capacity expansion is allowed only for clusters in the running or scaling-error state.

URIl Format:

PUT /v1.1/{project_id}/cluster_infos/{cluster_id}l Parameter description




cluster_id Yes Cluster ID

Requestl Example:

{ "service_id": "", "plan_id": "", "parameters": { "order_id": "", "scale_type": "scale_out", "node_id": "node_orderadd", "instances": "1", "include_instances": [ ], "exclude_instances": [ ] }, "previous_values": { "plan_id": "" }}




35



Type Description

service_id No String Service IDReserve the parameter for expandingAPIs. The user does not need toconfigure the parameter.

plan_id No String Plan IDReserve the parameter for expandingAPIs. The user does not need toconfigure the parameter.

parameters

order_id Yes String The order ID that is obtained by thesystem when the node is added. Theuser does not need to configure theparameter.

scale_type Yes String l scale_in: Delete nodes (notsupported currently).

l scale_out: Add nodes.

node_id Yes String The parameter must be set tonode_orderadd.

instances Yes String Number of nodes to be added ordeletedl The maximum number of nodes

to be added is 100 minus thenumber of cluster Core nodes. Forexample, the current number ofcluster Core nodes is 3, thenumber of nodes to be addedmust be less than or equal to 97.A maximum of 100 Core nodesare supported by default. If morethan 100 Core nodes are required,contact technical supportengineers or invoke a backgroundinterface to modify the database.

l Only added nodes can be deletedfor capacity reduction.



36


Type Description

include_instances Yes Array l If scale_type is set to scale_in,the node in the square bracket [ ]needs to be deleted (not supportedcurrently).

l If scale_type is set to scale_out,leave the square bracket [ ]empty.

For the parameter, leave the squarebracket [ ] empty.

exclude_instances Yes Array l If scale_type is set to scale_in,the node in the square bracket [ ]needs to be retained (notsupported currently).

l If scale_type is set to scale_out,leave the square bracket [ ]empty.

For the parameter, leave the squarebracket [ ] empty.

previous_values

plan_id No String Reserve the parameter for expandingAPIs. The user does not need toconfigure the parameter.

Responsel Example:

{ "result": "succeeded"}




Type Description

result Yes String Operation resultl succeeded: operation succeededl Table 4-11 describes the error

codes returned upon operationfailures.

The following table lists the error codes that may be returned if capacity expansion fails.



37

Table 4-11 Error code

Error Code Message

12000002 The parameter is invalid.

12000003 The cluster does not exist.

12000009 The method parameter is invalid.

12000013 Reducing the capacity for the XX cluster failed.

12000014 Expanding the capacity for the XX cluster failed.

12000017 Capacity expansion/reduction is not allowed for clustersthat are not in the [Running] state.

12000018 The cluster capacity is being expanded/reduced. Newcapacity expansion/reduction operations on the cluster arenot allowed.

12000028 The cluster has a maximum of N Core nodes.

12000029 Obtaining the quota failed.

12000030 The number of nodes in the cluster exceeds the availablequota.

12000031 The number of CPU cores in the cluster exceeds theavailable quota.

12000032 The total memory of the cluster exceeds the availablequota.

12000033 The number of disks in the cluster exceeds the availablequota.

12000034 The total disk capacity of the cluster exceeds the availablequota.

12000054 The operation is not supported.

Status Code




200 The Core nodes are successfully added.




38

4.1.3 Querying a Cluster List

FunctionThis API is used to query information about clusters created by a user. This API isincompatible with Sahara.

URIl Format:

GET /v1.1/{project_id}/cluster_infosl Parameter description





Requestl Example:

None.l Parameter description



clusterName No Cluster nameThe value is globally uniqueand contains only 1 to 64letters, digits, andunderscores (_).

pageSize No Maximum number ofclusters displayed on a pageThe value ranges from 1 to100.

currentPage No Current page number



39


clusterState No Cluster statusl existingl historyl startingl runningl terminatedl failedl abnormall terminatingl rebootingl shutdownl frozenl scaling-outl scaling-inl scaling-error

Responsel Example:

{ "clusterTotal": 1, "clusters": [ { "clusterId": "bc134369-294c-42b7-a707-b2036ba38524", "clusterName": "mrs_D0zW", "masterNodeNum": "2", "coreNodeNum": "3", "clusterState": "terminated", "createAt": "1498272043", "updateAt": "1498636753", "chargingStartTime": "1498273733", "billingType": "Metered", "dataCenter": "cn-north-1", "vpc": null, "duration": "0", "fee": null, "hadoopVersion": null, "masterNodeSize": null, "coreNodeSize": null, "componentList": [ { "componentId": "MRS 1.3.0_001", "componentName": "Hadoop", "componentVersion": "2.7.2", "componentDesc": "A framework that allows for the distributed processing of large data sets across clusters." }, { "componentId": "MRS 1.3.0_003", "componentName": "HBase", "componentVersion": "1.0.2", "componentDesc": "A scalable, distributed database that supports structured data storage for large tables." },



40

{ "componentId": "MRS 1.3.0_002", "componentName": "Spark", "componentVersion": "1.5.1", "componentDesc": "A fast and general engine for large-scale data processing." }, { "componentId": "MRS 1.3.0_004", "componentName": "Hive", "componentVersion": "1.2.1", "componentDesc": "A data warehouse infrastructure that provides data summarization and ad hoc querying." } ], "externalIp": null, "externalAlternateIp": null, "internalIp": null, "deploymentId": null, "remark": "", "orderId": null, "azId": null, "masterNodeProductId": null, "masterNodeSpecId": null, "coreNodeProductId": null, "coreNodeSpecId": null, "azName": "az10.dc1", "instanceId": null, "vnc": "v2/5a3314075bfa49b9ae360f4ecd333695/servers/e2cda891-232e-4703-995e-3b1406add01d/action", "tenantId": null, "volumeSize": 0, "volumeType": null, "subnetId": null, "subnetName": null, "securityGroupsId": null, "slaveSecurityGroupsId": null, "safeMode": 0, "clusterVersion": null, "nodePublicCertName": null, "masterNodeIp": "unknown", "privateIpFirst": null, "errorInfo": "", "clusterType": 0 } ]}




Type Description

clusterTotal Yes String Total numbers of clusters in a list

clusters Yes Array Cluster list parametersSee Table 4-16.



41

Table 4-16 clusters parameter description


Type Description

clusterId Yes String Cluster ID

clusterName Yes String Cluster name

masterNodeNum Yes String Number of Master nodes deployed ina cluster

coreNodeNum Yes String Number of Core nodes deployed in acluster

clusterState Yes String Cluster statusValid values include:l existingl historyl startingl runningl terminatedl failedl abnormall terminatingl rebootingl shutdownl frozenl scaling-outl scaling-inl scaling-error

createAt Yes String Cluster creation time

updateAt Yes String Cluster updated time

billingType Yes String Cluster charging mode

dataCenter Yes String Cluster work region

vpc Yes String VPC name

duration Yes String Cluster subscription duration

fee Yes String Cluster creation fee, which isautomatically calculated.

hadoopVersion Yes String Hadoop version

masterNodeSize Yes String Instance specification of a Masternode

coreNodeSize Yes String Instance specification of a Core node



42


Type Description

componentList Yes Array Component listSee Table 4-17.

externalIp Yes String External IP address

externalAlternateIp Yes String Backup external IP address

internalIp Yes String Internal IP address

deploymentId Yes String Deployment ID of a cluster

remark Yes String Remarks of a cluster

orderId Yes String Order ID for creating clusters

azId Yes String ID of an available zone

masterNodeProductId

Yes String Product ID of a Master node

masterNodeSpecId Yes String Specification ID of a Master node

coreNodeProductId Yes String Product ID of a Core node

coreNodeSpecId Yes String Specification ID of a Core node

azName Yes String Name of an availability zone

instanceId Yes String Instance ID

vnc Yes String URI address for remote login of theelastic cloud server

tenantId Yes String Project ID

volumeSize Yes Integer Size of a volume

volumeType Yes String Volume type

subnetId Yes String Subnet ID

clusterType Yes String Cluster type

subnetName Yes String Subnet name

securityGroupsId Yes String Security group ID

slaveSecurityGrou-psId

Yes String Standby security group ID

safeMode Yes String Running mode of an MRS clusterPossible values include:l 0: common model 1: safe mode



43


Type Description

clusterVersion Yes String Cluster version

nodePublicCertName

Yes String Name of the key file

masterNodeIp Yes String IP address of a Master node

privateIpFirst Yes String Primary private IP address

errorInfo Yes String Error information

chargingStartTime Yes String Time when charging starts

Table 4-17 componentList parameter description


Type Description

componentId Yes String Component ID

componentName Yes String Component name

componentVersion Yes String Component version

componentDesc Yes String Component description

Status Code




200 The cluster list information is successfully queried.


4.1.4 Viewing Cluster Details

Function

This API is used to query detailed information about a cluster. This API is incompatible withSahara.



44

URIl Format:

GET /v1.1/{project_id}/cluster_infos/{cluster_id}l Parameter description





Requestl Example:


None.

Responsel Example:

{ "cluster":{ "clusterId":"bdb064ff-2855-4624-90d5-e9a6376abd6e", "clusterName":"c17022001", "masterNodeNum":"2", "coreNodeNum":"3", "clusterState":"scaling-in", "createAt":"1487570757", "updateAt":"1487668974", "billingType":"Metered", "dataCenter":"southchina", "vpc":"20161218", "duration":"0", "fee":"0", "hadoopVersion":"", "masterNodeSize":"c2.2xlarge.linux.mrs", "coreNodeSize":"c2.2xlarge.linux.mrs", "componentList":[ { "componentId":"MRS 1.3.0_001", "componentName":"Hadoop", "componentVersion":"2.7.2", "componentDesc":"A framework that allows for the distributed processing of large data sets across clusters." }, { "componentId":"MRS 1.3.0_002", "componentName":"Spark", "componentVersion":"1.5.1", "componentDesc":"A fast and general engine for large-scale data processing." },



45

{ "componentId":"MRS 1.3.0_004", "componentName":"Hive", "componentVersion":"1.2.1", "componentDesc":"A data warehouse infrastructure that provides data summarization and ad hoc querying." }, { "componentId":"MRS 1.3.0_003", "componentName":"HBase", "componentVersion":"1.0.2", "componentDesc":"A scalable, distributed database that supports structured data storage for large tables." } ], "externalIp":"100.64.49.9", "externalAlternateIp":"100.64.49.13", "internalIp":"192.168.1.242", "deploymentId":"4ac46ca7-a488-4b91-82c2-e4d7aa9c40c2", "remark":"", "orderId":"null", "azId":"1d7b939b382c4c3bb3481a8ca10da768", "masterNodeProductId":"b35cf2d2348a445ca74b32289a160882", "masterNodeSpecId":"8ab05e503b4c42abb304e2489560063b", "coreNodeProductId":"dc970349d128460e960a0c2b826c427c", "coreNodeSpecId":"cdc6035a249a40249312f5ef72a23cd7", "azName":"az1.dc1", "instanceId":"4ac46ca7-a488-4b91-82c2-e4d7aa9c40c2", "vnc":null, "tenantId":"3f99e3319a8943ceb15c584f3325d064", "volumeSize":100, "volumeType":"SATA", "subnetName":"subnet-f-test", "securityGroupsId":"930e34e2-195d-401f-af07-0b64ea6603f8", "slaveSecurityGroupsId":"2ef3343e-3477-4a0d-80fe-4d874e4f81b8", "safeMode":1, "clusterVersion":"FusionInsight V100R002C61", "nodePublicCertName":"myp", "masterNodeIp":"192.168.1.242", "privateIpFirst":"192.168.1.234", "errorInfo":null, "chargingStartTime":"0" }}




Type Description

clusterId Yes String Cluster ID

clusterName Yes String Cluster name

masterNodeNum Yes String Number of Master nodes deployed ina cluster

coreNodeNum Yes String Number of Core nodes deployed in acluster



46


Type Description

clusterState Yes String Cluster statusValid values include:l existingl historyl startingl runningl terminatedl failedl abnormall terminatingl rebootingl shutdownl frozenl scaling-outl scaling-inl scaling-error

createAt Yes String Cluster creation time, which is a 10-bit timestamp.

updateAt Yes String Cluster update time, which is a 10-bittimestamp.

billingType Yes String Cluster charging mode

dataCenter Yes String Cluster work region

vpc Yes String VPC name

duration Yes String Cluster subscription duration

fee Yes String Cluster creation fee, which isautomatically calculated.

hadoopVersion Yes String Hadoop version

masterNodeSize Yes String Instance specification of a Masternode

coreNodeSize Yes String Instance specification of a Core node

componentList Yes Array Component listSee Table 4-21.

externalIp Yes String External IP address

externalAlternateIp Yes String Backup external IP address

internalIp Yes String Internal IP address



47


Type Description

deploymentId Yes String Deployment ID of a cluster

remark Yes String Remarks of a cluster

orderId Yes String Order ID for creating clusters

azId Yes String ID of an available zone

masterNodeProductId

Yes String Product ID of a Master node

masterNodeSpecId Yes String Specification ID of a Master node

coreNodeProductId Yes String Product ID of a Core node

coreNodeSpecId Yes String Specification ID of a Core node

azName Yes String Name of an availability zone

instanceId Yes String Instance ID

vnc Yes String URI address for remote login of theelastic cloud server

tenantId Yes String Project ID

volumeSize Yes Integer Size of a volume

subnetName Yes String Subnet name

securityGroupsId Yes String Security group ID

slaveSecurityGrou-psId

Yes String Standby security group ID

safeMode Yes String Running mode of an MRS clusterPossible values include:l 0: common model 1: safe mode

clusterVersion Yes String Cluster version

nodePublicCertName

Yes String Name of the key file

masterNodeIp Yes String IP address of a Master node

privateIpFirst Yes String Primary private IP address

errorInfo Yes String Error information

chargingStartTime Yes String Time when charging starts



48

Table 4-21 componentList parameter description


Type Description

componentId Yes String Component ID

componentName Yes String Component name

componentVersion Yes String Component version

componentDesc Yes String Component description

Status Code




200 Details about a cluster are queried successfully.


4.1.5 Terminating a Cluster

Function

This API is used to terminate a cluster after data processing and analysis are completed or thecluster is abnormal. This API is compatible with Sahara.

Clusters in any of the following state cannot be terminated:

l scaling-out

l scaling-in

l starting

l terminating

l terminated

l failed

l scaling-error

URIl Format:

DELETE /v1.1/{project_id}/clusters/{cluster_id}




49




clusterId Yes Cluster ID

Requestl Example:


None.

Responsel Example:


None.

Status CodeTable 4-24 describes the status code of this API.



204 The cluster is successfully terminated.


4.2 Job Object InterfaceA job object lists the binaries that a job needs to run. To run a job, you must specify datasources and job parameters. You can run a job on an existing or new transient cluster.

4.2.1 Adding a Job and Executing the Job

FunctionThis API is used to add a job to an MRS cluster and execute the job. This API is incompatiblewith Sahara.



50

URIl Format:

POST /v1.1/{project_id}/jobs/submit-jobl Parameter description


Parameter Mandatoryor Not

Description

project_id Yes Project ID. For details on how to obtainthe project ID, see Obtaining a ProjectID.

Requestl Example:

The request example of MapReduce job:{ "job_type": 1, "job_name": "mrs_test_jobone_20170602_141106", "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", "jar_path": "s3a://mrs-opsadm/jarpath/hadoop-mapreduce-examples-2.7.2.jar", "arguments": "wordcount", "input": "s3a://mrs-opsadm/input/", "output": "s3a://mrs-opsadm/output/", "job_log": "s3a://mrs-opsadm/log/", "file_action": "", "hql": "", "hive_script_path": ""}

The request example of Spark job:{ "job_type": 2, "job_name": "mrs_test_sparkjob_20170602_141106", "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", "jar_path": "s3a://mrs-opsadm/jarpath/spark-test.jar", "arguments": "org.apache.spark.examples.SparkPi 10", "input": "", "output": "s3a://mrs-opsadm/output/", "job_log": "s3a://mrs-opsadm/log/", "file_action": "", "hql": "", "hive_script_path": ""}

The request example of Hive Script job:{ "job_type": 3, "job_name": "mrs_test_SparkScriptJob_20170602_141106", "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", "jar_path": "s3a://mrs-opsadm/jarpath/Hivescript.sql", "arguments": "", "input": "s3a://mrs-opsadm/input/", "output": "s3a://mrs-opsadm/output/", "job_log": "s3a://mrs-opsadm/log/", "file_action": "", "hql": "", "hive_script_path": "s3a://mrs-opsadm/jarpath/Hivescript.sql"}



51

The request example of DistCp job for import:{ "job_type": 5, "job_name": "mrs_test_importjob_20170602_141106", "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", "input": "s3a://mrs-opsadm/jarpath/hadoop-mapreduce-examples-2.7.2.jar", "output": "/user", "file_action": "import"}

The request example of DistCp job for export:{ "job_type": 5, "job_name": "mrs_test_exportjob_20170602_141106", "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", "input": "/user/hadoop-mapreduce-examples-2.7.2.jar", "output": "s3a://mrs-opsadm/jarpath/", "file_action": "export"}

The request example of Spark Script job:{ "job_type": 6, "job_name": "mrs_test_sparkscriptjob_20170602_141106", "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", "jar_path": "s3a://mrs-opsadm/jarpath/sparkscript.sql", "arguments": "", "input": "s3a://mrs-opsadm/input/", "output": "s3a://mrs-opsadm/output/", "job_log": "s3a://mrs-opsadm/log/", "file_action": "", "hql": "", "hive_script_path": "s3a://mrs-opsadm/jarpath/sparkscript.sql"}




52



Type Description


currently)l 5: DistCp, importing and

exporting data. For details, seeTable 4-27.

l 6: Spark Scriptl 7: Spark SQL, submitting Spark

SQL statement. For details, seeTable 4-28. (not supported in thisAPI currently)NOTE

Spark and Hive jobs can be added toonly clusters including Spark andHive components.

job_name Yes String Job nameContains only 1 to 64 letters, digits,and underscores (_).NOTE


cluster_id Yes String Cluster ID

jar_path Yes String Path of the .jar package or .sql filefor program executionThe parameter must meet thefollowing requirements:l Contains a maximum of 255


l Starts with / or s3a://.l Spark Script must end with .sql;

while MapReduce and Spark Jarmust end with .jar. sql and jarare case-insensitive.



53


Type Description

arguments No String Key parameter for programexecution. The parameter is specifiedby the function of the user's program.MRS is only responsible for loadingthe parameter.The parameter contains a maximumof 255 characters, excluding specialcharacters such as ;|&>'<$, and canbe empty.

input No String Path for inputting data, which muststart with / or s3a://. A correct OBSpath is required.The parameter contains a maximumof 255 characters, excluding specialcharacters such as ;|&>'<$, and canbe empty.

output No String Path for outputting data, which muststart with / or s3a://. A correct OBSpath is required. If the path does notexist, the system automaticallycreates it.The parameter contains a maximumof 255 characters, excluding specialcharacters such as ;|&>'<$, and canbe empty.

job_log No String Path for storing job logs that recordjob running status. This path muststart with / or s3a://. A correct OBSpath is required.The parameter contains a maximumof 255 characters, excluding specialcharacters such as ;|&>'<$, and canbe empty.

hive_script_path Yes String SQL program path, needed by SparkScript jobs only. The parameter mustmeet the following requirements:l Contains a maximum of 255


l Starts with / or s3a://.l Ends with .sql. sql is case-

insensitive.



54


Type Description

is_protected No Bool Whether is protectedl truel falseThe current version does not supportthis function.

is_public No Bool Whether is publicl truel falseThe current version does not supportthis function.

Table 4-27 DistCp parameter description


Type Description

job_name Yes String Job nameContains only 1 to 64 letters, digits,and underscores (_).NOTE


input No String Data source pathl When you import data, the

parameter is set to an OBS path.l When you export data, the

parameter is set to an HDFS path.

output No String Data receiving pathl When you import data, the

parameter is set to an HDFS path.l When you export data, the

parameter is set to an OBS path.

file_action Yes String Types of file operations, including:l export: Export data from HDFS

to OBS.l import: Import data from OBS to

HDFS.



55

Table 4-28 Spark SQL parameter description

Parameter Mandatoryor Not

Type Description

hql Yes String Spark SQL statement, which needsBase64 encoding and decoding.ABCDEFGHIJKLMNOPQR-STUVWXYZabcdefghijklmnopqr-stuvwxyz0123456789+/ is a standardcoding table. MRS usesABCDEFGHILKJMNOPQR-STUVWXYZabcdefghijklmnopqr-stuvwxyz0123456789+/ to encodethe Spark SQL statement into Base64strings. The value of the Hqlparameter is generated by adding anyletter to the beginning of the encodedcharacter string. The Spark SQLstatement is generated by decodingthe value in the background.Example:1. On the Web interface, enter the

Spark SQL statement showtables;.

2. UseABCDEFGHILKJMNOPQR-STUVWXYZabcdefghijklm-nopqrstuvwxyz0123456789+/toencode the Spark SQL statementinto a character stringc2hvdyB0YWLsZXM7.

3. At the beginning ofc2hvdyB0YWLsZXM7, add anycharacter, for examplegc2hvdyB0YWLsZXM7, togenerate the value of the Hqlparameter.

4. The Spark SQL statement showtables; is generated byautomatically decoding in thebackground.

job_name Yes String Job name, which contains only 1 to64 letters, digits, and underscores (_).NOTE




56

Responsel Example:

{ "job_execution": { "templated": false, "created_at": 1496387588913, "updated_at": 1496387588913, "id": "12ee9ae4-6ee1-48c6-bb84-fb0b4f76cf03", "tenant_id": "c71ad83a66c5470496c2ed6e982621cc", "job_id": "", "job_name": "mrs_test_jobone_20170602_141106", "input_id": null, "output_id": null, "start_time": 1496387588913, "end_time": null, "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", "engine_job_id": null, "return_code": null, "is_public": null, "is_protected": null, "group_id": "12ee9ae4-6ee1-48c6-bb84-fb0b4f76cf03", "jar_path": "s3a://mrs-opsadm/jarpath/hadoop-mapreduce-examples-2.7.2.jar", "input": "s3a://mrs-opsadm/input/", "output": "s3a://mrs-opsadm/output/", "job_log": "s3a://mrs-opsadm/log/", "job_type": 1, "file_action": "", "arguments": "wordcount", "hql": "", "job_state": 2, "job_final_status": 0, "hive_script_path": "", "create_by": "b67132be2f054a45b247365647e05af0", "finished_step": 0, "job_main_id": "", "job_step_id": "", "postpone_at": 1496387588911, "step_name": "", "step_num": 0, "task_num": 0, "update_by": "b67132be2f054a45b247365647e05af0", "credentials": "", "user_id": "b67132be2f054a45b247365647e05af0", "job_configs": null, "extra": null, "data_source_urls": null, "info": null }}




Type Description

templated Yes Bool Whether job execution objects aregenerated by job templates

created_at Yes Integer Creation time, which is a 13-bittimestamp.



57


Type Description

updated_at Yes Integer Update time, which is a 13-bittimestamp.

id Yes String Job ID

tenant_id Yes String Project ID

job_id Yes String Job application ID

job_name Yes String Job name

input_id Yes String Null

output_id Yes String Null

start_time Yes Integer Start time of job execution, which isa 13-bit timestamp.

end_time Yes Integer End time of job execution, which is a13-bit timestamp.

cluster_id Yes String Cluster ID

engine_job_id Yes String Oozie work flow ID

return_code Yes Integer Returned code for an execution result

is_public Yes Bool Whether a job is publicl truel falseThe current version does not supportthis function.

is_protected Yes Bool Whether a job is protectedl truel falseThe current version does not supportthis function.

group_id Yes String Group ID of a job

jar_path Yes String Path of the .jar package for programexecution

input Yes String Address for inputting data

output Yes String Address for outputting data

job_log Yes String Address for storing job logs



58


Type Description


temporarily)l 5: DistCpl 6: Spark Scriptl 7: Spark SQL (not supported in

this API currently)

file_action Yes String Data input and output

arguments Yes String Key parameter for programexecution. The parameter is specifiedby the function of the user's internalprogram. MRS is only responsiblefor loading the parameter. Thisparameter can be empty.

job_state Yes Integer Job statusl -1: Terminatedl 1: Startingl 2: Runningl 3: Completedl 4: Abnormall 5: Error

job_final_status Yes Integer Final job statusl 0: unfinishedl 1: terminated due to an execution

errorl 2: executed successfullyl 3: canceled

hive_script_path Yes String Hive script address

create_by Yes String User ID for creating jobsThis parameter is not used in thecurrent version, but is retained forcompatibility with earlier versions.



59


Type Description

finished_step Yes Integer Number of completed stepsThis parameter is not used in thecurrent version, but is retained forcompatibility with earlier versions.

job_main_id Yes String Main ID of a jobThis parameter is not used in thecurrent version, but is retained forcompatibility with earlier versions.

job_step_id Yes String Step ID of a jobThis parameter is not used in thecurrent version, but is retained forcompatibility with earlier versions.

postpone_at Yes Integer Delay time, which is a 13-bittimestamp.This parameter is not used in thecurrent version, but is retained forcompatibility with earlier versions.

step_name Yes String Step name of a jobThis parameter is not used in thecurrent version, but is retained forcompatibility with earlier versions.

step_num Yes Integer Number of stepsThis parameter is not used in thecurrent version, but is retained forcompatibility with earlier versions.

task_num Yes Integer Number of tasksThis parameter is not used in thecurrent version, but is retained forcompatibility with earlier versions.

update_by Yes String User ID for updating jobs

credentials Yes String TokenThe current version does not supportthis function.

user_id Yes String User ID for creating jobsThis parameter is not used in thecurrent version, but is retained forcompatibility with earlier versions.

job_configs Yes String Key-value pair set for saving jobrunning configurations



60


Type Description

extra Yes String Authentication informationThe current version does not supportthis function.

data_source_urls Yes String Data source URL of a job

info Yes String Key-value pair set, containing jobrunning information returned byOozie

Status Code


Table 4-30 Status Code


200 The job is successfully added.


4.2.2 Querying the exe Object List of Jobs

Function

This API is used to query the exe object list of all jobs. This API is incompatible with Sahara.

URIl Format:

GET /v1.1/{project_id}/job-exesl Parameter description



Description


cluster_id No Cluster ID

id No Job ID



61


Description

page_size No Maximum number of jobs displayed on apageThe value ranges from 1 to 100.

current_page No Current page number

job_name No Job name

state No Job status codel -1: Terminatedl 1: Startingl 2: Runningl 3: Completedl 4: Abnormall 5: Error

Requestl Example:

GET/v1.1/{project_id}/job-exes?page_size=10&current_page=1&state=3&job_name=myfirstjob

l Parameter descriptionNone.

Responsel Example:

{ "totalRecord": 14, "job_executions": [ { "id": "669476bd-89d2-45aa-8f1a-872d16de377e", "create_at": 1484641003707, "update_at": 1484641003707, "tenant_id": "3f99e3319a8943ceb15c584f3325d064", "job_id": "669476bd-89d2-45aa-8f1a-872d16de377e", "job_name": "myfirstjob", "start_time": 1484641003707, "end_time": null, "cluster_id": "2b460e01-3351-4170-b0a7-57b9dd5ffef3", "group_id": "669476bd-89d2-45aa-8f1a-872d16de377e", "jar_path": "s3a://jp-test1/program/hadoop-mapreduce-examples-2.4.1.jar", "input": "s3a://jp-test1/input/", "output": "s3a://jp-test1/output/", "job_log": "s3a://jp-test1/joblogs/", "job_type": 1, "file_action": "", "arguments": "wordcount", "hql": "", "job_state": 2, "job_final_status": 1, "hive_script_path": null,



62

"create_by": "3f99e3319a8943ceb15c584f3325d064", "finished_step": 0, "job_main_id": "", "job_step_id": "", "postpone_at": 1484641003174, "step_name": "", "step_num": 0, "task_num": 0, "update_by": "3f99e3319a8943ceb15c584f3325d064", "spend_time": null, "step_seq": 222, "progress": "first progress" } ]}




Type Description

totalRecord Yes Integer Total numbers of jobs in a list

job_executions Yes Array Job list parameterFor details, see Table 4-33.

Table 4-33 job_executions parameter description


Type Description


create_at Yes Integer Creation time, which is a 13-bittimestamp.

update_at Yes Integer Update time, which is a 13-bittimestamp.


job_id Yes String Job ID




cluster_id Yes String Cluster ID of a job

group_id No String Group ID of a job

jar_path No String Path of the .jar package or .sql filefor program execution



63


Type Description

input No String Address for inputting data

output No String Address for outputting data

job_log No String Address for storing job logs

job_type No Integer Job type codel 1: MapReducel 2: Sparkl 3: Hive Scriptl 4: HiveQL (not supported

temporarily)l 5: DistCpl 6: Spark Scriptl 7: Spark SQL (Not supported in

this api currently)

file_action No String Data input and output

arguments No String Key parameter for programexecution. The parameter is specifiedby the function of the user's internalprogram. MRS is only responsiblefor loading the parameter. Thisparameter can be empty.


job_state Yes Integer Job status codel -1: Terminatedl 1: Startingl 2: Runningl 3: Completedl 4: Abnormall 5: Error

job_final_status Yes Integer Job final statusl 0: unfinishedl 1: terminated due to an execution


hive_script_path Yes String Address of the Hive script

create_by No String User ID for creating jobs



64


Type Description

finished_step No Integer Number of completed steps

job_main_id No String Main ID of a job

job_step_id No String Step ID of a job

postpone_at No Integer Delay time, which is a 13-bittimestamp.

step_name No String Step name of a job

step_num No Integer Number of steps

task_num No Integer Number of tasks

update_by No String User ID for updating jobs

spend_time No Integer Duration of job execution (unit: s)

step_seq No Integer Step sequence of a job

progress No String Job execution progress

Status Code




200 The exe object list of jobs is queried successfully.


4.2.3 Querying exe Object Details

Function

This API is used to query detailed information about the exe object of a job. This API isincompatible with Sahara.

URIl Format:

GET /v1.1/{project_id}/job-exes/{job_exe_id}




65



Description


job_exe_id Yes exe object ID

Requestl Example:

None.


None.

Responsel Example:

{ "job_execution": { "id": "632863d5-15d4-4691-9dc1-1a72340cb097", "create_at": 1484240559176, "update_at": 1484240559176, "tenant_id": "3f99e3319a8943ceb15c584f3325d064", "job_id": "632863d5-15d4-4691-9dc1-1a72340cb097", "job_name": "hive_script", "start_time": 1484240559176, "end_time": null, "cluster_id": "8b1d55f6-150e-45e2-8347-b2ca608d366b", "group_id": "632863d5-15d4-4691-9dc1-1a72340cb097", "jar_path": "s3a://jp-test1/program/Hivescript.sql", "input": "s3a://jp-test1/input/", "output": "s3a://jp-test1/output/", "job_log": "s3a://jp-test1/joblogs/", "job_type": 3, "file_action": "", "arguments": "wordcount", "hql": null, "job_state": 3, "job_final_status": 1, "hive_script_path": "s3a://jp-test1/program/Hivescript.sql", "create_by": "3f99e3319a8943ceb15c584f3325d064", "finished_step": 0, "job_main_id": "", "job_step_id": "", "postpone_at": 1484240558705, "step_name": "", "step_num": 0, "task_num": 0, "update_by": "3f99e3319a8943ceb15c584f3325d064", "spend_time": null, "step_seq": 222, "progress": "first progress" }}




66



Type Description


create_at Yes Integer Creation time, which is a 13-bittimestamp.

update_at Yes Integer Update time, which is a 13-bittimestamp.


job_id Yes String Job ID




cluster_id Yes String Cluster ID of a job

group_id No String Group ID of a job

jar_path No String Path of the .jar package or .sql filefor program execution

input No String Address for inputting data

output No String Address for outputting data

job_log No String Address for storing job logs

job_type No Integer Job type codel 1: MapReducel 2: Sparkl 3: Hive Scriptl 4: HiveQL (not supported

temporarily)l 5: DistCpl 6: Spark Scriptl 7: Spark SQL (Not supported in

this api currently)

file_action No String Data input and output



67


Type Description

arguments No String Key parameter for programexecution. The parameter is specifiedby the function of the user's internalprogram. MRS is only responsiblefor loading the parameter. Thisparameter can be empty.


job_state Yes Integer Job status codel -1: Terminatedl 1: Startingl 2: Runningl 3: Completedl 4: Abnormall 5: Error

job_final_status Yes Integer Final job statusl 0: unfinishedl 1: terminated due to an execution


hive_script_path Yes String Address of the Hive script

create_by No String User ID for creating jobs

finished_step No Integer Number of completed steps

job_main_id No String Main ID of a job

job_step_id No String Step ID of a job

postpone_at No Integer Delay time, which is a 13-bittimestamp.

step_name No String Step name of a job

step_num No Integer Number of steps

task_num No Integer Number of tasks

update_by No String User ID for updating jobs

spend_time No Integer Duration of job execution (unit: s)

step_seq No Integer Step sequence of a job

progress No String Job execution progress



68

Status Code



Status code Description

200 The exe object details are queried successfully.


4.3 Job Execution Object InterfaceA job execution object represents a Hadoop job that runs on a cluster. A job execution pollsthe status of a running job and reports it to the user. Also a user can cancel a running job.

4.3.1 Deleting a Job Execution Object

Function

This API is used to delete a job execution object. This API is compatible with Sahara.

URIl Format:

DELETE /v1.1/{project_id}/job-executions/{job_execution_id}l Parameter description



Description


job_execution_id Yes Job execution object ID

Requestl Example:


None.



69

Responsel Example:


None.

Status CodeTable 4-39 describes the status code of this API.



204 The job execution object is deleted successfully.




70

A Change History

Release Date What's New

2017-07-14 This issue is the third official release.Modified the following contents:l Creating a Cluster and Running a Jobl Querying a Cluster List

2017-06-23 This issue is the second official release.l Added the following content:

Obtaining a Project IDl Modified the following contents:

– Creating a Cluster and Running a Job– Creating a Cluster and Running a Job– Adding Nodes to a Cluster– Viewing Cluster Details– Terminating a Cluster– Job Object Interface– Adding a Job and Executing the Job– Querying the exe Object List of Jobs– Querying exe Object Details– Job Execution Object Interface– Deleting a Job Execution Object

l Deleted the following contents:– Log Management Interface– File Management Interface

2016-08-25 This issue is the first official release.

MapReduce ServiceAPI Reference A Change History


71

api reference · mapreduce service api reference issue 03 date 2017-07-14 huawei technologies co.,...

Documents