aws batch private beta getting started guide · aws batch private beta getting started guide...

AmazonConfidential–SharedUnderNDA 1

AWSBatchPrivateBetaGettingStartedGuideNovember15,2016

AmazonConfidential–SharedunderNDA

ThankyouforparticipatingintheAWSBatchPrivateBetaProgram.Yourwillingnesstotestandprovidefeedbackonanunreleasedserviceisgreatlyappreciatedbytheteam.Afewcaveatsforyoutotakeintoconsiderationbeforeyoubeginusingthisservice.First,thisisaprivatebetaandtheserviceisnotyetreadyforproductionworkloads.Wewillmakereasonableattemptstoensurethattheservicefunctionsasexpectedduringthisperiod,butdonotbesurprisedifthingsbreakorbehavedifferentlythantheyshouldfromtimetotime.Second,youshouldassumethattheresourcescreatedaspartofthisbetawillnotpersistaftertheconclusionofthePrivateBeta.AnyJobQueues,ComputeEnvironments,JobDefinitions,andotherAWSBatchPrivateBetaresourcescanandlikelywillbedeletedattheendofthePrivateBetaperiod.Third,pleaserespecttheconfidentialityofthisPrivateBetaprogram.AllaspectsofthisservicearesharedwithyouunderthetermsofyouragreementswithAWS,includingthenon-disclosureagreement.

Questions&Issues:AsyouareworkingtestingAWSBatch,[email protected].

TableofContents: Questions&Issues:................................................................................................................................................1

TableofContents:..................................................................................................................................................1

Introduction:..........................................................................................................................................................2

CurrentcapabilitiesandKnownIssues:..................................................................................................................3ComputeEnvironments(CEs):...........................................................................................................................................3JobQueues(JQs):...............................................................................................................................................................3JobDefinitionsandJobs:...................................................................................................................................................3AWSBatchWebConsole:..................................................................................................................................................3

Downloadfilesusedinthisexample:.....................................................................................................................3

AWSCommand-LineInterface(CLI)Configuration:...............................................................................................3

AccountSetupActivities:.......................................................................................................................................4CreateanECSInstanceRole:.............................................................................................................................................4CreateaServiceRole:........................................................................................................................................................7

Step1:CreateanIAMPolicy.........................................................................................................................................7Step2:CreateanIAMRole............................................................................................................................................8Step3:AttachtheIAMPolicytoyourIAMRole............................................................................................................9Step4:EnsurethatyourVPCsupportsexternalnetworkaccess..................................................................................9

TestingtheHappyPath:.........................................................................................................................................9CreateaComputeEnvironment:.....................................................................................................................................10DescribeYourComputeEnvironment:............................................................................................................................10


CreateaJobQueue:.........................................................................................................................................................11DescribeYourJobQueue:................................................................................................................................................11RegisteraJobDefinition:.................................................................................................................................................12DescribeYourJobDefinition:...........................................................................................................................................12SubmitaJob:....................................................................................................................................................................13ListYourRunningJob:......................................................................................................................................................13DescribeYourRunningJob:.............................................................................................................................................14

CleanupYourHappyPathEnvironment:..............................................................................................................14DeregisterYourJobDefinition:........................................................................................................................................14UpdateYourJobQueue:..................................................................................................................................................14DeleteYourJobQueue:...................................................................................................................................................14UpdateYourComputeEnvironment:..............................................................................................................................15DeleteYourComputeEnvironment:................................................................................................................................15

Additionaltestsyoumaywanttoconduct:.........................................................................................................15AJobQueuewithMultipleComputeEnvironments:......................................................................................................15MultipleJobQueuesMappedtotheSameComputeEnvironment:...............................................................................15CreateanUnmanagedComputeEnvironment:...............................................................................................................15SubmitaCollectionofJobswithVaryingResourceRequirements:................................................................................16

Concepts:..............................................................................................................................................................17Jobs:.................................................................................................................................................................................17JobDefinitions:................................................................................................................................................................17JobQueues:.....................................................................................................................................................................17ComputeEnvironments:..................................................................................................................................................18

ManagedComputeEnvironments:..............................................................................................................................18UnmanagedComputeEnvironment:...........................................................................................................................18

Workflows,Pipelines,andJobDependencies:................................................................................................................18Scheduler:........................................................................................................................................................................18

Introduction:Batchcomputingisacommonwayfordevelopers,scientistsandengineerstoaccesslargeamountsofcomputeresources.AWSBatchremovestheundifferentiatedheavyliftingofconfiguringandmanagingtherequiredinfrastructure,makingiteasierthanevertorunbatchcomputingworkloadsontheAWScloud.Theservicealsomakesiteasyforadministrators,developers,andindependentsoftwarevendors(ISVs)toprovidevaluable,time-savingapplicationtemplatesfortheirusers.AWSBatchwillbefamiliartousersoftraditionalbatchcomputingsoftwareandcanefficientlyprovisionresourcesinresponsetojobssubmittedinordertoeliminatecapacityconstraints,reducecomputecosts,anddeliverresultsfasterthanever.Asafully-managedservice,AWSBatchautomaticallyprovisionscomputeresourcesandoptimizestheworkloaddistributionbasedonthequantityandscaleofthejobssubmitted.WithAWSBatch,thereisnoneedtoinstallormanagebatchcomputingsoftware,allowingyoutofocusonanalyzingresultsandsolvingproblems.AWSBatchplans,schedules,andexecutesyourbatchcomputingworkloadsacrossthefullrangeofAWScomputeservices,suchasAmazonEC2,AmazonEC2SpotwithLambdasupportarrivingshortlyafterlaunch.AWSBatchreducesoperationalcomplexities,savingtimeandreducingcosts,makingiteasyfordevelopers,scientists,andengineerstoruntheirbatchjobsintheAWScloud.


CurrentcapabilitiesandKnownIssues:ComputeEnvironments(CEs):ManagedComputeEnvironmentsdonotyetscaleupordown.ThenumberofinstanceslaunchedisdeterminedbythedesiredvCpusspecifiedfortheCE. JobQueues(JQs):JobQueueoperationsfunctionasexpected. JobDefinitionsandJobs:WedonotyetsupportLambdaJobsorArrayJobs(“copies”>1).JobDefinitionscanonlybeusedtospecifyacontainerizedjob.Jobdependenciesarenotyetsupported.AWSBatchWebConsole:ThePrivateBetadoesnotyetincludeaccesstotheAWSBatchWebConsole.AlloperationsmustbesubmittedviatheAWSCLI.

Downloadfilesusedinthisexample:Youcandownloadthesamplecodeforthistutorialfrom:https://s3.amazonaws.com/batch-preview/docs/AWS-Batch-Private-Beta-GSG-commands.txt

AWSCommand-LineInterface(CLI)Configuration:TheAWSCLIisabotocorebasedpackagethatsupportsdynamicadditionofadditionalservices.Ifyouhaven’talreadyinstalledtheAWSCLI,pleasefollowtheinstructionspublishedat:https://aws.amazon.com/cli/InordertorunAWSBatchcommandsviaAWSCLI,youwillneedtoinstalltheC2Jservicemodel.Pleasedownloadthefollowingfiletoyourcomputer:https://s3.amazonaws.com/batch-preview/model/aws-batch-private-model.jsonToaddAWSBatchtoyourAWSCLI,usethefollowingcommand:aws configure add-model --service-name batch --service-model file://aws-batch-private-model.json

Iftherearenoerrors,therewillbenooutput.Youcantestthatthemodelhasbeenaddedwiththefollowingcommand:aws batch help

Theoutputshouldlooksimilartothefollowing:BATCH() BATCH() NAME


batch - DESCRIPTION AWS Batch makes it easier than ever to run batch computing workloads on the AWS cloud. Batch computing is a common way for developers, scientists and engineers to access large amounts of compute resources, and AWS Batch removes the undifferentiated heavy lifting of configuring and managing the required infrastructure. It also makes it easy for admin- istrators, developers and independent software vendors to provide valu- able, time-saving application templates for their users. AWS Batch will be familiar to users of traditional batch computing software and can efficiently provision resources in response to jobs submitted in order to eliminate capacity constraints, reduce compute costs, and deliver results faster than ever. As a fully-managed service, AWS Batch enables developers, scientists, and engineers to easily and efficiently run batch computing workloads of any scale. AWS Batch automatically provisions compute resources and optimizes the workload distribution based on the quantity and scale of the workloads. With AWS Batch, there is no need to install or manage batch computing software, allowing you to focus on analyzing results and solving problems. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute ser- vices, such as Amazon EC2, Amazon EC2 Spot, and AWS Lambda. AWS Batch reduces operational complexities, saving time and reducing costs, mak- ing it easy for developers, scientists, and engineers to run their batch jobs in the AWS cloud. AVAILABLE COMMANDS o cancel-job o create-compute-environment o create-job-queue o create-resource

YoucanconfiguresecuritycredentialsandothersettingsfortheAWSCLIbyfollowingtheseinstructions:http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

AccountSetupActivities:BeforeyoucanworkwithAWSBatchduringtheprivatebeta,youwillneedtocreatesomeAWSIdentityandAccessManagement(IAM)roles.Specifically,youwillneedanECSInstanceRolethatthatenablesinstanceswithinyourComputeEnvironmentstointeractwithotherAWSServices.YouwillalsoneedaServiceRolethatdefineswhichservicesAWSBatchmayinvokeonyourbehalfinordertooperatetheserviceandcreate,modify,ordeleteresourcessuchasECSClusters,LaunchConfigurations,AutoScalingGroups,andCloudWatchLogGroups. CreateanECSInstanceRole:Inordertocreateyourinstancerole,logintotheAWSWebConsoleforIAMandnavigatetotheRolespage:https://console.aws.amazon.com/iam/home#roles


Yourbrowserwindowshouldlooksomethinglikethis:

Clickonthe“CreateNewRole”button,enterthename“AWSBatchECSInstanceRole”andclick“NextStep”


ScrolldownwithintheAWSServiceRolecategoryandselecttherolecalled“AmazonEC2RoleforEC2ContainerService”andclickthe“Select”button.

Choosethemanagedpolicythatispresentedinthelistandclick“NextStep”


Thepageshouldnowlooksomethinglikethis.Reviewtheroleandclick“CreateRole”

Findyourroleontheresultingpageandconfirmthatitwassuccessfullycreated. CreateaServiceRole:NowyouneedtocreateaServiceRolethatwillgiveAWSBatchpermissiontocreateandmanageAWSresourcesonyourbehalfinordertooperatetheservice.Atlaunch,AWSBatchwillcreatethisServiceRoleforyouautomatically.DuringthePrivateBetayouwillneedtocreateitmanuallyusingathreestepprocess:

1. CreateanIAMPolicy2. CreateanIAMRole3. AttachtheIAMPolicytotheIAMRole

Note:Inordertocompletethisstep,youneedtohavetheAWSCLIproperlyinstalledandconfiguredtouseAWScredentialswithiam:*permissions.Step1:CreateanIAMPolicyFirstcreateafilecalledAWSBatchBetaServicePolicy.jsonwiththefollowingcontents:{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "autoscaling:*", "ecs:*", "ec2:*", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "iam:PassRole" ],


"Resource": "*" } ] }

NextcreateanIAMPolicycalledAWSBatchBetaServicePolicywiththefollowingAWSCLIcommand: aws iam create-policy --policy-name AWSBatchBetaServicePolicy --policy-document file://AWSBatchBetaServicePolicy.json Theoutputshouldlooksimilartothefollowing:{ "Policy": { "PolicyName": "AWSBatchBetaServicePolicy", "CreateDate": "2016-11-06T20:17:26.627Z", "AttachmentCount": 0, "IsAttachable": true, "PolicyId": "AMPSIQB76QBFYGEZ5I2PM", "DefaultVersionId": "v1", "Path": "/", "Arn": "arn:aws:iam::012345678912:policy/AWSBatchBetaServicePolicy", "UpdateDate": "2016-11-06T20:17:26.627Z" } }

Step2:CreateanIAMRoleNowthatanIAMPolicyhasbeencreated,thenextstepistocreateanIAMRole.ThisrequiresapolicydocumentthatisspecifictotheAWSBatchPrivateBetaenvironment.FirstcreateafilecalledAWSBatchBetaServiceRole.jsonwiththefollowingcontents:{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::398747504704:root", "arn:aws:iam::398264837155:root" ] }, "Action": "sts:AssumeRole" } ] }

Next,createanIAMRolecalledAWSBatchBetaServiceRolewiththefollowingAWSCLIcommand: aws iam create-role --role-name AWSBatchBetaServiceRole --assume-role-policy-document file://AWSBatchBetaServiceRole.json

Theoutputshouldlooksimilartothefollowing:{ "Role": { "AssumeRolePolicyDocument": { "Version": "2012-10-17",


"Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::398747504704:root", "arn:aws:iam::398264837155:root" ] } } ] }, "RoleId": "AROAIZAGRNO45LX666252", "CreateDate": "2016-11-06T20:44:18.566Z", "RoleName": "AWSBatchBetaServiceRole", "Path": "/", "Arn": "arn:aws:iam::012345678912:role/AWSBatchBetaServiceRole" } }

PleasenotetheARNforyourAWSBatchBetaServiceRole,asyouwillneeditlater.TheARNlistedaboveissimplyanexample.Step3:AttachtheIAMPolicytoyourIAMRoleToattachtheAWSBatchBetaServicePolicyIAMPolicytotheAWSBatchBetaServiceRoleIAMRole,usethefollowingAWSCLIcommand:aws iam attach-role-policy --role-name AWSBatchBetaServiceRole --policy-arn arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSBatchBetaServicePolicy

Iftherearenoerrors,therewillbenooutput.Step4:EnsurethatyourVPCsupportsexternalnetworkaccessInstancesinyourcomputeenvironments(bothmanagedandunmanaged)requireexternalnetworkaccessinorderfortheECSAgenttocommunicatewiththeAmazonECSserviceendpoint.ifyourcontainerinstancesdonothavepublicIPaddresses,thentheymustusenetworkaddresstranslation(NAT)oranHTTPproxytoprovidethisaccess.Formoreinformation,seeNATInstancesintheAmazonVPCUserGuideandHTTPProxyConfigurationinthisguide.

TestingtheHappyPath:Congratulations!YouarenowreadytobeginusingAWSBatch.BycompletingtheAWSBatchHappyPath,youwillbecomefamiliarwiththeprimarycapabilitiesoftheservicebycreatingandtestingasimple,yetfully-capableAWSBatchconfiguration.Uponcompletionofthisexercise,youwillhaveperformedthefollowing:

1. CreateanddescribeaComputeEnvironment2. CreateanddescribeaJobQueue3. RegisteraJobDefinition4. SubmityourfirstAWSBatchjob


CreateaComputeEnvironment:AComputeEnvironmentistheresourcewithinAWSBatchthatdefinestheComputeResourcesthatwillbeavailableforcontainerjobs.Forthehappypath,theComputeEnvironmentwillbeMANAGED,usingEC2on-demandinstances.IMPORTANT!YouwillneedtousethefollowingattributesfromyourownAWSaccountinordertosuccessfullycallCreateComputeEnvironment:

§ AmazonVPCSubnetId(s)§ AmazonEC2SecurityGroupId(s)§ AmazonEC2KeypairName§ AmazonECSContainerInstanceIAMRoleName§ AWSBatchServiceRoleARN

Usingyourownvaluesfortheseattributes,thefollowingAWSCLIcommand,shouldcreateaVALIDComputeEnvironment:aws batch create-compute-environment --compute-environment-name demo1 --type MANAGED --state ENABLED --compute-resources type=EC2,minvCpus=0,maxvCpus=1024,desiredvCpus=16,instanceTypes=optimal,subnets=subnet-367ab30a,subnet-db3c9ff6,subnet-ff3691a4,subnet-3173e178,securityGroupIds=sg-8ec710f3,ec2KeyPair=aws-batch-beta-keypair,instanceRole=AWSBatchECSInstanceRole --service-role arn:aws:iam::123456789012:role/AWSBatchBetaServiceRole

Theoutputshouldlooksomethinglikethis: { "computeEnvironmentName": "demo1", "computeEnvironmentArn": "arn:aws:batch:us-east-1:0123456789012:compute-environment/demo1" }DescribeYourComputeEnvironment:WithinAWSBatch,theDescribeComputeEnvironmentAPIcanbeusedtolistComputeEnvironmentsortodescribeaspecificComputeEnvironment.Inaddition,youcanpasseithertheComputeEnvironmentNameorARN,withinalist.Assumingyoufollowedthestepabove,theAWSCLIcommandtodescribethefreshlycreatedComputeEnvironmentis:aws batch describe-compute-environments --compute-environments demo1

Theoutputshouldlooksimilartothefollowing:{ "computeEnvironments": [ { "status": "VALID", "serviceRole": "arn:aws:iam::123456789012:role/AWSBatchBetaServiceRole", "computeEnvironmentArn": "arn:aws:batch:us-east-1:123456789012:compute-environment/demo1", "computeResources": { "subnets": [ "subnet-367ab30a", "subnet-db3c9ff6",


"subnet-ff3691a4", "subnet-3173e178" ], "tags": {}, "desiredvCpus": 16, "minvCpus": 0, "instanceTypes": [ "optimal" ], "securityGroupIds": [ "sg-8ec710f3" ], "instanceRole": "AWSBatchECSInstanceRole", "maxvCpus": 1024, "type": "EC2", "ec2KeyPair": "aws-batch-beta-keypair" }, "ecsClusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/demo1_Batch_c1f45197-ca4a-39a2-8ce9-a366bc38feb6", "state": "ENABLED", "computeEnvironmentName": "demo1", "type": "MANAGED", "resources": [] } ] }

Youcanalsotrywithoutspecifyingthe--compute-environmentsparameterandperformaliststylecall.OnceyourComputeEnvironmentisVALID,pleaseproceedtoJobQueuecreation.CreateaJobQueue:AWSBatchJobsaresubmittedtoaJobQueue.YoucanhavemultipleJobQueueswithinyourAWSaccount,eachlinkedtomultipleComputeEnvironments.WhencreatingaJobQueue,youassociateitwithaVALIDComputeEnvironment.Assumingyoufollowedthestepabove,theAWSCLIcommandtocreateaJobQueueis:aws batch create-job-queue --job-queue-name demo1 --state ENABLED --compute-environment-order order=0,computeEnvironment=demo1 --priority 500

Theoutputshouldlooksimilartothefollowing:{ "jobQueueArn": "arn:aws:batch:us-east-1:123456789012:job-queue/demo1", "jobQueueName": "demo1" }

DescribeYourJobQueue:WithinAWSBatch,theDescribeJobQueueAPIcanbeusedtolistJobQueuesordescribeaspecificJobQueue.Additionally,youcanpasseithertheJobQueueNameorARN,withinalist.Assumingyoufollowedthestepabove,theAWSCLIcommandtodescribethefreshlycreatedJobQueueis:aws batch describe-job-queues --job-queue demo1

Theoutputshouldlooksimilartothefollowing:


{ "jobQueues": [ { "priority": 500, "state": "ENABLED", "jobQueueArn": "arn:aws:batch:us-east-1:123456789012:job-queue/demo1", "computeEnvironmentOrder": [ { "computeEnvironment": "arn:aws:batch:us-east-1:123456789012:compute-environment/demo1", "order": 0 } ], "jobQueueName": "demo1" } ] }

Youcanalsotrywithoutspecifyingthe--job-queuesparameterandperformaliststylecall.RegisteraJobDefinition:AnAWSBatchJobDefinitionenablesyoutoregisteralloftheattributesofaJobinadvance.TheJobDefinitioncanthenbeusestoeasilysubmitJobs.Thefollowingisasamplejob,thatwilluseabusyboxDockerimageandrunsleep120.TheAWSCLIcommandtoregistertheJobDefinitionis:aws batch register-job-definition --job-definition-name sleep --type container --container-properties '{ "image": "busybox", "vcpus": 1, "memory": 128, "command": [ "sleep", "120"]}'

Theoutputshouldlooksimilartothefollowing:{ "jobDefinitionArn": "arn:aws:batch:us-east-1:123456789012:job-definition/sleep", "jobDefinitionName": "sleep" }

DescribeYourJobDefinition:TheAWSBatchDescribeJobDefinitionsAPIcanbeusedtolistJobDefinitionsordescribeaspecificJobDefinition.YoucanoptionallypasseithertheJobDefinitionNameorARN,withinalist.Assumingyoufollowedthestepabove,theAWSCLIcommandtodescribeyourJobDefinitionis:aws batch describe-job-definitions --job-definitions sleep

Theoutputshouldlooksimilartothefollowing:{ "jobDefinitions": [ { "containerProperties": { "mountPoints": [], "parameters": {}, "image": "busybox",


"environment": {}, "vcpus": 2, "command": [ "sleep", "120" ], "ulimits": [], "volumes": [], "memory": 2048, "resources": [] }, "jobDefinitionArn": "arn:aws:batch:us-east-1:123456789012:job-definition/sleep", "type": "container", "jobDefinitionName": "sleep" } ] }

SubmitaJob:YouarenowreadytosubmityourfirstAWSBatchjob.Todoso,runthefollowingcommand:aws batch submit-job --job-name MySleep --job-definition sleep --job-queue demo1

Theoutputshouldlooksimilartothefollowing:{ "jobName": "MySleep", "arraySize": 1, "jobId": "arn:aws:batch:us-east-1:123456789012:job/1eac6559-18a1-4fc6-b3aa-5d1b2e204923" }

ListYourRunningJob:Nowthatyouhavesubmittedyourfirstjob,youcanwatchit’sprogressasitisscheduledandexecuted.Todoso,runthefollowingcommand:aws batch list-jobs --job-queue demo1 --job-status RUNNING

Theoutputshouldlooksimilartothefollowing(assumingyouhavesubmittedmultiplejobs):{ "jobSummaryList": [ { "jobId": "7fb78800-1c37-4940-9ac3-983dbff76ea8" }, { "jobId": "7fb78800-1c37-4940-9ac3-983dbff76e00" }, { "jobId": "7fb78800-1c37-4940-9ac3-983dbff76e03" } ] }

Tryrunningthesamecommandusingdifferentstates(e.g.RUNNABLE,PENDING,andSUCCEEDED).


DescribeYourRunningJob:Ifyouwouldliketoexamineoneofyourjobsinmoredetail,youcanusetheAWSBatchDescribeJobsAPI:aws batch describe-jobs --jobs 1eac6559-18a1-4fc6-b3aa-5d1b2e204923

Theoutputshouldlooksimilartothefollowing:{ "jobs": [ { "container": { "mountPoints": [], "image": "busybox", "environment": [], "vcpus": 2, "command": [ "sleep", "120" ], "volumes": [], "memory": 128, "ulimits": [] }, "parameters": [], "jobDefinition": "sleep", "jobQueue": "arn:aws:batch:us-east-1:123456789012:job-queue/demo1", "jobId": "1eac6559-18a1-4fc6-b3aa-5d1b2e204923", "dependsOn": [], "jobName": "MySleep" } ] }

CleanupYourHappyPathEnvironment:NowthatyouhavesuccessfullyrunyourfirstAWSBatchjob,let’scleanupeverythingthatwehavecreated.DeregisterYourJobDefinition:Usingthefollowingcommand,youcanderegistertheJobDefinitioncreatedearlier:aws batch deregister-job-definition --job-definition arn:aws:batch:us-east-1:560348900601:job-definition/sleep

UpdateYourJobQueue:BeforeyoucandeleteyourJobQueue,youfirstneedtochangeitfromENABLEDtoDISABLEDviathefollowingcommand:aws batch update-job-queue --job-queue demo1 --state DISABLED

DeleteYourJobQueue:NowthatyourJobQueueisDISABLED,youcandeleteit:aws batch delete-job-queue --job-queue demo1


UpdateYourComputeEnvironment:AswithJobQueues,beforeyoucandeleteyourComputeEnvironment,youfirstneedtochangeitfromENABLEDtoDISABLEDviathefollowingcommand:aws batch update-compute-environment --compute-environment demo1 --state DISABLED

DeleteYourComputeEnvironment:NowthatyourComputeEnvironmentisDISABLED,youcandeleteit:aws batch delete-compute-environment --compute-environment demo1

AllAWSresourcescreatedinyouraccount,shouldnowbegone.

Additionaltestsyoumaywanttoconduct:AJobQueuewithMultipleComputeEnvironments:WhathappenswhenyoucreateaJobQueuewithmultipleComputeEnvironments?Wherearejobsexecuted?HowareresourceswithineachComputeEnvironmentutilized? MultipleJobQueuesMappedtotheSameComputeEnvironment:WhathappenswhenyoucreatemultipleJobQueuesmappedtothesameComputeEnvironment?WhichJobQueue’sjobsareexecutedfirst?CreateanUnmanagedComputeEnvironment:TrytocreateanUnmanagedcomputeenvironmentusingthefollowingcommand:aws batch create-compute-environment --type UNMANAGED --compute-environment-name UnmanagedDemo1 --service-role arn:aws:iam::123456789012:role/DilithiumBetaServiceRole

Theresponseshouldlooklikethis: { "computeEnvironmentName": "UnmanagedDemo1", "computeEnvironmentArn": "arn:aws:batch:us-east-1:123456789012:compute-environment/UnmanagedDemo1" }

NowdescribeyourUnmanagedComputeEnvironmentandretrievetheARNoftheECSclusterthatwascreatedforyou:aws batch describe-compute-environments --compute-environment UnmanagedDemo1

Theresponsewilllooklikethis: { "computeEnvironments": [ { "status": "VALID", "serviceRole": "arn:aws:iam::123456789012:role/DilithiumBetaServiceRole", "computeEnvironmentArn": "arn:aws:batch:us-east-1:123456789012:compute-environment/UnmanagedDemo1", "statusReason": "ComputeEnvironment Healthy",


"ecsClusterArn": "arn:aws:ecs:us-east-1:123456789012:cluster/UnmanagedDemo1_Batch_c3b9955f-9f66-3c0a-9d0f-55ceb9e5edad", "state": "ENABLED", "computeEnvironmentName": "UnmanagedDemo1", "type": "UNMANAGED", "resources": [] } ] }

Inthisexample,yourECSclusterARNis: arn:aws:ecs:us-east-1:123456789012:cluster/UnmanagedDemo1_Batch_c3b9955f-9f66-3c0a-9d0f-55ceb9e5edad

andtheECSclusternameis:UnmanagedDemo1_Batch_c3b9955f-9f66-3c0a-9d0f-55ceb9e5edad

NowfollowtheseinstructionstolaunchinstancesoftheECS-optimizedAMIintoyourUnmanagedCE’sECSCluster.http://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.htmlSubmitaCollectionofJobswithVaryingResourceRequirements:HowwelldoestheAWSBatchSchedulerwork?ArejobsscheduledinamannerthatefficientlyutilizesresourceswithinyourComputeEnvironments?ThefollowingPythonscriptcanbeusedtosubmit50jobstoaJobQueuecalleddemo1,eachjobrunningthesleepcommandforsomewherebetween60and360secondswithvaryingvCPUandmemoryrequirements.#!/usr/bin/python import botocore.session import botocore.exceptions import random region = 'us-east-1' endpoint_url = 'https://dilithium-beta.us-east-1.amazonaws.com' session = botocore.session.get_session() client = session.create_client('batch', region, endpoint_url=endpoint_url) y=50 for x in range(0, y): vcpus=random.randrange(1,32+1,2) memory=random.randrange(512,2048+512,512) sleep=str(random.randrange(60,360,60)) command=['sleep', sleep] request = client.submit_job(jobQueue='demo1', jobName='sleep', jobDefinition='sleep', arraySize=1, containerOverrides=dict(vcpus=vcpus,memory=memory*vcpus,command=command)) print(request['jobId'])


Concepts: Jobs:JobsaretheunitofworkexecutedbyAWSBatch.JobscanbeexecutedasLambdafunctionsorcontainerizedapplicationsrunningonAmazonEC2.Containerizedjobscanreferenceacontainerimage,command,andparametersoruserscansimplyprovidea.ziporscript(bash,python,perl,etc…)thatrunswithinadefaultAmazonLinuxcontainer. Userscansubmitalargenumberofindependent“simplejobs”aswellas“arrayjobs”thatrunmanycopiesofanapplicationagainstanarrayofelements.Arrayjobsareanefficientwaytorun:

• Parametricsweeps• MonteCarlosimulations• Processingalargecollectionofobjects

Jobscanhavethefollowingstates:SUBMITTED:Ajobthathasbeensubmittedtothequeue,buthasnotyetbeenevaluatedbythescheduler.PENDING:Ajobthatresidesinthequeueandisnotyetabletorunduetoadependencyonanotherjoborresource.RUNNABLE:Ajobwhichresidesinthequeue,hasnooutstandingdependencies,andisthereforereadytobescheduledtoahost.JobsinthisstatearestartedassoonassufficientresourcesareavailableinoneoftheComputeEnvironment’smappedtotheJob’sQueue.STARTING:Thesejobshavebeenscheduledtoahostandtherelevantcontainerinitiationoperationsareunderway.RUNNING:ThejobisrunningeitherasaLambdafunctionorcontainerjobonacomputeresourcewithinaComputeEnvironment.SUCCEEDED:Thejobhassuccessfullycompleted.FAILED:Thejobhasfailed. JobDefinitions:SimilartoECSTaskDefinitions,JobDefinitionsspecifyhowjobsaretoberun.Whileeachjobmustreferenceajobdefinition,manyparameterscanbeoverridden.Someoftheattributesspecifiedinajobdefinitioninclude:

• IAMroleassociatedwiththejob• vCPUandmemoryrequirements• Mountpoints• ContainerorLambdaproperties• Environmentvariables

JobQueues:JobsaresubmittedtoaJobQueue,wheretheyresideuntiltheyareabletobescheduledtoacomputeresource.AnAWSaccountcanhavemultipleJobQueues.Forexample,youmightcreateaqueueforhighpriorityjobsandanotherqueueforlow-priorityjobs.JobshaveaprioritythatisusedbytheSchedulertodeterminewhichJobQueue’sJobsshouldbeevaluatedforexecutionfirst.


ComputeEnvironments:JobqueuesaremappedtooneormoreComputeEnvironment.ComputeEnvironmentscontaintheEC2instancesusedtoruncontainerizedbatchjobs.AgivenComputeEnvironmentcanalsobemappedtooneormanyJobQueues.WithinaJobQueue,theassociatedComputeEnvironmentseachhaveanorderthatisusedbytheSchedulertodeterminewheretoplacejobsthatarereadytobeexecuted.IfthefirstComputeEnvironmenthasfreeresources,thejobwillbescheduledtoahostwithinthatComputeEnvironment.IftheComputeEnvironmentisunabletoprovideasuitablecomputeresource,theSchedulewillattempttorunthejobonthenextComputeEnvironment. ManagedComputeEnvironments:ManagedComputeEnvironmentsenableyoutodescribeyourbusinessrequirements(e.g.instancetypes,VPCSubnets,min/max/desiredaggregatevCPUsacrossallinstances,andspotbidasa%ofOn-Demand)andAWSBatchlaunchesandscalesresourcesonyourbehalf.UnmanagedComputeEnvironment:Alternatively,youcanlaunchandmanageyourownresourceswithinanUnmanagedComputeEnvironment.TheinstancethatyoulaunchneedtoincludetheECSagentandrunasupportedversionofLinuxandDocker.UnmanagedComputeEnvironmentsareusefulifyouneedtoconfigureyourcomputeresourcesinawaythatisdifferentfromthedefaultconfigurationprovidedbyManagedComputeEnvironments.SeethispagetolearnhowtoconfigureEC2instanceswithinanUnmanagedComputeEnvironment:http://docs.aws.amazon.com/AmazonECS/latest/developerguide/container_instance_AMIs.htmlWorkflows,Pipelines,andJobDependencies:Jobscanexpressadependencyonthesuccessfulcompletionofotherjobs,specificelementsofanarrayjob,orAWSBatchresources(countersorEFSfilesystems).AWSBatchdoesnothaveaSubmitWorkflowAPI.Instead,customersusetheirpreferredworkflowengineandlanguagetosubmitjobs.Flow-basedsystemssimplysubmitjobsinserialwhileDAG-basedsystemssubmitmanyjobsatonce,identifyinginter-jobdependenciesbyJobnameorID.Scheduler:TheSchedulerevaluateswhen,where,andhowtorunjobsthathavebeensubmittedtoajobqueue.OurinitialalgorithmisFIFOwithdependencyawareness,meaningthatwerunjobsinapproximatelytheorderinwhichtheyaresubmittedaslongasalldependenciesonotherjobsresourceshavebeenmet.

aws batch private beta getting started guide · aws batch private beta getting started guide...

Documents