Download - Everything as a code
Everything as a CodeНепривычно о привычном
@aatarasoff
@aatarasoff
@aatarasoff
No warranty guarantee
Matrix has you
4
5
6
Выйти за пределы...
7
...поняв, что всё есть код
8
Как вы представляете себе код?
9
@Configuration@EnableConfigurationProperties(OneServerProperties.class)public class OneServerConfiguration implements ApplicationContextAware { ApplicationContext applicationContext;
@Autowired OneServerProperties properties;
@Bean public HttpServer httpServer() throws IOException { HttpServer httpServer = new RelativePathHttpServer();
for (String beanName : context.getBeans(HttpController.class)) { httpServer.addRequestHandlers(context.getBean(beanName)); }
return httpServer; }}
10
Наверное как-то так
Что такое код?
● Коллекция инструкций
● Человекочитаемый формат
○ plain text
● Может быть понят и “проигран”
○ после компиляции
○ интерпретатором
11
Да я тоже пишу код!
12
Вооружи себя
13
Настройка рабочего окружения
14
Install apps Code Checkout Configure workspace First Blood
15
Install apps Code Checkout Configure workspace First Blood
brew install
16
Install apps Code Checkout Configure workspace First Blood
brew installpip install
17
Install apps Code Checkout Configure workspace First Blood
git
18
Install apps Code Checkout Configure workspace First Blood
customize*.properties
19
Install apps Code Checkout Configure workspace First Blood
customize*.propertiestemplate
.gradle
20
Install apps Code Checkout Configure workspace First Blood
customize*.propertiesinstall
certificates
21
Install apps Code Checkout Configure workspace First Blood
first build
22
Install apps Code Checkout Configure workspace First Blood
mass buildfirst deploy
23
ansible-playbook -i local-inv setup.yml --tags configure
24
Интерпретатор
ansible-playbook -i local-inv setup.yml --tags configure
25
Конфигурация
ansible-playbook -i local-inv setup.yml --tags configure
26
Алгоритм
#checkout repositories from git- include: checkout.yml
#configure your local environment- include: configure.yml
#add useful mappings to your hosts file- include: hosts.yml
#add gradle support- include: gradle.yml
#clean and build all projects- include: build.yml
#delpoy all services to dev- include: build.yml
27
Алгоритм
ansible-playbook -i local-inv setup.yml --tags configure
28
Входные параметры
- name: checkout services git: repo: "{{ git.root }}/{{ item.name }}.git" dest: "{{ work_dir }}/{{ item.name }}" update: yes with_items: - "{{ services }}" tags: - services
29
Массовый checkout/pull
- name: checkout services git: repo: "{{ git.root }}/{{ item.name }}.git" dest: "{{ work_dir }}/{{ item.name }}" update: yes with_items: - "{{ services }}" tags: - services
Переменные
30
- name: checkout services git: // with_items: - "{{ services }}"
services: - { name: one-instant-messenger , sourceDir: src } - { name: one-discussions, sourceDir: src } - { name: one-attachment, sourceDir: src, testDir: test, local: true }
Циклы
31
services: - { name: one-instant-messenger, sourceDir: src } - { name: one-discussions, sourceDir: src } - { name: one-attachment, sourceDir: src, testDir: test }
Коллекции и структуры данных
32
- name: create gradle build file template: src: build.gradle.j2 dest: "{{ work_dir }}/build.gradle" mode: 0644
- name: create gradle settings file template: src: settings.gradle.j2 dest: "{{ work_dir }}/settings.gradle" mode: 0644
Шаблоны
33
{% if services is not none %}{% for service in services %} if (project.name == '{{ service.name }}') {{% if 'sourceDir' in service %} main.java.srcDir '{{ service.sourceDir }}'{% endif %}{% if 'testDir' in service %} test.java.srcDir '{{ service.testDir }}'{% endif %} }
Условные операторы
34
{% if services is not none %}{% for service in services %} if (project.name == '{{ service.name }}') {{% if 'sourceDir' in service %} main.java.srcDir '{{ service.sourceDir }}'{% endif %}{% if 'testDir' in service %} test.java.srcDir '{{ service.testDir }}'{% endif %} }
Опять циклы
35
- stat: path={{ username }}.key register: certkey
- name: generate private key shell: openssl genrsa -out {{ username }}.key -aes256 4096 when: not certkey.stat.exists
Идемпотентность
36
Install apps Code Checkout Configure workspace Multiplie Use
37
Петля обратной связи
git
Что получили
● Автоконфигурация рабочего
пространства
○ повторяемая
○ немутабельная
● Можно дать скрипт новичку
● Можно и нужно пользоваться этим
каждый день
38
Код есть код
39
Искусство сборки
40
Compile code Unit tests Package Publishing
41
Compile code Unit tests Package
compiler
Publishing
42
Compile code Unit tests Package
javacresourceprocessing
Publishing
43
Compile code Unit tests Package
javacresources copyingdependency resolving
Publishing
44
Compile code Unit tests Package
junit
Publishing
45
Compile code Unit tests Package
jar
Publishing
46
Compile code Unit tests Package
jarnpm, deb, ...
Publishing
47
Compile code Unit tests Package
jarnpm, so, ...docker image
Publishing
48
Compile code Unit tests Package
ivy
Publishing
49
Compile code Unit tests Package
maven, ivymaven
Publishing
50
Compile code Unit tests Package Publishing
maven, ivylocal or dev deploydocker registry
51
Системы сборки
● Ant + Ivy
● Maven
● Gradle
● Docker
● npm
● ...
52
FROM golang:1.7-alpineMAINTAINER [email protected]
VOLUME /dataWORKDIR /data
RUN apk update && \ apk upgrade && \ apk add git bash
RUN go get github.com/aatarasoff/apistress && \ go install github.com/aatarasoff/apistress
CMD [ "apistress" ]
Dockerfile. Наследование
53
FROM golang:1.7-alpineMAINTAINER [email protected]
VOLUME /dataWORKDIR /data
RUN apk update && \ apk upgrade && \ apk add git bash
RUN go get github.com/aatarasoff/apistress && \ go install github.com/aatarasoff/apistress
CMD [ "apistress" ]
Dockerfile. Инструкции
54
FROM golang:1.7-alpineMAINTAINER [email protected]
ARG maindir=/data
VOLUME $maindirWORKDIR $maindir
RUN apk update && apk upgrade && apk add git bash
RUN go get github.com/aatarasoff/apistress && \ go install github.com/aatarasoff/apistress
CMD [ "apistress" ]
Dockerfile. Переменные
55
docker build … && docker push …
56
publishing { publications { mavenJava(MavenPublication) { artifactId 'spring-one-nio-autoconfigure'
from components.java
artifact sourceJar { classifier "sources" } } }}
Gradle. DSL
57
compile(ivyDependencies(projectDir, 'runtime'))
def ivyDependencies(ivyPath, conf) { def dep = [] def ivyModule = new XmlParser().parse(file("${ivyPath}/ivy.xml"))
ivyModule.dependencies.dependency.each dep.add([group: (null == it.@org ? 'ru.odnoklassniki' : it.@org), name: it.@name, version: it.@rev, configuration: (it.@conf =~ /->(\w+)/)[0][1]]) }
return dep}
Gradle. Ну просто код
58
<macrodef name="docker-build-image" description="Build docker image"> <attribute name=" buildcommand"
default="build -t @{repo}/@{image}:@{tag} ."/>
<sequential> <exec executable="docker"> <arg line=" @{buildcommand}"/> </exec> </sequential></macrodef>
И даже в Ant-е есть жизнь
59
./gradlew build test ant-docker-build-image publish
60
Фреймворки для автоматизации
● Ant + Ivy
● Maven
● Gradle
● Docker
● npm
● ...
61
Что получили
● Сборка это код
○ XML
○ DSL
○ Groovy
○ etc
● Системы сборки не только для
сборки
62
Ликвидация багов
63
Unit tests API tests Stress tests UI tests
64
Unit tests API tests Stress tests UI tests
TDD
65
Unit tests API tests Stress tests UI tests
BDD Specs
66
def "Return error code 400, if User-ID header is not presented"() { given: def request = post("/rent")
when: def result = this.mvc.perform(request)
then: result.andExpect(status().isBadRequest()) .andDo(document("rent-user-id-is-absent"))}
Дружелюбный BDD
67
def "Return error code 400, if User-ID header is not presented"() { given: def request = post("/rent")
when: def result = this.mvc.perform(request)
then: result.andExpect( status().isBadRequest()) .andDo(document("rent-user-id-is-absent"))}
Простые проверки
68
Unit tests API tests Stress tests UI tests
JMeter, wrk, vegeta
69
Unit tests API tests Stress tests UI tests
JMeterproduction
70
{ "baseUrl": "http://host:9000/rent-service", "tests": [ { "rps": 10, "duration": 5, "target": { "method": "GET", "path": "/rent", "headers": [ ... ] }, "sla": { "latency": 1000, "successRate": 99.999 } }, ... ]}
config.json
71
{ "baseUrl": "http://host:9000/rent-service", "tests": [ { "rps": 10, "duration": 5, "target": { "method": "GET", "path": "/rent", "headers": [ ... ] }, "sla": { "latency": 1000, "successRate": 99.999 } }, ... ]}
config.json
72
{ "baseUrl": "http://host:9000/rent-service", "tests": [ { "rps": 10, "duration": 5, "target": { "method": "GET", "path": "/rent", "headers": [ ... ] }, "sla": { "latency": 1000, "successRate": 99.999 } }, ... ]}
config.json
73
cat config.json | docker run -i apistress -config=stdin
74
Где-то мы такое видели
Requests [total, rate] 50, 10.20Duration [total, attack, wait] 5.022872793s, 4.899943287s, 122.929506msLatencies [mean, 50, 95, 99, max] 143.772484ms, ..., 290.101831msBytes In [total, mean] 4842, 96.84Bytes Out [total, mean] 950, 19.00Success [ratio] 100.00%Status Codes [code:count] 200:50Error Set:
Test#1
75
> cat config.json | docker run -i apistress -config=stdin> echo $?0
76
Код возврата
Requests [total, rate] 200, 10.05Duration [total, attack, wait] 23.04784s, 19.899754743s, 3.148093811sLatencies [mean, 50, 95, 99, max] 3.023677499s, ..., 11.832287083sBytes In [total, mean] 6874, 34.37Bytes Out [total, mean] 1349, 6.75Success [ratio] 35.50%Status Codes [code:count] 0:129 200:71Error Set:Get http://host:9000/rent-service/rent: EOFGet http://host:9000/rent-service/rent: http: server closed idle connection...
Test#2
77
> cat config.json | docker run -i apistress -config=stdin> echo $?1
78
Код возврата
Unit tests API tests Stress tests UI tests
Selenium, Selenide
79
Что получили
● Все тестовые сценарии в коде
● Можно встроить в процесс сборки
или доставки ПО
● Если хотите, то можно
генерировать отчёты для разбора
полётов
80
Кододокументация
81
Analyst, PM Developer Tester Docs
Word, PDF...
82
Analyst, PM Developer Tester Docs
Word, PDF... Code + Tests
83
Analyst, PM Developer Tester Docs
Word, PDF... Code + Tests Test cases
84
Analyst, PM Developer Tester Docs :(
Word, PDF... Code + Tests Test cases
85
Analyst, PM Developer Tester Docs :)
Markdown/Asciidoctor
Docs :)
86
= Hippo Rent ServiceThis is documentation for Open API of our hippo renting service
== Methods
=== Rent
==== Request specification===== Headers//тут опишем http-заголовки===== Example//а здесь будут примеры вызова
==== Response specification===== Response fields//здесь описание полей ответа===== Example//ну и пример того, что вообще ожидать в ответе
Вот такой документ
87
./gradlew ... asciidoc publishDocs
88
Ну, вы поняли да? :)
def "Rent a hippo"() { given: def request = post("/rent").header("User-ID", "aatarasoff")
when: def result = this.mvc.perform(request)
then: result.andExpect(status().isOk()) .andDo(document( "rent-hippo", preprocessResponse(prettyPrint()), requestHeaders( headerWithName("User-ID").description("User unique identifier")), responseFields( fieldWithPath("hippoRemain").description("Hippo remain count"), fieldWithPath("parrot_fee").description("Fee in virtual parrots"), fieldWithPath("ins").description("Insurance number. Yeah, we sell it"), fieldWithPath("hash").description("Blockchain block hash")) ))}
…и снова тесты
89
def "Rent a hippo"() { given: def request = post("/rent").header("User-ID", "aatarasoff")
when: def result = this.mvc.perform(request)
then: result.andExpect(status().isOk()) .andDo(document( "rent-hippo", preprocessResponse(prettyPrint()), requestHeaders( headerWithName("User-ID").description("User unique identifier")), responseFields( fieldWithPath("hippoRemain").description("Hippo remain count"), fieldWithPath("parrot_fee").description("Fee in virtual parrots"), fieldWithPath("ins").description("Insurance number. Yeah, we sell it"), fieldWithPath("hash").description("Blockchain block hash")) ))}
А не документация ли это?
90
document( "rent-hippo", preprocessResponse(prettyPrint()), requestHeaders( headerWithName("User-ID").description("User unique identifier")), responseFields( fieldWithPath("hippoRemain").description("Hippo remain count"), fieldWithPath("parrot_fee").description("Fee in virtual parrots"), fieldWithPath("ins").description("Insurance number. Yeah, we sell it"), fieldWithPath("hash").description("Blockchain block hash")))
Имя сниппета
91
document( "rent-hippo", preprocessResponse(prettyPrint()), requestHeaders( headerWithName("User-ID").description("User unique identifier")), responseFields( fieldWithPath("hippoRemain").description("Hippo remain count"), fieldWithPath("parrot_fee").description("Fee in virtual parrots"), fieldWithPath("ins").description("Insurance number. Yeah, we sell it"), fieldWithPath("hash").description("Blockchain block hash")))
Тестируем заголовки
92
document( "rent-hippo", preprocessResponse(prettyPrint()), requestHeaders( headerWithName("User-ID").description("User unique identifier")), responseFields( fieldWithPath("hippoRemain").description("Hippo remain count"), fieldWithPath("parrot_fee").description("Fee in virtual parrots"), fieldWithPath("ins").description("Insurance number. Yeah, we sell it"), fieldWithPath("hash").description("Blockchain block hash")))
Тестируем поля ответа
93
generated-snippetsrent-hippo
curl-request.adochttp-request.adochttp-response.adochttpie-request.adocrequest-headers.adocresponse-fields.adoc
Получаем сниппеты
94
= Hippo Rent ServiceThis is documentation for Open API of our hippo renting service
== Methods
=== Rent
==== Request specification===== Headersinclude::{snippets}/rent-hippo/request-headers.adoc[]===== Exampleinclude::{snippets}/rent-hippo/http-request.adoc[]
==== Response specification===== Response fieldsinclude::{snippets}/rent-hippo/response-fields.adoc[]===== Exampleinclude::{snippets}/rent-hippo/http-response.adoc[]
Вставляем их в документ
95
= Hippo Rent ServiceThis is documentation for Open API of our hippo renting service
== Methods
=== Rent
==== Request specification===== Headersinclude::{snippets}/rent-hippo/request-headers.adoc[]===== Exampleinclude::{snippets}/rent-hippo/http-request.adoc[]
==== Response specification===== Response fieldsinclude::{snippets}/rent-hippo/response-fields.adoc[]===== Exampleinclude::{snippets}/rent-hippo/http-response.adoc[]
96
97
Что получили?
● Документация как код
○ лежит в репозитории с кодом
○ встроена в цикл сборки
○ рендерится в html, pdf и т.д.
○ почти всегда актуальна
● Синергия с тестами
98
Инфракод
99
Hardware
Containers
Application
PaaS
Mesos/Kubernetes/Private cloud
100
Hardware + OS System Libs PaaS Application
101
Hardware + OS System Libs PaaS Application
Ansible
102
Hardware + OS System Libs PaaS Application
AnsiblePuppet/Chef
103
Ansible. Inventory
[datacenter]api-server-1api-server-2api-server-3
[datacenter:vars]magicvar = 42
104
Ansible. Playbook
- hosts: datacenter roles: - role: docker - role: rsyslog
105
ansible-playbook -i datacenter1 bootstrap.yml
106
Без комментариев
Hardware + OS System Libs PaaS Application
Ansible
107
Hardware + OS System Libs PaaS Application
AnsiblePuppet/Chef
108
Hardware + OS System Libs PaaS Application
Manifest
109
Docker compose
services: zk: image: zookeeper network_mode: bridge ports: - 2181:2181 environment: ZK_CONFIG: tickTime=2000,initLimit=10,clientPort=2181 ZK_ID: 1
110
Docker compose
services: zk: image: zookeeper network_mode: bridge ports: - 2181:2181 environment: ZK_CONFIG: tickTime=2000,initLimit=10,clientPort=2181 ZK_ID: 1
111
Конфигурация сервиса
services: zk: image: zookeeper network_mode: bridge ports: - 2181:2181 environment: ZK_CONFIG: tickTime=2000,initLimit=10,clientPort=2181 ZK_ID: 1
112
Mesos/Marathon{ "id": "/api/rent-service", "cpus": 1, "mem": 1024, "instances": 3, "container": { "docker": { "image": "rent-service:0.0.1", "portMappings": [ { "containerPort": 8080 } ] } }}
113
curl -X POST ... http://marathon/v2/apps?force=true
114
Вот и весь деплоймент
Конфигурация приложений
https://your_repo/rent-service-config/routes.yml
routes: payment: path: /payment-service/** serviceId: payment-service
115
Ещё конфигурация
● Zookeeper
● Consul
● Vault
● configo by Zeroturnaround
● ...
116
configo
117
docker run \ -e CONFIGO_SOURCE_0='{"type": "http", "format": "yaml", "url":
"https://my.server.com/common.yaml"}' \ rent-service
//внутри приложенияgetEnvironmentVariable("MY_ENV_VAR")
https://github.com/zeroturnaround/configo
Что получили?
● Инфрастуктура может быть легко
описана в виде кода
● Деплоймент и конфигурация
приложений в виде конфигов и
манифестов
118
Неубиваемый CI
119
120
Install Master Configure Slaves
121
Install Master Configure Slaves
Ansible
122
Install Master Configure Slaves
Ansible
Jenkins Docker Cloud plugin
<——— Хост с докером
<——— Сколько контейнеров можно запустить
123
Автоконфигурация
<clouds> {% for group in ['build', 'test', 'production'] %} {% for node in groups[group + '-slaves'] %} <com.github.kostyasha.yad.DockerCloud plugin="[email protected]"> <name>{{ node }}</name>
... <templates> <com.github.kostyasha.yad.DockerSlaveTemplate> <id>mycloud-template</id> <dockerContainerLifecycle> <image>{{ group }}-jenkins-slave</image> ...
</templates> <connector> <serverUrl>tcp://{{ node.hostname }}:2375</serverUrl> <apiVersion>1.20</apiVersion> </connector> </com.github.kostyasha.yad.DockerCloud> {% endfor %} {% endfor %} </clouds>
124
Код доставки
125
126
pipeline-template.groovy
127
//checkout and definition stagenode('build') { // Mark the code checkout 'stage' stage 'Checkout' git credentialsId: 'jenkins-git', url: "${git_url}/${repo}.git"
// Mark build 'stage' stage 'Build'
sh ('./gradlew clean build final')}
//next steps128
//checkout and definition stagenode('build') { // Mark the code checkout 'stage' stage 'Checkout' git credentialsId: 'jenkins-git', url: "${git_url}/${repo}.git"
// Mark build 'stage' stage 'Build'
sh ('./gradlew clean build final')}
//next steps129
//checkout and definition stagenode('build') { // Mark the code checkout 'stage' stage 'Checkout' git credentialsId: 'jenkins-git', url: "${git_url}/${repo}.git"
// Mark build 'stage' stage 'Build'
sh ('./gradlew clean build final')}
//next steps130
//checkout and definition stagenode('build') { // Mark the code checkout 'stage' stage 'Checkout' git credentialsId: 'jenkins-git', url: "${git_url}/${repo}.git"
// Mark build 'stage' stage 'Build'
sh ('./gradlew clean build final')}
//next steps131
//checkout and definition stagenode('build') { // Mark the code checkout 'stage' stage 'Checkout' git credentialsId: 'jenkins-git', url: "${git_url}/${repo}.git"
// Mark build 'stage' stage 'Build'
sh ('./gradlew clean build final')}
//next steps132
//checkout and definition stagenode('build') { // Mark the code checkout 'stage' stage 'Checkout' git credentialsId: 'jenkins-git', url: "${git_url}/${repo}.git"
// Mark build 'stage' stage 'Build'
sh ('./gradlew clean build final')}
//next steps133
//deploy artifact to testnode('test') { sh('ansible-galaxy install -r requirements.yml')
ansiblePlaybook( credentialsId: 'ansible', installation: 'ansible', playbook: 'deploy.yml', inventory: 'test' )}
134
//deploy artifact to testnode('test') { sh('ansible-galaxy install -r requirements.yml')
ansiblePlaybook( credentialsId: 'ansible', installation: 'ansible', playbook: 'deploy.yml', inventory: 'test' )}
135
//deploy artifact to testnode('test') { sh('ansible-galaxy install -r requirements.yml')
ansiblePlaybook( credentialsId: 'ansible', installation: 'ansible', playbook: 'deploy.yml', inventory: 'test' )}
136
//deploy artifact to testnode('test') { sh('ansible-galaxy install -r requirements.yml')
ansiblePlaybook( credentialsId: 'ansible', installation: 'ansible', playbook: 'deploy.yml', inventory: 'test' )}
137
//deploy artifact to testnode('test') { sh('ansible-galaxy install -r requirements.yml')
ansiblePlaybook( credentialsId: 'ansible', installation: 'ansible', playbook: 'deploy.yml', inventory: 'test' )}
jiraComment ( issueKey: issue_id, body: "Artifact has been deployed")
138
node('build') { def repos = fetchRepos(project) for (repo in repos) { build(repo) }}
def fetchRepos(String project) { def url = new URL("https://repo/projects/${project}/repos?limit=1000")
def conn = url.openConnection() conn.setRequestMethod("GET") def responseCode = conn.getResponseCode()
final slurper = new groovy.json.JsonSlurper()
def repos = slurper.parse(conn.getInputStream()).values
for (repo in repos) { if (repo.slug.contains('one-')) result << repo.slug }
return result}
139
node('build') { def repos = fetchRepos(project) for (repo in repos) { build(repo) }}
def fetchRepos(String project) { def url = new URL("https://repo/projects/${project}/repos?limit=1000")
def conn = url.openConnection() conn.setRequestMethod("GET") def responseCode = conn.getResponseCode()
final slurper = new groovy.json.JsonSlurper()
def repos = slurper.parse(conn.getInputStream()).values
for (repo in repos) { if (repo.slug.contains('one-')) result << repo.slug }
return result}
140
node('build') { def repos = fetchRepos(project) for (repo in repos) { build(repo) }}
def fetchRepos(String project) { def url = new URL("https://repo/projects/${project}/repos?limit=1000")
def conn = url.openConnection() conn.setRequestMethod("GET") def responseCode = conn.getResponseCode()
final slurper = new groovy.json.JsonSlurper()
def repos = slurper.parse(conn.getInputStream()).values
for (repo in repos) { if (repo.slug.contains('one-')) result << repo.slug }
return result}
141
Микросервисы
142
143
Install Master Configure Slaves
Create meta job
Ansible
144
Install Master Configure Slaves
Create meta job
AnsiblecURL
145
Install Master Configure Slaves
Create meta job
Create pipelines
jobs.each { job -> pipelineJob("${basePath}/${job}") { //define SCM
definition { cps { script(readFileFromWorkspace('pipeline-template.groovy')) sandbox() } } }}
JobDSL plugin
146
jobs.each { job -> pipelineJob("${basePath}/${job}") { //define SCM
definition { cps { script(readFileFromWorkspace('pipeline-template.groovy')) sandbox() } } }}
JobDSL plugin
147
jobs.each { job -> pipelineJob("${basePath}/${job}") { //define SCM
definition { cps { script(readFileFromWorkspace('pipeline-template.groovy')) sandbox() } } }}
JobDSL plugin
148
149
Install Master Configure Slaves
Create meta job
Create pipelines
git
ansible-playbook -i jenkins-for-my-team jenkins.yml
150
Это последний раз
Что получили?
● Пайплайн как код
● Неубиваемый CI
○ без бэкапов
○ всё хранится как код
○ разворачивается за X минут
151
Выводы
● Почти любой процесс можно
формализовать, представить в
виде кода и автоматизировать
● Мы все пишем код, хотя можем
думать, что это не так
152
Спасибо, что выбрали красную
@aatarasoff
@aatarasoff
@aatarasoff
QA