inside sqale's backend at rubyconf taiwan 2012
DESCRIPTION
TRANSCRIPT
Inside Sqale’s BackendRubyConf Taiwan 2012
Gosuke Miyashitapaperboy&co., Inc., Japan
cpan:mizzygithub.com/mizzy
mizzy.org@gosukenator
Kentaro KuribayashiLiving on the Edge Rails
Kensuke NagaeIntroducing nonopaste-cli
Shinya TsunematsuBuilding production server
environment of ruby in modern way
Hiroshi SHIBATAHow to discover the Ruby's defects
with web application
Gosuke MiyashitaInside Sqale’s Backend
We are from paperboy&co., Japan
What is Sqale?
Cloud Application Platform like Heroku
Architecture Overview
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
Containers
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
Virtual Environments Assigned To Users
Similar to Dynos of Heroku
Containers made by LXC (Linux Containers)
EC2 Instance (1 Virtual Machine)
Container for
user B
Container for
user A
Container for
user A
Container for
user B
Container for
user B
Container for
user D
Container for
user D
Container for
user C
Container for
user E
Container for
user E
Container for
user F
Container for
user F
Container for
user E
Container for
user F
Container for
user F
NginxUnicorn
sshdsupervisrod
on each container
Amazon Linux+
Patched kernel(3.2.16)
grsecurity kernel patchfor various restrictions
original kernel patchesto restrict tcp port
bind and fork bomb
Anti fork bomb patch makes some changes to cgroup and fork process
Seepaperboy-sqale/sqale-patches
on GitHub
Web Proxy
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
nginx
Container for
user A
Container for
user B
Container for
user B
Container for
user C
Container for
user C
Container for
user C
ELB
nginx
HTTP/HTTPS
nginxlua-nginx-module
redis2-nginx-module
Container for
lokka-mizzy
Container for
lokka-mizzy
Container for
i4pc-mizzy
Container for
i4pc-mizzy
nginx
http://www.i4pc.jp/
Redis
nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084
Which containers?
host001:8083, host001:8084
host001
or
location / { set $container ""; set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
nginx.conf (excerpt)
local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)end
local containers, type = parser.parse_reply(reply.body)
dynamic-proxy.lua (excerpt)
while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
dynamic-proxy.lua (excerpt)
ngx.var.container = containerngx.var.next_containers = luabins.save(containers)
dynamic-proxy.lua (excerpt)
location / { set $container ""; set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
nginx.conf (again)
location @failover { error_page 502 = @failover;
rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
nginx.conf (excerpt)
failover.lua (excerpt)
local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end
failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
if not container then ngx.exit(503)end
ngx.var.container = containerngx.var.next_containers = luabins.save(containers)
failover.lua (excerpt)
location @failover { error_page 502 = @failover;
rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
nginx.conf (agin)
SSH Router
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
SSH Router
File Repositories(Git Server)
Git SSH Login
File Repositories(File Server)
Containers
SFTP
How implement this routing?
OpenSSH with script authentication patch
Seemizzy/openssh-script-auth
on GitHub
Change routes by SSH_ORIGNAL_COMMAND
In case of SSH_ORIGINAL_COMMAND
is “git-*”
SSH Router
File Repository(Git Server)
git push(ssh [email protected] git-recieve-pack ‘/mizzy/lokka.git’)
MySQL
Run AuthorizedKeys Script
Verify the public keyand get the user’s git server
command=“ssh [email protected] git-recieve-pack ‘/var/repos/mizzy/lokka.git’”
In case of SSH_ORIGINAL_COMMAND
is “sftp-server”
SSH Router
File Repository
(File Server)
sftp [email protected](ssh [email protected] sftp-server)
MySQL
File Repository(Git Server)
git push
Run AuthorizedKeys Script
Verify the public keyand get the user’s file server command=“ssh [email protected] sftp-server”
In case of SSH_ORIGINAL_COMMAND
is empty
SSH Router
Container
MySQL
Run AuthorizedKeys Script
Verify the public keyand get the user’s cotainers list
command=“ssh sqale@ users001.sqale.lan -p 8081”
Display the user’s containers list and wait the user’s selection
Ruby On RailsPuppet/Chef
UnicornResque
And many gems
多謝