SKYMIND INTELLIGENCE LAYER (SKIL)REFERENCE ARCHITECTURE

Overview

● Why am I up here?● Reinforcement Learning● Use cases● Demo!● Deep Reinforcement Learning● Rl4j● Dl4j● Spark/RL - why?

Why am I up here?

Wrote this -->Book Giveaway!

Reinforcement Learning

● Learn a “policy” with repeated trial and error

● An agent explores a search space● Learns from rewards and penalties

each time it takes a step● Think of win/lose scenarios● Rewards/punishment set by an

“environment”

Credit: http://ai.berkeley.edu/reinforcement.html

Use cases (not games!)

● Risk analysis (loans)● Network Intrusion● Learning patterns from

simulations (MCMC)

Demo!

Cartpole (Hello world of RL)

Deep Reinforcement Learning

● Teach a neural net from environment● Policy determines gradient descent steps ● Most work has been based on raw frames

from games (pixel input)● Various techniques (A3C,Policy Gradients,Deep

Q,..)● Core idea: Neural net has a softmax

(probability distribution) mapped to actions to take in an environment

RL4j

● Deep Reinforcement Learning library for Java

● Openai Gym Intregration● Deep Reinforcement Learning

with DL4j● Implementations of A3C,DeepQ,

Policy Gradients● Openai Gym Java Bindings

Dl4j

Dl4j

● Import keras models● Focus on running in production● Integrate with existing big data ecosystem● Transparent usage of cpus and gpus● End to end ecosystem for building data

products (not just algorithms!)

Spark/RL Why?

● Spark is distributed compute● A lot of simulations and

environments to run● Distributed workers running

experiments in parallel● Data Parallelism with neural nets

Summary

● Spark for orchestrating simulations● Spark for distributed training● Integrated storage with HDFS● Orchestrate GPU based spark jobs● Easy to hook in to production (java/scala)● Great streaming ecosystem for incremental

updates