Upload File

regression and gradient descent - github pages · 2020. 12. 20. · linear regression with batch...

  • Home
  • Documents
  • Regression and Gradient Descent - GitHub Pages · 2020. 12. 20. · Linear regression with Batch Gradient Descent Repeat { (for every ) } Learning rate is typically held constant
49
Gradient Descent Source: Intro. to Machine Learning By Andrew Ng, Stanford, Coursera Regression and

Upload: others

Post on 28-Jan-2021

4 views

Category:

Documents


0 download

Report

TRANSCRIPT

  • GradientDescent

    Source:Intro.toMachineLearningByAndrewNg,Stanford,Coursera

    Regressionand

  • Notation:m=Numberoftrainingexamplesx’s=“input”variable/featuresy’s=“output”variable/“target”variable

    Sizeinfeet2(x) Price($)in1000's(y)2104 4601416 2321534 315852 178… …

    Trainingsetof

    housingprices

    (Portland,OR)

  • 0

    100

    200

    300

    400

    500

    0 500 1000 1500 2000 2500 30000

    100

    200

    300

    400

    500

    0 500 1000 1500 2000 2500 3000

    HousingPrices

    (Portland,OR)

    Price(in1000sofdollars)

    Size(feet2)RegressionProblem

    Predictreal-valuedoutput

  • Howtochoose‘s?

    TrainingSet

    Hypothesis:‘s:Parameters

    Sizeinfeet2(x) Price($)in1000's(y)2104 4601416 2321534 315852 178… …

  • Hypothesis:

    Parameters:

    CostFunction:

    Goal:

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

    0

    100

    200

    300

    400

    500

    0 1000 2000 3000

    Price($)in1000’s

    Sizeinfeet2(x)

  • Gradientdescent

  • Havesomefunction

    Want

    Outline:

    •  Startwithsome

    •  Keepchangingtoreduce

    untilwehopefullyendupataminimum

  • !1!0

    J(!0,!1)

  • !0

    !1

    J(!0,!1)

  • Ifαistoosmall,gradientdescentcanbeslow.

    Ifαistoolarge,gradientdescentcanovershoottheminimum.Itmayfailtoconverge,orevendiverge.

  • atlocaloptima

    Currentvalueof

  • Gradientdescentalgorithm

  • Gradientdescentalgorithm

    Correct:Simultaneousupdate Incorrect:

  • Gradientdescentalgorithm LinearRegressionModel

  • Gradientdescentalgorithm

    updateand

    simultaneously

  • !1!0

    J(!0,!1)

  • !0

    !1

    J(!0,!1)

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

  • (forfixed,thisisafunctionofx) (functionoftheparameters)

  • Linearregressionwithgradientdescent

    Repeat{(forevery)}

  • LinearregressionwithBatchGradientDescent

    Repeat{(forevery)}

  • Learningrateistypicallyheldconstant.Canslowlydecreaseovertimeifwewanttoconverge.(E.g. )const1 iterationNumber + const2

    Stochasticgradientdescent

    1.  Randomlyshuffledataset.2.  Repeat{

    for {

    (for)}}

  • Adviceforapplyingmachinelearning

    Diagnosingbiasvs.variance

    MachineLearning

  • Bias/variance

    Highbias(underfit)

    “Justright” Highvariance(overfit)

    Price

    Size

    Price

    Size

    Price

    Size

  • Bias/variance

    Highbias(underfit)

    “Justright” Highvariance(overfit)

    Price

    Size

    Price

    Size

    Price

    Size

  • Bias/variance

    degreeofpolynomiald

    error

    Trainingerror:

    Crossvalidationerror:

    size

    price

  • Bias/variance

    degreeofpolynomiald

    error

    Trainingerror:

    Crossvalidationerror:

    size

    price

  • Diagnosingbiasvs.variance

    degreeofpolynomiald

    error

    Supposeyourlearningalgorithmisperforminglesswellthanyouwerehoping.(orishigh.)Isitabiasproblemoravarianceproblem?

    (crossvalidationerror)

    (trainingerror)

    Bias(underfit):

    Variance(overfit):

  • Diagnosingbiasvs.variance

    degreeofpolynomiald

    error

    Supposeyourlearningalgorithmisperforminglesswellthanyouwerehoping.(orishigh.)Isitabiasproblemoravarianceproblem?

    (crossvalidationerror)

    (trainingerror)

    Bias(underfit):

    Variance(overfit):

  • Adviceforapplyingmachinelearning

    Regularizationandbias/variance

    MachineLearning

  • Linearregressionwithregularization

    LargexxHighbias(underfit)

    Intermediatexx“Justright”

    SmallxxHighvariance(overfit)

    Model:Price

    Size

    Price

    Size

    Price

    Size

  • Linearregressionwithregularization

    LargexxHighbias(underfit)

    Intermediatexx“Justright”

    SmallxxHighvariance(overfit)

    Model:Price

    Size

    Price

    Size

    Price

    Size

  • Choosingtheregularizationparameter

  • Choosingtheregularizationparameter

  • Model:

    Choosingtheregularizationparameter

    Pick(say).Testerror:

  • 1.  Try2.  Try3.  Try4.  Try5.  Try12. Try

    Model:

    Choosingtheregularizationparameter

    Pick(say).Testerror:

  • Bias/varianceasafunctionoftheregularizationparameter

  • Bias/varianceasafunctionoftheregularizationparameter


Gradient Descent Optimization

Linear Regression & Gradient Descentbboots3/CS4641-Fall2018/...Linear Regression & Gradient Descent Robot Image Credit: ViktoriyaSukhanova© 123RF.com These slides were assembled by

Gradient Descent · 2015. 7. 6. · Example: Linear regression, big n and huge d • Gradient descent • O(d) communication slow with hundreds of millions parameters • Distribute

A Gentle Introduction to Gradient Boosting Boosting for Regression How is this related to gradient descent? For regression with square loss, residual ,negative gradient t h to residual

Gradient Descent Easy version

Optimization/Gradient Descent

Machine Learning - Stephan Robert€¦ · Lesson 1: Introduction to Machine Learning, Gradient Descent (cost function, many variables) Lesson 2: Linear regression, logistic regression

Gradient Descent Algorithm in Machine Learning · Examples of algorithms with coefficients that can be optimized using gradient descent are: Linear Regression Logistic Regression

Efficient Logistic Regression with Stochastic Gradient Descent William Cohen

Lecture 2: Linear Regression - GitHub Pages · Lecture 2: Linear Regression 1 Supervised Learning: Regression and Classi cation 2 Linear Regression 3 Gradient Descent Algorithm 4

Linear Regression & Gradient Descent

Machine Learning: Linear Regression - Jarrar€¦ · qPart 1: Motivation (Regression Problems) qPart 2: Linear Regression qPart 3: The Cost Function qPart 4: The Gradient Descent

Gradient descent method

07 logistic regression and stochastic gradient descent

Gradient descent GAN optimization is locally stablepapers.nips.cc/paper/7142-gradient-descent-gan-optimization-is... · Gradient descent GAN optimization is locally stable ... similarities

Getting to the Bottom of Regression with Gradient Descent...Getting to the Bottom of Regression with Gradient Descent Jocelyn T. Chi and Eric C. Chi January 10, 2014 This article on

Proximal Gradient Descent › ~aarti › Class › 10725_Fall17 › Lecture_Slides › ...Proximal gradient descent has convergence rate O(1=k), or O(1= ) Same as gradient descent!

Classification and Prediction: Regression Via Gradient Descent Optimization Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University

Learning to learn by gradient descent by gradient descent · Learning to learn by gradient descent by gradient descent Liyan Jiang July 18, 2019 1 Introduction The general aim of

Gradient Descent for Linear Regressioncs229.stanford.edu/notes-spring2019/Gradient_Descent_Viz.pdf · 2019-04-03 · Gradient Descent for Linear Regression This is meant to show you

An Overview of Gradient Descent Optimization Algorithms ... · Outline 1 Introduction Basics 2 Gradient Descent Variants Basic Gradient Descent Algorithms Limitations 3 Gradient Descent

Deep Learning I: Gradient Descentgd.pdf · Roadmap Intro,model,cost Gradient descent Gradient descent (GD) Gradient descent is a learning algorithm. Given: a hypothesis space (or

Intro Logistic+Regression Gradient+Descent+++SGD...9 SGD:+Stochastic+Gradient+Ascent+(or+Descent) • “True”gradient: • Samplebasedapproximation: • Whatifweestimategradientwithjustonesample???

Intro Logistic Regression Gradient Descent + SGDIntro Logistic Regression Gradient Descent + SGD Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade

Efficient Logistic Regression with Stochastic Gradient Descent

Intro Logistic Regression Gradient Descent + SGD AdaGradcourses.cs.washington.edu/courses/cse547/14wi/slides/...1 1 Intro Logistic Regression Gradient Descent + SGD AdaGrad Machine

Linear Regression & Gradient Descent · 2020. 1. 22. · Linear Regression & Gradient Descent Robot Image Credit: ViktoriyaSukhanova© 123RF.com These slides were assembled by Byron

Package ‘gradDescent’ · Description An implementation of various learning algorithms based on Gradient Descent for deal-ing with regression tasks. The variants of gradient descent

Lecture 6: stochastic gradient descent Sanjeev Arora Elad …...• Gradient descent algorithm, linear classification • Stochastic gradient descent Title 402-lec6 Created Date 10/6/2016

Machine Learning: Logistic Regression Lecture 04oucsace.cs.ohiou.edu/~razvan/courses/ml4900/lecture04a.pdf · Machine Learning: Logistic Regression Lecture 04. ... gradient descent

ECE595 / STAT598: Machine Learning I Lecture 15 Logistic ... · Lecture 14 Logistic Regression 1 Lecture 15 Logistic Regression 2 This lecture: Logistic Regression 2 Gradient Descent

1 Lecture 10: descent methods Gradient descent (reminder)

Gradient Descent and Artificial Neural Networks · Regression, Gradient Descent and Neural Networks. Lesson 5 5-11 The Neural Network model A neural network is a multi-layer assembly

Non-Convex Projected Gradient Descent for Generalized Low ... · Non-Convex Projected Gradient Descent for Generalized Low-Rank Tensor Regression Han Chen , Garvesh Raskuttiyand Ming

Gradient descent