Package

com.thoughtworks

deeplearning

Permalink

package deeplearning

This is the documentation for the DeepLearning.Scala

Overview

BufferedLayer, DifferentiableAny, DifferentiableNothing, Layer, Poly and Symbolic are base packages which contains necessary operations , all other packages dependent on those base packages.

If you want to implement a layer, you need to know how to use base packages.

Imports guidelines

If you want use some operations of Type T, you should import:

import com.thoughtworks.deeplearning.DifferentiableT._

it means: If you want use some operations of INDArray, you should import:

import com.thoughtworks.deeplearning.DifferentiableINDArray._

If you write something like this:

def softmax(implicit scores: INDArray @Symbolic): INDArray @Symbolic = {
  val expScores = exp(scores)
  expScores / expScores.sum(1)
}

If compiler shows error :

Could not infer implicit value for com.thoughtworks.deeplearning.Symbolic[org.nd4j.linalg.api.ndarray.INDArray]

you need add import this time :

import com.thoughtworks.deeplearning.DifferentiableINDArray._

If you write something like this:

def crossEntropyLossFunction(
  implicit pair: (INDArray :: INDArray :: HNil) @Symbolic)
: Double @Symbolic = {
 val score = pair.head
 val label = pair.tail.head
 -(label * log(score * 0.9 + 0.1) + (1.0 - label) * log(1.0 - score * 0.9)).mean
}

If the compiler shows error:

value * is not a member of com.thoughtworks.deeplearning.Layer.Aux[com.thoughtworks.deeplearning.Layer.Tape.Aux[org.nd4j.linalg.api.ndarray.INDArray,org.nd4j.linalg.api.ndarray.INDArray],com.thoughtworks.deeplearning.DifferentiableINDArray.INDArrayPlaceholder.Tape]val bias = Nd4j.ones(numberOfOutputKernels).toWeight * 0.1...

you need add import :

import com.thoughtworks.deeplearning.Poly.MathMethods.*
import com.thoughtworks.deeplearning.DifferentiableINDArray._

If the compiler shows error:

not found: value log -(label * log(score * 0.9 + 0.1) + (1.0 - label) * log(1.0 - score * 0.9)).mean...

you need add import:

import com.thoughtworks.deeplearning.Poly.MathFunctions.*
import com.thoughtworks.deeplearning.DifferentiableINDArray._

Those + - * / and log exp abs max min are defined at MathMethods and MathFunctions,those method are been implemented at DifferentiableType,so you need to import the implicit of DifferentiableType.

Composability

Neural networks created by DeepLearning.scala are composable. You can create large networks by combining smaller networks. If two larger networks share some sub-networks, the weights in shared sub-networks trained with one network affect the other network.

See also

Compose

Linear Supertypes
Content Hierarchy
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. deeplearning
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. trait CumulativeLayer extends Layer

    Permalink

    A Layer that minimizes the computation during both forward pass and backward pass.

    A Layer that minimizes the computation during both forward pass and backward pass.

    For forward pass, the result will be cached.

    For backward pass, the Tape is accumulated until flush.

    Author:

    杨博 (Yang Bo) <[email protected]>

    See also

    Layer.Output

  2. trait Layer extends AnyRef

    Permalink

    一个Layer表示一个神经网络。每个Layer可以作为子网络被包含在其它Layer中,构成更复杂的神经网络。Layer的嵌套结构可以用来表示数学公式或粗粒度神经网络结构。 当神经网络被编写完成后,其中大部分元素都是占位符,当网络开始训练时数据才真正进入到网络。

    一个Layer表示一个神经网络。每个Layer可以作为子网络被包含在其它Layer中,构成更复杂的神经网络。Layer的嵌套结构可以用来表示数学公式或粗粒度神经网络结构。 当神经网络被编写完成后,其中大部分元素都是占位符,当网络开始训练时数据才真正进入到网络。

    Layer 的树结构
    val myLayer: Layer.Aux[Tape.Aux[Double, Double], Tape.Aux[Double, Double]] = {
      Times(
        Plus(
          Literal(1.0),
          Identity[Double, Double]()
        ),
        Weight(2.0)
      )
    }

    以上代码等价的数学公式可以用Symbolic写作:(1.0 + x) * 2.0.toWeight2.0.toWeight表示一个变量,其初始值是2,在神经网络迭代时,其值会更新。 TimesPlus都是 case class, 因此myLayer是一个case class构成的嵌套结构的树。TimesPlus都是占位符。

    Weight是一个包含权重的Layer,初始值是2.0Identity是一个输入和输出相同的Layer,它会将输入原样返回。Identity在这里是Input的占位符。 Literal是一个包含常量的Layer

    迭代

    网络每次训练称为一个迭代,分为forwardbackward两个阶段,构成一次完整的反向传播流程。

    forward

    Layer.Aux[A,B]中调用forward时,A是输入类型,B是输出类型,AB都是Tape。下面开始逐段解释代码:

    例如:

    val inputTape: Tape.Aux[Double, Double] = Literal(a)
    val outputTape = myLayer.forward(inputTape)

    当调用myLayer.forward(inputData)时,首先调用Timesforward,其伪代码如下:

    final case class Times(operand1: Layer, operand2: Layer) extends Layer {
      def forward(inputData: Tape): Output = {
        val upstream1 = operand1.forward(input)
        val upstream2 = operand2.forward(input)
        new Output(upstream1, upstream2)//这里忽略具体实现,而关注递归细节
      }
      final class Output(upstream1: Tape, upstream2: Tape) extends Tape { ... }
    }

    myLayer.operand1Plus,myLayer.operand2Weight,因此,upstream1upstream2分别是operand1operand2 forward 的结果。

    以此类推,Plusforward代码与Timesforward类似,当调用Plusforward时,operand1Literaloperand2Identity,这时会各自调用LiteralIdentityforward

    当调用Identityforward时会原样返回输入, Identityforward的伪代码如下:

    def forward(inputTape: Tape.Aux[Double, Double]) = inputTape

    所以Input即 数学公式(1.0 + x) * 2.0.toWeight 中的x,这样Input就被传递给了神经网络。

    myLayer.forward的返回值outputTapeTape类型,所以最终会生成一棵Tape构成的树,结构和myLayer一样。 因此,通过层层传播 myLayer.forward(inputTape)最终被Identity原样返回,组合进新生成的Tape树。

    outputTape 的包含forward 的计算结果,计算结果可以用来 backward 比如:

    try {
      val loss = outputTape.value
      outputTape.backward(loss)
      loss
    } finally {
      outputTape.close()
    }

    outputTape.value 是数学公式 (1.0 + x) * 2.0.toWeight 的计算结果。

    backward

    outputTape.backwardTimes.Outputbackward ,伪代码如下:

    case class Times(operand1: Layer, operand2: Layer) extends Layer {
      def forward = ...
      class Output(upstream1, upstream2) extends Tape {
        private def upstreamDelta1(outputDelta: Double) = ???
        private def upstreamDelta2(outputDelta: Double) = ???
        override protected def backward(outputDelta: Double): Unit = {
          upstream1.backward(upstreamDelta1(outputDelta))
          upstream2.backward(upstreamDelta2(outputDelta))
        }
      }
    }

    outputTape.upstream1outputTape.upstream2分别是operand1operand2 forward 的结果。然后outputTape.upstream1outputTape.upstream2分别进行backward

    以此类推,Plusbackward代码与Timesbackward类似,当调用Plusbackward时,upstream1upstream2分别是LiteralIdentity forward的结果,这时会各自调用upstream1upstream2backward

    Weightbackward时会更新自己,参考updateDouble

    Aux & Symbolic API

    Layer.Aux[A,B]表示Input的类型是AOutput的类型是BTape.Aux[C,D]表示Data的类型是CDelta的类型是DLayer.AuxType.Aux可以组合起来使用,比如Layer.Aux[Tape.Aux[A,B],Tape.Aux[C,D]]可以用来表示一个layer的输入类型是一个Tape,这个Tape的数据类型为Adelta类型为Blayer的输出类型是一个Tape,这个Tape的数据类型为Cdelta类型为D

    Aux是一种实现了type refinement的设计模式,可以用来限制类型参数的范围。

    通常我们不会手写Aux类型,因为我们可以使用Symbolic实现同样的功能,例如在用于符号方法内部变量和返回值时:Layer.Aux[Tape.Aux[INDArray, INDArray], Tape.Aux[INDArray, INDArrayINDArray @Symbolic 是等价的,所以我们经常使用Symbolic来替代Aux的写法。

    See also

    Symbolic

    Backpropagation

    type refinement

    aux pattern evolution

    aux pattern

  3. trait Symbolic[NativeOutput] extends AnyRef

    Permalink

    Provides @Symbolic annotation to create symbolic methods, in which you can create Layers from mathematical formulas.

    Provides @Symbolic annotation to create symbolic methods, in which you can create Layers from mathematical formulas.

    Symbolic is a dependent type class that calculates a specific Layer type according to NativeOutput. Combining with implicit-dependent-type compiler plugin, it can be treated as a type annotation in the form of NativeOutput @Symbolic, converting NativeOutput to a specific Layer type.

    @Symbolic 的三种用法

    用于符号方法的隐式参数类型

    如果某个方法的隐式类型参数标注了@Symbolic,那么这个方法就是符号方法,@Symbolic所标注的隐式参数类型是这个符号方法的输入类型。 这种情况下,NativeOutput @Symbolic会被展开为:

    Identity[NativeOutput, NativeOutput的导数类型]

    例如:

    def sumNetwork(implicit scores: INDArray @Symbolic): Double = {
      exp(scores).sum
    }

    上述代码中,由于INDArray导数类型也是INDArray,所以sumNetwork的输入类型INDArray @Symbolic展开后是Identity[INDArray, INDArray]

    用于符号方法内部变量和返回值

    在符号方法内部和返回值处,NativeOutput @Symbolic会被展开为:

    Layer.Aux[Tape.Aux[输入类型的值类型, 输入类型的导数类型], Tape.Aux[NativeOutput, NativeOutput的导数类型]]

    例如:

    def sumNetwork(implicit scores: INDArray @Symbolic): Double @Symbolic = {
      val expScores: INDArray @Symbolic = exp(scores)
      val result: Double @Symbolic = expScores.sum
      result
    }

    上述代码中,expScores的类型INDArray @Symbolic展开后是:

    Layer.Aux[Tape.Aux[INDArray, INDArray], Tape.Aux[INDArray, INDArray]]

    result的类型Double @Symbolic展开后是:

    Layer.Aux[Tape.Aux[INDArray, INDArray], Tape.Aux[Double, Double]]
    用于符号方法之外

    在符号方法之外,(NativeInput => NativeOutput) @Symbolic会被展开为:

    Layer.Aux[Tape.Aux[NativeInput, NativeInput的导数类型], Tape.Aux[NativeOutput, NativeOutput的导数类型]]

    例如:

    val predictor: (INDArray => Double) @Symbolic = sumNetwork

    上述代码中,predictor的类型(INDArray => Double) @Symbolic展开后是:

    Layer.Aux[Tape.Aux[INDArray, INDArray], Tape.Aux[Double, Double]]

    定制符号类型

    @Symbolic通过检查Symbolic.ToLiteral隐式值来确定原生类型和导数之间的映射关系。 因此,只要定义Symbolic.ToLiteral类型的隐式值,@Symbolic就可以支持定制符号类型。

    比如,假如你希望支持Short @Symbolic,其中使用Float作为Short的导数类型,那么可以这样做:

    implicit object ShortToLiteral extends ToLiteral[Short] {
      override type Data = Short
      override type Delta = Float
      override def apply(data: Short) = Literal(data)
    }
    
    def makeShortNetwork(implicit input: Short @Symbolic): Short @Symbolic = {
      input
    }
    
    val shortNetwork: (Short => Short) @Symbolic = makeShortNetwork

    这样一来shortNetwork的类型就会展开为:

    Layer.Aux[Tape.Aux[Short, Float], Tape.Aux[Short, Float]]
    Annotations
    @implicitNotFound( ... )
    See also

    Layer.Tape#Delta

    Symbolic.ToLiteral

    Symbolic.Layers.Identity

Value Members

  1. object CumulativeLayer

    Permalink
  2. object DifferentiableAny

    Permalink

    A namespace of common operators for any layers.

    A namespace of common operators for any layers.

    After importing DifferentiableAny._, the following methods will be available on any layers.

    Author:

    杨博 (Yang Bo) <[email protected]>

  3. object DifferentiableBoolean

    Permalink

    A namespace of common operators for Boolean layers.

    A namespace of common operators for Boolean layers.

    After importing DifferentiableBoolean._, the following methods will be available on Boolean layers.

    Author:

    杨博 (Yang Bo) <[email protected]>

  4. object DifferentiableCoproduct

    Permalink

    A namespace of common operators for Coproduct layers.

    A namespace of common operators for Coproduct layers.

    After importing DifferentiableCoproduct._, the following methods will be available on Coproduct layers.

    Author:

    杨博 (Yang Bo) <[email protected]>

  5. object DifferentiableDouble

    Permalink

    A namespace of common operators for Double layers.

    A namespace of common operators for Double layers.

    Author:

    杨博 (Yang Bo) <[email protected]>

  6. object DifferentiableFloat

    Permalink

    A namespace of common operators for Float layers.

    A namespace of common operators for Float layers.

    Author:

    杨博 (Yang Bo) <[email protected]>

  7. object DifferentiableHList

    Permalink

    A namespace of common operators for HList layers.

    A namespace of common operators for HList layers.

    After importing DifferentiableHList._, the following methods will be available on HList layers.

    Author:

    杨博 (Yang Bo) <[email protected]>

  8. object DifferentiableINDArray

    Permalink

    A namespace of common operators for INDArray layers.

    A namespace of common operators for INDArray layers.

    After importing DifferentiableINDArray._,

    You will able to use MathFunctions,like

    You will able to use MathMethods,like

    You will able to use INDArrayLayerOps,like

    You will able to use some methods like conv2d

    Author:

    杨博 (Yang Bo) <[email protected]>

  9. object DifferentiableInt

    Permalink

    A namespace of common operators for Int layers.

    A namespace of common operators for Int layers.

    Author:

    杨博 (Yang Bo) <[email protected]>

  10. object DifferentiableNothing

    Permalink

    A namespace of common operators for all layers.

    A namespace of common operators for all layers.

    Author:

    杨博 (Yang Bo) <[email protected]>

  11. object DifferentiableSeq

    Permalink

    A namespace of common operators for Seq layers.

    A namespace of common operators for Seq layers.

    After importing DifferentiableSeq._, the following methods will be available on Seq layers.

    Author:

    杨博 (Yang Bo) <[email protected]>

  12. object Layer

    Permalink
  13. object Poly

    Permalink

    A namespace of common math operators.

    A namespace of common math operators.

    MathMethods and MathFunctions provide functions like +, -, *, /, log, abs, max, min and exp, those functions been implements in specific Differentiable Object such as DifferentiableINDArray

    Author:

    杨博 (Yang Bo) <[email protected]>

    See also

    DifferentiableINDArray.exp(INDArray)

    DifferentiableINDArray.Double+INDArray

  14. object Symbolic extends LowPrioritySymbolic

    Permalink

    There are two ways to convert a value to Layer.

    There are two ways to convert a value to Layer.

    The first way is invoke toLayer, such as:

    def createMyNeuralNetwork(implicit input: Float @Symbolic): Float @Symbolic = {
      val floatLayer: Float @Symbolic = 1.0f.toLayer
      floatLayer
    }

    The second way is autoToLayer, such as:

    def createMyNeuralNetwork(implicit input: Float @Symbolic): Float @Symbolic = {
      val floatLayer: Float @Symbolic = 1.0f
      floatLayer
    }

    In order to compile the above code through, you will need:

    import com.thoughtworks.deeplearning.Symbolic._
    import com.thoughtworks.deeplearning.Symbolic
    import com.thoughtworks.deeplearning.DifferentiableFloat._

Inherited from AnyRef

Inherited from Any

Ungrouped