W5 V2 - Surrogate gradients

"Leaky Integrate and Fire" neurons are non-differentiable functions: the "Heaviside function" creates a step, which has an undefined derivative, therefore the "if voltage > threshold" always returns a delta of nothing.

A "surrogate gradient" keeps the Heaviside function in the forward pass and substitutes a differentiable step function for the backward pass.

The surrogate gradient in turn participates in backpopagation through time--so copies of the network have to be held from historical passes.

Requires cheesing your auto-gradient/differentiation framework. The differentiable function is *not* the derivative of the forward function, but the derivative of the surrogate function.