_backward as lambdas?

[ondras]

Hi @karpathy,

congratulations on this repo/talk. The educational value is truly immense. Good job!

Can you please explain the main motivation for _backward methods implemented as lambdas, as opposed to (one) regular method that starts with a hypothetical switch (self._op) and contains implementation for all arithmetic cases?

[steve-z-seattle]

@ondras, That is a very interesting question. IMHO, it is less about lambda, and more about closure. It seems that the alternative approach will need some tedious unpacking, while the implementation here has the advantage to be more concise, as all the variables are conveniently available (typically self/x, other/y, and out) for gradient updates.

For the alternative implementation, please refer to my repo (WIP): https://github.com/steve-z-seattle/undergrad. The alternative implementation does not use lambdas/closure. Working in progress though.

At this point, I do think the closure implementation is more elegant.

[ondras]

Hi @steve-z-seattle,

thanks for your opinion. I will have a look at your repo.

[ElHadi73]

def _backward():
self.grad = (1 - t**2) * out.grad
out._backward = _backward
return out

can anyone explain how this one work, it really confused me, i understand the math, but i didnt understand the access to it, and stuff, like how it is being accessed to and what variables the function will have access to later