Potential error in the derivation for backpropagation through through time in RNNs

[faerryn]

We believe there is an error with section 9.7.2: Backpropagation Through Time in Detail, and would like to ask for clarification, and we would like to help fix the error if it indeed is present. This issue was researched by myself, @ishaanharry, and @Calcu-dev, with guidance from our course instructor. The text of this issue was primarily authored by @ishaanharry.

The text notes that

Thus as an intermediate step to computing dL/dW_hx and dL/dW_hh, dh_t/dW_hx and dh_t/dW_hh must be computed.

The text equates the above expressions to these, which implies that dh_t/dW_hx = x_t^T and dh_t/dW_hh = h_{t-1}^T

However, this cannot be correct, as can be seen upon reviewing the expression for h_t.

Observe that h_t contains the term h_{t-1}, so this is a telescopic function that rolls out until the initial value. What's important to note is that each telescoping term contains another W_hx and W_hh, so the derivative of h_t w.r.t either of the weight matrices must have multiple terms, not simply x_t^T or h_{t-1}^T.

The textbook even states this in a prior section. In 9.7.1: Analysis of Gradients in RNNs, the text derived dh_t/dw_h using the rollout I described earlier, with h_t being a generalized function of x_t, h_{t-1}, and w_h.

So applying this to the earlier derivation of dL/dW_hx and dL/dW_hh, we should get