Neural Moore machines

Elman (1990)'s simple recurrent net, a widely-used Moore NSM, is described by a next-state function identical to the next-state function of Robinson and Fallside (1991), eq. (3.10), and an output function ${\bf h}({\bf x}[t])$ whose $i$-th component ( $i=1,\ldots,n_Y$) is given by

$$h_i({\bf x}[t])=g\left(\sum_{j=1}^{n_X} W_{ij}^{yx} x_j[t] + W^y_i\right).$$ (4.15)

However, an even simpler DTRNN is the one used by Williams and Zipser (1989c,a), which has the same next-state function but an output function that is simply a projection of the state vector $y_i[t]=x_i[t]$ for $i=1,\ldots,n_Y$ with $n_Y\le n_X$. This architecture is also used in the encoder (or compressor) part of Pollack (1990)'s RAAM (see page ) when encoding sequences.

The second-order counterpart of Elman (1990)'s simple recurrent net has been used by Blair and Pollack (1997) and Carrasco et al. (1996). In that case, the $i$-th coordinate of the next-state function is identical to eq. (3.8), and the output function is identical to eq. (3.15).

The second-order DTRNN used by Giles et al. (1992) , Watrous and Kuhn (1992), Pollack (1991) , Forcada and Carrasco (1995), and Zeng et al. (1993) may be formulated as a Moore NSM in which the output vector is simply a projection of the state vector $h_i({\bf x}[t])=x_i[t]$ for $i=1,\ldots,n_Y$ with $n_Y\le n_X$, as in the case of Williams and Zipser (1989c) and Williams and Zipser (1989a). The classification of these second-order networks as Mealy or Moore NSM depends on the actual configuration of feedback weights used by the authors. For example, Giles et al. (1992) use one of the units of the state vector ${\bf x}[t]$ as an output unit; this would be a neural Moore machine in which $y[t]=x_1[t]$ (this unit is part of the state vector because its value is also fed back to form ${\bf x}[t-1]$ for the next cycle).