A Generalized Multinomial Distribution from Dependent Categorical Random Variables

# A Generalized Multinomial Distribution from Dependent Categorical Random Variables

## Generalized Multinomial Distribution

From the construction in Section 2, we derive a generalized multinomial distribution in which all categorical variables are identically distributed but no longer independent.

Theorem 2: Generalized Multinomial Distribution.  Let $\epsilon_{1},\ldots,\epsilon_{N}$ be categorical random variables with categories $1,\ldots,K$, constructed as in Section 2. Let $X_{i} = \sum_{j=1}^{N}[\epsilon_{j} = i], i = 1,\ldots,K$, where $[\cdot]$ is the Iverson bracket. Denote $\mathbf{X} = (X_{1},\ldots,X_{K})$, with observed values $\mathbf{x} = (x_{1},\ldots,x_{K})$. Then

$$P(\mathbf{X} = \mathbf{x}) = \sum_{i=1}^{K}p_{i}\frac{(N-1)!}{(x_{i}-1)!\prod_{j \neq i}x_{j}!}\left(p_{i}^{+}\right)^{x_{i}-1}\prod_{j \neq i}\left(p_{j}^{-}\right)^{x_{j}}.$$

### Properties

This section details some useful properties of the Generalized Multinomial Distribution and the dependent categorical random variables.

#### Marginal Distributions

Theorem 3: Univariate Marginal Distribution.  The univariate marginal distribution of the Generalized Multinomial Distribution is the Generalized Binomial Distribution. That is,

$P(X_{i} = x_{i}) = q{N-1 \choose x_{i}}\left(p_{i}^{-}\right)^{x_{i}}\left(q^{-}\right)^{N-1-x_{i}} + p_{i}{N-1 \choose x_{i}-1}\left(p_{i}^{+}\right)^{x_{i}-1}\left(q^{+}\right)^{N-1-(x_{i}-1)},$
where $q = \sum_{j \neq i}p_{j}$, $q^{+} = q + \delta p_{i}$, and $q^{-} = q - \delta q$.
The above theorem shows another way the generalized multinomial distribution is an extension of the generalized binomial distribution.

Theorem 4: Moment Generating Function.  The moment generating function of the generalized multinomial distribution with K categories is given by
$$M_{\mathbf{X}}(\mathbf{t}) = \sum_{i=1}^{K}p_{i}e^{t_{i}}\left(p_{i}^{+}e^{t_{i}} + \sum_{j \neq i}p_{j}^{-}e^{t_{j}}\right)^{n-1}$$

where $\mathbf{X} = (X_{1},...,X_{K})$ and $\mathbf{t} = (t_{1},...,t_{K})$.

#### Moments of the Generalized Multinomial Distribution

Using the moment generating function in the standard way, the mean vector $\mu$ and the covariance matrix $\Sigma$ may be derived.

Expected Value. The expected value of $\mathbf{X}$ is given by $\mu = n\mathbf{p}$ where $\mathbf{p} = (p_{1},...,p_{K})$

Covariance Matrix The entries of the covariance matrix are given by
$$\Sigma_{ij} = \left\{\begin{array}{lr}p_{i}(1-p_{i})(n + \delta(n-1) + \delta^{2}(n-1)(n-2)), & i = j \\p_{i}p_{j}(\delta(1-\delta)(n-2)(n-1)-n), & i \neq j\end{array}\right.$$ Note that if $\delta = 0$, the generalized multinomial distribution reduces to the standard multinomial distribution and $\Sigma$ becomes the familiar multinomial covariance matrix. The entries of the corresponding correlation matrix are given by
$$\rho(X_{i},X_{j}) = -\sqrt{\frac{p_{i}p_{j}}{(1-p_{i})(1-p_{j})}}\left(\frac{n-\delta(n-1)(n-2)}{n+\delta(n-1)+\delta^{2}(n-1)(n-2)}\right)$$

If $\delta = 1$, the variance of $X_{i}$ tends to $\infty$ with n. This is intuitive, as $\delta = 1$ implies perfect dependence of $\epsilon_{2},...,\epsilon_{n}$ on the outcome of $\epsilon_{1}$. Thus, $X_{i}$ will either be 0 or n, and this spread increases to $\infty$ with n.