average surprise about samples from the distribution; quantifies the “informativeness” of a distribution. A smaller entropy means is more concentrated and a large entropy indicates a diffuse distribution (we are less certain that the samples )
1 min read
average surprise about samples from the distribution; quantifies the “informativeness” of a distribution. A smaller entropy means p is more concentrated p and a large entropy indicates a diffuse distribution (we are less certain that the samples x∼p)