SKA Entropy Gradient Explorer
Compare the entropy and cosine alignment trajectories with and without the ΔD term in the SKA gradient. Same architecture, same data, same weights — only the gradient formulation differs.
The Two Entropy Gradient Variants
| Variant | Entropy Gradient ∂H/∂z |
|---|---|
| With ΔD | -(1/ln2) [ z·D(1-D) + ΔD ] |
| Without ΔD | -(1/ln2) [ z·D(1-D) ] |
Reference Paper
Abstract
We introduce the Structured Knowledge Accumulation (SKA) framework, which reinterprets entropy as a dynamic, layer-wise measure of knowledge alignment in neural networks. Instead of relying on traditional gradient-based optimization, SKA defines entropy in terms of knowledge vectors and their influence on decision probabilities across multiple layers. This formulation naturally leads to the emergence of activation functions such as the sigmoid as a consequence of entropy minimization. Unlike conventional backpropagation, SKA allows each layer to optimize independently by aligning its knowledge representation with changes in decision probabilities. As a result, total network entropy decreases in a hierarchical manner, allowing knowledge structures to evolve progressively. This approach provides a scalable, biologically plausible alternative to gradient-based learning, bridging information theory and artificial intelligence while offering promising applications in resource-constrained and parallel computing environments.
SKA Explorer Suite
About this App
The ΔD term captures the discrete change in decision probability between two consecutive forward passes. Removing it reduces the gradient to a continuous-like form. This app runs both variants side by side — same weights, same data, same architecture — so the role of ΔD is immediately visible in the entropy and cosine trajectories.