Fractal Analysis of GPT-2 Token Embedding Spaces: Stability and Evolution of Correlation Dimension

This paper explores the fractal properties of token embedding spaces in GPT-2 language models by analyzing the stability of the correlation dimension, a measure of geometric complexity. Token embeddings represent words or subwords as vectors in a high-dimensional space. We hypothesize that the correlation dimension $D_2$ remains consistent across different vocabulary subsets, revealing fundamental structural characteristics of language representation in GPT-2. Our main objective is to quantify and analyze the stability of $D_2$ in these embedding subspaces, addressing the challenges posed by their high dimensionality. We introduce a new theorem formalizing this stability, stating that for any two sufficiently large random subsets $S_1,S_2\subset E$, the difference in their correlation dimensions is less than a small constant $\epsilon$. We validate this theorem using the Grassberger–Procaccia algorithm for estimating $D_2$, coupled with bootstrap sampling for statistical consistency. Our experiments on GPT-2 models of varying sizes demonstrate remarkable stability in $D_2$ across different subsets, with consistent mean values and small standard errors. We further investigate how the model size, embedding dimension, and network depth impact $D_2$. Our findings reveal distinct patterns of $D_2$ progression through the network layers, contributing to a deeper understanding of the geometric properties of language model representations and informing new approaches in natural language processing.