vec2vec-mlp-silu-not-cnn-no-spatial-structure
IN premise — summaries/2026/08/24/jha-2025-vec2vec-s2-problem-formulation-unsupervised-embedding-translation.md
Created 2026-08-24T17:10:57+00:00
vec2vec uses MLPs with residual connections, layer normalization, and SiLU activations (not CNNs) because text embeddings lack pixel-grid spatial structure; discriminators omit residual connections.
Summary
vec2vec deliberately chose fully-connected layers over convolutional layers because embedding vectors don't have the row-and-column spatial layout that images do, so convolving across "rows" of numbers would be meaningless. This means the system's architecture is tuned to treat each embedding as a bag of independent features, and the discriminators are slightly simpler (no skip connections) since they only need to judge, not reconstruct.