sae-training-objective-mse-plus-l1

Summary

When training a Sparse Autoencoder, you balance two competing goals — faithfully reconstructing the input and keeping most features switched off — but there is no objective rule for how much of each to prioritize. That means the number and granularity of features the SAE discovers depend largely on a manually chosen knob, so any interpretation of those features carries an extra layer of uncertainty that isn't grounded in the data itself.