Full Changelog: https://github.com/NTDLS/NTDLS.Determinet/compare/1.2.1...2.1.0

Nuget Package

https://www.nuget.org/packages/NTDLS.Determinet/ 📦

This release is a correctness overhaul. Backpropagation has been rewritten, and every activation function and loss path is now verified against numerical gradients. Several bugs that silently degraded training are fixed, training and inference are both faster, and model files from 1.x still load.

⚠️ Breaking changes

• New model file layout. Files saved by 2.x can't be opened by 1.x. Files saved by 1.x load normally, with identical predictions.
• DniSynapse.Weights is now a flat double[] (one row per output node) instead of double[,]. Use GetWeight(input, output) / SetWeight(...) for indexed access.
• Renamed or removed parameters:
  • Network.UseAdamBatchOptimization → Network.UseAdamOptimization. It now applies to both Train() and TrainBatch().
  • Layer.UseBatchNorm → Layer.UseLayerNorm. See "Training math" below for why.
  • Layer.BatchNormMomentum and SoftMax.MaxLogit have been removed.
• Network.GradientClip now limits the total size of the gradient (default 5.0; 0 disables it). Previously it clamped each value separately, which changed the update's direction.
• IDniActivationFunction.UsesCrossEntropy has been removed. SoftMax functions now implement IDniSoftMaxFunction.
• SoftMax / SimpleSoftMax are rejected on hidden layers. They have no element-wise derivative, so training through them was never mathematically valid.
• Non-SoftMax outputs now report squared-error loss. Previously they reported cross-entropy while training on squared-error gradients.

Bug fixes

Training math

• Batch normalization was actually a broken layer normalization. It normalized across a single sample's neurons, but:
  • backpropagation ignored the normalization step entirely;
  • γ/β were updated using the wrong values;
  • TrainBatch() never trained γ/β;
  • inference used running averages that training never saw. It is now a correct layer normalization that behaves the same in training and inference.
• SoftMax temperature is now included in the gradient. The loss is computed exactly from the pre-SoftMax values (logits), and MaxLogit clamping is gone because it distorted outputs.
• Activation None returned the wrong derivative (the pre-activation value instead of 1).
• Train() and TrainBatch() had drifted into two different implementations. TrainBatch() also ran the forward pass twice per sample. Train() clipped the step (learning rate × gradient) instead of the gradient, so clipping effectively never happened. There is now one shared path.
• The target array length is now validated. A mismatch previously caused an out-of-range crash or silently ignored values.
• Divergence (NaN/∞) now raises a clear exception. Previously those values were silently replaced with 0.
• Activation derivatives are evaluated at pre-activation values, and InputLabels reads the input layer instead of the output layer. (Both were fixed in 090d2ed.)

Activation functions

• PiecewiseLinear is continuous at its range edges; it used to jump whenever Alpha ≠ 1.
• SoftPlus and Mish no longer overflow to ∞ for large inputs. Sigmoid and Swish use a numerically stable form.
• SELU now uses the full-precision constants from the paper.

Infrastructure

• Parameters and DniRange are stored independent of the system locale, without precision loss. The old format truncated values below 1e‑17, and on comma-decimal locales it broke DniRange outright. Old files still parse.
• Reading a parameter as a different numeric type (e.g. set as int, read as double) no longer throws.
• DniUtility.Random is thread-safe, now backed by Random.Shared.
• DniConfiguration.LearningRate defaults to 0.005. It used to default to 0, which silently trained nothing.

New features

• Adam (AdamW) is available for both single-sample and mini-batch training. Optimizer state is saved with the model, so resumed training continues exactly where it stopped.
• TrainBatch(IEnumerable<(double[] inputs, double[] expected)>) overload.
• ComputeLoss(inputs, expected) evaluates the loss without modifying the network.
• Save(Stream) / Load(Stream) alongside the file-based methods.
• Weight initialization matched to each layer's activation: He for the ReLU family, LeCun for SELU, Glorot/Xavier otherwise. Biases now start at zero.
• Forward() no longer modifies the network, so inference is safe to call from multiple threads.
• Configuration is validated: node counts, label counts, where SoftMax and layer normalization are allowed, and SoftMax temperature.

Performance

Measured on a 4096→2048→512→128→62 network:

• Inference is about 5–7× faster, from a memory-friendly weight layout and SIMD dot products.
• Training is about 1.3–1.9× faster. Weight gradients are now applied in a single pass over the weights instead of being built as a full-size matrix first.

Quality

• New NTDLS.Determinet.Tests project (104 tests), covering:
  • gradient checks for every activation, both loss types, SoftMax temperature and layer normalization;
  • batch averaging and gradient clipping;
  • save/load round trips, including Adam state;
  • loading of 1.x files;
  • locale-independent parameter storage.

Test harness and samples

• ImageSharp has been replaced by Magick.NET in every sample app (Train, Validate, Draw, GenImages, OCR).
• The Draw app no longer leaks memory on every 250 ms refresh.
• The training harness now uses Adam and a 32×32 input with a smaller network (1024→512→256).
• Harness fixes from 717a87a: resumed runs no longer overwrite the best checkpoint too early, early stopping counts correctly after a learning-rate change, and loading training samples is faster (O(N log N) instead of O(N²) per epoch).
• The README is rewritten with usage samples and a parameter reference.

Upgrading

1. Rename the parameters listed under Breaking changes.
2. If you use Adam, lower the learning rate (0.0001–0.001 is typical) and raise weight decay to around 0.01.
3. If you access Synapse.Weights directly, switch to GetWeight / SetWeight.

academia ai machine-learning neural-network