Job instance 271925

Stderr output

<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
14:30:34 (4024): wrapper (7.7.26016): starting
14:30:34 (4024): wrapper: running ../../projects/axiom.heliex.net/axiom_worker_2.17_windows_x86_64.exe (wu.json result.json)
[CPU] Using NumPy
[Worker] Saved 5 gradient files to contribute folder (gradient sharing)
INFO: Loaded model from cache: expert_245_v2.0.bin (42580994 params)
INFO: Found 5 local files to train on.
INFO: Converted 2621760 bits from local files.
[Worker] Running 30 batches for expert 245...
[Worker] FATAL ERROR: MemoryError: Unable to allocate 12.0 MiB for an array with shape (64, 64, 768) and data type float32
Traceback (most recent call last):
  File "worker_boinc_app.py", line 1699, in main
  File "worker_boinc_app.py", line 1322, in step
  File "simple_ml.py", line 1176, in backward
    return self.expert.backward_softmax_cross_entropy(x, y_idx, logits)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "simple_ml.py", line 649, in backward_softmax_cross_entropy
    self._backward_chunk(x, y_idx, logits, sc['ctx'], grads, B_total)
  File "simple_ml.py", line 795, in _backward_chunk
    d_h_pre_ln1, d_lg1, d_lb1 = _layernorm_backward(
                                ^^^^^^^^^^^^^^^^^^^^
  File "simple_ml.py", line 337, in _layernorm_backward
    - x_hat * np.mean(dx_hat * x_hat, axis=-1, keepdims=True))
                      ~~~~~~~^~~~~~~
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 12.0 MiB for an array with shape (64, 64, 768) and data type float32

14:33:20 (4024): axiom_worker_2.17_windows_x86_64.exe exited; CPU time 192.468750
14:33:20 (4024): called boinc_finish(0)

</stderr_txt>
]]>