Job instance 271211

Stderr output

<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
14:24:08 (21160): wrapper (7.7.26016): starting
14:24:08 (21160): wrapper: running ../../projects/axiom.heliex.net/axiom_worker_2.17_windows_x86_64.exe (wu.json result.json)
[CPU] Using NumPy
[Worker] Saved 5 gradient files to contribute folder (gradient sharing)
INFO: Downloading model: https://axiom.heliex.net/pyhelix/model/expert_18_v2.0.bin
INFO: Downloaded and cached model: expert_18_v2.0.bin (42580994 params)
[Worker] Running 30 batches for expert 18...
[Worker] FATAL ERROR: MemoryError: Unable to allocate 2.25 MiB for an array with shape (768, 768) and data type float32
Traceback (most recent call last):
  File "worker_boinc_app.py", line 1699, in main
  File "worker_boinc_app.py", line 1322, in step
  File "simple_ml.py", line 1176, in backward
    return self.expert.backward_softmax_cross_entropy(x, y_idx, logits)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "simple_ml.py", line 637, in backward_softmax_cross_entropy
    grads = self._zero_grads()
            ^^^^^^^^^^^^^^^^^^
  File "simple_ml.py", line 675, in _zero_grads
    lg['d_' + key] = np.zeros_like(param)
                     ^^^^^^^^^^^^^^^^^^^^
  File "numpy\_core\numeric.py", line 128, in zeros_like
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 2.25 MiB for an array with shape (768, 768) and data type float32

14:24:51 (21160): axiom_worker_2.17_windows_x86_64.exe exited; CPU time 44.421875
14:24:51 (21160): called boinc_finish(0)

</stderr_txt>
]]>