<core_client_version>8.0.4</core_client_version> <![CDATA[ <stderr_txt> 2026-03-20 06:36:46 (21876): Streaming wrapper: running ../../projects/axiom.heliex.net/axiom_experiment_639_gpu.exe (wu.json result.json) 2026-03-20 06:36:46 (21876): Streaming wrapper: created child process 20832 [GPU] CuPy CUDA: 16375MB total, 15048MB free [BOINC] Starting BOINC mode [BOINC] init_data.xml gpu_device_num=0 [BOINC] Set CUDA device to GPU 0 [BOINC] GPU detected: 16375MB VRAM, 15048MB free [BOINC] Work unit: wu.json [BOINC] Result file: result.json [BOINC] Current directory: C:\ProgramData\BOINC\slots\0 [BOINC] Directory contents: ['.cupy_cache', 'axiom_experiment_639_gpu.exe', 'boinc_lockfile', 'init_data.xml', 'job.xml', 'result.json', 'stderr.txt', 'streaming_wrapper_259_windows_x86_64.exe', 'streaming_wrapper_checkpoint.txt', 'wu.json', '_MEI208322'] [BOINC] Resolved soft link: ../../projects/axiom.heliex.net/exp_exp_neurovascular_endfoot_pericyte_bridge_threshold_gpu_8aefc7.json [BOINC] Resolved soft link: ../../projects/axiom.heliex.net/exp_exp_neurovascular_endfoot_pericyte_bridge_threshold_gpu_8aefc7_0_r1512997413_0 [BOINC] Actual WU file: ../../projects/axiom.heliex.net/exp_exp_neurovascular_endfoot_pericyte_bridge_threshold_gpu_8aefc7.json [BOINC] Successfully loaded work unit [BOINC] Experiment container mode detected [EXP] Experiment mode: exp_exp_neurovascular_endfoot_pericyte_bridge_threshold_gpu_8aefc7 [EXP] Script URL: https://axiom.heliex.net/experiments/exp_neurovascular_endfoot_pericyte_bridge_threshold_gpu.py [EXP] Run duration: 1800s [EXP] Downloading script... [EXP] Downloaded 24331 bytes [EXP] Executing experiment 'exp_exp_neurovascular_endfoot_pericyte_bridge_threshold_gpu_8aefc7'... [EXP] GPU available: NVIDIA GeForce RTX 4080 (16375MB) [GPU] Running on NVIDIA GeForce RTX 4080 with 16375MB VRAM [GPU] Measured 1.0 TFLOPS Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Traceback (most recent call last): File "experiment_script.py", line 431, in sample_metrics eigvals, eigvecs = xp.linalg.eigh(operator) ^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\linalg\_eigenvalue.py", line 141, in eigh File "cupy\linalg\_eigenvalue.py", line 53, in _syevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3475, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 3489, in cupy_backends.cuda.libs.cusolver.xsyevd File "cupy_backends\\cuda\\libs\\cusolver.pyx", line 1079, in cupy_backends.cuda.libs.cusolver.check_status cupy_backends.cuda.libs.cusolver.CUSOLVERError: CUSOLVER_STATUS_EXECUTION_FAILED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__' Traceback (most recent call last): File "cupy_backends\\cuda\\api\\runtime.pyx", line 600, in cupy_backends.cuda.api.runtime.free File "cupy_backends\\cuda\\api\\runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered [EXP] Execution error: CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered Traceback (most recent call last): File "axiom_streaming_client.py", line 2754, in _run_experiment_mode File "experiment_script.py", line 616, in <module> main() File "experiment_script.py", line 578, in main stats = sample_metrics(current_n, mid, DISORDER_SAMPLES, sample_seed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "experiment_script.py", line 427, in sample_metrics operator, left_pericyte, left_endfoot, perivascular_core, right_endfoot, right_pericyte, bridge_mask, outer_tissue = build_operator( ^^^^^^^^^^^^^^^ File "experiment_script.py", line 272, in build_operator coords = xp.linspace(-1.0, 1.0, size_n, dtype=xp.float64) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\_creation\ranges.py", line 161, in linspace return _linspace_scalar(start, stop, num, endpoint, retstep, dtype) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy\_creation\ranges.py", line 105, in _linspace_scalar _linspace_ufunc(start, step, ret) File "cupy\\_core\\_kernel.pyx", line 1376, in cupy._core._kernel.ufunc.__call__ File "cupy\\cuda\\function.pyx", line 237, in cupy.cuda.function.Function.linear_launch File "cupy\\cuda\\function.pyx", line 205, in cupy.cuda.function._launch File "cupy_backends\\cuda\\api\\driver.pyx", line 273, in cupy_backends.cuda.api.driver.launchKernel File "cupy_backends\\cuda\\api\\driver.pyx", line 63, in cupy_backends.cuda.api.driver.check_status cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered [TELEMETRY] Error reported [EXP] Read experiment_result.json successfully [EXP] Experiment 'exp_exp_neurovascular_endfoot_pericyte_bridge_threshold_gpu_8aefc7' finished in 494.6s [EXP] Status: error 2026-03-20 06:45:24 (21876): axiom_experiment_639_gpu.exe exited; CPU time 407.843750 2026-03-20 06:45:24 (21876): called boinc_finish(0) </stderr_txt> ]]>