Fatal Python error: Illegal instruction

[VincentRouvreau]

Select Topic Area

Bug

Body

I have 2 distincts projects https://github.com/GUDHI/gudhi-deploy and https://github.com/GUDHI/gudhi-devel

gudhi-deploy is the one that I use to build the docker images (with GitHub actions) that are required by gudhi-devel in its CI (also with GitHub actions - for the buggy part).

I was quite happy with it, but now I have a Fatal Python error: Illegal instruction that I cannot explain because it means something like the docker images build is done with GitHub actions on some hardware that is different from the one that is used for the gudhi-devel CI, that is also done on GitHub actions...

The docker image ghcr.io/gudhi/gudhi-deploy/ci_for_gudhi:2026.03.02 was built from a GitHub actions.

Then when I use it in gudhi-devel CI (cf. job/67447355806):

../../../src/python/test/test_knn.py .Fatal Python error: Illegal instruction

Current thread 0x00007f1496023080 (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1289 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 921 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/__w/gudhi-devel/gudhi-devel/build/src/python/gudhi/point_cloud/knn.py", line 140 in fit
  File "/__w/gudhi-devel/gudhi-devel/src/python/test/test_knn.py", line 125 in test_knn_compare
  File "/opt/venv/lib/python3.12/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/venv/lib/python3.12/site-packages/_pytest/python.py", line 1720 in runtest
  File "/opt/venv/lib/python3.12/site-packages/_pytest/runner.py", line 179 in pytest_runtest_call
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/venv/lib/python3.12/site-packages/_pytest/runner.py", line 245 in <lambda>
  File "/opt/venv/lib/python3.12/site-packages/_pytest/runner.py", line 353 in from_call
  File "/opt/venv/lib/python3.12/site-packages/_pytest/runner.py", line 244 in call_and_report
  File "/opt/venv/lib/python3.12/site-packages/_pytest/runner.py", line 137 in runtestprotocol
  File "/opt/venv/lib/python3.12/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/venv/lib/python3.12/site-packages/_pytest/main.py", line 396 in pytest_runtestloop
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/venv/lib/python3.12/site-packages/_pytest/main.py", line 372 in _main
  File "/opt/venv/lib/python3.12/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/opt/venv/lib/python3.12/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 199 in main
  File "/opt/venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 223 in console_main
  File "/opt/venv/lib/python3.12/site-packages/pytest/__main__.py", line 9 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, charset_normalizer.md, charset_normalizer.cd, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._batched_linalg, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation_cy, scipy.spatial.transform._rigid_transform_cy, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, sklearn.__check_build._check_build, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpacklib, scipy.sparse.linalg._propack, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._rcont.rcont, scipy.stats._qmvnt_cy, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, pandas._libs._cyutility, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.properties, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, sklearn._cyutility, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.neighbors._partition_nodes, sklearn.neighbors._ball_tree, sklearn.neighbors._kd_tree, sklearn.utils.arrayfuncs, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.linear_model._sag_fast, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast (total: 186)

So I created a personnal access token and rebuild the docker image on a dedicated virtual machine, and push it as ghcr.io/gudhi/gudhi-deploy/ci_for_gudhi:2026.03.01, and now it works fine, cf.job/67540391573...

I thought that GitHub actions was preventing to mix hardware setup and was ensuring the reproducibility of builds, but it seems that this is not the case.

For me it is clearly a bug, but if you have a workaround, I would take it. Building on a virtual machine is so annoying...

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[mecodeatlas]

Thanks for posting in the GitHub Community, @VincentRouvreau!

We're happy you're here. You are more likely to get a useful response if you post your question in the most relevant category. This question is less related to Packages and more about a Python/runtime issue across environments.

This question should be in the Programming Help category. I've gone ahead and moved it for you. Good luck! 🙌

[ghazy001]

This looks very much like a CPU instruction mismatch rather than a GitHub Actions bug.

The "Illegal instruction" error typically occurs when native code is built with CPU-specific optimizations (e.g. AVX/AVX2), and then executed on a machine that does not support those instructions.

In your case:

• The Docker image is built on one GitHub Actions runner
• Then used on another runner (potentially with different CPU features)

Since GitHub-hosted runners do not guarantee identical CPU models, this can lead to non-reproducible builds when native extensions are compiled.

The fact that rebuilding on your VM fixes the issue strongly supports this hypothesis.

Things to check

• Whether any build uses flags like -march=native or similar
• Whether some dependencies are compiled from source (numpy/scipy/torch/sklearn)
• Compare CPU flags between environments (lscpu)

Possible workarounds

• Force a generic target (e.g. -march=x86-64)
• Prefer prebuilt wheels (pip install --only-binary=:all:)
• Ensure build and test run on the same runner type

So this may be more a limitation of heterogeneous runners than a strict bug, but I’d be interested if others can confirm.