Clarification on which `cgan_2026` benchmark version is used

[xiangruzh]

Hi,

I noticed a possible version mismatch for the cgan2026 benchmark. The current instance set is said to be generated from commit 793dbaab3ab4fbbf526bcb3acb3ca20a6bb7b176 (e3dcc19) of the benchmark repository https://github.com/Eli-Pregerson/cgan_benchmark2026.

However, the latest commit in that repository seems to be ea4974d02c98bdac1da08ddc6a650fca72c89d3d (Eli-Pregerson/cgan_benchmark2026@ea4974d).

Could you clarify which version of the benchmark is intended to be used for the competition?

Thanks!

[stanleybak]

@xiangruzh you are correct that it should have been the latest commit. Not sure if it's too late now to fix. If so, we can just update it next year

[toladnertum]

Generally, I agree to use the latest version; @emanino seems to have asked @Eli-Pregerson in the other issue about the latest commit, but there was no response..

[stanleybak]

@toladnertum Eli's my student... he's traveling at the moment. latest version is ok.

[toladnertum]

@toladnertum Eli's my student... he's traveling at the moment. latest version is ok.

Sorry, I didn't realize.

I reran the submission, but it seems the benchmark is missing the onnx in requirements.txt

  File "generate_properties.py", line 7, in <module>
  import onnx
  ModuleNotFoundError: No module named 'onnx'
``

[stanleybak]

Apologies @toladnertum , try this one: https://github.com/stanleybak/cgan_benchmark2026 with commit 14ac1ef

[toladnertum]

@stanleybak the benchmark still has an error: ERROR: Could not find a version that satisfies the requirement onnx==1.22.0 (from versions: 0.1, 0.2, 0.2.1, 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.1.2, 1.2.1, 1.2.2, 1.2.3, 1.3.0, 1.4.0, 1.4.1, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.13.0, 1.13.1, 1.14.0, 1.14.1, 1.15.0, 1.16.0, 1.16.1, 1.16.2, 1.17.0)

[stanleybak]

Sorry again @toladnertum , I think we might be on on different python versions. I committed 8d2cecd now which removes the version requirements and tested on python 3.8 and 3.12 to be identical.

[toladnertum]

@stanleybak np, updated it. Please double check here: https://github.com/VNN-COMP/vnncomp2026_benchmarks/tree/main/benchmarks/cgan2026