ScanCode contains too many data files

[pombredanne]

The src/licensedcode/data directory contains 68K+ files and 64k just for the rules.
These rule files are not used much at runtime because they are baked into the index in a compressed form that is used at runtime. The same applies to the licenses files that are fully included in the index in an object form.

These are only needed when the index is rebuilt.
Another issue is that handling so many files makes any filesystem operation (unbearably) slow including during development time and at installation time.

It also creates side issues as #2427 (comment) and linkedin/shiv#224

I suggest some of these to fix the issue:

• we can half the number of files by combining the YAML data file and the license/rule text file in a single file with minimal code changes either a as combined YAML or YAML front matter https://jekyllrb.com/docs/front-matter/
• we can split the files in multiple sub-directories to limit the number of files to some sensible number (say under 5K per dir)
• at runtime, and as part of the build we could replace the many files by a single (larger) file that could be a big JSON or YAML file or a zip with the actual files accessed as a filesystem https://github.com/PyFilesystem/pyfilesystem2/blob/master/fs/zipfs.py or path https://github.com/jaraco/zipp

Combining either these three actions or just the last two should make this OK and workable both for development, installation and runtime.

[mjherzog]

The first action seems to be the simplest solution.

[AyanSinhaMahapatra]

@pombredanne :

YAML or YAML front matter https://jekyllrb.com/docs/front-matter/

I think YAML front-matter would be much better. There are a lot of issues in YAML when it comes to text with whitespace/symbols as we were discussing, and plain YAMLs are causing a lot of YAML read errors.

[AyanSinhaMahapatra]

we can split the files in multiple sub-directories to limit the number of files to some sensible number (say under 5K per dir)

are we going ahead with this?

Then the options are:

• Generate some kind of content based hash (no need for nested dirs, just one level of directories, so maybe just one character of the hash is required)
• give the major license keys a folder of their own (the various gpls, apache, lgpl and so on)
• make directories based on characters from the license-key (if there's a lot of them under one character, we can use option 2. or the later characters)

[pombredanne]

IMHO let's wait a bit for this

[milahu]

licensedcode/data/rules could be compressed from 150 to 3 MB

$ du -sh scancode-toolkit-32.1.0/lib/python3.11/site-packages/licensedcode/data/rules/
149M    scancode-toolkit-32.1.0/lib/python3.11/site-packages/licensedcode/data/rules/

$ ls -U scancode-toolkit-32.1.0/lib/python3.11/site-packages/licensedcode/data/rules/ | wc -l
34859

$ tar cf scancode-toolkit-rules.tar  scancode-toolkit-32.1.0/lib/python3.11/site-packages/licensedcode/data/rules/
$ zstd -19 scancode-toolkit-rules.tar -o scancode-toolkit-rules.tar.19.zst

$ du -sh scancode-toolkit-rules.tar*
64M     scancode-toolkit-rules.tar
3.0M    scancode-toolkit-rules.tar.19.zst

[pombredanne]

@milahu Thanks!
See also #3761 where we are building smaller wheels .... this may help too