Correction of Baserecordingsnppets._set_probes

[OlivierPeron]

Hello,

I noticed an issue in the _set_probes function.
To get the number of contacts of the probe-s, this operation is performed :

number_of_device_channel_indices = np.max(list(device_channel_indices) + [0])

This only works if the contacts range from 0 to max_nb_of_contacts, which is not necessarly the case. I suggest to do the following:

number_of_device_channel_indice = sum(len(sublist) if isinstance(sublist, np.ndarray) else 1 for sublist in device_channel_indices)

Unfortunatly I won't have time to do a PR, but I hope this will be usefull !

[dpeg22]

To add onto this: if any channels are missing from the recording, this line is not the only issue. My understanding of device_channel_indices are that they are the map from probe channels to recording channels. I'm unsure how using them as a true "index" into the list of recording object channels would work? If your probe has N channels and thus N device_channel_indices, and if your recording has N - n_bad_chans then you end up with a mismatch where your device_channel_indices can take up to value N-1, and your recording may have a channel "N", but not enough existing channels for using N-1 as an index to be functional.
This causes issues here

I'll work on a PR, but would appreciate any insight in case I misunderstand the current construction.

[alejoe91]

Hi @dpeg22

device_channel_indices are indices. By construction, the probe object should have less or equal the number of channels in the recording. If the probe has fewer entries than the recording channels, then the set_probes will slice the recording and only return the selected device_channel_indices.

[dpeg22]

Hi @alejoe91 thanks for clarifying!
Do I now understand correctly that the device_channel_indices should then be a mapping between the order of the channel list in the recording object to the probe channel ID? My previous thinking was it should be a mapping from the probe channel ID to the recording system's channel ID, not the recording object's channel list, in which case if you're missing a recording from a channel then you could still have a device_channel_indices value of N-1 if your config is consistent regardless of bad channels on probes.
I was wondering if you could help me understand just a bit more about how this works as I don't see much documentation about how to handle the situation if the recording is missing channels and thus has fewer entries than the physical probe electrode sites?

[alejoe91]

It should be a mapping from a probe contact to the index of the trace. For example, if the device channel index for contact 3 is 31, it means that the signal from contact 3 will be the 32nd trace - 0-based (so it's a channel index, the channel ID is the "name" of the channel).

You can check out a tutorial video here:
https://youtu.be/pHze_8s4Qak?si=8a_NudyPXokky-DM
(The probe mapping is covered around min. 17)

[zm711]

Hello,

I noticed an issue in the _set_probes function.
To get the number of contacts of the probe-s, this operation is performed :

number_of_device_channel_indices = np.max(list(device_channel_indices) + [0])
This only works if the contacts range from 0 to max_nb_of_contacts, which is not necessarly the case. I suggest to do the following:

number_of_device_channel_indice = sum(len(sublist) if isinstance(sublist, np.ndarray) else 1 for sublist in device_channel_indices)
Unfortunatly I won't have time to do a PR, but I hope this will be usefull !

I don't think taking the len of a sublist works. At this stage we've already sliced out channels marked by disconnected:

spikeinterface/src/spikeinterface/core/baserecordingsnippets.py

Lines 161 to 164 in dd0c6d3

	# keep only connected contact ( != -1)
	keep = probe_as_numpy_array["device_channel_indices"] >= 0
	if np.any(~keep):
	warn("The given probes have unconnected contacts: they are removed")

This is why we don't deal with the length of the list. As Alessio said in this case we take the max device index that the user has listed because that needs to correspond to the max possible trace (-1 for the 0-indexing of python of course). So even in the case of multi-shank where the n_contacts are like 64/shank the device index would still have to go up by 64 * 4 -1 because in the unified matrix of your data one of the traces will be trace 64 * 4. Does that make sense @OlivierPeron ? Or could you explain more what your concern is. I think Alessio explained it well enough, but just wanted to add my two sense because I think users often don't realize that we have a mechanism for disconnects which makes taking the len unreliable for this type of indexing.

[OlivierPeron]

Hi !

Let say I have a tetrode, that for some reason has channels such has [128, 129, 130, 131].
In this case taking the max will not work. This is why I first created this issue.
Maybe I did not properly understand how to use this function, but has it is I cannot use it.

[alejoe91]

With channels you mean device_channel_indices? It will work, but your recording must have >= 132 channels. Otherwise your device channel indices [128, 129, 130, 131] are wrong.

If your recording has say 32 channels, then device channel indices must be max 31. That's how it works :)

[OlivierPeron]

Sure, my recording has the same number of channels as the max. But in the case where I channel slice the recording this is no longer true no ? Or should I adjust the device_channel_indices in this case ?
Exemple:
recording with 8 channels (2 tetrodes). If I split by tetrode, I will have 2 recording with 4 channels, for the first one the function will work without issue, but for the second one, the number of channel in the probe and the recording are the same but using max, if my channels are [4, 5, 6, 7], won't work.

[alejoe91]

Are you getting an error? In that case, can you paste the Traceback?

[OlivierPeron]

So I have a recording using 3 probes. First I channel slice the recording per probe, then I am trying to set the corresponding probe on the channel sliced recording.

Here is the traceback I got:

File "C:\Users\giradmin\.conda\envs\spykeline\lib\site-packages\spykeline\preprocessing\preprocess.py", line 137, in run_preprocessing
    recording_sliced = recording_sliced.set_probe(rec_probe)
  File "C:\Users\giradmin\.conda\envs\spykeline\lib\site-packages\spikeinterface\core\baserecordingsnippets.py", line 101, in set_probe
    return self._set_probes(probegroup, group_mode=group_mode, in_place=in_place)
  File "C:\Users\giradmin\.conda\envs\spykeline\lib\site-packages\spikeinterface\core\baserecordingsnippets.py", line 181, in _set_probes
    raise ValueError(error_msg
ValueError: The given Probe have 'device_channel_indices' that do not match channel count
127 vs 64
device_channel_indices are the following: [ 64  65  66  67  68  69  70  71  72  73  74  75  76  77  78  79  80  81
  82  83  84  85  86  87  88  89  90  91  92  93  94  95  96  97  98  99
 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117
 118 119 120 121 122 123 124 125 126 127]
recording channels are the following: [105 110 104 111 107 108 106 109  80  89  81  88  82  86  84  85  90  83
  92  87  93  91  94  95  77  68  73  66  69  67  65  64  71  78  70  79
  72  76  75  74 112 119 113 118 114 117 115 116 120 127 121 126 122 125
 123 124  97 102  96 103  99 100  98 101]

As you can notice, the channel ids are the same (one is sorted and the other isn't) and the number of channel as well.
To me these 2 conditions should be the one to check. And testing for the max doesn't make any sense, at least in my case.

I then replaced :

number_of_device_channel_indices = np.max(list(device_channel_indices) + [0])

by:

number_of_device_channel_indices = sum(len(sublist) if isinstance(sublist, np.ndarray) else 1 for sublist in device_channel_indices)

as I originally proposed.
Doing so didn't trigger the error, as expected, but instead raised another error:

File "C:\Users\giradmin\.conda\envs\spykeline\lib\site-packages\spykeline\preprocessing\preprocess.py", line 134, in run_preprocessing
    recording_sliced = recording_sliced.set_probe(rec_probe)
  File "C:\Users\giradmin\.conda\envs\spykeline\lib\site-packages\spikeinterface\core\baserecordingsnippets.py", line 101, in set_probe
    return self._set_probes(probegroup, group_mode=group_mode, in_place=in_place)
  File "C:\Users\giradmin\.conda\envs\spykeline\lib\site-packages\spikeinterface\core\baserecordingsnippets.py", line 182, in _set_probes
    new_channel_ids = self.get_channel_ids()[device_channel_indices]
IndexError: index 64 is out of bounds for axis 0 with size 64

So from there I concluded that the whole function is only suited for probes with channel ranging from 0 to nChannels. Which at least in my case is almost never the case.

[zm711]

What I would recommend is just set all three probes to the recording and then do a channel_slice or a grouping_property instead. The channel_slice at that point would handle the probe stuff automatically. Or what a lot of users do for multiprobe/shank stuff is use a grouping_property. Is there a reason why you don't want to do that work flow instead?

[OlivierPeron]

Yes, the reason is that I wanted to perform a whitening per probe. And your whitening function doesn't have the group option. So to cope with it I thought of doing this channel slice.
I was also planning to use the function run_sorter_jobs() and from what I understand I would need to slice the recording there anyway.

[alejoe91]

Channel slice is ok, but @zm711 is suggesting to just set a probe group initially and then split by probe and apply preprocessing. That should't result in any error because you set the probe group globally at the beginning.
Alternatively, you would need to modify te probe device channel indices for each channel slice..