Extending current hook

[AdrianSosic]

Hi @Tinche,

Probably I just can't see the wood for the trees, but I'm somehow stuck getting this simple scenario to work. Can you give a quick hint what is:

• generally the "right" way to extend an existing hook? That is, what is the best way to define some hook logic that internally uses the old logic without running into recursion errors? I mean, I know I can use things like make_dict_unstructure_fn but quick question on this one: does it use already registered hooks? I guess the answer is "yes", because we explicitly pass a converter object, right? So it would use the hooks of that converter, whatever they are?
• Assuming the above is true, I still don't know how to get it running for non-attrs classes, and specifically, protocols, for which I can't even do an isinstance check that would enable me to use hook factories. Below the example:

from typing import Protocol

import cattrs
from attrs import define

converter = cattrs.Converter()


class SomeGetter(Protocol):
    """A index-based mapping from strings to surrogates."""

    def __getitem__(self, key: str) -> int: ...


@define
class Container:
    getter: SomeGetter


# None of the following works, expectedly, because of endless recursion loops / non-attrs

@converter.register_unstructure_hook
def unstructure_getter(obj: SomeGetter) -> dict:
    return {"type": type(obj), **converter.unstructure(obj)}


@converter.register_unstructure_hook
def unstructure_getter(obj: SomeGetter) -> dict:
    fn = converter.get_unstructure_hook(SomeGetter)
    return {"type": type(obj), **fn(obj)}


@converter.register_unstructure_hook
def unstructure_getter(obj: SomeGetter) -> dict:
    from cattrs.gen import make_dict_unstructure_fn

    fn = make_dict_unstructure_fn(SomeGetter, converter)
    return {"type": type(obj), **fn(obj)}


container = Container({"a": 1})
dct = converter.unstructure(container)
print(dct)

[Tinche]

Hm, I tried:

from typing import Protocol

from attrs import define

import cattrs

converter = cattrs.Converter()


class SomeGetter(Protocol):
    """A index-based mapping from strings to surrogates."""

    def __getitem__(self, key: str) -> int: ...


@define
class Container:
    getter: SomeGetter


@converter.register_unstructure_hook
def unstructure_getter(obj: SomeGetter) -> dict:
    return {"type": type(obj), **converter.unstructure(obj)}


container = Container({"a": 1})
dct = converter.unstructure(container)
print(dct)

And it works for me:

{'getter': {'type': <class 'dict'>, 'a': 1}}

This on Python 3.11.

To answer some of your questions:

generally the "right" way to extend an existing hook?

Cattrs can't compose hooks for the same type (yet), you have to do that yourself. The usual way to do this is to get the base hook, wrap it with your hook, and register your hook.

I mean, I know I can use things like make_dict_unstructure_fn but quick question on this one: does it use already registered hooks

Yes, because of how make_dict_unstructure_fn is implemented. At hook creation time it will reach into the Converter and get hooks for all the fields, and store them for later use.

[AdrianSosic]

Hi @Tinche, thanks for the quick answer. I'll go through it later in detail (typing from my phone atm) but just want to briefly say that I forgot to add: it also works for me on Python 3.10 but I stumbled over it because it doesn't on 3.12. Perhaps you can confirm? Will get back later with more info

[Tinche]

Ok, looks like something changed in functools.singledispatch between versions.

Small reproducer:

from functools import singledispatch
from typing import Protocol

class SomeGetter(Protocol):
    """A index-based mapping from strings to surrogates."""

    def __getitem__(self, key: str) -> int: ...

@singledispatch
def test(val):
    raise Exception()


test.register(SomeGetter, lambda v: print(v))

test({"a": 1})

The hook registered for SomeGetter fires for dicts. It prints out the dict instead of raising.

I don't know why it works this way; it's probably a bug (especially since your protocol isn't even @runtime_checkable). You might want to report it to CPython if you feel like it.

Anyway, any time you find an oddity in singledispatch, just don't use it. Use a predicate hook instead.

from typing import Protocol

from attrs import define

import cattrs

converter = cattrs.Converter()


class SomeGetter(Protocol):
    """A index-based mapping from strings to surrogates."""

    def __getitem__(self, key: str) -> int: ...


@define
class Container:
    getter: SomeGetter


def unstructure_getter(obj: SomeGetter) -> dict:
    return {"type": type(obj), **converter.unstructure(obj)}


converter.register_unstructure_hook_func(lambda t: t is SomeGetter, unstructure_getter)

container = Container({"a": 1})
dct = converter.unstructure(container)
print(dct)

There is some documentation about it here: https://catt.rs/en/latest/customizing.html#custom-un-structuring-hooks. I plan on removing the singledispatch code path one day, since predicate hooks are much more predictable, and having two mechanisms for registering hooks can be confusing. (The API will remain the same, but register_unstructure_hook will just register an unstructure hook with an issubclass predicate.)

[AdrianSosic]

Hey @Tinche, as always: thanks a lot for your support, much appreciated. Now that you point it out, I think it might actually be related to #525, over which the two of us stumbled already a year ago 😅 So I guess avoiding singledispatch is a good idea then 👍🏼

Just one last question, for me to finally have this signed by the guru:
To extend an existing hook (i.e. reuse its current logic within the new hook), the intended way is to write a function that internally:

• either manually extracts existing hooks via get_(un)structure_hook and wraps them somehow
• or wraps utilities like make_dict_structure_fn that internally access these hooks

With that I should normally not run into recursion errors, right?

[Tinche]

Correct, you got it!

I wonder if we can implement something in cattrs one day to make this wrapping easier. For example if your hook factory has an argument typed as PreviousHook (which could be a type alias to Callable or something), cattrs could generate the hook while looking only at older factories, and just give you the hook, letting you wrap it.

[AdrianSosic]

Sounds like a neat idea!

[gwerbin-tive]

Sorry if this isn't quite the same situation as OP, but the problem does seem the same:

import dataclasses
import cattrs

@dataclasses.dataclass
class v00LineBufferBase:
    pass

@dataclasses.dataclass
class v00LineBufferMeanStddev(v00LineBufferBase):
    # Gaussian distribution: 95% of data is <= 2 std. dev from mean
    stddev_multiplier: float = 2.0

@dataclasses.dataclass
class v00LineBufferPercentileIQR(v00LineBufferBase):
    # Tukey boxplot standard default definition for a "flier"
    percentile_base: int = 75
    iqr_multiplier: float = 1.5


# Attempt 1: Intuitive guess
converter = cattrs.Converter()
converter.register_unstructure_hook(
    v00LineBufferBase,
    lambda o: {"_type": type(o).__name__,  **converter.unstructure(o)}
)
# RecursionError
converter.unstructure(v00LineBufferMeanStddev())
converter.unstructure(v00LineBufferMeanStddev(), unstructure_as=v00LineBufferBase)

# Attempt 2: Following advice above
converter = cattrs.Converter()
converter.register_unstructure_hook_func(
    lambda t: issubclass(t, v00LineBufferBase),
    lambda o: {"_type": type(o).__name__,  **converter.unstructure(o)}
)
# RecursionError
converter.unstructure(v00LineBufferMeanStddev())
converter.unstructure(v00LineBufferMeanStddev(), unstructure_as=v00LineBufferBase)

Both the type-based unstructure hook and the explicit predicate-based hook factory result in the RecursionError. This is using cattrs == 24.1.3 and Python 3.12.6.

I know I can use the Union technique described in the docs, but the base class is a lot more convenient. I also want to be able to unstructure these things nested inside a dict, and the unstructure hook doesn't seem to be applied here, even using the Union: converter.unstructure({"algorithm": v00LineBufferMeanStddev(), "another_thing": SomethingElseUnstructureable()}).