Advanced serialization for Pydantic models via Kedro and fsspec.
This package implements custom Kedro DataSet types for not only "pure" (JSON-serializable)
Pydantic models, but also models with
Keep reading for a basic tutorial, or check out the API Reference for auto-generated docs.
To simplify the documentation, we will refer to JSON-serializable Pydantic models as "pure" models, while all others will be "arbitrary" models.
We also assume you are familiar with Kedro's Data Catalog and Datasets.
Usage with Kedro¶
You can use the PydanticAutoDataSet
or any other dataset from
pydantic-kedro within your
to save your Pydantic models:
1 2 3 4
Then use it as usual within your Kedro pipelines:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
If you are using Kedro for the pipelines or data catalog, that should be enough.
If you want to use these datasets stand-alone, keep on reading.
The functions save_model and load_model can be used directly. See the relevant docs for more info.
"Pure" Pydantic Models¶
If you have a JSON-safe Pydantic model, you can use a
to save your model to any
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Note: YAML support is enabled by
Note that specifying custom JSON encoders will work as usual, even for YAML models.
However, if your custom type is difficult or impossible to encode/decode via JSON, read on to Arbitrary Types.
Automatic Saving of Pydantic Models¶
The easiest way to use
v0.2.0) is through the
You can use it in the place of any other dataset for reading or writing.
When reading, it will figure out what the actual dataset type is. When writing, it will try to save it as a pure model, or fallback to an arbitrary model, depending on the options set. Below you can see the default options: