WIP - Pydantic v2 support

[collerek]

Switching to Pydantic v2.X.X

Migration to 0.20.0 based on pydantic 2.X.X

Version 0.20.0 provides support for pydantic v2.X.X that provides significant speed boost (validation and serialization is written in rust) and cleaner api for developers,
at the same time it drops support for pydantic v.1.X.X. There are changes in ormar interface corresponding to changes made in pydantic.

Breaking changes

Migration to version >= 0.20.0 requires several changes in order to work properly.

ormar Model configuration

Instead of defining a Meta class now each of the ormar models require an ormar_config parameter that is an instance of the OrmarConfig class.
Note that the attribute must be named ormar_config and be an instance of the config class.

import databases
import ormar
import sqlalchemy

database = databases.Database("sqlite:///db.sqlite")
metadata = sqlalchemy.MetaData()

# ormar < 0.20
class Album(ormar.Model):
    class Meta:
        database = database
        metadata = metadata
        tablename = "albums"
    

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)
    favorite: bool = ormar.Boolean(default=False)

# ormar >= 0.20
class AlbumV20(ormar.Model):
    ormar_config = ormar.OrmarConfig(
        database=database,
        metadata=metadata,
        tablename="albums_v20"
    )

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)
    favorite: bool = ormar.Boolean(default=False)

OrmarConfig api/ parameters

The ormar_config expose the same set of settings as Meta class used to provide.
That means that you can use any of the following parameters initializing the config:

metadata: Optional[sqlalchemy.MetaData]
database: Optional[databases.Database]
engine: Optional[sqlalchemy.engine.Engine]
tablename: Optional[str]
order_by: Optional[List[str]]
abstract: bool
exclude_parent_fields: Optional[List[str]]
queryset_class: Type[QuerySet]
extra: Extra
constraints: Optional[List[ColumnCollectionConstraint]]

BaseMeta equivalent - best practice

Note that to reduce the duplication of code and ease of development it's still recommended to create a base config and provide each of the models with a copy.
OrmarConfig provides a convenient copy method for that purpose.

The copy method accepts the same parameters as OrmarConfig init, so you can overwrite if needed, but by default it will return already existing attributes, except for: tablename, order_by and constraints which by default are cleared.

import databases
import ormar
import sqlalchemy

base_ormar_config = ormar.OrmarConfig(
    database=databases.Database("sqlite:///db.sqlite"),
    metadata=sqlalchemy.MetaData()
)

class AlbumV20(ormar.Model):
    ormar_config = base_ormar_config.copy(
        tablename="albums_v20"
    )

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)

    
class TrackV20(ormar.Model):
    ormar_config = base_ormar_config.copy(
        tablename="tracks_v20"
    )

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)

choices Field parameter is no longer supported.

Before version 0.20 you could provide choices parameter to any existing ormar Field to limit the accepted values.
This functionality was dropped, and you should use ormar.Enum field that was designed for this purpose.
If you want to keep the database field type (i.e. an Integer field) you can always write a custom validator.

import databases
import ormar
import sqlalchemy

database = databases.Database("sqlite:///db.sqlite")
metadata = sqlalchemy.MetaData()

# ormar < 0.20
class Artist(ormar.Model):
    class Meta:
        database = database
        metadata = metadata
    

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)
    country: str = ormar.String(default=False, max_length=50, choices=["UK", "US", "Vietnam", "Colombia"])

# ormar >= 0.20
from enum import Enum

class Country(str, Enum):
    UK = "UK"
    US = "US"
    VIETNAM = "Vietnam"
    COLOMBIA = "Colombia"

class ArtistV20(ormar.Model):
    ormar_config = ormar.OrmarConfig(
        database=database,
        metadata=metadata,
        tablename="artists_v20"
    )

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)
    country: Country = ormar.Enum(enum_class=Country)

pydantic_only Field parameter is no longer supported

pydantic_only fields were already deprecated and are removed in v 0.20. Ormar allows defining pydantic fields as in ordinary pydantic model.

import databases
import ormar
import sqlalchemy

database = databases.Database("sqlite:///db.sqlite")
metadata = sqlalchemy.MetaData()

# ormar < 0.20
class Dish(ormar.Model):
    class Meta:
        database = database
        metadata = metadata
        tablename = "dishes"
    

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)
    cook: str = ormar.String(max_length=40, pydantic_only=True, default="sam")

# ormar >= 0.20
class DishV20(ormar.Model):
    ormar_config = ormar.OrmarConfig(
        database=database,
        metadata=metadata,
        tablename="dishes_v20"
    )

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)
    cook: str = "sam"  # this is normal pydantic field

property_field decorator is no longer supported

property_field decorator was used to provide a way to pass calculated fields that were included in dictionary/ serialized json representation of the model.
Version 2.X of pydantic introduced such a possibility, so you should now switch to the one native to the pydantic.

import databases
import ormar
import sqlalchemy
import pydantic

database = databases.Database("sqlite:///db.sqlite")
metadata = sqlalchemy.MetaData()

# ormar < 0.20
class Employee(ormar.Model):
    class Meta:
        database = database
        metadata = metadata
    

    id: int = ormar.Integer(primary_key=True)
    first_name: str = ormar.String(max_length=100)
    last_name: str = ormar.String(max_length=100)
    
    @ormar.property_field()
    def full_name(self) -> str:
        return f"{self.first_name} {self.last_name}"

# ormar >= 0.20
class EmployeeV20(ormar.Model):
    ormar_config = ormar.OrmarConfig(
        database=database,
        metadata=metadata,
    )

    id: int = ormar.Integer(primary_key=True)
    first_name: str = ormar.String(max_length=100)
    last_name: str = ormar.String(max_length=100)

    @pydantic.computed_field()
    def full_name(self) -> str:
        return f"{self.first_name} {self.last_name}"

Deprecated methods

All methods listed below are deprecated and will be removed in version 0.30 of ormar.

dict() becomes the model_dump()

import databases
import ormar
import sqlalchemy

database = databases.Database("sqlite:///db.sqlite")
metadata = sqlalchemy.MetaData()

class Album(ormar.Model):
    ormar_config = ormar.OrmarConfig(
        database=database,
        metadata=metadata,
        tablename="albums"
    )

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)
    favorite: bool = ormar.Boolean(default=False)

album = Album(name="Dark Side of the Moon")
    
# ormar < 0.20
album_dict = album.dict()

# ormar >= 0.20
new_album_dict = album.model_dump() 

Note that parameters remain the same i.e. include, exclude etc.

json() becomes the model_dump_json()

import databases
import ormar
import sqlalchemy

database = databases.Database("sqlite:///db.sqlite")
metadata = sqlalchemy.MetaData()

class Album(ormar.Model):
    ormar_config = ormar.OrmarConfig(
        database=database,
        metadata=metadata,
        tablename="albums"
    )

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)
    favorite: bool = ormar.Boolean(default=False)

album = Album(name="Dark Side of the Moon")
    
# ormar < 0.20
album_json= album.json()

# ormar >= 0.20
new_album_dict = album.model_dump_json() 

Note that parameters remain the same i.e. include, exclude etc.

construct() becomes the model_construct()

import databases
import ormar
import sqlalchemy

database = databases.Database("sqlite:///db.sqlite")
metadata = sqlalchemy.MetaData()

class Album(ormar.Model):
    ormar_config = ormar.OrmarConfig(
        database=database,
        metadata=metadata,
        tablename="albums"
    )

    id: int = ormar.Integer(primary_key=True)
    name: str = ormar.String(max_length=100)
    favorite: bool = ormar.Boolean(default=False)
    
params = {
    "name": "Dark Side of the Moon",
    "favorite": True,
}
# ormar < 0.20
album = Album.construct(**params)

# ormar >= 0.20
album = Album.model_construct(**params)

To read more about construct please refer to pydantic documentation.

[TouwaStar]

@collerek Is this your up-to-date wip branch? are you open to getting some help with this?

[collerek]

@TouwaStar Now it is 😅 I would not only be open but super grateful. :)

Apart from fixing typing/lining/renaming methods/etc. which I left for the end there are 2 main issues:

• base64 encoding bytes in nested models in fastapi - I guess this one was fixed already in pydantic TypeAdapter containing Annotated model does not use ser_json_bytes setting pydantic/pydantic#8214
• serialization of response models in fastapi is using TypeAdapter and calling dict directly so there is no chance to invoke a logic to avoid circural references, was thinking about ugly workaround with getattributes hijacking dict calls if the frame is coming from fastapi/typedict. Much better would be to rework ormar into plugin but for now the the plugin is only around validation and not serializations Plugins Roadmap pydantic/pydantic#7643

[collerek]

Checklist:

• Fix _evaluate in python 3.8
• Check coverage - remove code if not needed anymore, verify it is not really needed
• Fix types
• Fix linting
• Rename methods to v2, deprecate the old ones
• Check warnings - a lot of warnings
  • field names shadowing
  • validation dict instead of model for circular refs
  • fastapi lifecycle warnings
  • deprecated pydantic methods
  • pydantic only fields
  • pydantic_extra_types
• Code cleanup
  • Remove Meta references in code and docstrings
  • Remove choices code left
  • Remove pydantic_only code left
  • Remove property_fields code left
• Make docs versionable to display docs for v1 and v2 in github pages
• Make example in docs self-contained(?)
• Fix line numbers highlighted in docs examples after reformat
• Switch from on_startup and shutdown e to lifecycle (tests, docs)
• Fix benchmarks
• Update docs to reflect breaking changes
  • (i.e. class Meta is now ormar_config = OrmarConfig)
  • removed choices - use Enum field instead
  • removed pydantic_only - use normal pydantic fields
  • removed property_field -> use computed_field from pydantic
  • add migration guide
• Proofread the docs and fix issues (no longer valid parts, errors, formatting issues etc.)

[codspeed-hq]

CodSpeed Performance Report

Merging #1238 will degrade performances by 23.36%

_{Comparing pydantic_v2 (d8db468) with master (3a206dd)}

Summary

⚡ 18 improvements
❌ 2 (👁 2) regressions
✅ 64 untouched benchmarks

Benchmarks breakdown

	Benchmark	master	pydantic_v2	Change
⚡	test_count[1000]	24.7 ms	20.7 ms	+19.62%
⚡	test_max[1000]	16.3 ms	13 ms	+25.11%
⚡	test_making_and_inserting_models_in_bulk[10]	24.1 ms	21.2 ms	+13.35%
⚡	test_making_and_inserting_models_in_bulk[20]	34.6 ms	28.7 ms	+20.89%
⚡	test_making_and_inserting_models_in_bulk[40]	64.6 ms	52.5 ms	+23.03%
⚡	test_get_all[1000]	920.6 ms	789.3 ms	+16.63%
⚡	test_get_all[250]	234.2 ms	201.9 ms	+15.98%
⚡	test_get_all[500]	463.2 ms	397.7 ms	+16.46%
⚡	test_get_all_with_related_models[10]	68.2 ms	60.3 ms	+13.01%
⚡	test_get_or_none[1000]	14.6 ms	12.7 ms	+15.29%
👁	test_get_or_none[250]	10 ms	13 ms	-23.36%
⚡	test_initializing_models[1000]	461.2 ms	355.5 ms	+29.74%
⚡	test_initializing_models[250]	118.7 ms	93.5 ms	+26.9%
⚡	test_initializing_models[500]	230.8 ms	178.2 ms	+29.46%
⚡	test_initializing_models_with_related_models[10]	19.7 ms	11.8 ms	+66.41%
⚡	test_initializing_models_with_related_models[20]	38.6 ms	31 ms	+24.69%
👁	test_initializing_models_with_related_models[40]	76.6 ms	93.4 ms	-17.96%
⚡	test_iterate[1000]	969.3 ms	843 ms	+14.98%
⚡	test_iterate[250]	248.1 ms	217.6 ms	+14.02%
⚡	test_iterate[500]	487.6 ms	424.6 ms	+14.84%