Scala Schemas with Shapeless

# Scala Schema ## with Shapeless --- ## Disclaimers -- ### All the code does actually compile #### Thanks to mdoc (kudo to @olafurpg & @jvican) -- > This is a work of fiction. Any resemblance to actual persons, living or dead, or actual events is purely coincidental. --- ### What this talk is not about ? Recursion Schemes  -- ### What this talk about ? Manipulating constructs at the type level  Note: - ground-up introduction - realworld/concrete example -- ## Agenda - motivation - schema - case classes manipulation at the type level --- ### The project ##### relatively modern stack  - microservices  - JSON/REST APIs  - NoSQL database  - Kafka for events  -- ### The project ![](ms-diagram.png) Note: - lots of communication - lots of (de)serialization -- ### User - first / last name - age - creation date - permissions -- ### User ```scala case class User( firstName: String, lastName: String, age: Int, creationDate: Long, permissions: List[String] ) ``` -- #### Read from DB / Write to API ```scala import play.api.libs.json._ val userWrites = new Writes[User] { override def writes(u: User): JsValue = Json.obj( "firstName" -> u.lastName, "lastName" -> u.firstName, "age" -> u.creationDate, "creationDate" -> u.creationDate, "permisions" -> u.permissions ) } ``` -- ```scala import java.time.Instant.now val donald = User("Donald", "Duck", 85, now.toEpochMilli, Nil) ``` ### Oops  ```scala println(Json.prettyPrint(userWrites.writes(donald))) // { // "firstName" : "Duck", // "lastName" : "Donald", // "age" : 1651670154078, // "creationDate" : 1651670154078, // "permisions" : [ ] // } ```  -- ### Generated Writes ```scala val userAutoWrites = Json.writes[User] ``` ```scala println(Json.prettyPrint(userAutoWrites.writes(donald))) // { // "firstName" : "Donald", // "lastName" : "Duck", // "age" : 85, // "creationDate" : 1651670154078, // "permissions" : [ ] // } ```  -- #### What if we remove a field from the code ? ##### We can break compatibility  -- #### What if we stored our passwords #### in our user ? -- No problem ```scala case class UserWithPassword( firstName: String, lastName: String, age: Int, creationDate: Long, permissions: List[String], password: String ) ``` -- ```scala val mickey = UserWithPassword("Mickey", "Mouse", 91, now.toEpochMilli, Nil, "azerty") val userWPWAutoWrites = Json.writes[UserWithPassword] ``` #### Oops  ```scala println(Json.prettyPrint(userWPWAutoWrites.writes(mickey))) // { // "firstName" : "Mickey", // "lastName" : "Mouse", // "age" : 91, // "creationDate" : 1651670154536, // "permissions" : [ ], // "password" : "azerty" // } ```  -- #### Automatically generated code is evil ! ##### Or is it ?  -- ### Coupling is evil ![](decoupling-diagram.png)  -- ### Maybe we need a schema ? --- ## Schema -- > The word schema comes from the Greek word σχήμα (skhēma), which means shape, or more generally, plan. -- ### In databases > In a relational database, the schema defines the tables, the fields in each table, and the relationships between fields and tables. -- ### Schema So, let's simplify, a schema is: - A way to define data with constraints - A way to define relationships in our data -- #### With types, we can - model and constraint our data - case classes - base types (String, Int, Boolean, ...) - define relationships - inheritances - composition -- #### Types are a way to define schema ![](decoupling-scala-diagram.png)  -- ### Schema Evolution (in Avro) -- #### Operations - add (optional) fields  - delete (optional) field  - modify (optional) field  -- #### Backward compatible > Backward compatibility means that consumers using the new schema can read data produced with the previous schema. -- #### Backward compatible We can automatically find a function v1 => v2 -- #### Backward compatible operations - delete fields - add, delete or modify optional fields -- #### Forward compatible > Forward compatibility means that data produced with a new schema can be read by consumers using the previous schema -- #### Forward compatible compatible We can automatically find a function v2 => v1 -- #### Forward compatible operations - add field - add, delete or modify optional fields -- ### Decoupling Business and API ```scala case class UserDomain( firstName: String, lastName: String, age: Int, creationDate: Long, permissions: List[String], password: String ) ```  ```scala case class GetUserAPI( firstName: String, lastName: String, age: Int, creationDate: Long, permissions: List[String] ) ```  -- ```scala def toUserDomain(u: UserDomain): GetUserAPI = GetUserAPI( u.firstName, u.lastName, u.age, u.creationDate, u.permissions ) val UAPIWrites = Json.writes[GetUserAPI] ``` -- ```scala val minnie = UserDomain( "Minnie", "Mouse", 91, now.toEpochMilli, Nil, "azerty" ) ``` ```scala println( Json.prettyPrint(UAPIWrites.writes(toUserDomain(minnie))) ) // { // "firstName" : "Minnie", // "lastName" : "Mouse", // "age" : 91, // "creationDate" : 1651670154558, // "permissions" : [ ] // } ``` -- ### It works ! #### with tons of boilerplate  -- ![](sad-panda.png) -- #### Why can't the compiler do that for us ? --- ## Transforming Data with shapeless -- ### Goal - Done at compile time - Typesafe - Minimal amount of boilerplate -- #### Compile time "runtimes" in scala - macros  - implicits  -- ### Implicits - context - extension method - composition -- ### Recursion -- #### List ```scala sealed trait MyList[T] case object MyNil extends MyList[Nothing] case class MyCons[T](h: T, t: MyList[T]) extends MyList[T] ``` -- #### List Length ```scala def length[T](l: List[T]): Int = l match { case Nil => 0 case h :: t => 1 + length(t) } ``` ```scala length(List(1, 2, 3)) ```  ```scala 1 + length(List(2, 3)) ```  ```scala 1 + 1 + length(List(3)) ```  ```scala 1 + 1 + 1 + length(Nil) ```  ```scala val r = 1 + 1 + 1 + 0 // r: Int = 3 ```  -- #### Can we do it at compile time ? Note: - no we can't, not like this - the compiler does not have enough information -- #### Statically sized list ```scala sealed trait SList[+H] case object SNil extends SList[Nothing]{ def ::[H](h: H) = new ::(h, this) } case class ::[H, T <: SList[H]](h: H, t: T) extends SList[H]{ def ::(h: H) = new ::(h, this) } ``` ```scala 1 :: 2 :: 3 :: SNil // res10: Int :: Int :: Int :: SNil.type = 1 :: 2 :: 3 :: SNil ```  ```scala 1 :: 2 :: 3 :: Nil // res11: List[Int] = List(1, 2, 3) ```  -- #### Typelevel length ```scala case class Size[T](size: Int) object Size { def apply[T](implicit s: Size[T]) = s } ``` -- ##### Typelevel length ```scala implicit val SNilSize: Size[SNil.type] = Size(0) ``` ```scala implicit def sconsSize[H, T <: SList[H]]( implicit l: Size[T] ): Size[H :: T] = Size(1 + l.size) ``` ```scala Size[Int :: Int :: Int :: SNil.type].size // res12: Int = 3 ```  ```scala sconsSize[Int, Int :: Int :: SNil.type]( Size[Int :: Int :: SNil.type] ) ```  ```scala sconsSize[Int, Int :: Int :: SNil.type]( sconsSize(Size[Int :: SNil.type]) ) ```  ```scala sconsSize(sconsSize(sconsSize(Size[SNil.type]))) ```  ```scala sconsSize(sconsSize(sconsSize(SNilSize))) ```  -- #### Implicits composition behaves like recursion and pattern matching on types ![](exploding-brain.gif)  -- ### Transform Typeclass ```scala trait Transform[A, B] { def to(a: A): B } object Transform{ def apply[A, B](implicit t: Transform[A, B]) = t } ``` -- ### Let's start simple - Only field deletions - Same field types -- ### How would we do it at runtime ? -- ### Let's define our types ```scala sealed trait RType case object RInt extends RType case object RString extends RType case object RBoolean extends RType type RTuple = List[RType] ``` -- ### Now let's generate our fake transform ```scala def to(a: RTuple, b: RTuple): RTuple = (a, b) match { case (_, Nil) => Nil case (ha :: ta, hb :: tb) if ha == hb => ha :: to(ta, tb) case (ha :: ta, b) => to(ta, b) } ``` 1. Base case, b is complete  2. Heads do match  2. Heads do not match  -- ### Does it work ? ```scala to(RInt :: Nil, RInt :: Nil) // res18: RTuple = List(RInt) to(RInt :: RString :: Nil, RInt :: Nil) // res19: RTuple = List(RInt) ``` ```scala to(RString :: Nil, RInt :: Nil) // scala.MatchError: (List(),List(RInt)) (of class scala.Tuple2) // at repl.MdocSession$App17.to(scala-schema-wshapeless.md:330) // at repl.MdocSession$App17.to(scala-schema-wshapeless.md:333) // at repl.MdocSession$App17$$anonfun$30.apply(scala-schema-wshapeless.md:350) // at repl.MdocSession$App17$$anonfun$30.apply(scala-schema-wshapeless.md:350) ```  #### Yes !  -- ### Let's do this at the type level ? -- ### HList to the rescue ```scala sealed trait HList case class ::[H, T <: MyHList](head: H, tail: T) extends HList case object HNil extends HList ``` ```scala import shapeless._ 1 :: "hello" :: true :: HNil // res20: Int :: String :: Boolean :: HNil = 1 :: "hello" :: true :: HNil Generic[(Int, String, Boolean)] .to((1, "hello", true)) // res21: Int :: String :: Boolean :: HNil = 1 :: "hello" :: true :: HNil ```  -- ### Our base case ```scala import shapeless._ implicit def HNilTransform[T]: Transform[T, HNil] = _ => HNil ``` ```scala Transform[String :: Int :: HNil, HNil] .to("hello" :: 1 :: HNil) // res22: HNil = HNil ```  -- ### Head matches ```scala implicit def HeadMatchTransform[ Head, TailOfA <: HList, TailOfB <: HList ]( implicit t: Transform[TailOfA, TailOfB] ): Transform[Head :: TailOfA, Head :: TailOfB] = a => a.head :: t.to(a.tail) ``` ```scala Transform[String :: Int :: HNil, String :: Int :: HNil] .to("Bob" :: 42 :: HNil) // res23: String :: Int :: HNil = "Bob" :: 42 :: HNil ```  -- ### Head do not match ```scala import shapeless._ implicit def HeadDoNotMatchTransform[ HeadOfA, TailOfA <: HList, HeadOfB, TailOfB <: HList ]( implicit t: Transform[TailOfA, HeadOfB :: TailOfB] ): Transform[HeadOfA :: TailOfA, HeadOfB :: TailOfB] = a => t.to(a.tail) ``` ```scala Transform[String :: Int :: HNil, Int :: HNil] .to("Bob" :: 42 :: HNil) // res24: Int :: HNil = 42 :: HNil ```  -- ### Great ! But what about case classes ? -- #### Well they are not so special ```scala case class Tag(tag: String, t: RType) extends RType ``` ```scala to( Tag("age", RInt) :: Tag("name", RString) :: Nil, Tag("name", RString) :: Nil ) // res25: RTuple = List(Tag("name", RString)) ```  ```scala to( Tag("age", RInt) :: Tag("name", RString) :: Nil, Tag("nam", RString) :: Nil ) // scala.MatchError: (List(),List(Tag(nam,RString))) (of class scala.Tuple2) // at repl.MdocSession$App17.to(scala-schema-wshapeless.md:330) // at repl.MdocSession$App17.to(scala-schema-wshapeless.md:333) // at repl.MdocSession$App17.to(scala-schema-wshapeless.md:333) // at repl.MdocSession$App17$$anonfun$38.apply(scala-schema-wshapeless.md:451) // at repl.MdocSession$App17$$anonfun$38.apply(scala-schema-wshapeless.md:449) ```  -- ## Labelled Generics ```scala case class Labelled(param1: Int, param2: String) LabelledGeneric[Labelled] .to(Labelled(1, "Hello")) // res26: Int with labelled.KeyTag[Symbol with tag.Tagged[param1], Int] :: String with labelled.KeyTag[Symbol with tag.Tagged[param2], String] :: HNil = 1 :: "Hello" :: HNil ``` -- ### Case classes transformation ```scala implicit def ccTransform[A, ReprA, B, ReprB]( implicit genA: LabelledGeneric.Aux[A, ReprA], genB: LabelledGeneric.Aux[B, ReprB], transform: Transform[ReprA, ReprB] ): Transform[A, B] = a => genB.from(transform.to(genA.to(a))) ``` -- ```scala case class UserDomain( firstName: String, lastName: String, age: Int, creationDate: Long, permissions: List[String], password: String ) ```  ```scala case class GetUserAPI( firstName: String, lastName: String, age: Int, creationDate: Long, permissions: List[String] ) ```  -- ```scala import java.time.Instant.now val donald = UserDomain( "Donald", "Duck", 85, now.toEpochMilli, Nil, "azerty" ) ``` ```scala Transform[UserDomain, GetUserAPI].to(donald) // res27: GetUserAPI = GetUserAPI("Donald", "Duck", 85, 1651670154690L, List()) ``` -- ### Automatic wraping / unwraping ```scala implicit def optionTransform[A]: Transform[A, Option[A]] = a => Option(a) case class UserId(value: String) implicit val userIdTransform: Transform[UserId, String] = a => a.value ``` -- ### Automatic wraping / unwraping ```scala implicit def HeadMatchWTransform[ HeadOfA, TailOfA <: HList, HeadOfB, TailOfB <: HList, ]( implicit hT: Transform[HeadOfA, HeadOfB], tT: Transform[TailOfA, TailOfB] ): Transform[HeadOfA :: TailOfA, HeadOfB :: TailOfB] = a => hT.to(a.head) :: tT.to(a.tail) ``` ```scala Transform[String :: Int :: HNil, Option[Int] :: HNil] .to("hello" :: 1 :: HNil) // res28: Option[Int] :: HNil = Some(1) :: HNil ``` --- ## So right to production ? -- ### With this you will have... - a compiler to keep you warm in winter  - overhead going to and from HLists  - definitely helpful error messages  ```scala case class UserMisorderedBis( age: Int, lastName: String, creationDate: Long, ) Transform[User, UserMisorderedBis] // error: not found: type User // Transform[User, UserMisorderedBis] // ^^^^ ```  -- ### What about if we mess with field order ```scala case class UserMisordered( age: Int, lastName: String, creationDate: Long, permissions: List[String] ) Transform[UserDomain, UserMisordered].to(donald) // error: could not find implicit value for parameter t: App2.this.Transform[App2.this.UserDomain,App2.this.UserMisordered] // Transform[UserDomain, UserMisordered].to(donald) // ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ```  -- ### Use chimney ! -- ```scala case class UserMisorderedBis( age: Int, fullName: String, creationDate: Long, legacyUser: Boolean ) ``` ```scala import io.scalaland.chimney.dsl._ donald.transformInto[UserMisorderedBis] // error: Chimney can't derive transformation from App2.this.UserDomain to App2.this.UserMisorderedBis // // repl.MdocSession.App2.UserMisorderedBis // fullName: java.lang.String - no accessor named fullName in source type repl.MdocSession.App2.UserDomain // legacyUser: scala.Boolean - no accessor named legacyUser in source type repl.MdocSession.App2.UserDomain // // Consult https://scalalandio.github.io/chimney for usage examples. // // // donald.transformInto[UserMisorderedBis] // ^ ```  -- ```scala import io.scalaland.chimney.dsl._ donald.into[UserMisorderedBis] .withFieldComputed( _.fullName, u => u.firstName + " " + u.lastName ) .withFieldConst(_.legacyUser, true) .transform // res32: UserMisorderedBis = UserMisorderedBis( // 85, // "Donald Duck", // 1651670154690L, // true // ) ``` --- ### Wrap Up - Avoid coupling  - Write more types  - Derive all the things !  --- ### Going Further - [The Type Astronaut's Guide to Shapeless](https://books.underscore.io/shapeless-guide/shapeless-guide.html) - [Massaging case classes with shapeless](https://www.signifytechnology.com/blog/2018/10/massaging-case-classes-with-shapeless-by-chris-birchall-at-scala-in-the-city) - [Chimney](https://scalalandio.github.io/chimney/) --- ## Thank you ! Any questions ? ###### Slides: [bit.ly/2TL6QJu](https://bit.ly/2TL6QJu) Kévin Rauscher • [`@tomahna`](https://twitter.com/tomahna) --- ### Bonus -- ### Handling field order ```scala import shapeless.ops.hlist.Selector implicit def ContainTransform[ A <: HList, HeadOfB, TailOfB <: HList ]( implicit select: Selector[A, HeadOfB], t: Transform[A, TailOfB] ): Transform[A, HeadOfB :: TailOfB] = a => select(a) :: t.to(a) ``` -- ```scala case class UserMisordered( age: Int, lastName: String, creationDate: Long, permissions: List[String] ) Transform[UserDomain, UserMisordered].to(donald) // res33: UserMisordered = UserMisordered(85, "Duck", 1651670154690L, List()) ``` ## Thank you ! Any questions ? ###### Slides: [bit.ly/2TL6QJu](https://bit.ly/2TL6QJu) Kévin Rauscher • [`@tomahna`](https://twitter.com/tomahna) ---