The Unfulfilled Promise of Schema Driven Development

Schema driven development is a software development approach that has been gaining popularity in recent years. The promise of schema driven development is that it can help us create more robust and maintainable applications by emphasizing the importance of data schemas in the design and development process. I have been searching for a solution to schema management that is integrated across my whole application stack, but despite the hype, schema driven development has yet to fulfill its promise.

For me, the biggest benefit of schema driven development is the promise of type-safety across each layer of your application. We have great languages that ensure type safety right now, but each time data is transferred from one layer or service to another, the type safety is lost. Maybe your database enforces a NOT NULL constraint that your API does not. Maybe your web client only has access to a subset of fields that your API does. Don’t even get me started on communication between micro-services… These are common problems today, but don’t have to be.

Then we get to standards… There’s SQL, JSON Schema, GraphQL, Swagger, TypeScript… and the list goes on…

“The wonderful thing about standards is that there are so many of them to choose from.”

― Grace Murray Hopper

Each one of these specifications has their place, but in practice that means we need to manually translate back and forth between them, leaving our application prone to human error and the ultimate promise of schema driven development out of reach.

Furthermore, we are not all 10x developers who can manifest the perfect schema on the first try. Taking an iterative and informal approach to developing new applications and features is not going away. Any new systems we adopt need to be able to accommodate this process. Managing prototyping, migrations and versioning is a huge problem that’s yet to be solved in this space.

What is the dream?

I believe we can do better. We are not at the final form of application development and the future of coding is bright with the possibility of more robust applications and more developer friendly tooling. We don’t need yet another standard, we need a better way to interact with the plethora of options that we already have at our disposal.

What if there was a full-stack type compiler that could ensure type-safety across each layer of an application? What would it look like? What would it need to be capable of? Here is my wish list:

1. It must be flexible. I should be able to adopt it into my existing workflow without restricting how I work. It should not force a specific standard, but instead empower me to integrate the tools that I already use.
2. It must support schema changes. Code generation is not enough. I need to be able to modify my schema and have the changes cascade throughout the different services that are affected by the change.
3. It must be built on open standards. I should not be reliant on one vendor’s cloud platform. Ideally it works with my existing version control system.
4. It must be extensible. It should support many of the existing popular standards and it should allow me to add support for my technology-of-choice if it is not already supported.
5. It should work across the whole stack. How can I extend my schema so that one layer of the stack can have access to slightly different information? How do I protect sensitive information from not being exposed where it shouldn’t?

The challenges of maintaining consistency across different layers of the stack, as well as detecting and preventing unexpected bugs, are all problems that schema driven development can help solve. We’re not there yet, but I believe we will get there with a new generation of developer tooling. Existing standards like GraphQL and Swagger are on the right track. I’m working on trying to fill in some of the gaps. If you’re interested in this space please reach out or follow along by subscribing below!