So it sounds like you are looking at an Entity Component System as applied to a DB for data modeling. Thanks for taking the time to explain. I do agree it's a valid approach that has many benefits - However, I do disagree that it is a strictly better approach.
I've been around a decent while and have seen a lot of tech, and it's a trade-off that has come about in many different forms. It's static typing vs dynamic typing, it's fixed schema vs flexible schema, it's compile-time checks vs run-time checks. It's SQL vs NOSQL. Overall it gives a lot of flexibility in that any entity can take on any behavior with minimal changes; however it also comes with the flip side of it being harder to catch modeling errors. It's the same argument of "make impossible states unrepresentable" vs "constraints should be enforced in the app along with business logic". It's also Typescript vs Javascript. But there isn't a true winner in approach. Typescript beats JS in my book, but dynamically typed Clojure is also a great language. It's "is-a" [1] vs "has-a"[2] relationships and it's squarely in the "has-a" approach which I (and most people) feel is the better approach. So again, no clear winner in approach.
I do think it's going to continue to gain steam and more people will adopt it simply because ECS is popular. I also think down the line people will realize it has similar / the same short-comings as NoSQL and will re-adopt relational for its benefits.
If you are thinking of writing a blog, it never hurts, that said there was a post to HN just a week or two ago of someone discussing the approach. They were making a different point, but their underlying assumption was the same. Namely: "You can due ECS in a DB and benefit from it." Note though that they also called out the trade-off of "normal" ECS and how you lose mode enforced / constraint checking. (Their improvement was to go to DB approach, although that doesn't solve it). If you hadn't seen it, you might want to check it out [3]. And if you squint ECS (or ECA) I believe is the non-OOP version of mixins. If you haven't done any reading about that it might be of interest [4].
But thanks for sharing your thinking and yes I do agree it's a valid approach for design - just as with anything make sure you know the tradeoffs (what you gain and what it costs).
I have so much to say about all of this, but the indentation on this comment thread is getting extreme. Everything you said is correct in some general sense of understanding software engineering, and yet ... somehow too fence-sitting. Static Typing vs. Dynamic Typing is not some 50/50 argument: evidence and the experience of people who use both is strongly in favor of Static Typing being close to "strictly better" for most teams. SQL vs. NoSQL is less clear, but you can extrapolate known "good practices" to show that the relational model is "strictly better" for the kinds of applications that most people actually want to build.
There's No Free Lunch, sure, but that's defined over all problems, which is an unbelievably vast set of things that we mostly never care about. The problems that human beings ever actually care about rounds to 0.000% of "all problems", and you could keep going with 100 more 0s after that. When you look at the industry and the problems people are actually trying to solve, sometimes "strictly better" starts looking like the right term to use for some of these tradeoffs; or at least "almost always better".
Ultimately Software Engineering is still a juvenile discipline compared to other kinds of Engineering, and I think we can look to our big siblings for some insights. It's obviously true that in Aeronautical Engineering, everything is a tradeoff and nothing is "strictly better", yet we don't see many biplanes flying around anymore. It is possible for basically the whole field to advance. (Although old ideas have a way of resurrecting, too -- maybe biplanes will suddenly become very useful for certain drones, or in the Martian atmosphere or something!)
> dynamically typed Clojure is also a great language
I'm sure it is! And I'm also sure that statically typed Clojure, with a sufficiently strong type system to represent the kind of code you want to write with it, would be an even better language. Although perhaps "sufficiently strong" is not yet achievable.
> I also think down the line people will realize [ECS] has similar / the same short-comings as NoSQL and will re-adopt relational for its benefits.
Entity-Component is more relational than standard table-per-noun architecture (at least the way I do it), but I understand why this isn't obvious and that it's something I'd need to prove.
> Note though that they also called out the trade-off of "normal" ECS and how you lose mode enforced / constraint checking.
I encourage you to sit down and think through all of the constraints on your domain logic, and then see how many are enforced at the database level. Let's look at the old classic https://wiki.c2.com/?WhyIsPayrollHard -- how many of these constraints are enforced with ACID guarantees at the database level? Something like this:
> After being sick for 3 days (when your pay comes from your normal account), you go on disability, where your pay continues but comes from another account. After N days you only get 60% of your pay, unless you come back and go out on a different disability.
This is a complex domain-logic rule that you could never implement in the database (unless you're using T-SQL as your application development language, in which case your Triggers are simply your application and you're still doing this in the application layer). All foreign key constraints can be phrased in terms of (often volatile) business validation rules -- there is nothing special about them, except the fact that they happen to be easy to enforce by the database. Look at that list of Payroll requirements and then ask again: how do you prevent invalid data states? There's no reason to single out database-enforced foreign key constraints as the only data validation check that you absolutely must enforce at the database layer. It's completely fine to drop foreign key constraints if you have other ways to validate your model -- and you must! (Yes, at the dreaded "application layer"! That's exactly what your application layer is!)
> any entity can take on any behavior with minimal changes;
Unfortunately we haven't even started talking about how behavior and domain logic works its way in here. It's not really like mixins -- mixins is like trying to add the concept of addition onto the number 7. 7 is just a number, and addition exists outside of it. Think more of the "Arithmetic System" as defining a set of operations that make sense to do on (collections of) numbers, and checking/enforcing constraints on that (no division by 0). (Obviously this doesn't explicitly exist since programming languages and databases already give us arithemetic for free.) Then the "Unit Conversion" system relies on the Arithmetic System to do its jobs, defining more complex aggregate data types, further narrowing the scope of valid operations, further enforcing checks. Then the "Chemical Reaction" system relies on the Unit Conversion System (and others), defines larger, stricter aggregates, further narrows the scope of valid operations on those aggregates, etc. It's built in successively strict layers working on successively more complicated "aggregate components" -- not really anything like mixins.
Thanks for the links. SpaceTimeDB is not really what I'm talking about but it's sorta close-ish.
I will say, based on what you've shared, you and I appear to have fairly similar approaches / philosophies to design - which I don't often find. I also agree this thread is way deeper than I planned, but it's a topic I really enjoy so here's another reply. :)
Regarding the choices I listed above I'm firmly in the Static Typing / SQL / TypeScript / Fixed Schema / "has-a" / make-the-impossible-unrepresentable camps. Which, excluding the last one, I believe aligns with your views of the word. (I do also agree that a statically / optionally typed clojure would probably be my preference. And I haven't spent much/any time yet with clojure's spec). I mainly listed them with alternates as those alternates are valid choices at times even if they aren't my choice and I think I can defend why those I listed are better (and it seems you agree). But the other reason I listed them is that ECS clearly falls in with the dynamically typed / NoSQL / ... camps. It is structural typing (or duck typing). But if I go that route how do I enforce that my modeling is correct? Or in a computer game example, if I model my chat messages as entities, how do I ensure that a chat message can never "hold a grenade launcher" via a component? Clearly an error in modeling. And if the answer is "write good code", then that's fine, but it's definitely the dynamically typed approach to solving the problem which isn't my usual preference.
For me that's an important part of modeling - making invalid state unrepresentable. So if I were actually going with ECS (or ECA), I'd actually go ahead and build a type system on top of it. I'd build a meta-component which is a name of a type and it would specify via statements (using your terminology) this type named X must have these components and can optionally have these other components. Any other components by default it can't have. A type (specified by the type meta-component) would be a collection of components. And entity must have an entry in the type component specifying which other components it can and must have. But then it brings the next question, can a type refer to other types? If I declare a type A that has 30 components and a type B that has 20, and I now want to declare a type C, do I have to list out all 50 components? Or can I say type C has all of Type A's and Type B's components? Once I've now decided to include these types that reference types (or higher order / higher-kinded types), and I include the most basic two ways of combining them sum/product, or enum/struct or tuple/discriminatedUnion or which ever names you want to call them by and I have a full support for ADTs I'd be happy. But then the pragmatist in me would ask now that you've added a fully structured type system on top of your ECS/ECA have you now lost all of the flexibility that ECA gave you? I don't know the answer to that. At this point, we've reached thought exercise for me so I'd have to do it and see. But It would be very tempted to explore to to answer those questions and decide do I stick with ECS or include typing on top to better ensure my system adheres to my business domain constraints.
So jumping back up the stack a bit, the C2 Payroll list is a great real-world list of the style of things that people often ignore. However, I think things go deeper than that. That list is a collection of condition action pairs. If an entity has this situation then treat it like the following. That is a big part of of business logic. But it ignores the other part. Which entities are allowed to have which situations. For example I want to ensure that in my system a non-union worker can never have the UPGWA old-timer pay. Funnily enough that's actually one of the things I was playing around with in clojure from concepts of Out of the Tar Pit.
> After being sick for 3 days (when your pay comes from your normal account), you go on disability, where your pay continues but comes from another account. After N days you only get 60% of your pay, unless you come back and go out on a different disability.
You see, for this hard to describe constraint I see the same problem as you do with it. But my approach was the exact opposite. Defining constraints in the data layer is great - but only a subset of constraints can be easily defined in a relational data layer. Your approach was push everything to the application layer. My approach was add support for way more complex constraints in the data layer.
I'd model the payroll constraint above as: have your only base table be statements of employee X was sick on Y day. Have a derived table / sql view that takes the last 3 days of entries and sees if it was 3 days ago, if so in an 'out of the tar pit' derived-data sense or a Functional Reactive Programming sense enable a field on this derived table that says they are now on disability. Another set of code or table that says 'if that field is enabled then reference the alternate account'. To do this you need support for so many more constraints than basic SQL support now. So that was my approach build in support all of these complex row level, multi-row, and full table constraints into your data system (or just above it) so that your domain language could be expressed as constraints on either you base statement table, or constraints on your derived data / materialized view tables. And yup with each insert, modify, delete in an FRP/trigger style all of your relevant derived tables would be updated and their constraints checked. It turned out that once you start deriving data the constraints aren't too complicated, lots of map, filter, reduce and the rest across your table rows (yes you could express the same things in SQL but I was playing with clojure and the functional approach here seemed cleaner). But this project never really made it past 1 weekend of playing around. It was promising, but I will also say it was solving a slightly different problem which is "how do you get state out of code?" - and that was via dynamically updating derived data (similar enough to FRP).
So all in all, if I'm understanding you correctly, you and I may both be seeing the same problem and approached it slightly differently. But I would agree it is a area where there is a lot of potential for improving software quality and modeling. But maybe you went a different way with your ECA and building relational types on top of it? I don't quite follow that final paragraph you wrote about arithmetic, but will think it over some more.
(I loved this conversation by the way, so thanks!)
Just a couple last major things:
> make-the-impossible-unrepresentable ... ECS clearly falls in with the dynamically typed / NoSQL camps
I see why you think that, but I disagree: you still make the impossible unrepresentable. The difference is that "impossible" depends on your perspective -- on the "domain layer" you're working in. Lower-level domain layers are less strict than higher-level ones. Negative numbers are really useful -- your Number System should allow it; it's up to the higher-level consumers of your Number System to decide whether negative numbers should be possible in their ruleset. So type InventoryReading { date, product, amount } can verify that amount is not negative. It can even define amount to be a PositiveInteger if it wants; that's fine. But what is or is not "impossible" turns out to be extremely dependent on your perspective.
> So if I were actually going with ECS (or ECA), I'd actually go ahead and build a type system on top of it...
You're actually overthinking a bit here. No need for higher-kinded types. Just make structs-of-structs-of-structs. If you have a "Facility System" that works with "Facility Models" like { USPostalAddress address, IEnumerable<NaicsCode> naicsCodes, ContactInfo operatorContact } (all strongly typed), then the Facility System can define a bunch of operations by relying on the USPostalSystem, NaicsSystem, and ContactSystem. It can do its own checks and enforce its own rules, do its own translations, etc., but it does that on top of passing each smaller object to its (generally dependency-injected) subsystems.
Those subsystems can in turn rely on other subsystems. The lower level subsystems are less strict -- maybe the USPostalSystem allows PO Boxes, but your Facility System does not consider that a valid address for its uses. So USPostalAddress could even have a method like IsPOBox() -- although even better might be something like IsPhysicalBuilding() or something, so nobody else has to even know what a "PO Box" is. Or to go further into making invalid states unrepresentable, you could have "PhysicalAddress" as the restrictive type here, and your AddressSystem implementation(s) would know how to translate that into their less-strict type after their own checks (can't be a PO Box).
Of course there's an obvious question if we loop way back to what started this whole thing: "but aren't you categorizing things as a Facility right now?"
Naming things is hard. Really hard. The important thing is how I think about what "Facility" represents -- a (rather complicated!) statement that can be made about an entity. Deep-rooted in the philosophy of this design is the idea that I never consider that list to be the comprehensive list of all things anyone might call a "Facility", nor must everything in that list meet all the requirements that anyone might have on a "Facility". It's really just a collection of statements I want to make about some entities. I'm very aware that this definition of "Facility" is highly opinionated and volatile, and that many other people will have a different definition. In fact, that's largely the point of the architecture. So nowhere still have I defined hard boundaries for a "Facility" category. Nobody familiar with this design should be surprised if they encounter a competing definition of "Facility" -- we can handle both! It's just the names that are hard.
Thanks for sharing thoughts, it's been a great discussion. I'm gonna call it an end here (otherwise I could go on forever), but if you do ever end up putting up a blog post on it, please do share!
I've been around a decent while and have seen a lot of tech, and it's a trade-off that has come about in many different forms. It's static typing vs dynamic typing, it's fixed schema vs flexible schema, it's compile-time checks vs run-time checks. It's SQL vs NOSQL. Overall it gives a lot of flexibility in that any entity can take on any behavior with minimal changes; however it also comes with the flip side of it being harder to catch modeling errors. It's the same argument of "make impossible states unrepresentable" vs "constraints should be enforced in the app along with business logic". It's also Typescript vs Javascript. But there isn't a true winner in approach. Typescript beats JS in my book, but dynamically typed Clojure is also a great language. It's "is-a" [1] vs "has-a"[2] relationships and it's squarely in the "has-a" approach which I (and most people) feel is the better approach. So again, no clear winner in approach.
I do think it's going to continue to gain steam and more people will adopt it simply because ECS is popular. I also think down the line people will realize it has similar / the same short-comings as NoSQL and will re-adopt relational for its benefits.
If you are thinking of writing a blog, it never hurts, that said there was a post to HN just a week or two ago of someone discussing the approach. They were making a different point, but their underlying assumption was the same. Namely: "You can due ECS in a DB and benefit from it." Note though that they also called out the trade-off of "normal" ECS and how you lose mode enforced / constraint checking. (Their improvement was to go to DB approach, although that doesn't solve it). If you hadn't seen it, you might want to check it out [3]. And if you squint ECS (or ECA) I believe is the non-OOP version of mixins. If you haven't done any reading about that it might be of interest [4].
But thanks for sharing your thinking and yes I do agree it's a valid approach for design - just as with anything make sure you know the tradeoffs (what you gain and what it costs).
1. https://en.wikipedia.org/wiki/Is-a 2. https://en.wikipedia.org/wiki/Has-a 3. https://spacetimedb.com/blog/databases-and-data-oriented-des... 4. https://en.wikipedia.org/wiki/Mixin