Wow, I struck a nerve. I'm happy to address these points however.
> The diagram looks like an ad
Lol, Its an architecture diagram. You could swap the AWS-specific icons for generic ones I suppose, and it wouldn't change anything. It is fulfilling its purpose of explaining how all the services connect together to deliver the product. Just because it is an AWS Lambda icon doesn't mean you couldn't make it an Azure Function icon instead and perform the same goal. You're just too focused on hating AWS here to see the forest through the trees.
> Do you actually need all of this or do you use it because Amazon tells you to? I know for instance you cannot use Amazon SES without also using S3 and Lambda
This is just a standard event-driven architecture. There's really nothing exciting to see here. Data comes in, it gets stored somewhere (S3), that triggers an event (SNS), a compute service (lambda in this case, but could be anything, even a standard VM, bare metal or anything else) picks up the task and processes it or performs a job on the data and stores it, it triggers another event, something else picks it up, and so forth. This isn't an AWS design, its just an event driven architecture design and this is how they work.
SES can be used standalone. It doesn't require Lambda or S3 like you postulate. There are only a few times AWS requires something else and its usually Cloudwatch or S3 and these will sometimes be the destinations required for specific types of logging or auditing and so forth.
AWS is forcing you to do nothing here. The creator chose this stuff. But I assume they chose it to keep it free. Most of this will survive under the free tier until the project becomes massive. If it inches over the free tier, it will still be cheap. That's probably the incentive for a lot of this.
> You're saying that I cannot move to other cloud provider without my existing code becoming useless?
Correct your code is your code. Think about a lambda. In this scenario data comes from NOAA and is put in a bucket. You write a serverless function that takes that data out of the bucket, reformats the NOAA data into your proprietary format and puts it in another bucket. The code that does that is written in Go, Python, C++, Java, or whatever you want. If written correctly, it accepts data, processes it, and outputs it. So if you move to another cloud provider your code would still work. It might run in an Azure Function instead of an AWS Lambda, but that doesn't matter. Your code does the same thing you don't need to throw it out.
> What about the data pipelines and business logic?
Hard to say without more information. But its possible BI dashboards need to be changed to point to the new service and stuff like that. Sure. Again, you're not switching clouds in an afternoon. But the point its the next cloud system is eerily similar. Its not like you have to rebuild, it would be more of a refactor.
> There is: avoiding vendor-specific APIs altogether
Possibly. But if built correctly your code doesn't need to be aware of the environment it is in. But there might be cases where you interact with the services directly, like downloading from S3. This would change with the next provider possibly (although most actually have S3-compatible APIs). But most of your application will not directly interact with the cloud, it will interact with the services. So for example you use RDS to host a managed Postgres db, but your application is just interacting with postgres, not AWS here. But you're right there might be some scenarios that use vendor-specific APIs.
> I don't have to. There are things like Railway, Fly.io, PlanetScale, Supabase, Upstash, Minio, which can work without locking me in
I fail to see how these are any different than tying yourself to a product like S3 or Lambda. In many ways, these solutions are TRUE vendor lock-in, with all the vendor specific APIs that you live in fear of. Fly.io is a PaaS, which is going to be way harder to move away from than switching from AWS to Azure. PlanetScale, Minio, and Upstash are literally no different than equivalent products in AWS/GCP/Azure. I guess you could host the instance that runs these products ondifferent clouds and it would be the same, but you're still tying yourself to something. The risk of tying yourself to a startup is higher than tying yourself to Amazon/Microsoft/Google. You're trading one evil for another, in most ways you are actually losing freedom with these not gaining it.
> You are contradicting yourself
My point is that there isn't as much vendor lock-in that people fear. Yes it exists, but don't live your life in fear of it. Yes you would need to refactor stuff here and there. But the same architecture diagram we saw for AWS is basically the same one that would exist in Azure or GCP. The underlying tools don't change. The marketing names and logos change, which clearly bothers you, but the underlying system doesn't change.
You clearly know way more about cloud implementations than I do, so I really appreciate the time you took to explain that out! Since I am the dev here though, the one things I can confirm is that you're 100% correct about the setup methodology- almost every decision was based on "how can I do this in a way that's cost effective". In particular, the underlying data was already on AWS, so it just made sense to build it there.
I think one thing that gets lost in discussion is the advantages of serverless approaches for people without a ton of technical background. I built 90% of this without knowing anything about servers or APIs, but the cloud tools (from whoever) let me ignore all of that and just write a bunch of Python scripts that do cool things. I know it ends up sounding a bit like an AWS ad (I wish I was sponsored by them, but am not), but there really are perks to the approach
> The diagram looks like an ad
Lol, Its an architecture diagram. You could swap the AWS-specific icons for generic ones I suppose, and it wouldn't change anything. It is fulfilling its purpose of explaining how all the services connect together to deliver the product. Just because it is an AWS Lambda icon doesn't mean you couldn't make it an Azure Function icon instead and perform the same goal. You're just too focused on hating AWS here to see the forest through the trees.
> Do you actually need all of this or do you use it because Amazon tells you to? I know for instance you cannot use Amazon SES without also using S3 and Lambda
This is just a standard event-driven architecture. There's really nothing exciting to see here. Data comes in, it gets stored somewhere (S3), that triggers an event (SNS), a compute service (lambda in this case, but could be anything, even a standard VM, bare metal or anything else) picks up the task and processes it or performs a job on the data and stores it, it triggers another event, something else picks it up, and so forth. This isn't an AWS design, its just an event driven architecture design and this is how they work.
SES can be used standalone. It doesn't require Lambda or S3 like you postulate. There are only a few times AWS requires something else and its usually Cloudwatch or S3 and these will sometimes be the destinations required for specific types of logging or auditing and so forth.
AWS is forcing you to do nothing here. The creator chose this stuff. But I assume they chose it to keep it free. Most of this will survive under the free tier until the project becomes massive. If it inches over the free tier, it will still be cheap. That's probably the incentive for a lot of this.
> You're saying that I cannot move to other cloud provider without my existing code becoming useless?
Correct your code is your code. Think about a lambda. In this scenario data comes from NOAA and is put in a bucket. You write a serverless function that takes that data out of the bucket, reformats the NOAA data into your proprietary format and puts it in another bucket. The code that does that is written in Go, Python, C++, Java, or whatever you want. If written correctly, it accepts data, processes it, and outputs it. So if you move to another cloud provider your code would still work. It might run in an Azure Function instead of an AWS Lambda, but that doesn't matter. Your code does the same thing you don't need to throw it out.
> What about the data pipelines and business logic?
Hard to say without more information. But its possible BI dashboards need to be changed to point to the new service and stuff like that. Sure. Again, you're not switching clouds in an afternoon. But the point its the next cloud system is eerily similar. Its not like you have to rebuild, it would be more of a refactor.
> There is: avoiding vendor-specific APIs altogether
Possibly. But if built correctly your code doesn't need to be aware of the environment it is in. But there might be cases where you interact with the services directly, like downloading from S3. This would change with the next provider possibly (although most actually have S3-compatible APIs). But most of your application will not directly interact with the cloud, it will interact with the services. So for example you use RDS to host a managed Postgres db, but your application is just interacting with postgres, not AWS here. But you're right there might be some scenarios that use vendor-specific APIs.
> I don't have to. There are things like Railway, Fly.io, PlanetScale, Supabase, Upstash, Minio, which can work without locking me in
I fail to see how these are any different than tying yourself to a product like S3 or Lambda. In many ways, these solutions are TRUE vendor lock-in, with all the vendor specific APIs that you live in fear of. Fly.io is a PaaS, which is going to be way harder to move away from than switching from AWS to Azure. PlanetScale, Minio, and Upstash are literally no different than equivalent products in AWS/GCP/Azure. I guess you could host the instance that runs these products ondifferent clouds and it would be the same, but you're still tying yourself to something. The risk of tying yourself to a startup is higher than tying yourself to Amazon/Microsoft/Google. You're trading one evil for another, in most ways you are actually losing freedom with these not gaining it.
> You are contradicting yourself
My point is that there isn't as much vendor lock-in that people fear. Yes it exists, but don't live your life in fear of it. Yes you would need to refactor stuff here and there. But the same architecture diagram we saw for AWS is basically the same one that would exist in Azure or GCP. The underlying tools don't change. The marketing names and logos change, which clearly bothers you, but the underlying system doesn't change.