Designing a software architecture that scales well isn’t too hard, but designing an affordable/low budget one is the real challenge.
Luckily AWS provides a myriad of services – when cleverly combined – can cut deep in the costs, provided you are willing to undergo some learning curves along the way.
In this and next posts I cover a few combos of AWS services that scale and are cheap.
S3 and Lambda
S3 is probably the most known AWS service, for good reasons, it’s cheap, reliable, you can use it for… well the list is long and only limited by your imagination. You can host websites on it, use it as backup service for your personal NAS and this is only the ‘storage’ functionality which is of course still it’s primary purpose. But add Lambda to the equation and you have a totally different ballgame. Lambda functions are those ‘serverless’ functions you can trigger from virtually anywhere… and of course also from S3.
Let it sink in a bit, you can trigger a Lambda function whenever a key is added, deleted, updated and these invocations of the Lambda functions are all concurrent.
- You want to achieve massive parallelism somewhere in a sequential kind of workflow, an S3 bucket can act as the mediator/state storage to trigger Lambda function and hence massively parallelize those parts of your workflow. A typical example would be a service that uploads tons of files (images/video/docs) that needs conversion. You will want the conversion to run in parallel I hope 🙂
- Or you may be trying to achieve a fire and forget scenario on a web server, for example during a web service/REST call. The solution: put the workload in the bucket and let a Lambda function handle it… Forget about background threads/processes and other ‘hacks’. Heck, you can even let a Lambda PUT the result of the workload processing on the bucket again and be accessible by your web application so it can provide updates to the web service callers.
The problems this combo can solve is virtually unlimited and is CHEAP. Go ahead to the AWS cost calculator and see for yourself.
I successfully applied it for massive parallelization of event processing (coming from another great AWS service called Kinesis).
As of December 1 2016 it’s possible to use .NET Core to build Lambda functions and serverless applications. More on it later but you can still read the announcement here: https://aws.amazon.com/blogs/compute/announcing-c-sharp-support-for-aws-lambda/