Wondering how to create a scalable solution using Step Functions? You’re in the right place!

Step Functions is a serverless orchestrator that makes it easier to sequence different AWS services into business-critical applications. It provides quite a bit of convenient functionality: automatic retry handling, orchestration, monitoring and alerting. On Gousto Recipes — Building Pipelines with Step Functions, you can see how we have been using Step Functions at Gousto.

In this blog post, we will go into more details on how to take advantage of Dynamic Parallelism (some people might know that as Fan-out).

What problem were we trying to solve?

Site Specific Forecast pipeline is one…

When we create a project, we are so focused on the project itself that we forget to think about its Github repository configuration. Actually, I believe we shouldn’t bother at all, but create a tool to automatise that process and ensure our zillion of projects follow a standard setup.

One small mistake such as configuring a company repository as public could expose loads of private and critical content to the whole world. In this post, I would like to share how you can create a tool to handle repository configuration using PyGithub and show some useful configurations.

PyGithub Library

PyGitHub is a…

Gousto is on a mission to become the UK’s favourite way to eat dinner. Essentially, customers can select from an ever-changing range of recipes every week and receive fresh ingredients in a box on their doorstep with easy-to-follow recipes. The sheer number of UK families placing their trust in Gousto every week means we need to build a supply chain that is supported by a reliable tech stack, and AWS technologies are one of the most important ingredients of our tech products.

We have recently adopted AWS Step Functions, a serverless orchestrator that makes it easier to coordinate different AWS…

dbt is a command line tool which does the T in ELT (Extract, Load, Transform) processes and it is a powerful technology to help engineers and data scientists to transform data by simply writing select statements and SQL validations in a structured fashion. In my previous job at Deliveroo, I was working with dbt (data build tool) and I’d love to share some learnings and why we should treat SQL as software such as any other micro service or Lambda function implemented using a programming language.

dbt

The diagram below shows how dbt fits into a tech stack. Essentially, it takes…

Adriano Medeiros Dos Santos

Software Engineer at Gousto

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store