Sign in Subscribe

Olga Braginskaya

The Data Engineering Docker-Compose Starter Kit

We’ve all been there. Starting a new data engineering project and, instead of diving right into the fun stuff, we’re stuck setting up configurations. Again. So, I’ve put together a Docker Compose config with a bunch of services you’ll often see in data engineering projects. Just

The Golang Saga: A Coder’s Journey There and Back Again. Part 3: The Graphing Conundrum

Welcome back to the third part of “The Golang Saga: A Coder’s Journey There and Back Again.” In the first part of this series, I started work on a personal project, the Climate Change Visualizer, using Go as my chosen programming language. In the second part, we delved into

The Golang Saga: A Coder’s Journey There and Back Again. Part 2: The Data Expedition

Welcome back to the second part of “The Golang Saga: A Coder’s Journey There and Back Again.” In the first part of this series, I started working on a personal project, the Climate Change Visualizer, using Golang as my chosen programming language. Just a quick recap, I had already

The Golang Saga: A Coder’s Journey There and Back Again. Part 1: Leaving the Shire

The idea of learning Golang has been circulating in my head for quite some time. However, like many of us, I constantly found excuses to put it off. Just the other day, though, I was reading this compelling article that talked about how the key to effective learning lies in

Unleash Your Pipeline Creativity: Local Development with Argo Workflows and MinIO on Minikube

As a data engineer, you know the joy of wrangling massive datasets and navigating complex data pipelines. Argo Workflows, a popular workflow engine for Kubernetes, is your trusty companion in this data-driven journey, allowing you to define, run, and manage data pipelines as code. However, like any adventure, there are

From Kafka to Amazon S3: Partitioning Outputs

So you’ve got a ton of data that needs to be processed in real-time, huh? Don’t worry, in this tutorial I’ll show you how to stream data from Kafka compatible streaming platform to Amazon S3. But wait, there’s more! I’ll also cover how to create

How we mastered dbt: A true story

Staying sane in the modern world of data engineering is a non-trivial mission. As data engineers we ask ourselves the same question several times a day: is everything okay with my data? If you know what I’m talking about, you also know that, let’s be honest, our primary

PubSub to BigQuery: How to Build a Data Pipeline Using Dataflow, Apache Beam, and Java

I’ve recently worked on a project that required me to collect data from Google PubSub and load it into different BigQuery tables. I’ve faced many challenges during this process so I would like to share my experience building a complete data pipeline in Google Cloud Platform. Problem statement