Why we need to talk more openly about privilege — This one’s going to be short and to the point, because I’m actually quite upset. I’m upset about privilege. Or, to be more precise, I’m upset about how little we talk about privilege in the data industry. A lot of us act like the tech industry is this golden land…

Everything you need to know about the recent changes — As of May 13, 2022, access to the NYC Taxi data has changed. Parquet has now become the new default file format, instead of CSV. Practically, this means you will need to change two things in your code: Change the path to the S3 bucket Use the dd.read_parquet() method instead…

Comparing Programming Languages for Data Science — This article compares Julia to Python in terms of general performance, package availability and adoption and gives guidance on whether you should consider learning it. Know Your Programming Languages for Data Science In 2021 Python achieved #1 ranking in the TIOBE Index of programming languages for the second year in a row. This should come as no…

Strategies for Successful Distributed Computing in Python — Using Dask for the first time can be a steep learning curve. This post presents the 5 most common mistakes people make when using Dask — and strategies for how you can avoid making them. Let’s jump in. 1. “Dask is basically pandas, right?” The single-most important thing to do before starting to build things with…

Scale your Airflow pipelines to the cloud — Airflow DAGs for (Really!) Big Data Apache Airflow is one of the most popular tools for orchestrating data engineering, machine learning, and DevOps workflows. But it has one important drawback. Out-of-the-box, Airflow will run your computations locally, which means you can only process datasets that fit within the resources of your machine. To use Airflow for…

Read and Write Arrays Faster with Dask — tl;dr This post tells you why and how to use the Zarr format to save your NumPy arrays. It walks you through the code to read and write large NumPy arrays in parallel using Zarr and Dask. Here’s the code if you want to jump right in. If you have questions…

And why you should consider other file formats — This post explains how to write NumPy arrays to CSV files. We will look at: the syntax for writing different NumPy arrays to CSV the limitations of writing NumPy arrays to CSV alternative ways to save NumPy arrays Let’s get to it. Writing NumPy Arrays to CSV You can use the np.savetxt() method to save…

Machine Learning that Scales to Very Large Datasets — tl;dr Out of the box, XGBoost cannot be trained on datasets larger than your computer memory; Python will throw a MemoryError. This tutorial will show you how to go beyond your local machine limitations by leveraging distributed XGBoost with Dask with only minor changes to your existing code. Specifically, you will…