7 Fundamental Concepts to Succeed With Distributed Computing in Python — Enter the Distributed Universe More and more data scientists are venturing into the world of distributed computing to scale up their computations and process larger datasets faster. But starting your distributed computing journey can feel a bit like entering an alternate universe: overwhelming, intimidating and confusing. But here’s the good news: you don’t need…

Identify Patterns in Time-Series Data with Overlapping Window Techniques — Windowing techniques enable data analysts to identify valuable patterns in time-series data. Sliding windows are particularly powerful because they allow you to spot patterns earlier than other techniques. This is an important feature in situations when making a key decision a few minutes (or seconds) earlier can save you money.

Why we need to talk more openly about privilege — This one’s going to be short and to the point, because I’m actually quite upset. I’m upset about privilege. Or, to be more precise, I’m upset about how little we talk about privilege in the data industry. A lot of us act like the tech industry is this golden land…

Everything you need to know about the recent changes — As of May 13, 2022, access to the NYC Taxi data has changed. Parquet has now become the new default file format, instead of CSV. Practically, this means you will need to change two things in your code: Change the path to the S3 bucket Use the dd.read_parquet() method instead…

Comparing Programming Languages for Data Science — This article compares Julia to Python in terms of general performance, package availability and adoption and gives guidance on whether you should consider learning it. Know Your Programming Languages for Data Science In 2021 Python achieved #1 ranking in the TIOBE Index of programming languages for the second year in a row. …

Strategies for Successful Distributed Computing in Python — Using Dask for the first time can be a steep learning curve. This post presents the 5 most common mistakes people make when using Dask — and strategies for how you can avoid making them. Let’s jump in. 1. “Dask is basically pandas, right?” The single-most important thing to do before starting to build things with…

Scale your Airflow pipelines to the cloud — Airflow DAGs for (Really!) Big Data Apache Airflow is one of the most popular tools for orchestrating data engineering, machine learning, and DevOps workflows. But it has one important drawback. Out-of-the-box, Airflow will run your computations locally, which means you can only process datasets that fit within the resources of your machine. To use Airflow for…

Read and Write Arrays Faster with Dask — tl;dr This post tells you why and how to use the Zarr format to save your NumPy arrays. It walks you through the code to read and write large NumPy arrays in parallel using Zarr and Dask. Here’s the code if you want to jump right in. If you have questions…

And why you should consider other file formats — This post explains how to write NumPy arrays to CSV files. We will look at: the syntax for writing different NumPy arrays to CSV the limitations of writing NumPy arrays to CSV alternative ways to save NumPy arrays Let’s get to it. Writing NumPy Arrays to CSV You can use the np.savetxt() method to save…