9. Data aggregation

The aggregation pipeline lets you transform and combine documents. It is like doing a GROUP BY in SQL, but much more powerful.

Pipeline structure

An aggregation is an array of stages. Each stage transforms the documents and passes them to the next one:

db.unicorns.aggregate([
    { $match: { ... } },    // Stage 1: filter
    { $group: { ... } },    // Stage 2: group
    { $sort: { ... } }      // Stage 3: sort
])

The $group stage

The most important stage is $group. It groups documents and computes values:

// Count unicorns by gender
db.unicorns.aggregate([
    {
        $group: {
            _id: '$gender',
            total: { $sum: 1 }
        }
    }
])

Result:

[
    { _id: 'f', total: 4 },
    { _id: 'm', total: 4 }
]

The _id field indicates which field to group by. The $ symbol before a field name means "the value of this field".

Accumulation operators

Inside $group you can use these operators:

$sum: sums values

db.unicorns.aggregate([
    {
        $group: {
            _id: '$gender',
            totalWeight: { $sum: '$weight' }
        }
    }
])

$avg: calculates the average

db.unicorns.aggregate([
    {
        $group: {
            _id: '$gender',
            avgVampires: { $avg: '$vampires' }
        }
    }
])

$min: minimum value

db.unicorns.aggregate([
    {
        $group: {
            _id: '$gender',
            minWeight: { $min: '$weight' }
        }
    }
])

$max: maximum value

db.unicorns.aggregate([
    {
        $group: {
            _id: '$gender',
            maxVampires: { $max: '$vampires' }
        }
    }
])

$addToSet: array with unique values

db.unicorns.aggregate([
    {
        $group: {
            _id: '$gender',
            names: { $addToSet: '$name' }
        }
    }
])

The $match stage

Filters documents before grouping. It is like find:

// Average weight of unicorns under 600
db.unicorns.aggregate([
    { $match: { weight: { $lt: 600 } } },
    {
        $group: {
            _id: '$gender',
            avgWeight: { $avg: '$weight' }
        }
    }
])

The $sort stage

Sorts the results:

db.unicorns.aggregate([
    {
        $group: {
            _id: '$gender',
            total: { $sum: 1 }
        }
    },
    { $sort: { total: -1 } }
])

The $limit stage

Limits the number of results:

// Top 3 genders with the most unicorns
db.unicorns.aggregate([
    {
        $group: {
            _id: '$gender',
            total: { $sum: 1 }
        }
    },
    { $sort: { total: -1 } },
    { $limit: 3 }
])

The $unwind stage

Breaks down an array into separate documents. Very useful for analyzing arrays:

// Which food is the most popular?
db.unicorns.aggregate([
    { $unwind: '$loves' },
    {
        $group: {
            _id: '$loves',
            count: { $sum: 1 }
        }
    },
    { $sort: { count: -1 } },
    { $limit: 1 }
])

Complete example

Let's find the favorite food of the heavy unicorns (over 500):

db.unicorns.aggregate([
    // 1. Filter heavy unicorns
    { $match: { weight: { $gt: 500 } } },

    // 2. Break down the array of foods
    { $unwind: '$loves' },

    // 3. Group by food and count
    {
        $group: {
            _id: '$loves',
            total: { $sum: 1 },
            unicorns: { $addToSet: '$name' }
        }
    },

    // 4. Sort by popularity
    { $sort: { total: -1 } },

    // 5. Only the top 1
    { $limit: 1 }
])
Activity 1
  1. Calculate the total number of listings per neighborhood (neighbourhood_group_cleansed)
  2. Calculate the number of listings per room type (room_type)
  3. Find how many superhosts there are in total (count where host_is_superhost is "t")
  4. Count how many listings there are per property type (property_type), showing only the 10 most common
  5. Calculate how many listings have more than 50 reviews

Pro:

  1. Calculate the average of accommodates (capacity) per neighborhood, sorted from highest to lowest
  2. Find the top 5 neighborhoods with the most listings (group by neighbourhood_group_cleansed, count and sort)
  3. Calculate the maximum number of reviews (number_of_reviews) for each room type (room_type)

Solutions

Building SPAs with Django and HTML Over the Wire: Learn to build real-time single page applications with Python

Building SPAs with Django and HTML Over the Wire: Learn to build real-time single page applications with Python

The HTML over WebSockets approach simplifies single-page application (SPA) development and lets you bypass learning a JavaScript rendering framework such as React, Vue, or Angular, moving the logic to Python. This web application development book provides you with all the Django tools you need to simplify your developments with real-time results.

Buy the book

Help me keep writing

Every coffee gives me a push toward the next article.

Comments

There are no comments yet.