Dashboards with Jupyter
and Quarto

J.J. Allaire — Founder & CEO, Posit

Hello!

  • J.J. Allaire: Founder & CEO of Posit, PBC

  • RStudio Posit (July 2022)

  • Goal: Become a multi-language data science tools company

  • Early in the journey, but have made lots of investments already

  • Today I’ll share some of our work and talk about what’s next

Overview

  • Quarto Basics & Workflow

  • Introducing Quarto Dashboards

  • Posit and Python: History and Future

Quarto Basics

Quarto — https://quarto.org

An open-source scientific and technical publishing system that builds on standard markdown with features essential for scientific communication.

  • Pandoc Markdown

  • Jupyter Kernels

  • Dozens of Output Formats

  • Specialized Project Types

How does Quarto work?

  • Computations: Jupyter1
  • Markdown: Pandoc w/ many enhancements
  • Output: Documents, presentations, websites, books, blogs

Render Notebook to HTML (default options)

Render Notebook to HTML (document level options)

Render Notebook to HTML (document and cell level options)

Render Notebook to PDF https://quarto.org/docs/output-formats/pdf-basics.html

Render Notebook to Revealjs https://quarto.org/docs/presentations/revealjs/

Workflow

Render and Preview

Render to output formats:

# ipynb notebook
quarto render notebook.ipynb
quarto render notebook.ipynb --to docx

# plain text qmd
quarto render notebook.qmd 
quarto render notebook.qmd --to pdf

Live preview server (re-render on save):

# ipynb notebook
quarto preview notebook.ipynb
quarto preview notebook.ipynb --to docx

# plain text qmd
quarto preview notebook.qmd
quarto preview notebook.qmd --to pdf

Plain Text Notebooks w/.qmd Files

penguins.qmd
---
title: "Palmer Penguins"
author: Norah Jones
date: March 12, 2023  
format: html
jupyter: python3
---

```{python}
#| echo: false

import pandas as pd
df = pd.read_csv("palmer-penguins.csv") 
df = df[["species", "island", "year", \
         "bill_length_mm", "bill_depth_mm"]]
```

## Exploring the Data

See @fig-bill-sizes for an exploration of bill sizes.

```{python}
#| label: fig-bill-sizes
#| fig-cap: Bill Sizes by Species

import matplotlib.pyplot as plt
import seaborn as sns
g = sns.FacetGrid(df, hue="species", height=3)
g.map(plt.scatter, "bill_length_mm", "bill_depth_mm") \
  .add_legend()
```
  • Editable with any text editor (extensions for VS Code, Neovim, and Emacs)

  • Cells always run in the same order

  • Integrates well with version control

  • Cache output with Jupyter Cache or Quarto freezer

  • Lots of pros and cons visa-vi traditional .ipynb format/editors, use the right tool for each job

Rendering Pipeline

Notebook workflow (no execution occurs by default):

Plain text workflow (.qmd => .ipynb then execute cells):

Quarto Dashboards

A new output format for easily creating
dashboards from Jupyter Notebooks

Notebook ➝ Dashboard

 

Notebook ➝ Dashboard

:::

Notebook ➝ Dashboard

:::

Dashboard Components

  1. Navigation Bar and Pages — Icon, title, and author along with links to sub-pages (if more than one page is defined).

  2. Sidebars, Rows & Columns, and Tabsets — Rows and columns using markdown heading (with optional attributes to control height, width, etc.). Sidebars for interactive inputs. Tabsets to further divide content.

  3. Cards (Plots, Tables, Value Boxes, Content) — Cards are containers for cell outputs and free form markdown text. The content of cards typically maps to cells in your notebook or source document.

All of these components can be authored and customized within notebook UI or plain text qmd.

Layout: Rows

---
title: "Focal (Top)"
format: dashboard
---
    
## Row {height=70%}

```{python}
```

## Row {height=30%}

```{python}
```

```{python}
```

Layout: Columns

---
title: "Focal (Top)"
format: 
  dashboard:
    orientation: columns
---
    
## Column {width=60%}

```{python}
```

## Column {width=40%}

```{python}
```

```{python}
```

Tabsets

---
title: "Palmer Penguins"
format: dashboard
---
    
## Row

```{python}
```

## Row {.tabset}

```{python}
#| title: Chart 2
```

```{python}
#| title: Chart 3
```

Plots

plotly

```{python}
#| title: GDP and Life Expectancy
import plotly.express as px
df = px.data.gapminder()
px.scatter(
  df, x="gdpPercap", y="lifeExp", 
  animation_frame="year", animation_group="country",
  size="pop", color="continent", hover_name="country", 
  facet_col="continent", log_x=True, size_max=45, 
  range_x=[100,100000], range_y=[25,90]
)
```

ipyleaflet

```{python}
#| title: "World Map"
#| padding: 0px
from ipyleaflet import Map, basemaps, basemap_to_tiles
Map(basemap=basemap_to_tiles(basemaps.OpenStreetMap.Mapnik),
    center=(48.204793, 350.121558), zoom=2)
```

Tables

tabulate

```{python}
from tabulate import tabulate
from IPython.display import Markdown
Markdown(tabulate(penguins, showindex=False)}
```

itables

```{python}
from itables import show
show(penguins)
```

Value Boxes

## Row

```{python}
#| component: valuebox
#| title: "Current Price"
dict(icon = "currency-dollar",
     color = "secondary",
     value = get_price(data))
```

```{python}
#| component: valuebox
#| title: "Change"
change = get_change(data)
dict(value = change['amount'],
     icon = change['icon'],
     color = change['color']) 
```

Text Content

## Column

```{python}
#| title: Population
px.area(df, x="year", y="pop", 
        color="continent", 
        line_group="country")
```

```{python}
#| title: Life Expectancy
px.line(df, x="year", y="lifeExp", 
        color="continent", 
        line_group="country")
```

::: {.card}
Gapminder combines data from multiple sources
into unique coherent time-series that can’t be
found elsewhere. Learn more about the Gampminder
dataset at <https://www.gapminder.org/data/>.
:::

Expanding Cards

Cards provide an Expand button which appears at bottom right on hover:

Dashboard Deployment

Dashboards are typically just static HTML pages so can be deployed to any web server or web host.

Static Rendered a single time (e.g. when underlying data won’t ever change)
Scheduled Rendered on a schedule (e.g. via cron job) to accommodate changing data.
Parameterized Variations of static or scheduled dashboards based on parameters.
Interactive Fully interactive dashboard using Shiny (requires a server for deployment).

Parameterized Dashboards

Add a parameters tag to the first cell (based on papermill) :

```{python}
#| tags: [parameters]
ticker = "BA"
```

Use the -P command line option to vary the parameter:

quarto render dashboard.qmd -P ticker:GOOG

Interactive Dashboards

https://quarto.org/docs/dashboards/interactivity/shiny-python/

  • For interactive exploration, some dashboards can benefit from a live Python backend

  • To do this with Quarto Dashboards, add interactive Shiny components

  • Note that this requires a server for deployment

Hello, Shiny

---
title: "Penguin Bills"
format: dashboard
server: shiny
---

```{python}
import seaborn as sns
penguins = sns.load_dataset("penguins")
```

## {.sidebar}

```{python}
from shiny import render, ui
ui.input_select("x", "Variable:",
                choices=["bill_length_mm", "bill_depth_mm"])
ui.input_select("dist", "Distribution:", choices=["hist", "kde"])
ui.input_checkbox("rug", "Show rug marks", value = False)
```

## Column

```{python}
@render.plot
def displot():
    sns.displot(
        data=penguins, hue="species", multiple="stack",
        x=input.x(), rug=input.rug(),kind=input.dist())
```

Shiny Deployment

Shiny for Python applications are built on Starlette and ASGI, and can deployed in server environments that support WebSockets and sticky sessions.

Alternatively, deploy serverless using Pyodide. See the Retirement Simulation example for details.

Posit and Python

Posit, PBC

Founded 14 years ago to create open source software for data science.

Posit is a Public Benefit Corporation with a mission to create open source software for data science, scientific research, and technical communication.

This is built into our charter, and our directors and officers have a fiduciary duty to pursue these public benefits along with balancing the needs of all our stakeholders.

Posit Incentives

  • Posit is an independent company and is committed to always being one

  • Corporate control lies within the company (not with outside investors)

  • Our imperative is not growth at all costs but rather to build something that is organically sustainable and still here fulfilling its mission in 100 years

  • Our core aspiration is to be a durable, trustworthy provider of open source software for science.

PyData Projects

quarto Scientific and technical publishing
quartodoc Python package documentation
shiny Interactive PyData web applications
vetiver Deploy and monitor ML models
suiba Data manipulation for pandas, duckdb, etc.
plotnine Grammer of graphics for Python
  • Principle Corporate Sponsor of NumFOCUS

  • Working on standards for scientific notebook publishing in NotebooksNow

Learning More

Quarto v1.4 Pre-Release

https://quarto.org/docs/download/prerelease.html

pip install git+https://github.com/quarto-dev/quarto-cli

Resources

Slides

https://jjallaire.github.io/pydata-quarto-dashboards/