xboard.dev

Tip: set depends_on_past=True in Airflow when creating a forecasting model pipeline

Flavio Regis de Arruda — Thu, 27 Jul 2023 03:00:00 GMT

Tip

When creating forecasting model pipeline in Airflow set depends_on_past=True.

Why?

Forecasting models help us predict future data based on patterns and trends observed in historical data. These models output inherently depend on past observations/data, hence any deviation from the expected sequence can affect their accuracy.

One way to ensure the integrity of the sequence of data while creating forecasting models in Apache Airflow is to set the depends_on_past argument to ‘True’. This way each subsequent task in the DAG will only run once its preceding task has completed successfully. This approach guarantees that the forecasting model is fed with the correct inputs in the correct order and that the historical data is correctly sequenced.

If you don’t set depends_on_past=True and you have a problem with one of the partitions of the data used by your model, one of two things will happen in the following days:

Worst case scenario: your workflow will run without any issues being reported. This is a sign that you forgot to add sensors to test the availability of historical data and, chances are, your features are using incomplete data, to say the least.
Alternatively, subsequent executions of your workflow will also fail, bombarding your on call colleague’s PagerDuty with messages. And when the problem is finally fixed, you still have to remember to clear and restart all the previous failed executions.

How?

For a specific task, most probably a sensor:

task = SomeSensor(
    task_id='task',
    depends_on_past=True,
    # ...
)

or in your dag default_args to enable depends_on_past for all your tasks

default_args = {
    'depends_on_past': True,
    # ...
}

dag = DAG(
    dag_id="time_series_pipeline",
    default_args=default_args,
    # ...
)

Using pip-tools to manage project dependencies in Python

Flavio Regis de Arruda — Thu, 20 Jul 2023 03:00:00 GMT

Improve Your Python Dependency Management with pip-tools

As an engineer who loves to solve problems using Python and creates tens of projects by year, keeping track of different packages and their versions can be complex. I often have to test and deploy my projects in different environments (development/testing/production machines) and on different Cloud or PaaS providers and need to be sure that all of them will use exactly the same Python packages and versions as I have used in the development, to be sure no problems were introduced by an unexpected package upgrade.

A naive approach would be to install the packages you need using pip and at the end generate a requirements.txt file with

pip freeze > requirements.txt

to persist all dependencies (with their versions) that got installed.We can later install into an empty virtual environment using that requirements file gets us the same packages and versions.

But this approch is problematic beause is virtually impossible to know which packages are needed by our application and which were pulled in as dependencies:

suppose I choose to switch from pandas to polars in my project. In that scenario, the requirements.txt file may have many pandas dependencies that are superfluous to polars. Unfortunately, we are uncertain of who they are.
if you need to upgrade some of the packages (i.e Django version from 3.2 to 4.2), which Django dependencies need to be upgrated and to which version? Which new dependencies the new Django version have?

That is why I use the pip-tools project to simplify the process of package dependency management. In this blog post, we will discuss what pip-tools does, the problem it solves and why developers should consider using it, and provide five examples using pip-tools for managing Python project dependencies. Additionally, we will explore some alternatives to pip-tools.

What is pip-tools?

pip-tools is an open-source project created to simplify the requirements files management used in Python projects. It takes a project’s dependencies and recursively generates pinned version requirement files.

Moreover, pip-tools allows you to easily manage your project dependencies with minimum effort and ensures you have a reproducible development environment. It allows other developers that are working on the same project to have identical dependencies to yours, with no version conflicts.

This can help reduce time spent debugging issues related to package conflicts and ensure that code runs consistently across different environments.

It streamlines the generation and maintenance of requirements files
It ensures that all packages are pinned to a specific version to avoid version conflicts
It provides an easier way to manage different environments, like production, development, and testing
It permits simple package updating and upgrading as new versions become available

Example of using `pip-tools`

Here are five examples of how developers can use ppip-tools to manage the package requirements of their Python projects:

Basic usage of pip-tools with requirements files

Developers can use pip-tools to manage their project dependencies by creating a requirements.in file with the required packages and their respective versions.

Listing 1: requirements.in

pandas>=2.0
scikit-learn
xgboost==1.7.6

Developers can then run the following command to generate the required requirements.txt file:

$ pip-compile requirements.in

This generates a pinned version requirement file with all the packages and their respective dependencies:

Listing 2: Generated requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile requirements.in
#
joblib==1.3.1
    # via scikit-learn
numpy==1.25.1
    # via
    #   pandas
    #   scikit-learn
    #   scipy
    #   xgboost
pandas==2.0.3
    # via -r requirements.in
python-dateutil==2.8.2
    # via pandas
pytz==2023.3
    # via pandas
scikit-learn==1.3.0
    # via -r requirements.in
scipy==1.11.1
    # via
    #   scikit-learn
    #   xgboost
six==1.16.0
    # via python-dateutil
threadpoolctl==3.2.0
    # via scikit-learn
tzdata==2023.3
    # via pandas
xgboost==1.7.6
    # via -r requirements.in

pip-tools with multiple environments

We can define several files with libraries to use in multiple environments, such as development, production, and testing. We can create corresponding dev-requirements.in, prod-requirements.in, and test-requirements.in files and then use the following commands to generate their respective files:

$ pip-compile dev-requirements.in
$ pip-compile prod-requirements.in
$ pip-compile test-requirements.in

These commands will generate dev-requirements.txt, prod-requirements.txt, and test-requirements.txt files with the corresponding dependencies.

pip-tools with custom package indexes

We can also use pip-tools with package indexes different from the official PyPI index. To do this, we can specify a custom index in their requirements.in file, like this:

--index-url https://custompackageindex.com/
django==3.2.5

Then, running pip-compile requirements.in will output a requirements.txt file with the packages pinned to versions on the custom package index.

pip-tools with hash dependencies

If you want to pin your dependencies to an specific binary wheel compilation hash (and not only to a package-version) you can use following command:

$ pip-compile --generate-hashes requirements.in

This will add a hash value to each package in the requirements file, including the transitive dependencies. This is usefull to increase security as PyPI has “made the deliberate choice to allow wheel files to be added to old releases”

Show requirements.txt file with generated hashes

Migrating my blog from fastpages to Quarto

Flavio Regis de Arruda — Mon, 17 Jul 2023 03:00:00 GMT

Why migrate?

In a post published on September 3, 2022, Hamel Husain announced that he has rebuilt nbdev on top of Quarto. In the blog post, he explained that there have been new pieces of technology like Jupyter Book and Quarto, which are both fabulous. Husain explained that he ended up using Quarto for nbdev, because of its shared vision with nbdev and the fantastic ongoing support from JJ Allaire and the rest of the Posit team. He also recommended that people trying to blog with notebooks should be using Quarto, instead of Fastpages.

What is Quarto?

Quarto is an open-source, web-based platform designed to simplify the creation and publication of scientific documents, including data-driven reports, technical articles, and more, with features such as:

Compatibility with a wide range of programming languages, including R, Python, and Julia;
Integration with popular web publishing platforms such as GitHub pages, Netlify among others;
Support for a broad range of document outputs, including LaTeX, PDF, and HTML;
Extensive customizability with built-in templates and reusable components; and
Built-in support for code snippets and data visualizations.

Installing Quarto

Go to the get started page on the quarto website to download the installer file. I’m on Ubuntu, so for me that’s a .deb file. I install it from the command line:

sudo dpkg -i quarto-1.3.433-linux-amd64.deb

Create my blog project, called quarto-blog:

quarto create-project quarto-blog --type website:blog

then enter the project and preview it:

cd quarto-blog/
quarto preview quarto-blog

Generated quarto blog preview

Migrating fastpages’s posts to Quarto

Follow instructions in nbdev’s Migrating from Fastpages tutorial.

In my case:

cd quarto-blog
cp -r ../fastpages-blog/_notebooks/* posts
cp -r ../fastpages-blog/_posts/* posts
cp ../fastpages-blog/images/* posts
cp -r ../fastpages-blog/images/copied_from_nb/* posts
nbdev_migrate --path posts

Setup general blog configuration

Rename blog title, description and add favicon

Edited section website in file _quarto.yml updating title and description and then adding site-url, favicon and google analytics configuration:

_quarto.yml

website:
  title: "xboard.dev"
  favicon: favicon.ico
  site-url: https://xboard.dev
  description: "Everything data and programming."
  google-analytics: "G-5FL5R0TTQ6"

Setup navigation menu and footer

Maintained About page and added github, twitter and rss buttons in the right section of the navigation bar and centered content license in the footer.

_quarto.yml

website:
  navbar:
    right:
      - about.qmd
      - icon: github
        href: https://github.com/xboard
      - icon: twitter
        href: https://twitter.com/xboard_
      - icon: rss
        href: index.xml
  page-footer:
    center: |
      Made with Data, ❤️, 🐍 and [Quarto](https://quarto.org). License: [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0/).

Setup social metadata

To enables richer sharing of links to articles on the web improving the previews of my content when a link to it is pasted into applications like Twitter, Slack, Discord, Facebook, Linkedin, and more.

_quarto.yml

website:
  twitter-card:
    creator: "@xboard_"
    card-style: summary_large_image
  open-graph: true

Setup themes for light and dark mode

After playing a bit with some of the default themes available I have chosen litera and superhero for light and dark modes respectively.

_quarto.yml

theme:
  light: litera
  dark: superhero
css: styles.css

Light mode with litera theme

Dark mode with superhero theme

Adding a `robots.txt` policy file

My final robots.txt file ended up as below:

robots.txt

User-Agent: *
Allow: /


User-agent: GPTBot
Disallow: /


User-agent: ChatGPT-User
Disallow: /

Sitemap: https://www.xboard.dev/sitemap.xml

It advise Open AI to not copy my site into their text database product and provide a reference to where other crawlers can find my generated sitemap.

Adding `canonical` tag into the section

I have followed this post from Matt Bowers adding the following python script:

add-canonicals.py

"""
This script adds canonical url tags to pages in a Quarto website.
Place the script in the root of the project and run it post render.

author: Matt Bowers - https://github.com/mcb00
from https://github.com/quarto-dev/quarto-cli/discussions/3976#discussioncomment-6376014
"""

from xml.dom.minidom import parse
from urllib.parse import urlparse
import warnings

site_dir = '_site'
sitemap_file = 'sitemap.xml'

document = parse(site_dir + '/' + sitemap_file)
locs = document.getElementsByTagName('loc')
urls = [l.firstChild.nodeValue for l in locs]

for url in urls:
    
    path = site_dir + urlparse(url).path
    canonical_tag = f'{url}" />'

    # Read in the file
    with open(path, 'r') as file :
      filedata = file.read()

    if filedata.__contains__('):
        warnings.warn(f'{path} already contains canonical tag. Skipping this file.')
    else:
        print(f'{path} adding canonical tag.')
        # Replace the target string
        filedata = filedata.replace('', canonical_tag +'\n')

        # Write the file out again
        with open(path, 'w') as file:
          file.write(filedata)

and them add a post-render configuration pointing to this script in my project:

_quarto.yml

project:
  post-render:
    - "python add-canonicals.py"

Git/Github

Create `.gitignore` file

Created .gitignore file with the following content:

.gitignore

/.quarto/
/_site/
/_book/
/_freeze/
.venv/

Initialize Git and upload to repository in Github

Initialize local repository:

git init
git add .
git commit -m "Migrated blog: first commit"

Then go to Github and create a new public repository named quarto-blog and in your local repository directory type in the terminal:

git remote add origin https://github.com/xboard/quarto-blog.git
git push -u -f origin master

Setup github pages

I have followed this article from the official quarto documentation on how to publish you content to a gh-pages branch.

My old blog was running in github-pages and I want to keep using it.

Add custom domain to github pages

First remove the domain in repository fastpages-blog

Remove domain from old blog repository

then add domain to quarto-blog repository github page.

Publish to github pages

Just type in project root:

quarto publish gh-pages --no-prompt

This will render your blog’s html in folder _site and publish its content into github pages. The --no-prompt option prevents it from asking for confirmation.

After the publish command succeeds deploying to github-pages it opens the browser to view your blog.

Creating Conda’s environment

Quarto supports executable Python code blocks within markdown. This allows you to create fully reproducible documents and reports—the Python code required to produce your output is part of the document itself, and is automatically re-run whenever the document is rendered.

Therefore it is crucial to have a reproducible Python execution environment and for such I have chosen Miniconda. This are the steps I did to create my virtual environment and to save its configuration in github.

Create and activate the environment

conda create -n 'quarto_env'
conda activate quarto_env

Install data-science libraries

conda install pandas seaborn matplotlib plotly pip scrapy statsmodels jupyter jupyterlab xgboost scikit-learn seaborn lightgbm "pymc>=5" jax[cpu] tsfresh sktime prophet gradio plotly  scipy statsmodels arviz xarray pytorch fastai  streamlit lime shap interpret yellowbrick pypdf langchain accelerate bitsandbytes transformers scrapy

Export all libraries installed in environment to environment.yml file

conda env export --name quarto_env --file environment.yml

Add and push to github

git add environment.yml
git commit -m 'Add conda environment.yml'

You can see the generated environment.yml file in github here.

Default configuration for blog posts

This setup properties that will be inherited by all posts.

posts/_metadata.yml


# options specified here will apply to all posts in this folder

# freeze computational output
# (see https://quarto.org/docs/projects/code-execution.html#freeze)
1freeze: true

# Enable banner style title blocks
2title-block-banner: true

# Default for table of contents
toc: true
3toc-title: Contents
toc-location: left
toc-expand: 2

# Default author
author:
  - name: Flavio Regis de Arruda
    url: https://xboard.dev

# Default fields for citation
4citation: false

1: computational documents should never be re-rendered during a global project render
2: create a banner style title block
3: table of content title
4: do not show citation metadata.

Enabling commenting

I have chosen Giscus app to leverage Github Discussions as commenting platform in my posts.

_quarto.yml

website:
  comments:
    giscus:
1      repo: xboard/quarto-blog
2      repo-id: R_kgDOJ7r2SQ
3      category: Announcements
4      category-id: DIC_kwDOJ7r2Sc4CX95f
      loading: lazy

1: the Github repo that will be used to store comments.
2: found using https://giscus.app/
3: discussion category in Github Discussions
4: found using https://giscus.app/

after this at the bottom of every post you will see the Giscus integration in action:

Giscus widget for commenting and reactions

Setup footer

Footer message to be displayed (centered) in all my pages:

_quarto.yml

website:
  page-footer:
    center: |
      Made with Data, ❤️, 🐍 and [Quarto](https://quarto.org). License: [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0/).

VSCode

I love to use Visual Studio Code to write my blogs posts and I was very pleased to discover it has an excellent Quarto extension.

Installing VS Code Quarto extension

To install the Quarto extension type (Ctrl+P), type:

ext install quarto.quarto

and press enter.

Some of the features this extensions includes:

Integrated render and preview for Quarto documents.
Syntax highlighting for markdown and embedded languages
Completion and diagnostics for YAML options
Completion for embedded languages (e.g. Python, R, Julia, etc.)
Commands and key-bindings for running cells and selected lines.
Live preview for LaTeX math as well as Mermaid and Graphviz diagrams

Adding snippet

Enabling inline suggestion

Set Snippet suggestions to inline as recommended here.

Adding spell checker

Installed Code Spell checker.

and them after clicking the spell button at the button at the editor bottom right I enabled its support to Quarto on my workspace.

Enabling Quarto for Code Spell Checker

Conclusion

In summary, Quarto provides me with a powerful and versatile platform to write my technical blog posts as a Data Scientist and Python programmer. Its integration with Jupyter notebooks, support for multiple programming languages, customization options, collaboration features, and rendering capabilities make it the ideal choice for me to create high-quality, detailed, and engaging technical articles. With Quarto, I can focus on delivering content while enjoying a seamless and efficient writing experience.

In this post I have described in details all the steps that I did to migrated my posts from fastpages and how I configured Quarto to support the blog you are reading now.

If you were also using fastpages and intend to migrate, I hope it not only helps you in your decision process but also speeds up the setup of your new Quarto blog.

Interrupted Time Series (ITS) in Python

Flavio Regis de Arruda — Sat, 01 Jan 2022 09:00:00 GMT

When A/B test is not an option

The gold standard for statistically asserting the effectiveness of an intervention is the randomized controlled experiment and its simplified online variant: the A/B test.

📝 During an A/B test there are two almost identical versions of a product, simultaneously running, that only differ by the hypothesis you want to test ( i.e can a red call to action button convert more than a blue one? ). Users are randomly chosen to experience one (and only one) of the two versions while the experiment is active.

They are easy to understand, easy to setup (great free tools easily available) and when correctly designed they rule out any covariate differences between the groups.

However, sometimes it’s just not possible to set up an A/B test:

Technical difficulties. Sometimes a change is so widespread and complex that it would be technically impossible to keep two different versions running simultaneously.
Business strategy. A new feature rollout will be available first to some countries and later for others.
Ethical concerns. Having a subset of customers having access to a feature or bug fix that gives them a competitive advantage over others that don’t.
Legal or regulatory requirements. A change in regulations becomes mandatory ( i.e. GPDR compliance ) and should be applied to all your customers of a given country at the same time.
Temporal infeasibility. You want to analyze an event that has already happened ( i.e. How last Google’s search algorithm update impacted your sales funnel? ).

Quasi Experiments

If you can’t do an A/B test then the second to best alternative are quasi experiments [1].

In a quasi experiment, your treatment and control group are not divided by a completely random process but by a natural process (i.e. time, location, etc) therefore there is a much larger chance for imbalance due to skewness and heterogeneous differences. The results of a quasi-experiment won’t be as precise as an A/B, but if carefully conducted could be considered close enough to compute estimates.

There are some scenarios, like some described in the previous section, where having a control group in parallel to a test group is just not possible, and this is when Interrupted Times Series comes in very handy.

Interrupted Time Series (ITS)

Interrupted time series (ITS) is a method of statistical analysis involving tracking a period before and after a intervention at a known point in time to assess the intervention’s effects within a single group/population. The time series refers to the data over the period, while the interruption is the intervention, which is a controlled external influence or set of influences. Effects of the intervention are evaluated by changes in the level and slope of the time series and statistical significance of the intervention parameters [2]. The more observations you have before and after the intervention, the more robust your model will be (typically). Because the evaluation is based on observing a single population over time, the ITS design is free from problems due to between-group difference but are susceptible to time-varying confounders like other interventions occurring around the time of the intervention that may also affect the outcome [3].

👍 Strengths of Interrupted Time Series include the ability to control for secular trends in the data (unlike a 2-period before-and-after -test), ability to evaluate outcomes using population-level data, clear graphical presentation of results, ease of conducting stratified analyses, and ability to evaluate both intended and unintended consequences of interventions.

👎 Limitations of Interrupted Time Series include the need for a minimum of 8 time periods before and 8 after an intervention to evaluate changes statistically, difficulty in analyzing the independent impact of separate components of a program that are implemented close together in time, and existence of a suitable control population.

In mathematical terms, it means that the time series equation includes four key coefficients:

Where:

is the outcome variable;

is a continuous variable which indicates the time passed from start of the observational period;

is a dummy variable indicating observation collected before () or after () the intervention;

is a continuous variable indicating time passed since the intervention has occurred (before intervention has occurred is equal to );

With representing a zero centered gaussian random error.

Counterfactual

What would have happened had Neo chosen the blue pill?

In an ITS it is important to understand the counterfactual. The counterfactual refers to what it would have occurred to Y, had the policy intervention not happened.

📝Counterfactuals are simply ways of comparing what happens given a change, versus what should have happened had some change not occurred in the first place.

In a randomized trial or A/B test we know the counterfactual average outcome because the experiment withheld the intervention from the control group (which by randomization is somewhat the same as the intervention group). A critical assumption in ITS is that the outcome of interest trend would remain unchanged in the absence of intervention.

A practical example

Bob runs a large and successful blog on personal finance. During a webinar he learns that making his web content load faster could reduce its bounce rate and therefore decides to sign up for a CDN service. It’s been 6 months since he added a CDN to his blog and he wants to know if the investiment he did reduced the bounce rate.

Dataset

Bob provides us with 💾 24 weeks of data before adding the CDN and 24 weeks after it (intervention). Therefore, weeks 1 to 24 have a bouncing rate before intervention and weeks 25 to 48 after it.

Visually, it looks like after enabling the CDN the bounce rate decreased, but by how much, and does it have statistical significance? To answer this question using interrupted time series analysis, we first need to prepare our data.

Dataset preparation

Using equation (1) notation we 💾 enrich this data with values for columns ( = before intervention, after) and (number of weeks since intervention started):

Bouncing rate (Y)	Week (T)	Intervention (D)	Intervention week (P)
12.92	1	0	0
13.03	2	0	0
13.06	3	0	0
13.17	4	0	0
…	…	…	…
12.04	45	1	21
12.45	46	1	22
12.74	47	1	23
12.57	48	1	24

Naive solution

Let’s implement an ordinary least squares (OLS) regression using statsmodels to measure the impact of our intervention:

import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf

df = pd.read_csv("enriched_data.csv")

model = smf.ols(formula='Y ~ T + D + P', data=df)
res = model.fit()
print(res.summary())

With output:

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      Y   R-squared:                       0.666
Model:                            OLS   Adj. R-squared:                  0.643
Method:                 Least Squares   F-statistic:                     29.18
Date:                Tue, 28 Dec 2021   Prob (F-statistic):           1.52e-10
Time:                        14:33:50   Log-Likelihood:                 4.8860
No. Observations:                  48   AIC:                            -1.772
Df Residuals:                      44   BIC:                             5.713
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
Intercept     12.9100      0.096    134.225      0.000      12.716      13.104
T              0.0129      0.007      1.920      0.061      -0.001       0.026
D             -0.5202      0.132     -3.942      0.000      -0.786      -0.254
P             -0.0297      0.010     -3.115      0.003      -0.049      -0.010
==============================================================================
Omnibus:                        3.137   Durbin-Watson:                   0.665
Prob(Omnibus):                  0.208   Jarque-Bera (JB):                1.995
Skew:                           0.279   Prob(JB):                        0.369
Kurtosis:                       2.172   Cond. No.                         125.
==============================================================================

The model estimates that the bounce rate decreased 🔻 0.52% and this effect is statistically significant ( is virtually zero).

It is also noteworth that the model estimates a small (on average 🔻 0.0297%) but with statistical significance trend of a decrease in bounce rate each week after intervention, which is unexpected since the CDN serves the whole website just a few hours after activation.

The figure below depicts how the model fits before and after intervention and how it project a counterfactual would be:


start = 24
end = 48
beta = res.params

# Get model predictions and 95% confidence interval
predictions = res.get_prediction(df)
summary = predictions.summary_frame(alpha=0.05)

# mean predictions
y_pred = predictions.predicted_mean

# countefactual assumes no interventions
cf_df = df.copy()
cf_df["D"] = 0.0
cf_df["P"] = 0.0

# counter-factual predictions
cf = res.get_prediction(cf_df).summary_frame(alpha=0.05)

# Plotting
plt.style.use('seaborn-whitegrid')
fig, ax = plt.subplots(figsize=(16,10))

# Plot bounce rate data
ax.scatter(df["T"], df["Y"], facecolors='none', edgecolors='steelblue', label="bounce rate data", linewidths=2)

# Plot model mean bounce rate prediction
ax.plot(df["T"][:start], y_pred[:start], 'b-', label="model prediction")
ax.plot(df["T"][start:], y_pred[start:], 'b-')

# Plot counterfactual mean bounce rate with 95% confidence interval
ax.plot(df["T"][start:], cf['mean'][start:], 'k.', label="counterfactual")
ax.fill_between(df["T"][start:], cf['mean_ci_lower'][start:], cf['mean_ci_upper'][start:], color='k', alpha=0.1, label="counterfactual 95% CI");

# Plot line marking intervention moment
ax.axvline(x = 24.5, color = 'r', label = 'intervention')

ax.legend(loc='best')
plt.ylim([10, 15])
plt.xlabel("Weeks")
plt.ylabel("Bounce rate (%)");

Problems with naive approach

OLS (Ordinary Least Squares) regression has seven main assumptions but for brevity in this article we will focus on two only:

Individual observations are independent.
Residuals follow a normal distribution.

Let’s first check for the normality of residuals:

We can apply the Jarque-Bera test on residuals to checks whether their skewness and kurtosis match a normal distribution (: residual distribution follows a normal distribution). Our statsmodels OLS summary output shows a Prob(JB): 0.369 which for a standard level of 0.05 doesn’t allow us discard null hypothesis ().

Let’s plot the distribution of residuals:

    res.resid.plot(kind="kde")

Which for a small dataset (less than 50 points) looks sufficiently gaussian.

Overall, the assumption of normality of residuals can’t be convincingly refuted. ✅

Checking independence of observations:

The Durbin-Watson statistic test if the residuals are correlated with its immediate predecessor, that is, if they have an autocorrelation at lag 1 or . Its value ranges from 0 to 4 and values smaller than 1.5 indicate a positive autocorrelation, while values greater than 2.5 signal a negative autocorrelation.

If we take a look again at our OLS summary output we will observe that the Durbin-Watson statistic has a value of 0.665 which signals a strong positive .

Let’s plot the residuals to see if we can observe this autocorrelation:

import altair as alt

rules = alt.Chart(pd.DataFrame({
  'residuals': [0.0],
  'color': ['black']
})).mark_rule().encode(
  y='residuals',
  color=alt.Color('color:N', scale=None)
)

residual_plot = alt.Chart(res_df).mark_point().encode(
    x=alt.X('Weeks'),
    y=alt.Y('residuals')
)

rules + residual_plot

Notice how residuals above/below zero have most points temporally close to it also above/below zero as well, which goes against the independence of observations assumption of OLS ❌.

📝In practice when analyzing time series data the presence of autocorrelation is the rule instead of the exception since in general the factors that contributed to a given observation tend to persist for a while.

Autoregressive model solution

The autoregressive model specifies that each observation depends linearly on previous observations.

Thus, an autoregressive model of order () can be written as

Where:

: observation at time ,

: coefficient of how much observation correlates to ,

: white noise ( ) at time .

Autocorrelation

To assess how much an observation correlates with past observations it is useful to do an autocorrelation plot as shown below:

sm.graphics.tsa.plot_acf(res.resid, lags=10)
plt.show()

Partial Autocorrelation

The partial autocorrelation at lag is the correlation that results after removing the effect of any correlations due to the terms at shorter lags.

sm.graphics.tsa.plot_pacf(res.resid, lags=10)
plt.show()

Model selection

The theory states that in an autoregressive model its autocorrelation plot should depict an exponential decay and the number of lags should be taken from the partial autocorrelation chart using its most relevant lags. Applying the theory to our plots above, we conclude that our model is autoregressive of lag 1 also known as AR(1).

ARIMA

In statistics ARIMA stands for autoregressive integrated moving average model and as can be inferred by the name AR models are as especial case of ARIMA therefore AR(1) is equivalent to ARIMA(1,0,0).

We can model an AR(1) process to our dataset using statsmodels ARIMA as below:

from statsmodels.tsa.arima.model import ARIMA

arima_results = ARIMA(df["Y"], df[["T","D","P"]], order=(1,0,0)).fit()
print(arima_results.summary())

Output:

                               SARIMAX Results                                
==============================================================================
Dep. Variable:                      Y   No. Observations:                   48
Model:                 ARIMA(1, 0, 0)   Log Likelihood                  18.574
Date:                Thu, 30 Dec 2021   AIC                            -25.148
Time:                        01:51:46   BIC                            -13.921
Sample:                             0   HQIC                           -20.905
                                 - 48                                         
Covariance Type:                  opg                                         
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
const         12.9172      0.279     46.245      0.000      12.370      13.465
T              0.0121      0.016      0.767      0.443      -0.019       0.043
D             -0.5510      0.273     -2.018      0.044      -1.086      -0.016
P             -0.0238      0.021     -1.155      0.248      -0.064       0.017
ar.L1          0.6635      0.138      4.803      0.000       0.393       0.934
sigma2         0.0267      0.006      4.771      0.000       0.016       0.038
===================================================================================
Ljung-Box (L1) (Q):                   1.00   Jarque-Bera (JB):                 0.15
Prob(Q):                              0.32   Prob(JB):                         0.93
Heteroskedasticity (H):               1.44   Skew:                            -0.05
Prob(H) (two-sided):                  0.47   Kurtosis:                         3.25
===================================================================================

The autoregressive model estimates that the bounce rate decreased 🔻 0.55% on average and this effect is statistically significant (, less than our ).

However, unlike the previous OLS model, the autoregressive model does not estimate a statistical significance trend of a decrease in bounce rate each week after intervention, which is in line with our expectations.

The models estimates (with counterfactual projections) can be seen in the chart below:


from statsmodels.tsa.arima.model import ARIMA

start = 24
end = 48

predictions = arima_results.get_prediction(0, end-1)
summary = predictions.summary_frame(alpha=0.05)

arima_cf = ARIMA(df["Y"][:start], df["T"][:start], order=(1,0,0)).fit()

# Model predictions means
y_pred = predictions.predicted_mean

# Counterfactual mean and 95% confidence interval
y_cf = arima_cf.get_forecast(24, exog=df["T"][start:]).summary_frame(alpha=0.05)

# Plot section
plt.style.use('seaborn-whitegrid')
fig, ax = plt.subplots(figsize=(16,10))

# Plot bounce rate data
ax.scatter(df["T"], df["Y"], facecolors='none', edgecolors='steelblue', label="bounce rate data", linewidths=2)

# Plot model mean bounce prediction
ax.plot(df["T"][:start], y_pred[:start], 'b-', label="model prediction")
ax.plot(df["T"][start:], y_pred[start:], 'b-')

# Plot counterfactual mean bounce rate with 95% confidence interval
ax.plot(df["T"][start:], y_cf["mean"], 'k.', label="counterfactual")
ax.fill_between(df["T"][start:], y_cf['mean_ci_lower'], y_cf['mean_ci_upper'], color='k', alpha=0.1, label="counterfactual 95% CI");


# Plot line marking intervention moment
ax.axvline(x = 24.5, color = 'r', label = 'intervention')

ax.legend(loc='best')
plt.ylim([10, 15])
plt.xlabel("Weeks")
plt.ylabel("Bounce rate (%)");

We can clearly see that the ARIMA(1, 0, 0) model fits our dataset better than the OLS model.

ARIMA residual analysis

The summary of our autoregressive model shows a Prob(JB): 0.93 which is compatible with the null-hypothesis of normaly distributed residuals. ✅

The Ljung-Box Q test verifies whether the residuals are independently distributed (they exhibit no serial autocorrelation) as (null-hypothesis). As the Prob(Q): 0.32 is way above the standard there is no evidence of serial autocorrelation in the ARIMA residuals. ✅

Let’s now take a look at residuals qqplot to check if they follow a normal distribution:


import scipy as sp
from statsmodels.graphics.gofplots import qqplot

fig, (ax1, ax2) = plt.subplots(1,2, figsize=(16,8))
sm.qqplot(res.resid, sp.stats.t, fit=True, line="45", ax=ax1);
ax1.set_title("OLS qqplot");

sm.qqplot(arima_results.resid, sp.stats.t, fit=True, line="45", ax=ax2);
ax2.set_title("ARIMA qqplot");
plt.show();

We may observe that the ARIMA(1,0,0) model residuals not only are in general normally distributed as they fit better than the OLS model the theoretical quantiles. ✅

Summary

A/B tests are a the most powerful and trustworthy method to do measure the impact of modifications/changes even before they are fully implemented, which is why they are so widely used.

However, there are some scenarios where A/B tests are not feasible and this is when the knowledge of quasi-experiments becomes valuable to get statistically sound measurements of change impact.

In this post we have shown why an ordinary least square (OLS) linear regression is not a good modeling approach for time series data since they usualy present non-negligible autocorrelation that violates some assumptions of OLS.

We demonstrated with an example how to use python (statsmodels, matplotlib, altair and pandas) to visualize residuals and plot autocorrelation and partial autocorrelations charts to figure out the lag of an autoregressive model and then implemented a ARIMA model using statsmodels to observed a more accurate and precise analysis and how to interpret statsmodels model output for OLS and ARIMA.

We also showed how to plot in a single chart the models estimates (mean and 95% confidence interval) for the time periods before and after intervention and its respective counterfactual.

References

[1] Shopify Engineering: How to Use Quasi-experiments and Counterfactuals to Build Great Products.

[2] Wikipedia: Interrupted Time Series.

[3] Campbell DT, Stanley JC. Experimental and Quasi-experimental Designs for Research. Boston, MA: Houghton Mifflin, 1963.

xboard.dev

Tip: set depends_on_past=True in Airflow when creating a forecasting model pipeline

Why?

How?

Using pip-tools to manage project dependencies in Python

Improve Your Python Dependency Management with pip-tools

What is pip-tools?

Example of using pip-tools

Basic usage of pip-tools with requirements files

pip-tools with multiple environments

pip-tools with custom package indexes

pip-tools with hash dependencies

Migrating my blog from fastpages to Quarto

Why migrate?

What is Quarto?

Installing Quarto

Migrating fastpages’s posts to Quarto

Setup general blog configuration

Setup navigation menu and footer

Setup social metadata

Setup themes for light and dark mode

Adding a robots.txt policy file

Adding canonical tag into the section

Git/Github

Create .gitignore file

Initialize Git and upload to repository in Github

Setup github pages

Add custom domain to github pages

Publish to github pages

Creating Conda’s environment

Default configuration for blog posts

Enabling commenting

Setup footer

VSCode

Installing VS Code Quarto extension

Adding snippet

Enabling inline suggestion

Adding spell checker

Conclusion

Interrupted Time Series (ITS) in Python

When A/B test is not an option

Quasi Experiments

Interrupted Time Series (ITS)

Counterfactual

A practical example

Dataset

Dataset preparation

Naive solution

Problems with naive approach

Let’s first check for the normality of residuals:

Checking independence of observations:

Autoregressive model solution

Autocorrelation

Partial Autocorrelation

Model selection

ARIMA

ARIMA residual analysis

Summary

References

Example of using `pip-tools`

Adding a `robots.txt` policy file

Adding `canonical` tag into the section

Create `.gitignore` file