Getting started with causal AI in Python – code examples and first steps

Many industry experts question the reliability of common prediction algorithms when making data-driven decisions. Spurious correlations, like that between chocolate consumption and Nobel laureates, highlight the crucial distinction between correlation and causation. While correlation exists, it doesn't automatically imply a causal relationship. Understanding root causes and using data analytics for impactful change is vital for businesses, leading companies like Microsoft and Amazon to invest heavily in causal AI.

This tutorial introduces fundamental causal AI concepts using Python's DoWhy library. While differing from standard machine learning, a basic grasp of regression analysis is helpful. DataCamp's "Introduction to Linear Modeling in Python" and "Machine Learning for Business" courses offer relevant background.

Causal AI Fundamentals

Causal inference requires a different approach than predictive analytics. Nancy Cartwright's "No causes in, no causes out" emphasizes the need for assumptions about the causal structure to obtain causal answers; a purely data-driven approach is insufficient. To determine causality, alternative explanations must be ruled out, requiring external knowledge beyond the data itself.

Consider a new work-from-home (WFH) policy's impact on employee productivity. Initial analysis might show higher task completion for WFH employees, but is this causal? Other factors, like employee personality or family situations, could influence both WFH preference and productivity, acting as common causes.

Note: Graph created by causalfusion.net

Causal graphs visually represent these relationships, making assumptions explicit and allowing for refinement. These assumptions can be strong, but the explicit nature of causal graphs enhances analysis credibility.

DoWhy in Python

Microsoft's DoWhy library (part of the PyWhy ecosystem) is a leading tool for causal analysis in Python. We'll simulate data to illustrate causal inference steps.

!pip install git+https://github.com/microsoft/dowhy.git
import numpy as np
import pandas as pd
import dowhy
from dowhy import CausalModel
import dowhy.datasets
import statsmodels.api as sm

# Set seed for reproducibility
np.random.seed(1)

# Simulate data
data = dowhy.datasets.linear_dataset(
    beta=1,
    num_common_causes=2,
    num_discrete_common_causes=1,
    num_instruments=1,
    num_samples=10000,
    treatment_is_binary=True)

df = data['df']

# ... (rest of the DoWhy code remains the same) ...

DoWhy uses labels as shown in Table 1 (original table remains unchanged). The causal graph is implicitly defined by data parameters. DoWhy uses the DOT language to represent graphs.

digraph {v0->y;W0-> v0; W1-> v0;Z0-> v0;W0-> y; W1-> y;}

A causal model is created combining data and graph:

model=CausalModel(
        data = df,
        treatment=data['treatment_name'],
        outcome=data['outcome_name'],
        graph=data['gml_graph']
        )

Causal Analysis and Bias Reduction

A simple linear regression shows a slope coefficient, but this can be biased due to common causes. DoWhy's backdoor criterion helps address this by controlling for variables influencing both treatment and outcome (introversion and number of kids in this example).

!pip install git+https://github.com/microsoft/dowhy.git
import numpy as np
import pandas as pd
import dowhy
from dowhy import CausalModel
import dowhy.datasets
import statsmodels.api as sm

# Set seed for reproducibility
np.random.seed(1)

# Simulate data
data = dowhy.datasets.linear_dataset(
    beta=1,
    num_common_causes=2,
    num_discrete_common_causes=1,
    num_instruments=1,
    num_samples=10000,
    treatment_is_binary=True)

df = data['df']

# ... (rest of the DoWhy code remains the same) ...

DoWhy provides various estimation methods; inverse probability weighting is used here for generality. The resulting estimate is close to the ground truth, demonstrating bias reduction.

Stress Testing and Robustness

DoWhy's refutation tests help assess assumption reliability. Adding an unobserved common cause significantly impacts the estimate's range, highlighting the influence of unobservable variables.

Instrumental Variables

Instrumental variables (like a subway closure affecting WFH but not directly impacting productivity) offer an alternative identification strategy. DoWhy automatically identifies suitable instruments, providing more robust, albeit potentially less precise, estimates.

Conclusion

DoWhy simplifies causal AI, providing a comprehensive pipeline. After mastering the basics, explore advanced techniques and other libraries. Causal inference requires domain expertise and collaboration to define suitable models and assumptions. The effort is worthwhile for obtaining causal answers crucial for informed business decisions. DataCamp's "Machine Learning for Business" course provides further learning opportunities.

The above is the detailed content of Getting started with causal AI in Python – code examples and first steps. For more information, please follow other related articles on the PHP Chinese website!