Data Analyst

Overview

This skill guides structured exploratory data analysis (EDA) and statistical interpretation. It provides a repeatable framework for understanding datasets—from first-look summaries through pattern detection, outlier identification, hypothesis testing, and insight communication—so analysis is rigorous, reproducible, and actionable for stakeholders.

When to Use

• You have a dataset (CSV, table, JSON) and need to understand what it contains
• A stakeholder asks "what's happening with our X metric?"
• You need to compare two groups or time periods
• You want to identify which variables correlate with an outcome
• You need to validate or challenge a business assumption with data
• You're preparing findings for a report, slide deck, or recommendation

When NOT to Use

• Building or optimizing ETL pipelines (use a data engineering skill)
• Real-time streaming analytics
• Designing a database schema or data warehouse
• Machine learning model training (use an ML skill)
• Writing complex production SQL stored procedures

Quick Reference

Instructions

Step 1: Understand the Business Question

Before touching data, clarify:

• What decision will this analysis inform?
• Who is the audience (executive, engineer, analyst)?
• What does "success" look like for this analysis?

Example: "We want to know if our new pricing page increased trial sign-ups."

Step 2: First-Look Exploration (EDA Phase 1)

import pandas as pd
import numpy as np

df = pd.read_csv("data.csv")

# Basic shape and types
print(df.shape)          # (rows, cols)
print(df.dtypes)         # column types
print(df.head(10))       # first rows

# Summary statistics
print(df.describe(include='all'))  # count, mean, std, min, quartiles, max

# Null values
print(df.isnull().sum())           # nulls per column
print(df.isnull().mean() * 100)    # % missing per column

# Duplicates
print(df.duplicated().sum())

Step 3: Univariate Analysis — One Variable at a Time

For numerical columns:

• Mean: arithmetic average, sensitive to outliers
• Median: middle value, robust to outliers
• Std: spread of values; 1 std covers ~68% of a normal distribution
• Skewness: >1 right-skewed (long tail of high values); <-1 left-skewed

col = df['revenue']
print(f"Mean:   {col.mean():.2f}")
print(f"Median: {col.median():.2f}")
print(f"Std:    {col.std():.2f}")
print(f"Skew:   {col.skew():.2f}")

For categorical columns:

print(df['region'].value_counts())
print(df['region'].value_counts(normalize=True) * 100)  # percentages

Step 4: Bivariate Analysis — Relationships Between Variables

Correlation (numerical vs numerical):

corr_matrix = df[['revenue', 'units_sold', 'discount_pct', 'marketing_spend']].corr()
print(corr_matrix.round(2))
# Interpretation: |r| > 0.7 strong, 0.4–0.7 moderate, < 0.4 weak

Group comparison (categorical vs numerical):

df.groupby('region')['revenue'].agg(['mean', 'median', 'std', 'count'])

Pivot for cross-tab:

pd.crosstab(df['product_category'], df['customer_segment'], values=df['revenue'], aggfunc='sum')

Step 5: Outlier Detection

IQR Method (robust, non-parametric):

Q1 = df['revenue'].quantile(0.25)
Q3 = df['revenue'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = df[(df['revenue'] < lower) | (df['revenue'] > upper)]
print(f"Outliers: {len(outliers)} rows ({len(outliers)/len(df)*100:.1f}%)")

Z-Score Method (assumes normal distribution):

from scipy import stats
z_scores = np.abs(stats.zscore(df['revenue']))
outliers_z = df[z_scores > 3]

Step 6: Hypothesis Testing Basics

When comparing two groups (e.g., control vs treatment):

from scipy import stats

control = df[df['group'] == 'control']['conversion_rate']
treatment = df[df['group'] == 'treatment']['conversion_rate']

# Two-sample t-test
t_stat, p_value = stats.ttest_ind(control, treatment)
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Statistically significant difference (p < 0.05)")
else:
    print("No statistically significant difference")

For proportions (e.g., click rates):

from statsmodels.stats.proportion import proportions_ztest
count = np.array([treatment_conversions, control_conversions])
nobs  = np.array([treatment_n, control_n])
stat, pval = proportions_ztest(count, nobs)

Step 7: Communicate Findings

Use the BLOT structure: Bottom Line On Top

1. Headline: The single most important finding
2. Context: Why this matters
3. Evidence: Data supporting the claim (keep it to 3 key numbers)
4. Recommendation: What should happen next

Examples

Example 1: Analyze Sales Data Trends

Input: Monthly sales CSV with columns: date, region, product, revenue, units_sold, discount_pct

Analysis approach:

df['date'] = pd.to_datetime(df['date'])
df['month'] = df['date'].dt.to_period('M')

# Monthly revenue trend
monthly = df.groupby('month')['revenue'].sum().reset_index()
monthly['pct_change'] = monthly['revenue'].pct_change() * 100

# Revenue by region
region_summary = df.groupby('region').agg(
    total_revenue=('revenue', 'sum'),
    avg_order_value=('revenue', 'mean'),
    total_units=('units_sold', 'sum')
).sort_values('total_revenue', ascending=False)

# Discount correlation with revenue
corr = df[['revenue', 'discount_pct']].corr().iloc[0, 1]
print(f"Discount vs Revenue correlation: {corr:.2f}")

Output insight: "Revenue grew 12% MoM in Q3, driven primarily by the West region (+34%). However, the North region is declining (−8% MoM). Discount rate correlates negatively with revenue per order (r = −0.42), suggesting aggressive discounting is not recovering volume losses."

Example 2: Interpret Survey Results

Input: NPS survey data with columns: respondent_id, score (0–10), segment, comment

df['nps_category'] = pd.cut(
    df['score'],
    bins=[-1, 6, 8, 10],
    labels=['Detractor', 'Passive', 'Promoter']
)

summary = df['nps_category'].value_counts(normalize=True) * 100
promoters = summary.get('Promoter', 0)
detractors = summary.get('Detractor', 0)
nps_score = promoters - detractors

print(f"NPS Score: {nps_score:.1f}")
print(f"Promoters: {promoters:.1f}%")
print(f"Detractors: {detractors:.1f}%")

# NPS by segment
df.groupby('segment').apply(
    lambda x: (
        (x['nps_category'] == 'Promoter').mean() -
        (x['nps_category'] == 'Detractor').mean()
    ) * 100
).sort_values()

Output insight: "Overall NPS is +42 (industry average: +31). Enterprise customers score +68, while SMB customers score only +18—suggesting an onboarding gap for smaller accounts. Top detractor theme: 'slow support response times' (mentioned in 38% of Detractor comments)."

Best Practices

• Always ask the business question before opening the data
• State your assumptions explicitly (e.g., "excluding rows where revenue < 0")
• Round numbers appropriately for your audience (executives: 1 decimal; engineers: 3+ decimals)
• Distinguish correlation from causation in all communications
• Include sample sizes in every comparison (n=30 vs n=30,000 tells a different story)
• Use median over mean for skewed distributions (salaries, order values, response times)
• Validate findings against business intuition; anomalies may be data quality issues

Common Mistakes

• Reporting averages without standard deviation or distribution context
• Comparing groups with very different sample sizes without noting it
• Treating correlation as causation (e.g., "discount causes lower revenue")
• Ignoring data freshness—always check the date range of your data
• Over-indexing on a single metric; always check at least 2–3 related metrics
• Not filtering out test/internal users from customer-facing metrics
• Forgetting to handle time zones when aggregating date-based data

Tips & Tricks

• Start every analysis with df.describe() and df.isnull().sum()—it catches 80% of data issues
• A box plot reveals outliers, IQR, and skewness in one glance—use it before any group comparison
• When a metric looks wrong, add a row count to the aggregation to check for sample size issues
• Use .value_counts(dropna=False) to catch NaN values hiding in categorical columns
• Period-over-period % change is almost always more meaningful than raw numbers for trends
• Keep a running "data diary" noting any filters, exclusions, or transformations applied

Related Skills

• chart-builder
• sql-analyst
• data-cleaner
• dashboard-designer