Welcome to etl_toolkit’s documentation!

The etl_toolkit contains many utilities to write better pyspark to simplify pipelines on Databricks. Navigate to each module of the etl_toolkit to learn about the functions provided and how to use them.

• expressions: Contains functions for deriving complex pyspark.Columns

• analyses: Contains functions for deriving complex pyspark.DataFrames that apply many data transformations.

Contents:

• E module (“expressions”)
  • Aggregate Expressions
    • sum_if()
    • avg_if()
  • Boolean Expressions
    • any()
    • all()
    • between()
  • Calculation Expressions
    • growth_rate_by_lag()
  • ID Expressions
    • uuid5()
  • Mapping Expressions
    • chain_assigns()
    • chain_cases()
  • Normalization Expressions
    • normalize_text()
    • try_cast()
  • Regex Expressions
    • rlike_any()
    • rlike_all()
  • Time Expressions
    • normalize_date()
    • normalize_timestamp()
    • parse_all_dates()
    • parse_date_range()
    • date_trunc()
    • date_end()
    • next_complete_period()
    • quarter_label()
• A module (“analyses”)
  • Calculation Analyses
    • add_lag_columns()
    • add_percent_of_total_columns()
  • Card Analyses
    • add_card_date_adjustment()
    • add_card_day_of_week_lag_adjustment()
    • add_card_paneling()
    • add_card_paneling_reweighted()
    • get_card_panels()
    • source_card_transactions()
    • card_coverage_metrics()
    • coverage_metric()
  • E-Receipt Analyses
    • source_ereceipts()
  • Dedupe Analyses
    • dedupe_by_condition()
    • dedupe_by_row_number()
  • Index Analyses
    • index_from_rolling_panel()
  • Investor Standard Analyses
    • revenue_mix_by_income_bucket()
  • Investor Reporting Analyses
    • earnings_results_from_gsheet()
    • earnings_rankings_from_gsheet()
    • earnings_results_with_backtests()
    • backtest_configuration()
    • add_unified_consensus_column()
  • Ordering Analyses
    • shift_columns()
  • Parser Analyses
    • parse_records()
    • parser()
  • Scalar Analyses
    • get_aggregates()
  • Time Analyses
    • fill_periods()
    • periods()
  • Comparison Analyses
    • get_schema_comparison()
    • get_rows_comparison()
  • Investor Standard Metrics Analyses
    • standard_metric_unified_kpi()
    • standard_metric_unified_kpi_derived()
    • standard_metric_data_download()
    • standard_metric_live_feed()
    • standard_metric_feed()
    • standard_metric_daily_growth()
    • standard_metric_quarter_month_pivot()
    • standard_metric_trailing_day_pivot()
    • standard_metric_net_adds()
    • standard_metric_weekly_qtd_progress()
    • standard_metric_monthly_qtd_progress()
    • standard_metric_daily_qtd_progress()
    • standard_metric_half_year_progress()
    • standard_metric_ui_metadata()
  • Calendar Analyses
    • calendar()
• Table creation functions
  • Batch Tables
    • create_table()
  • Standard Metrics Tables
    • create_standard_metric_kpi_table()
    • create_standard_metric_data_download_table()
    • create_standard_metric_feed_table()
    • create_standard_metric_live_feed_table()
• Mutable Table functions
  • Table Operations
    • create_mutable_table()
    • sync_mutable_table()

Indices and tables

• Index

• Module Index

• Search Page