Transform 1-6 of top ten RUS12 tables by aesharpe · Pull Request #4970 · catalyst-cooperative/pudl

aesharpe · 2026-01-22T23:18:05Z

Overview

Part of #4901 and #4884.

What problem does this address?

Adds transform functions for top RUS tables

What did you change?

Add dbt schema.yml file for transformed tables
Add transform function for transformed tables
Update alembic
Add transformed table schema to RESOURCE_METADATA from DRAFT_RESOURCE_METADATA

TO-DO

Finish migrating table schema to RESOURCE_METADATA from DRAFT_RESOURCE_METADATA
Keep updating alembic
Keep adding new fields to fields.py
Keep adding new dbt schemas and simple tests

Documentation

Make sure to update relevant aspects of the documentation:

Update the release notes: reference the PR and related issues.
Update relevant Data Source jinja templates (see docs/data_sources/templates).
Update relevant table or source description metadata (see src/metadata).
Review and update any other aspects of the documentation that might be affected by this PR.

Testing

How did you make sure this worked? How can a reviewer verify this?

Materialize the asset in dagster
Run unit tests

To-do list

If updating analyses or data processing functions: make sure to update row count expectations in dbt tests.
Run pixi run pre-commit-run to run linters and static code analysis checks.
Run pixi run pytest-ci locally to ensure that the merge queue will accept your PR.
Review the PR yourself and call out any questions or issues you have.
For PRs that change the PUDL outputs significantly, run the full ETL locally and then run the data validations using dbt. If you can't run the ETL locally then run the build-deploy-pudl GitHub Action manually and ensure that it succeeds.

aesharpe · 2026-01-24T00:09:10Z

src/pudl/package_data/rus12/column_maps/long_term_debt.csv

@@ -1,4 +1,4 @@
-year_index,borrower_id_rus,borrower_name_rus,report_year,report_month,debt_description,balance_end_of_report_year,interest,principal,total
+year_index,borrower_id_rus,borrower_name_rus,report_year,report_month,debt_description,debt_balance_end_of_report_year,debt_interest_billed,debt_principal_billed,total


I updated these columns so it would be more clear what they were. Take note @cmgosnell for rus7 if you think this is a good change.

i like the add of debt_ at the beginning (i added loan_ but i like this better i will mirror) but i don't love _billed. I'd rather go with ending_balance instead of balance_end_of_report_year bc that's what we use for ferc1 and this is super analogous. the form says "billed this year". I think in general it is expected that the values reported are reported during the report date period and idk billed just seems superfluous but i won't die on that hill.

total should have the same treatment!

I'd shoot towards this schema:

"fields": [ "report_date", "borrower_id_rus", "borrower_name_rus", "debt_description", "debt_ending_balance", "debt_interest", "debt_principal", "debt_total", ],

I excluded the debt_total field here because it was redundant, do you think I should add it back in?

i think we should keep the totals in! if we have time to validate that interest + principal = total then i could imagining removing it but unless/until then i think keeping it in is more safe (also what if there are things in total like fees or other cram that means you can't always add em up)

src/pudl/package_data/rus12/column_maps/balance_sheet.csv

src/pudl/package_data/rus7/column_maps/balance_sheet.csv

…table name

cmgosnell

this looks real good!! are you waiting on the unit conversion from rus7 for the last two of these four tables? (is that why some of the fields are commented out? seems like it but 🤷🏻 . the second phase of rus7 tables are merged now so that should be accessible now. (be wary of alembic migrations 🙃 - i tired to lump that all together so it'd be easier)

for the fully transformed tables i left some tiny nits + we should get lined up about debt column names which feels easy and minor!

also add release notes!

src/pudl/metadata/resources/rus12.py

cmgosnell · 2026-02-02T15:46:57Z

src/pudl/package_data/rus12/column_maps/long_term_debt.csv

@@ -1,4 +1,4 @@
-year_index,borrower_id_rus,borrower_name_rus,report_year,report_month,debt_description,balance_end_of_report_year,interest,principal,total
+year_index,borrower_id_rus,borrower_name_rus,report_year,report_month,debt_description,debt_balance_end_of_report_year,debt_interest_billed,debt_principal_billed,total


i like the add of debt_ at the beginning (i added loan_ but i like this better i will mirror) but i don't love _billed. I'd rather go with ending_balance instead of balance_end_of_report_year bc that's what we use for ferc1 and this is super analogous. the form says "billed this year". I think in general it is expected that the values reported are reported during the report date period and idk billed just seems superfluous but i won't die on that hill.

total should have the same treatment!

I'd shoot towards this schema:

"fields": [ "report_date", "borrower_id_rus", "borrower_name_rus", "debt_description", "debt_ending_balance", "debt_interest", "debt_principal", "debt_total", ],

src/pudl/metadata/fields.py

cmgosnell

three non-blocking suggestions: remove dollars from two column names, added enums and tinsy baby remove additional_details_text. honestly all so small so as to be non-blocking.

but i generated all of these tables locally and they look good!

src/pudl/metadata/fields.py

cmgosnell · 2026-02-04T19:19:56Z

src/pudl/metadata/fields.py

+    "primary_renewable_fuel_type": {
+        "type": "string",
+        "description": ("Primary renewable fuel type used by the plant."),
+    },
+    "primary_renewable_fuel_type_id": {
+        "type": "integer",
+        "description": ("Unique numeric identifier for each renewable fuel type."),
+    },


can or should these be enumed?

yea I can do that quickly!

cmgosnell · 2026-02-04T19:22:01Z

src/pudl/metadata/fields.py

+    "prime_mover_id": {
+        "type": "integer",
+        "description": "Unique numeric identifier for each prime mover type used by RUS borrowers.",
+    },
+    "prime_mover_type": {
+        "type": "string",
+        "description": "Type of prime mover (e.g. Hydro, Internal Combustion).",
+    },


can these also be enum-ed easily?

cmgosnell · 2026-02-04T19:24:34Z

src/pudl/metadata/resources/rus12.py

nbd but you left a handful of these empty text guys

I'm gonna leave the draft ones for now because they get handled in the other PR

aesharpe added 3 commits January 21, 2026 13:09

Add process_raw to extract/rus12 to add report_year to borrowers table

064739d

Add rus12 to etl_group

7a8371f

Transform rus12 meeting and board table

f5ffd2b

aesharpe self-assigned this Jan 22, 2026

aesharpe added this to Catalyst Megaproject Jan 22, 2026

aesharpe added new-data Requests for integration of new data. integrate integrate outside work into the PUDL repository rus12 USDA Rural Utilities Services Form 12 -- Financial and Operating Report: Electric Power Supply labels Jan 22, 2026

github-project-automation bot moved this to New in Catalyst Megaproject Jan 22, 2026

aesharpe moved this from New to In progress in Catalyst Megaproject Jan 22, 2026

aesharpe added 6 commits January 23, 2026 12:11

Transform rus12 balance sheet assets table

9a872bb

Add rus12 balance sheet liabilities table

e5a2809

Fix duplicate col name in raw rus7 balance sheet assets table

8f02515

Add draft version of rus12 borrowers table

1efed33

Add draft version of rus12 renewable plants table

21abf87

Add rus12 long term debt table

24793cb

aesharpe requested a review from cmgosnell January 24, 2026 00:07

aesharpe commented Jan 24, 2026

View reviewed changes

src/pudl/package_data/rus12/column_maps/balance_sheet.csv Show resolved Hide resolved

aesharpe commented Jan 24, 2026

View reviewed changes

src/pudl/package_data/rus7/column_maps/balance_sheet.csv Show resolved Hide resolved

Add rus12 long term debt table

ed410dc

aesharpe changed the title ~~Transform top ten RUS12 tables~~ Transform 1-5 of top ten RUS12 tables Jan 29, 2026

aesharpe marked this pull request as ready for review January 29, 2026 20:21

aesharpe changed the title ~~Transform 1-5 of top ten RUS12 tables~~ Transform 1-6 of top ten RUS12 tables Jan 29, 2026

Add yearly to core_rus12__yearly_lines_stations_labor_materials_cost …

8c2e9bb

…table name

cmgosnell requested changes Feb 2, 2026

View reviewed changes

aesharpe added 4 commits February 2, 2026 16:34

Remove old migrations

e8b449b

Merge branch 'main' into rus12-transform-2

447b233

Finish schema for rus12 long term debt and borrowers tables

f901c04

Add schema for renewable plants rus12 table

ce9e897

aesharpe added 3 commits February 3, 2026 15:38

Add rus12 lines and stations table schema

68e8601

Add release notes

be89ff7

Add first rus12 alembic migrations

b534958

cmgosnell approved these changes Feb 4, 2026

View reviewed changes

aesharpe added 2 commits February 4, 2026 13:44

Final touches on 1-6 rus12 tables

6c80eb4

Merge branch 'main' into rus12-transform-2

78f7c1a

aesharpe enabled auto-merge February 4, 2026 23:41

aesharpe added this pull request to the merge queue Feb 4, 2026

Merged via the queue into main with commit 28d4f13 Feb 5, 2026
14 checks passed

aesharpe deleted the rus12-transform-2 branch February 5, 2026 00:44

github-project-automation bot moved this from In progress to Done in Catalyst Megaproject Feb 5, 2026

		@@ -1,4 +1,4 @@
		year_index,borrower_id_rus,borrower_name_rus,report_year,report_month,debt_description,balance_end_of_report_year,interest,principal,total
		year_index,borrower_id_rus,borrower_name_rus,report_year,report_month,debt_description,debt_balance_end_of_report_year,debt_interest_billed,debt_principal_billed,total

Uh oh!

Conversation

aesharpe commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What problem does this address?

What did you change?

Documentation

Testing

To-do list

Uh oh!

aesharpe Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cmgosnell left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cmgosnell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aesharpe commented Jan 22, 2026 •

edited

Loading

aesharpe Jan 24, 2026 •

edited

Loading

cmgosnell left a comment •

edited

Loading