GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

rok · 2025-12-22T00:15:41Z

Rationale for this change

This is the first in series of PRs adding type annotations to pyarrow and resolving #32609.

What changes are included in this PR?

This PR establishes infrastructure for type checking:

Adds CI workflow for running mypy, pyright, and ty type checkers on linux, macos and windows
Configures type checkers to validate stub files (excluding source files for now)
Adds PEP 561 py.typed marker to enable type checking
Updates wheel build scripts to include stub files in distributions
Creates initial minimal stub directory structure
Updates developer documentation with type checking workflow

Are these changes tested?

No. This is mostly a CI change.

Are there any user-facing changes?

This does not add any actual annotations (only py.typed marker) so user should not be affected.

GitHub Issue: [Python] Type checking support #32609

…d script for including docstrings into stubfiles before building wheels.

AlenkaF

Thank you @rok for working on this! Splitting up the main PR sounds like a great idea to me.

I only have one small question, otherwise the changes look good to me.

ci/scripts/python_test_type_annotations.sh

rok · 2025-12-23T10:26:37Z

Thanks for review @AlenkaF ! I think we probably want @raulcd to look at this too before merging?

AlenkaF · 2025-12-23T10:32:04Z

Yes, let's wait for Raul 👍

raulcd

Thanks for the PR @rok sorry it took me a while to review with holidays, release, etcetera

ci/scripts/python_test_type_annotations.bat

ci/scripts/python_test_type_annotations.sh

docs/source/developers/python/development.rst

ci/scripts/python_wheel_validate_contents.py

ci/scripts/python_wheel_xlinux_build.sh

raulcd · 2026-01-13T19:27:53Z

@github-actions crossbow submit wheel-manylinux-2-28-cp311-cp311-amd64

github-actions · 2026-01-13T19:30:08Z

Revision: 28fba3a

Submitted crossbow builds: ursacomputing/crossbow @ actions-8a459250fe

Task	Status
wheel-manylinux-2-28-cp311-cp311-amd64

Co-authored-by: Raúl Cumplido <[email protected]>

rok · 2026-01-14T10:41:33Z

Thanks for review @raulcd ! I've responded to your points.

raulcd · 2026-01-14T11:41:13Z

compose.yaml

      ARROW_S3: "OFF"
      ARROW_SUBSTRAIT: "OFF"
      ARROW_WITH_OPENTELEMETRY: "OFF"
+      PYARROW_TEST_ANNOTATIONS: "ON"


I think is worth it if instead of using a separate ci/scripts/python_test_type_annotations.sh we use ci/scripts/python_test.sh and we use the PYARROW_TEST_ANNOTATIONS variable to manage whether we want annotations to be tested or not as part of python testing.
If we decide we want to have two separate scripts I think this env var should be removed.
I lean towards having a single python test script that can allow us to also test annotations but I can find arguments for both so not an issue.

raulcd · 2026-01-14T11:42:42Z

ci/scripts/python_wheel_macos_build.sh

+# We first populate stub docstrings and then build the wheel
+python setup.py build_ext --inplace
+python -m pip install griffe libcst
+python ../dev/update_stub_docstrings.py pyarrow-stubs


this is not necessary, right?

Same as #48618 (comment). It's a noop and we don't have to introduce python logic in stub PRs if we do here.

ci/scripts/python_wheel_validate_contents.py

raulcd · 2026-01-14T11:44:04Z

ci/scripts/python_wheel_windows_build.bat

+@REM We first populate stub docstrings and then build the wheel
+%PYTHON_CMD% setup.py build_ext --inplace
+%PYTHON_CMD% -m pip install griffe libcst


Until we have real annotations this isn't necessary, right?

I added running dev\update_stub_docstrings.py, so I suppose it's a noop now and we can keep it to make sure it doesn't fail before introducing actual stubs? This will keep CI logic out of stub PRs.

…g check

rok · 2026-01-14T15:48:44Z

@github-actions crossbow submit wheel-manylinux-2-28-cp311-cp311-amd64

github-actions · 2026-01-14T15:50:55Z

Revision: ee72560

Submitted crossbow builds: ursacomputing/crossbow @ actions-cd0acc6696

Task	Status
wheel-manylinux-2-28-cp311-cp311-amd64

rok · 2026-01-14T16:15:51Z

@github-actions crossbow submit wheel-manylinux-2-28-cp311-cp311-amd64

github-actions · 2026-01-14T16:18:46Z

Revision: 19b54bc

Submitted crossbow builds: ursacomputing/crossbow @ actions-f406282276

Task	Status
wheel-manylinux-2-28-cp311-cp311-amd64

rok · 2026-01-14T17:31:52Z

@github-actions crossbow submit wheel-manylinux-2-28-cp311-cp311-amd64

github-actions · 2026-01-14T17:34:42Z

Revision: c23cb9c

Submitted crossbow builds: ursacomputing/crossbow @ actions-286c26b719

Task	Status
wheel-manylinux-2-28-cp311-cp311-amd64

rok · 2026-01-14T17:48:25Z

@raulcd I've factored out griffe and click, but we still need libcst (using ast would not provide stable syntax here). I think wheel builds should now pass. Please take another look when you have time.

rok · 2026-01-14T18:47:36Z

@github-actions crossbow submit wheel-manylinux-2-28-cp311-cp311-amd64

github-actions · 2026-01-14T18:50:26Z

Revision: 491ff2c

Submitted crossbow builds: ursacomputing/crossbow @ actions-b6331706da

Task	Status
wheel-manylinux-2-28-cp311-cp311-amd64

change bat lint add a popd and nicer logging for windows ReplaceElipsis -> DocstringInserter simplify remove sphinx

rok · 2026-01-14T19:30:01Z

@github-actions crossbow submit wheel-manylinux-2-28-cp311-cp311-amd64

github-actions · 2026-01-14T19:33:54Z

Revision: ba07ac8

Submitted crossbow builds: ursacomputing/crossbow @ actions-6c5a9f1e7e

Task	Status
wheel-manylinux-2-28-cp311-cp311-amd64

rok requested review from AlenkaF, assignUser, jonkeane, kou and raulcd as code owners December 22, 2025 00:15

github-actions bot added Component: Python Component: Documentation awaiting committer review Awaiting committer review labels Dec 22, 2025

Add GitHub Actions workflow for type checking (mypy, pyright, ty). Ad…

30017ff

…d script for including docstrings into stubfiles before building wheels.

rok force-pushed the pyarrow-stubs-pr1-infrastructure branch from afe4699 to 30017ff Compare December 22, 2025 00:20

This was referenced Dec 22, 2025

GH-32609: [Python] Add type annotations to PyArrow #47609

Open

GH-32609: [Python] Add internal type system stubs (_types, error, _stubs_typing) #48622

Draft

AlenkaF approved these changes Dec 23, 2025

View reviewed changes

ci/scripts/python_test_type_annotations.sh Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Dec 23, 2025

Update ci/scripts/python_test_type_annotations.sh

28fba3a

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Dec 23, 2025

raulcd reviewed Jan 13, 2026

View reviewed changes

Apply suggestion from @raulcd

5858f47

Co-authored-by: Raúl Cumplido <[email protected]>

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jan 14, 2026

review feedback

59ce2dd

github-actions bot removed the awaiting change review Awaiting change review label Jan 14, 2026

raulcd reviewed Jan 14, 2026

View reviewed changes

include dev/update_stub_docstrings.py

0de99a0

rok changed the title ~~GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotions~~ GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations Jan 14, 2026

Use PYARROW_TEST_ANNOTATIONS in windows build, disable wheel docstrin…

c327d26

…g check

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jan 14, 2026

rok added 2 commits January 14, 2026 16:14

work on dev/update_stub_docstrings.py

090185e

further work on dev/update_stub_docstrings.py

5c13084

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jan 14, 2026

lint

ee72560

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jan 14, 2026

add click for docstring population

19b54bc

Remove dependencies on click and griffe

c23cb9c

rok requested a review from raulcd January 14, 2026 17:48

fix import paths

491ff2c

add PYARROW_TEST_ANNOTATIONS to AMD64 Windows 2022 Python 3.13

ba07ac8

change bat lint add a popd and nicer logging for windows ReplaceElipsis -> DocstringInserter simplify remove sphinx

GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

Are you sure you want to change the base?

GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

Conversation

rok commented Dec 22, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

AlenkaF left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rok commented Dec 23, 2025

Uh oh!

AlenkaF commented Dec 23, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd commented Jan 13, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

rok commented Jan 14, 2026

Uh oh!

raulcd Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

raulcd Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

rok Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raulcd Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

rok Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

rok commented Jan 14, 2026

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

rok commented Jan 14, 2026

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

rok commented Jan 14, 2026

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

rok commented Jan 14, 2026

Uh oh!

rok commented Jan 14, 2026

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

rok commented Jan 14, 2026

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rok commented Dec 22, 2025 •

edited by github-actions bot

Loading