Unexperimentalize operators #6134

szalpal · 2025-12-16T16:06:39Z

Category:

Refactoring (Redesign of existing code that doesn't affect functionality)

Description:

This PR takes out of experimental the following operators:
experimental.audio_resample
experimental.debayer
experimental.equalize
experimental.filter
experimental.tensor_resize
nvidia.dali.plugin.numba.fn.experimental.numba_function

The experimental flavours of the operators is kept to maintain backwards compatibility.

Tests are configured to test both: experimental and regular flavours of the operators. Since the backwards compatibility is maintained, both of these operators need to be tested.

TODO:

Adjust documentation (hide experimental flavours)
Unexperimentalize decoders.video

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: N/A

JIRA TASK: DALI-4504

dali/test/python/type_annotations/test_typing_pipelines.py

dali/test/python/test_utils.py


-def check_numba_compatibility_gpu(if_skip=True):
-    import nvidia.dali.plugin.numba.experimental as ex
+def check_numba_compatibility_gpu(if_skip=True, use_experimental: bool = False):


szalpal · 2025-12-16T16:18:43Z

!build

dali-automaton · 2025-12-16T16:21:36Z

CI MESSAGE: [40306038]: BUILD STARTED

dali-automaton · 2025-12-16T21:24:51Z

CI MESSAGE: [40306038]: BUILD FAILED

dali/python/nvidia/dali/plugin/numba/__init__.py

+            # try importing cuda.core as it can be used later to check the compatibility
+            # it is okay to fail as it may not be installed, the check later can handle this
+            import cuda.core
+        except ImportError:


szalpal · 2025-12-17T07:16:22Z

!build

dali-automaton · 2025-12-17T07:20:58Z

CI MESSAGE: [40349534]: BUILD STARTED

szalpal · 2025-12-17T07:41:31Z

!build

dali-automaton · 2025-12-17T07:45:38Z

CI MESSAGE: [40350589]: BUILD STARTED

greptile-apps · 2025-12-17T11:15:13Z

Greptile Summary

This PR promotes several operators from the experimental namespace to the main DALI namespace while maintaining backwards compatibility through deprecated aliases.

Operators moved out of experimental:

fn.audio_resample
fn.debayer
fn.equalize
fn.filter
fn.tensor_resize
fn.decoders.video
nvidia.dali.plugin.numba.fn.numba_function

Key changes:

C++ operator schemas renamed from experimental__* to main names, with deprecated aliases added
Both experimental and non-experimental operator registrations kept for backwards compatibility
NumbaFunction class moved from numba.experimental to numba module
Tests updated to cover both experimental and non-experimental operator variants
Type stub files (.pyi) added for the numba plugin

Note: The type annotations for setup_fn parameter in the new .pyi files have an invalid Callable signature that was already present in the experimental version. This is a pre-existing issue that was copied over.

Confidence Score: 4/5

This PR is a safe refactoring that moves operators out of experimental namespace while preserving backwards compatibility.
Score of 4 reflects that this is a straightforward namespace migration with proper deprecation aliases. The type annotation issues in .pyi files are minor and don't affect runtime behavior. No logic changes to operator implementations.
The numba plugin type stub files have invalid Callable signatures for setup_fn, but this is a pre-existing issue from the experimental version.

Important Files Changed

Filename	Overview
dali/operators/image/color/debayer.cc	Moves Debayer schema from experimental to main namespace with deprecated alias for backwards compatibility.
dali/operators/image/color/equalize.cc	Moves Equalize schema from experimental to main namespace with deprecated alias for backwards compatibility.
dali/operators/image/convolution/filter.cc	Moves Filter schema from experimental to main namespace with deprecated alias for backwards compatibility.
dali/operators/video/decoder/video_decoder_cpu.cc	Moves video decoder schema from experimental to main namespace with deprecated alias.
dali/python/nvidia/dali/plugin/numba/init.py	Moves NumbaFunction class from experimental to main module. Contains pre-existing bug at line 187 where `num_outs` is used instead of `num_ins` for input shapes in setup function.
dali/python/nvidia/dali/plugin/numba/init.pyi	New type stub file for NumbaFunction. Has invalid Callable type annotation for `setup_fn` parameter.
dali/python/nvidia/dali/plugin/numba/fn/init.pyi	New type stub file for numba_function. Has invalid Callable type annotation for `setup_fn` parameter.
dali/test/python/test_dali_variable_batch_size.py	Adds tests for non-experimental operators alongside experimental versions, updates tested_methods list.

Sequence Diagram

sequenceDiagram
    participant User
    participant MainAPI as fn.* / numba.fn.*
    participant ExpAPI as fn.experimental.* / numba.fn.experimental.*
    participant Operator as C++ Operator Implementation

    Note over MainAPI,ExpAPI: Before PR: Only experimental namespace
    
    rect rgb(200, 230, 200)
        Note over MainAPI,ExpAPI: After PR: Both namespaces available
        User->>MainAPI: fn.debayer() / fn.filter() / etc.
        MainAPI->>Operator: Calls main schema (Debayer, Filter, etc.)
        Operator-->>User: Returns result
        
        User->>ExpAPI: fn.experimental.debayer() (deprecated)
        ExpAPI->>Operator: Calls deprecated alias (experimental__Debayer)
        Operator-->>User: Returns result (identical behavior)
    end
    
    Note over MainAPI,ExpAPI: Backwards compatibility preserved

greptile-apps

Additional Comments (6)

dali/operators/image/color/equalize.cc, line 37 (link)

style: The docstring reference should use :meth:nvidia.dali.fn.equalize`` for consistency with DALI's documentation standards

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}
dali/python/nvidia/dali/plugin/numba/__init__.pyi, line 35 (link)

logic: Return type annotation may be incorrect - setup_fn appears to return None but the signature suggests it takes parameters and returns None

Should the setup_fn callable return type be None or should it return something else based on the actual implementation?
dali/python/nvidia/dali/plugin/numba/fn/__init__.pyi, line 36 (link)

syntax: The setup_fn parameter type annotation appears malformed - it has None as a return type in the callable signature but should likely return something or be Optional[Callable[...]] without the trailing None
dali/operators/generic/resize/tensor_resize_cpu.cc, line 49-62 (link)

style: The deprecated schema definition duplicates parent metadata (NumInput, NumOutput, SupportVolumetric, AllowSequences) that is already inherited from "TensorResize". Since AddParent("TensorResize") inherits all properties, these redundant specifications are unnecessary.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}
dali/test/python/checkpointing/test_dali_stateless_operators.py, line 643-646 (link)

logic: inconsistent with other tests - this test only uses fn.audio_resample but the decorator includes both experimental and non-experimental versions
dali/python/nvidia/dali/plugin/numba/__init__.py, line 187 (link)

logic: Incorrect variable used: should be num_ins instead of num_outs for the input shapes calculation

_{25 files reviewed, 6 comments}

_{Edit Code Review Agent Settings | Greptile}

szalpal · 2025-12-17T17:05:14Z

!build

dali-automaton · 2025-12-17T17:11:34Z

CI MESSAGE: [40376844]: BUILD STARTED

dali-automaton · 2025-12-17T17:45:46Z

CI MESSAGE: [40376844]: BUILD FAILED

dali-automaton · 2025-12-17T20:58:13Z

CI MESSAGE: [40349534]: BUILD FAILED

szalpal · 2025-12-17T23:20:12Z

!build

dali-automaton · 2025-12-17T23:25:26Z

CI MESSAGE: [40397505]: BUILD STARTED

Signed-off-by: Michał Szołucha <[email protected]>

szalpal · 2026-01-14T11:10:58Z

!build

dali-automaton · 2026-01-14T11:16:35Z

CI MESSAGE: [41719186]: BUILD STARTED

greptile-apps

Additional Comments (1)

dali/operators/image/color/debayer.cc, line 51 (link)

style: Example in documentation still uses experimental namespace and should be updated to use fn.debayer

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

_{29 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-01-14T11:26:42Z

dali/python/nvidia/dali/plugin/numba/fn/__init__.pyi

+    in_types: List[DALIDataType],
+    outs_ndim: List[int],
+    ins_ndim: List[int],
+    setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]], None]]] = None,


logic: The setup_fn type annotation has an invalid callable signature with None as the third parameter type. Callable type parameters should only specify input argument types, not return types. Should be Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]]], None]] if it returns None, or just Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]]], Any]].

Suggested change

setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]], None]]] = None,

setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]]], None]] = None,

greptile-apps · 2026-01-14T11:26:49Z

dali/python/nvidia/dali/plugin/numba/__init__.py

+                in_shapes_np = _get_shape_view(in_shapes_ptr, in_ndims_ptr, num_outs, num_samples)
+                setup_fn(out_shapes_np, in_shapes_np)


logic: Bug in parameter usage: should be num_ins instead of num_outs for input shapes.

Suggested change

in_shapes_np = _get_shape_view(in_shapes_ptr, in_ndims_ptr, num_outs, num_samples)

setup_fn(out_shapes_np, in_shapes_np)

in_shapes_np = _get_shape_view(in_shapes_ptr, in_ndims_ptr, num_ins, num_samples)

setup_fn(out_shapes_np, in_shapes_np)

dali-automaton · 2026-01-14T20:10:13Z

CI MESSAGE: [41717403]: BUILD FAILED

dali-automaton · 2026-01-15T00:43:33Z

CI MESSAGE: [41719186]: BUILD FAILED

Signed-off-by: Michał Szołucha <[email protected]>

greptile-apps · 2026-01-15T11:42:12Z

Greptile's behavior is changing!

From now on, if a review finishes with no comments, we will not post an additional "statistics" comment to confirm that our review found nothing to comment on. However, you can confirm that we reviewed your changes in the status check section.

_{This feature can be toggled off in your Code Review Settings by deselecting "Create a status check for each PR".}

szalpal · 2026-01-16T12:00:39Z

!build

dali-automaton · 2026-01-16T12:05:30Z

CI MESSAGE: [41881093]: BUILD STARTED

dali-automaton · 2026-01-16T22:34:00Z

CI MESSAGE: [41881093]: BUILD FAILED

Signed-off-by: Michał Szołucha <[email protected]>

szalpal · 2026-01-19T12:09:25Z

!build

dali-automaton · 2026-01-19T12:15:23Z

CI MESSAGE: [42048338]: BUILD STARTED

dali-automaton · 2026-01-19T15:54:06Z

CI MESSAGE: [42048338]: BUILD FAILED

szalpal · 2026-01-19T22:21:29Z

!build

dali-automaton · 2026-01-19T22:25:35Z

CI MESSAGE: [42071458]: BUILD STARTED

dali-automaton · 2026-01-19T23:41:17Z

CI MESSAGE: [42071458]: BUILD FAILED

Signed-off-by: Michał Szołucha <[email protected]>

szalpal · 2026-01-20T11:14:13Z

!build

dali-automaton · 2026-01-20T11:20:29Z

CI MESSAGE: [42103524]: BUILD STARTED

dali-automaton · 2026-01-20T14:27:44Z

CI MESSAGE: [42103524]: BUILD PASSED

github-advanced-security bot found potential problems Dec 16, 2025

View reviewed changes

szalpal force-pushed the unexperimentalize branch from d372995 to 5aa731e Compare December 16, 2025 20:20

szalpal force-pushed the unexperimentalize branch 3 times, most recently from e630529 to 51e599d Compare December 17, 2025 07:12

github-advanced-security bot found potential problems Dec 17, 2025

View reviewed changes

szalpal force-pushed the unexperimentalize branch 2 times, most recently from a3c3ac6 to a424878 Compare December 17, 2025 07:41

szalpal marked this pull request as ready for review December 17, 2025 11:06

greptile-apps bot reviewed Dec 17, 2025

View reviewed changes

dali-automaton assigned szkarpinski and stiepan Dec 17, 2025

szalpal marked this pull request as draft December 17, 2025 13:13

szalpal force-pushed the unexperimentalize branch from a424878 to d5291dc Compare December 17, 2025 17:05

szalpal force-pushed the unexperimentalize branch from d5291dc to e1c19f9 Compare December 17, 2025 23:19

szalpal force-pushed the unexperimentalize branch from e1c19f9 to 0d5e199 Compare December 17, 2025 23:37

szalpal added 3 commits January 14, 2026 12:09

Adjust Filter tests

c2795d2

Signed-off-by: Michał Szołucha <[email protected]>

Adjust video decoder tests

58fc81f

Signed-off-by: Michał Szołucha <[email protected]>

Adjust stateless tests

5fc0aef

Signed-off-by: Michał Szołucha <[email protected]>

szalpal force-pushed the unexperimentalize branch from 93131c0 to 114a2cf Compare January 14, 2026 11:09

greptile-apps bot reviewed Jan 14, 2026

View reviewed changes

Adjust CPU only and variable batch size tests

611b3ae

Signed-off-by: Michał Szołucha <[email protected]>

szalpal force-pushed the unexperimentalize branch from 114a2cf to 611b3ae Compare January 15, 2026 11:38

stiepan approved these changes Jan 16, 2026

View reviewed changes

Adjust mypy test

25cbb03

Signed-off-by: Michał Szołucha <[email protected]>

Adjust checkpointing test

f940c22

Signed-off-by: Michał Szołucha <[email protected]>

szalpal force-pushed the unexperimentalize branch from 61436c5 to f940c22 Compare January 20, 2026 11:11

szalpal merged commit 08d2da1 into NVIDIA:main Jan 21, 2026
8 checks passed

szalpal deleted the unexperimentalize branch January 21, 2026 11:43

	setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]], None]]] = None,
	setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]]], None]] = None,

		in_shapes_np = _get_shape_view(in_shapes_ptr, in_ndims_ptr, num_outs, num_samples)
		setup_fn(out_shapes_np, in_shapes_np)

Unexperimentalize operators #6134

Unexperimentalize operators #6134

Uh oh!

Conversation

szalpal commented Dec 16, 2025 • edited by JanuszL Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Uh oh!

Uh oh!

Check notice

szalpal commented Dec 16, 2025

Uh oh!

dali-automaton commented Dec 16, 2025

Uh oh!

dali-automaton commented Dec 16, 2025

Uh oh!

Check notice

szalpal commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

szalpal commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

greptile-apps bot commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (6)

Uh oh!

szalpal commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

szalpal commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

szalpal commented Jan 14, 2026

Uh oh!

dali-automaton commented Jan 14, 2026

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

greptile-apps bot Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

dali-automaton commented Jan 14, 2026

Uh oh!

dali-automaton commented Jan 15, 2026

Uh oh!

greptile-apps bot commented Jan 15, 2026

Greptile's behavior is changing!

Uh oh!

szalpal commented Dec 16, 2025 •

edited by JanuszL

Loading

greptile-apps bot commented Dec 17, 2025 •

edited

Loading

greptile-apps bot left a comment •

edited

Loading

greptile-apps bot left a comment •

edited

Loading