Skip to content

Conversation

@rostan-t
Copy link
Collaborator

@rostan-t rostan-t commented Dec 17, 2025

Category: Documentation

Description:

Currently, most examples in the documentation only show pipeline mode. This PR adds dynamic mode examples.

  • The front page now uses tabs to show a both a pipeline mode and dynamic snippets. A sphinx plugin allows it to render properly both on GitHub and in the HTML documentation.
  • The getting started page now has variants for both pipeline mode and dynamic mode.
  • Some examples and tutorials now have both pipeline mode and dynamic mode variants. Those are stored as subpages:
    Capture d’écran 2026-01-07 à 18 01 58

Because of the third point, a lot of pages have been renamed (generally to pipeline_mode.ipynb) with little or no changes (when they contained links with relative paths, those needed to be changed).

Additional information:

Affected modules and functionalities:

Pages affected by this PR:

  • Front page
  • Getting started tutorial
  • Data loading:
    • LMDB Reader
    • MXNet RecordIO Reader
    • TFRecord Reader
    • Webdataset Reader
    • COCO Reader
    • NumPy Reader
  • Image processing:
    • Augmentation Gallery (already existed, now moved)
    • Brightness and contrast
    • CLAHE
    • Color Space Conversion
    • Image Decoder
    • HSV
    • Resize Operator
    • Warp Affine
  • Audio Processing:
    • Audio Decoder
    • Audio Spectogram
  • Video Processing
    • Simple video pipeline
    • Video pipeline with labels
    • Video pipeline with timestamps and frame numbers
    • Video with image processing operators
    • Optical flow

Key points relevant for the review:

The PR contains little code changes apart from documentation pages so it's best reviewed directly as the rendered HTML doc.

  • Does the new organization make sense?
  • Are links still working correctly?
  • Do the dynamic mode examples capture everything that the corresponding pipeline mode examples capture?
  • Do the examples cover every aspect of dynamic mode we want to cover? Note that integration with PyTorch and control features are separate tasks.

Tests:

  • Existing tests apply
  • New tests added
    • Python tests
    • GTests
    • Benchmark
    • Other
  • N/A

Checklist

Documentation

  • Existing documentation applies
  • Documentation updated
    • Docstring
    • Doxygen
    • RST
    • Jupyter
    • Other
  • N/A

DALI team only

Requirements

  • Implements new requirements
  • Affects existing requirements
  • N/A

REQ IDs: N/A

JIRA TASK: DALI-4515

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@rostan-t rostan-t marked this pull request as ready for review January 7, 2026 17:07
@rostan-t
Copy link
Collaborator Author

rostan-t commented Jan 7, 2026

!build

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [41293348]: BUILD FAILED

@greptile-apps
Copy link

greptile-apps bot commented Jan 7, 2026

Greptile Summary

Overview

This PR successfully adds comprehensive dynamic mode (NDD) examples to DALI's documentation, creating a parallel set of tutorials and examples alongside the existing pipeline mode documentation. The PR introduces 23 new dynamic mode notebooks, updates the front page with synchronized tabs, and implements a new Sphinx extension for seamless mode switching.

Key Changes

  • New Sphinx Extension: dali_tabs.py provides a .. container:: dali-tabs directive that transforms into synchronized .. tab-set:: for unified pipeline/dynamic mode presentation across GitHub and HTML documentation
  • Front Page Redesign: README.rst now uses tabs to show equivalent pipeline and dynamic mode examples side-by-side
  • Getting Started Guide: Split into separate pipeline_mode.ipynb and dynamic_mode.ipynb with a new index structure
  • 23 New Dynamic Mode Examples: Added across data loading (7 readers), image processing (9 operations), audio processing (2 operations), and video processing (4 examples)
  • Documentation Index Updates: All example index.py files updated with dual operator references (e.g., fn.readers.numpy and dynamic.readers.Numpy)
  • Badges: Configuration adds visual badges to identify which mode each notebook demonstrates
  • Fixed Operator References: Corrected inconsistent operator naming (e.g., dynamic.readers.numpydynamic.readers.Numpy)

Quality Assessment

Strengths:

  • Comprehensive coverage across all major operator categories with matching pipeline/dynamic examples
  • Clean Sphinx extension implementation with proper event handling and indentation preservation
  • Consistent documentation structure with Python-based index files for automatic RST generation
  • Proper handling of operator cross-references with automatic dynamic.*experimental.dynamic.* conversion
  • Badge system provides clear visual identification of example type
  • Excellent example quality (NumPy reader example is thorough and demonstrates advanced features)

Observations:

  • All operator references follow correct naming conventions: PascalCase for readers (dynamic.readers.Numpy, dynamic.readers.COCO) and lowercase for other operators (dynamic.resize, dynamic.clahe)
  • Test files correctly exclude examples requiring external data (DALI_EXTRA_PATH)
  • Generated RST index files properly gitignored
  • Documentation build system correctly integrated with doc_index.document_examples()

Test Coverage Note

Examples cannot be tested in standard CI without external data dependencies (DALI_EXTRA repository), which is handled appropriately in the test exclusions.

Confidence Score: 5/5

  • This PR is safe to merge with no critical issues or breaking changes. Documentation-only changes with proper Sphinx integration and comprehensive example coverage.
  • Score of 5 reflects: (1) Documentation-only changes with no production code modifications; (2) All examples follow consistent naming conventions and structure; (3) Sphinx extension is well-implemented with proper event handling; (4) Previous operator reference issue (lowercase numpy) has been fixed; (5) Test exclusions appropriately handle external data dependencies; (6) Configuration changes are additive and don't break existing functionality; (7) Documentation index generation is properly integrated. No syntax errors, logic issues, or security concerns identified.
  • No files require special attention. All changes have been properly reviewed and validated.

Important Files Changed

Filename Overview
docs/_extensions/dali_tabs.py Sphinx extension that transforms .. container:: dali-tabs blocks into .. tab-set:: directives for synchronized tab switching. Implementation is clean, well-tested regex patterns, proper indentation handling, and correct integration with Sphinx's include-read event. No issues found.
docs/conf.py Documentation configuration properly registers dali_tabs extension, configures nbsphinx badges for dynamic/pipeline modes, and calls doc_index.document_examples() to generate example indices and operator cross-references. Badge injection logic (lines 165-172) correctly adds Dynamic Mode/Pipeline Mode badges based on docname. No issues found.
docs/doc_index.py Module generates documentation indices and operator cross-references. OpReference class (lines 86-92) correctly converts dynamic.* operator references to experimental.dynamic.* by prepending experimental. prefix. Documentation correctly uses PascalCase for readers (e.g., dynamic.readers.Numpy) and lowercase for other operators. No critical issues, but the capitalization preservation in the operator string transformation is fragile.
docs/examples/getting_started/index.rst Manual RST index file provides clear overview of pipeline vs dynamic modes with good structure. References both pipeline_mode and dynamic_mode entries in toctree. Explanation of when to use each mode is helpful and accurate. No issues found.
docs/examples/general/data_loading/numpy_reader/index.py Example index properly defines both pipeline_mode and dynamic_mode entries with correct operator references. Previously had an issue with lowercase dynamic.readers.numpy (fixed in commit 3f96079 to uppercase Numpy). Now correctly uses dynamic.readers.Numpy matching the actual API. No issues found.
README.rst Updated front page to show both pipeline and dynamic mode examples using the new .. container:: dali-tabs directive. Examples are comparable and demonstrate equivalent functionality in both modes. Code formatting and structure are correct. No issues found.

Sequence Diagram

sequenceDiagram
    participant User
    participant Sphinx
    participant dali_tabs
    participant doc_index
    participant nbsphinx
    participant HTML

    User->>Sphinx: sphinx-build docs
    Sphinx->>doc_index: document_examples("examples/index.py")
    doc_index->>doc_index: Recursively process .py index files
    doc_index->>doc_index: Collect operator references
    doc_index-->>Sphinx: Return {operator → examples_list}
    
    Sphinx->>Sphinx: Read RST/notebook files
    Note over Sphinx: Include .rst files with dali-tabs
    Sphinx->>dali_tabs: include-read event
    dali_tabs->>dali_tabs: Transform dali-tabs to tab-set
    dali_tabs-->>Sphinx: Modified RST content
    
    Sphinx->>nbsphinx: Process .ipynb files
    nbsphinx->>nbsphinx: Add badges based on docname
    nbsphinx-->>Sphinx: Rendered notebook HTML
    
    Sphinx->>HTML: Generate final documentation
    HTML-->>User: Dynamic/Pipeline tabs with badges
Loading

@rostan-t
Copy link
Collaborator Author

rostan-t commented Jan 7, 2026

!build

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [41295926]: BUILD FAILED

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile Overview

Greptile Summary

This PR adds comprehensive dynamic mode examples to DALI documentation, creating parity with existing pipeline mode examples. A new Sphinx extension enables synchronized tabs for displaying both modes side-by-side.

Major Changes:

  • Added dali_tabs.py Sphinx extension to transform container blocks into synchronized tab-sets
  • Split getting started tutorial into separate pipeline_mode.ipynb and dynamic_mode.ipynb variants
  • Created dynamic_mode.ipynb examples for 20+ topics across data loading, image processing, audio processing, and video processing
  • Reorganized documentation structure: many examples moved from flat files into subdirectories with index.py files
  • Enhanced doc_index.py to map dynamic.* operator references to experimental.dynamic.* and handle Sphinx title syntax
  • Updated README.rst front page with tabbed code examples

Documentation Quality:

  • Dynamic mode examples are comprehensive and mirror pipeline mode functionality
  • Examples include proper error handling, context managers (EvalContext, EvalMode), and batch/tensor operations
  • Code demonstrates key dynamic mode patterns: direct operator calls, reader iteration, GPU/CPU data movement
  • Path references updated correctly for relocated notebooks (e.g., ../../data/images for deeper nesting)

Confidence Score: 5/5

  • This documentation PR is safe to merge with no code execution risks
  • Score reflects that this is a documentation-only change with well-structured code examples, proper organization, and correct path updates for all relocated files. The new Sphinx extension is clean and follows proper patterns.
  • No files require special attention - all changes are documentation enhancements

Important Files Changed

File Analysis

Filename Score Overview
docs/_extensions/dali_tabs.py 5/5 New Sphinx extension that transforms container:: dali-tabs blocks into proper tab-set RST for rendering pipeline/dynamic mode examples
README.rst 5/5 Front page updated with tabbed examples showing both pipeline mode and dynamic mode usage patterns
docs/conf.py 5/5 Configuration updated to register dali_tabs extension and pass references to dynamic mode documentation
docs/doc_index.py 5/5 Enhanced to support dynamic. operator prefix mapping to experimental.dynamic and handle sphinx title syntax in paths
docs/examples/getting_started/index.rst 5/5 New landing page explaining the difference between pipeline and dynamic modes with links to separate tutorials
docs/examples/getting_started/dynamic_mode.ipynb 5/5 Comprehensive dynamic mode tutorial showing readers, decoding, augmentations, GPU acceleration, and performance comparison

Sequence Diagram

sequenceDiagram
    participant User
    participant Sphinx
    participant dali_tabs
    participant doc_index
    participant Notebooks
    
    User->>Sphinx: Build documentation
    Sphinx->>doc_index: Process examples/index.py
    doc_index->>doc_index: Generate index.rst files
    doc_index->>doc_index: Map operator references<br/>(dynamic.* → experimental.dynamic.*)
    doc_index->>doc_index: Handle title syntax<br/>("Title <filename>")
    
    Sphinx->>dali_tabs: include-read event
    dali_tabs->>dali_tabs: Find "container:: dali-tabs"
    dali_tabs->>dali_tabs: Parse **Tab Name:** sections
    dali_tabs->>dali_tabs: Transform to tab-set RST<br/>with sync-group
    dali_tabs->>Sphinx: Return transformed RST
    
    Sphinx->>Notebooks: Render pipeline_mode.ipynb
    Notebooks->>Sphinx: Display pipeline examples
    
    Sphinx->>Notebooks: Render dynamic_mode.ipynb
    Notebooks->>Sphinx: Display dynamic examples
    
    Sphinx->>User: HTML documentation with<br/>synchronized tabs
Loading

@rostan-t
Copy link
Collaborator Author

rostan-t commented Jan 8, 2026

!build

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile Overview

Greptile Summary

This PR adds comprehensive dynamic mode examples to DALI documentation, complementing the existing pipeline mode examples. The implementation is well-structured and provides excellent coverage of dynamic mode functionality.

Key Changes:

  • Added dali_tabs.py Sphinx extension that transforms container:: dali-tabs blocks into synchronized tab-sets for dual-mode display
  • Updated front page (README.rst) with tabbed examples showing both pipeline and dynamic modes
  • Reorganized documentation structure: converted single notebooks into directories with pipeline_mode.ipynb and dynamic_mode.ipynb variants
  • Added 22 new dynamic mode notebooks covering data loading, image processing, audio processing, and video processing
  • Updated relative paths in refactored notebooks to account for new directory structure

Documentation Coverage:
The PR adds dynamic mode variants for all major functionality areas: file readers (LMDB, RecordIO, TFRecord, Webdataset, COCO, NumPy), image processing operations (decoders, augmentations, color transforms, resize, warp), audio processing (decoder, spectrogram), and video processing (simple pipelines, labels, timestamps, optical flow).

Technical Quality:

  • Dynamic mode examples correctly use ndd. (nvidia.dali.experimental.dynamic) prefix instead of fn.
  • Proper device handling: dynamic mode uses device="gpu" while pipeline mode uses device="mixed"
  • Relative paths properly updated (e.g., ../data/images to ../../data/images after directory restructure)
  • The dali_tabs.py extension is cleanly implemented with appropriate regex patterns and error handling
  • Tab synchronization via sync-group: dali-mode ensures consistent user experience across pages

Minor Observations:

  • The PR correctly deleted the old monolithic getting_started.ipynb in favor of the new split structure
  • Index files use the new doc_entry() structure to properly reference both pipeline and dynamic mode operators
  • The doc_index.py update correctly prefixes dynamic operators with "experimental." for internal references

Confidence Score: 5/5

  • This PR is safe to merge with very high confidence - it's a well-executed documentation enhancement with no functional code changes
  • Score reflects the documentation-only nature of changes (no production code modified), excellent implementation quality of the new Sphinx extension, proper handling of relative paths in refactored notebooks, correct usage of dynamic mode vs pipeline mode syntax, and comprehensive coverage of dynamic mode functionality across all example categories
  • No files require special attention - all changes are documentation enhancements with proper structure and correct syntax

Important Files Changed

File Analysis

Filename Score Overview
docs/_extensions/dali_tabs.py 5/5 New Sphinx extension that transforms container:: dali-tabs blocks into tab-set directives for dual-mode documentation display. Clean implementation with proper regex patterns and error handling.
README.rst 5/5 Updated front page with tabbed code examples showing both pipeline and dynamic modes. Uses container:: dali-tabs syntax that renders properly in both GitHub and HTML docs.
docs/conf.py 5/5 Added dali_tabs to extensions list and updated doc generation to include dynamic mode operator tables and autodoc.
docs/doc_index.py 5/5 Updated to handle dynamic mode operator references by prefixing "dynamic." with "experimental."
docs/examples/getting_started/index.rst 5/5 New landing page explaining pipeline vs dynamic mode with toctree to both tutorial variants.
docs/examples/getting_started/dynamic_mode.ipynb 5/5 Comprehensive dynamic mode tutorial covering readers, decoding, augmentations, GPU acceleration, and performance testing. Well-structured with clear explanations.
docs/examples/getting_started/pipeline_mode.ipynb 5/5 Refactored from original getting_started.ipynb with updated relative paths. Covers pipeline mode fundamentals comprehensively.
docs/examples/image_processing/decoder/dynamic_mode.ipynb 5/5 Dynamic mode examples for image decoders with proper device="gpu" usage (not "mixed"). Demonstrates decoder variants comprehensively.
docs/examples/image_processing/decoder/pipeline_mode.ipynb 5/5 Renamed from original with updated relative paths (../../data/images). Uses device="mixed" correctly for pipeline mode.
docs/examples/image_processing/decoder/index.py 5/5 New index structure with separate entries for pipeline and dynamic modes, each with appropriate operator references.

Sequence Diagram

sequenceDiagram
    participant User as User/GitHub
    participant Sphinx as Sphinx Build
    participant DaliTabs as dali_tabs.py Extension
    participant DocIndex as doc_index.py
    participant IndexPy as index.py Files
    
    User->>Sphinx: Build Documentation
    Sphinx->>Sphinx: Load conf.py configuration
    Sphinx->>Sphinx: Register dali_tabs extension
    
    Sphinx->>DocIndex: Execute doc_index.py
    DocIndex->>IndexPy: Process examples/index.py
    loop For each example category
        IndexPy->>DocIndex: Return Doc with entries
        DocIndex->>DocIndex: Generate index.rst files
        DocIndex->>DocIndex: Collect operator references
    end
    DocIndex-->>Sphinx: Return operator->example mapping
    
    Sphinx->>Sphinx: Process RST files
    loop For each RST file with dali-tabs
        Sphinx->>DaliTabs: include-read event
        DaliTabs->>DaliTabs: Find "container:: dali-tabs" blocks
        DaliTabs->>DaliTabs: Parse **Pipeline mode:** and **Dynamic mode:** sections
        DaliTabs->>DaliTabs: Transform to tab-set with sync-group
        DaliTabs-->>Sphinx: Return transformed RST
    end
    
    Sphinx->>Sphinx: Render HTML with sphinx_design tabs
    Sphinx-->>User: Generated Documentation
Loading

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [41342923]: BUILD FAILED

When a notebook has a pipeline mode and dynamic mode variants,
merge them and add a switcher button on the top.

Signed-off-by: Rostan Tabet <[email protected]>
'Interpolation method' and '3D Transforms' cannot be translated yet because of a bug.

Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
@szkarpinski
Copy link
Collaborator

szkarpinski commented Jan 12, 2026

Most notebooks still have "Defining and Running the Pipeline" section which is quite confusing with ndd:

image

options=":maxdepth: 1",
entries=[
doc_entry(
"Pipeline Mode <pipeline_mode.ipynb>",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: here we can choose if we want dynamic mode or pipeline mode, but then the actual notebook title doesn't indicate which one it is - you need to look at the menu on the left (or notebook contents).

Image

Maybe having a title like "Name Of The Notebook (Pipeline Mode)" would be better?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another related problem is that a user eager to learn about use cases can potentially skip the actual dynamic mode docs and land here without knowing what the dynamic mode is. Would it be possible to start each notebook with something like "This notebook describes how to [topic] with DALI dynamic mode. To learn more about DALI dynamic mode, visit [link]"?

Copy link
Collaborator Author

@rostan-t rostan-t Jan 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the first problem, it's possible to modify the breadcrumb or automatically add a badge to indicate the mode. Which one do you prefer? (or do we keep both?)
Capture d’écran 2026-01-12 à 19 08 28

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I'd keep both, but I don't have any strong opinions here.

@review-notebook-app
Copy link

review-notebook-app bot commented Jan 12, 2026

View / edit / reply to this conversation on ReviewNB

szkarpinski commented on 2026-01-12T15:32:19Z
----------------------------------------------------------------

The link directs the user to the pipeline fn.readers.numpy reader


Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 file reviewed, 1 comment

Edit Code Review Agent Settings | Greptile

@rostan-t
Copy link
Collaborator Author

Most notebooks still have "Defining and Running the Pipeline" section which is quite confusing with ndd:

Fixed by 7cf7324

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@rostan-t
Copy link
Collaborator Author

!build

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [41575994]: BUILD STARTED

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [41575994]: BUILD FAILED

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@rostan-t
Copy link
Collaborator Author

!build

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [41632777]: BUILD STARTED

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [41632777]: BUILD FAILED

Copy link
Member

@stiepan stiepan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@rostan-t
Copy link
Collaborator Author

!build

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [42620515]: BUILD STARTED

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [42620515]: BUILD FAILED

@rostan-t
Copy link
Collaborator Author

!build

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [42746344]: BUILD STARTED

@rostan-t
Copy link
Collaborator Author

!build

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [42805053]: BUILD STARTED

@dali-automaton
Copy link
Collaborator

CI MESSAGE: [42805053]: BUILD PASSED

@rostan-t rostan-t merged commit 59644c2 into NVIDIA:main Jan 29, 2026
8 checks passed
@rostan-t rostan-t deleted the ndd-examples branch January 29, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants