-
Notifications
You must be signed in to change notification settings - Fork 658
Create examples showing ndd usage #6140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Signed-off-by: Rostan Tabet <[email protected]>
|
!build |
|
CI MESSAGE: [41293348]: BUILD FAILED |
Greptile SummaryOverviewThis PR successfully adds comprehensive dynamic mode (NDD) examples to DALI's documentation, creating a parallel set of tutorials and examples alongside the existing pipeline mode documentation. The PR introduces 23 new dynamic mode notebooks, updates the front page with synchronized tabs, and implements a new Sphinx extension for seamless mode switching. Key Changes
Quality AssessmentStrengths:
Observations:
Test Coverage NoteExamples cannot be tested in standard CI without external data dependencies (DALI_EXTRA repository), which is handled appropriately in the test exclusions. Confidence Score: 5/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant Sphinx
participant dali_tabs
participant doc_index
participant nbsphinx
participant HTML
User->>Sphinx: sphinx-build docs
Sphinx->>doc_index: document_examples("examples/index.py")
doc_index->>doc_index: Recursively process .py index files
doc_index->>doc_index: Collect operator references
doc_index-->>Sphinx: Return {operator → examples_list}
Sphinx->>Sphinx: Read RST/notebook files
Note over Sphinx: Include .rst files with dali-tabs
Sphinx->>dali_tabs: include-read event
dali_tabs->>dali_tabs: Transform dali-tabs to tab-set
dali_tabs-->>Sphinx: Modified RST content
Sphinx->>nbsphinx: Process .ipynb files
nbsphinx->>nbsphinx: Add badges based on docname
nbsphinx-->>Sphinx: Rendered notebook HTML
Sphinx->>HTML: Generate final documentation
HTML-->>User: Dynamic/Pipeline tabs with badges
|
|
!build |
|
CI MESSAGE: [41295926]: BUILD FAILED |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Greptile Overview
Greptile Summary
This PR adds comprehensive dynamic mode examples to DALI documentation, creating parity with existing pipeline mode examples. A new Sphinx extension enables synchronized tabs for displaying both modes side-by-side.
Major Changes:
- Added
dali_tabs.pySphinx extension to transform container blocks into synchronized tab-sets - Split getting started tutorial into separate pipeline_mode.ipynb and dynamic_mode.ipynb variants
- Created dynamic_mode.ipynb examples for 20+ topics across data loading, image processing, audio processing, and video processing
- Reorganized documentation structure: many examples moved from flat files into subdirectories with
index.pyfiles - Enhanced
doc_index.pyto mapdynamic.*operator references toexperimental.dynamic.*and handle Sphinx title syntax - Updated README.rst front page with tabbed code examples
Documentation Quality:
- Dynamic mode examples are comprehensive and mirror pipeline mode functionality
- Examples include proper error handling, context managers (
EvalContext,EvalMode), and batch/tensor operations - Code demonstrates key dynamic mode patterns: direct operator calls, reader iteration, GPU/CPU data movement
- Path references updated correctly for relocated notebooks (e.g.,
../../data/imagesfor deeper nesting)
Confidence Score: 5/5
- This documentation PR is safe to merge with no code execution risks
- Score reflects that this is a documentation-only change with well-structured code examples, proper organization, and correct path updates for all relocated files. The new Sphinx extension is clean and follows proper patterns.
- No files require special attention - all changes are documentation enhancements
Important Files Changed
File Analysis
| Filename | Score | Overview |
|---|---|---|
| docs/_extensions/dali_tabs.py | 5/5 | New Sphinx extension that transforms container:: dali-tabs blocks into proper tab-set RST for rendering pipeline/dynamic mode examples |
| README.rst | 5/5 | Front page updated with tabbed examples showing both pipeline mode and dynamic mode usage patterns |
| docs/conf.py | 5/5 | Configuration updated to register dali_tabs extension and pass references to dynamic mode documentation |
| docs/doc_index.py | 5/5 | Enhanced to support dynamic. operator prefix mapping to experimental.dynamic and handle sphinx title syntax in paths |
| docs/examples/getting_started/index.rst | 5/5 | New landing page explaining the difference between pipeline and dynamic modes with links to separate tutorials |
| docs/examples/getting_started/dynamic_mode.ipynb | 5/5 | Comprehensive dynamic mode tutorial showing readers, decoding, augmentations, GPU acceleration, and performance comparison |
Sequence Diagram
sequenceDiagram
participant User
participant Sphinx
participant dali_tabs
participant doc_index
participant Notebooks
User->>Sphinx: Build documentation
Sphinx->>doc_index: Process examples/index.py
doc_index->>doc_index: Generate index.rst files
doc_index->>doc_index: Map operator references<br/>(dynamic.* → experimental.dynamic.*)
doc_index->>doc_index: Handle title syntax<br/>("Title <filename>")
Sphinx->>dali_tabs: include-read event
dali_tabs->>dali_tabs: Find "container:: dali-tabs"
dali_tabs->>dali_tabs: Parse **Tab Name:** sections
dali_tabs->>dali_tabs: Transform to tab-set RST<br/>with sync-group
dali_tabs->>Sphinx: Return transformed RST
Sphinx->>Notebooks: Render pipeline_mode.ipynb
Notebooks->>Sphinx: Display pipeline examples
Sphinx->>Notebooks: Render dynamic_mode.ipynb
Notebooks->>Sphinx: Display dynamic examples
Sphinx->>User: HTML documentation with<br/>synchronized tabs
|
!build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Greptile Overview
Greptile Summary
This PR adds comprehensive dynamic mode examples to DALI documentation, complementing the existing pipeline mode examples. The implementation is well-structured and provides excellent coverage of dynamic mode functionality.
Key Changes:
- Added
dali_tabs.pySphinx extension that transformscontainer:: dali-tabsblocks into synchronized tab-sets for dual-mode display - Updated front page (README.rst) with tabbed examples showing both pipeline and dynamic modes
- Reorganized documentation structure: converted single notebooks into directories with
pipeline_mode.ipynbanddynamic_mode.ipynbvariants - Added 22 new dynamic mode notebooks covering data loading, image processing, audio processing, and video processing
- Updated relative paths in refactored notebooks to account for new directory structure
Documentation Coverage:
The PR adds dynamic mode variants for all major functionality areas: file readers (LMDB, RecordIO, TFRecord, Webdataset, COCO, NumPy), image processing operations (decoders, augmentations, color transforms, resize, warp), audio processing (decoder, spectrogram), and video processing (simple pipelines, labels, timestamps, optical flow).
Technical Quality:
- Dynamic mode examples correctly use
ndd.(nvidia.dali.experimental.dynamic) prefix instead offn. - Proper device handling: dynamic mode uses
device="gpu"while pipeline mode usesdevice="mixed" - Relative paths properly updated (e.g.,
../data/imagesto../../data/imagesafter directory restructure) - The
dali_tabs.pyextension is cleanly implemented with appropriate regex patterns and error handling - Tab synchronization via
sync-group: dali-modeensures consistent user experience across pages
Minor Observations:
- The PR correctly deleted the old monolithic
getting_started.ipynbin favor of the new split structure - Index files use the new
doc_entry()structure to properly reference both pipeline and dynamic mode operators - The
doc_index.pyupdate correctly prefixes dynamic operators with "experimental." for internal references
Confidence Score: 5/5
- This PR is safe to merge with very high confidence - it's a well-executed documentation enhancement with no functional code changes
- Score reflects the documentation-only nature of changes (no production code modified), excellent implementation quality of the new Sphinx extension, proper handling of relative paths in refactored notebooks, correct usage of dynamic mode vs pipeline mode syntax, and comprehensive coverage of dynamic mode functionality across all example categories
- No files require special attention - all changes are documentation enhancements with proper structure and correct syntax
Important Files Changed
File Analysis
| Filename | Score | Overview |
|---|---|---|
| docs/_extensions/dali_tabs.py | 5/5 | New Sphinx extension that transforms container:: dali-tabs blocks into tab-set directives for dual-mode documentation display. Clean implementation with proper regex patterns and error handling. |
| README.rst | 5/5 | Updated front page with tabbed code examples showing both pipeline and dynamic modes. Uses container:: dali-tabs syntax that renders properly in both GitHub and HTML docs. |
| docs/conf.py | 5/5 | Added dali_tabs to extensions list and updated doc generation to include dynamic mode operator tables and autodoc. |
| docs/doc_index.py | 5/5 | Updated to handle dynamic mode operator references by prefixing "dynamic." with "experimental." |
| docs/examples/getting_started/index.rst | 5/5 | New landing page explaining pipeline vs dynamic mode with toctree to both tutorial variants. |
| docs/examples/getting_started/dynamic_mode.ipynb | 5/5 | Comprehensive dynamic mode tutorial covering readers, decoding, augmentations, GPU acceleration, and performance testing. Well-structured with clear explanations. |
| docs/examples/getting_started/pipeline_mode.ipynb | 5/5 | Refactored from original getting_started.ipynb with updated relative paths. Covers pipeline mode fundamentals comprehensively. |
| docs/examples/image_processing/decoder/dynamic_mode.ipynb | 5/5 | Dynamic mode examples for image decoders with proper device="gpu" usage (not "mixed"). Demonstrates decoder variants comprehensively. |
| docs/examples/image_processing/decoder/pipeline_mode.ipynb | 5/5 | Renamed from original with updated relative paths (../../data/images). Uses device="mixed" correctly for pipeline mode. |
| docs/examples/image_processing/decoder/index.py | 5/5 | New index structure with separate entries for pipeline and dynamic modes, each with appropriate operator references. |
Sequence Diagram
sequenceDiagram
participant User as User/GitHub
participant Sphinx as Sphinx Build
participant DaliTabs as dali_tabs.py Extension
participant DocIndex as doc_index.py
participant IndexPy as index.py Files
User->>Sphinx: Build Documentation
Sphinx->>Sphinx: Load conf.py configuration
Sphinx->>Sphinx: Register dali_tabs extension
Sphinx->>DocIndex: Execute doc_index.py
DocIndex->>IndexPy: Process examples/index.py
loop For each example category
IndexPy->>DocIndex: Return Doc with entries
DocIndex->>DocIndex: Generate index.rst files
DocIndex->>DocIndex: Collect operator references
end
DocIndex-->>Sphinx: Return operator->example mapping
Sphinx->>Sphinx: Process RST files
loop For each RST file with dali-tabs
Sphinx->>DaliTabs: include-read event
DaliTabs->>DaliTabs: Find "container:: dali-tabs" blocks
DaliTabs->>DaliTabs: Parse **Pipeline mode:** and **Dynamic mode:** sections
DaliTabs->>DaliTabs: Transform to tab-set with sync-group
DaliTabs-->>Sphinx: Return transformed RST
end
Sphinx->>Sphinx: Render HTML with sphinx_design tabs
Sphinx-->>User: Generated Documentation
|
CI MESSAGE: [41342923]: BUILD FAILED |
Signed-off-by: Rostan Tabet <[email protected]>
When a notebook has a pipeline mode and dynamic mode variants, merge them and add a switcher button on the top. Signed-off-by: Rostan Tabet <[email protected]>
'Interpolation method' and '3D Transforms' cannot be translated yet because of a bug. Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
…ction Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
| options=":maxdepth: 1", | ||
| entries=[ | ||
| doc_entry( | ||
| "Pipeline Mode <pipeline_mode.ipynb>", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another related problem is that a user eager to learn about use cases can potentially skip the actual dynamic mode docs and land here without knowing what the dynamic mode is. Would it be possible to start each notebook with something like "This notebook describes how to [topic] with DALI dynamic mode. To learn more about DALI dynamic mode, visit [link]"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! I'd keep both, but I don't have any strong opinions here.
|
View / edit / reply to this conversation on ReviewNB szkarpinski commented on 2026-01-12T15:32:19Z The link directs the user to the pipeline fn.readers.numpy reader |
Signed-off-by: Rostan Tabet <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1 file reviewed, 1 comment
Signed-off-by: Rostan Tabet <[email protected]>
Signed-off-by: Rostan Tabet <[email protected]>
Fixed by 7cf7324 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No files reviewed, no comments
|
!build |
|
CI MESSAGE: [41575994]: BUILD STARTED |
|
CI MESSAGE: [41575994]: BUILD FAILED |
Signed-off-by: Rostan Tabet <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No files reviewed, no comments
|
!build |
|
CI MESSAGE: [41632777]: BUILD STARTED |
|
CI MESSAGE: [41632777]: BUILD FAILED |
Signed-off-by: Rostan Tabet <[email protected]>
stiepan
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great!
|
!build |
|
CI MESSAGE: [42620515]: BUILD STARTED |
|
CI MESSAGE: [42620515]: BUILD FAILED |
|
!build |
|
CI MESSAGE: [42746344]: BUILD STARTED |
|
!build |
|
CI MESSAGE: [42805053]: BUILD STARTED |
|
CI MESSAGE: [42805053]: BUILD PASSED |



Category: Documentation
Description:
Currently, most examples in the documentation only show pipeline mode. This PR adds dynamic mode examples.
Because of the third point, a lot of pages have been renamed (generally to
pipeline_mode.ipynb) with little or no changes (when they contained links with relative paths, those needed to be changed).Additional information:
Affected modules and functionalities:
Pages affected by this PR:
Key points relevant for the review:
The PR contains little code changes apart from documentation pages so it's best reviewed directly as the rendered HTML doc.
Tests:
Checklist
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: DALI-4515