Skip to content

refactor(query): refactor parquet reader cache optimization #19255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
zhang2014 wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

refactor(query): refactor parquet reader cache optimization #19255

zhang2014 wants to merge 6 commits into from

Conversation

@zhang2014
Copy link
Member

@zhang2014 zhang2014 commented Jan 14, 2026 •
edited by drmingdrmer
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

refactor(query): refactor parquet reader cache optimization

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@zhang2014
refactor(query): optimize ReadParquetDataTransform to reduce thread s…
12fa71c 
…witching for cache hits

Move cache read operations from async runtime to processor runtime when all
columns are cached. This reduces unnecessary thread switching overhead.

Changes:
- Add CachedReader struct to hold sync cache read results
- Add sync_read_columns_data() to synchronously check and read from cache
- Add async_read_columns_data() to async read cache-miss columns
- Add build_block_result_from_cache() to build result from cached data
- Implement NewReadParquetDataTransform using Processor trait directly
- Use Event::Sync for cache hits, Event::Async for cache misses
- Replace ReadParquetDataTransform with NewReadParquetDataTransform
@zhang2014
z
dd0e5e1
@zhang2014 zhang2014 added the ci-cloud Build docker image for cloud test label Jan 14, 2026
@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Jan 14, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 14, 2026

Docker Image for PR

  • tag: pr-19255-36f79cf-1768387539

note: this image tag is only available for internal use.

@zhang2014 zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jan 14, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 14, 2026

Docker Image for PR

  • tag: pr-19255-f7c746a-1768397979

note: this image tag is only available for internal use.

@zhang2014 zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jan 15, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 15, 2026

Docker Image for PR

  • tag: pr-19255-300ef77-1768455121

note: this image tag is only available for internal use.

@zhang2014 zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Jan 15, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 15, 2026

Docker Image for PR

  • tag: pr-19255-3be7874-1768482589

note: this image tag is only available for internal use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ci-cloud Build docker image for cloud test pr-refactor this PR changes the code base without new features or bugfix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zhang2014