Rephrase external column data. #145
Conversation
| api-external-column-data: | ||
| status: full | ||
| api-parquet-summary-file: | ||
| status: none |
There was a problem hiding this comment.
@platypii I looked at hyparquet documentation quickly and didn't see this mentioned but please let me know if I missed it.
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thanks @emkornfield
Assuming apache/parquet-format#542 is correct I think this is a good change too.
I rendered it locally and it looks good to me
| note: "In parquet.thrift: ColumnChunk->file_path" | ||
| - id: api-parquet-summary-file | ||
| display_name: Parquet summary file | ||
| note: "Files named _metadata that consolidate footers from multiple parquet files." |
There was a problem hiding this comment.
I think the reference to the file_path field will help keep this specific. So perhaps something like
| note: "Files named _metadata that consolidate footers from multiple parquet files." | |
| note: "Files named _metadata that consolidate footers from multiple parquet files, stored in ColumnChunk->file_path" |
There was a problem hiding this comment.
Is "parquet summary file" really the same as supporting file_path?
Maybe summary files are the only known use case of actually using file_path? But nothing in the spec says file_path is necessarily associated with "summary files". In fact is there anything in the spec about summary file or is that just an implementation choice some people made?
I'm not opposed to having a row for support of summary files, but that seems like a different question than file_path support?
There was a problem hiding this comment.
Yeah, I'd err on not including that a reference to file_path here.
In fact is there anything in the spec about summary file or is that just an implementation choice some people made?
There is not apache/parquet-format#542 has some more details.
|
going to merge. Thanks for the reviews. |
3 participants
Based on support matrix I believe this was meant to capture Parquet summary files ("_metadata") and not reading external data.
Related format PR: apache/parquet-format#542