docs: Add documentation for metadata fields, feature flags, custom column types, and troubleshooting #2648
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add new Troubleshooting section under Using Estuary (guides) - Add article: Fixing Schema Inference Issues from Bad Upstream Data - Documents dataflow reset, castToString, and ddl override options - Fix Snowflake key-pair auth SQL to use identifier() function - Add .gitleaksignore for example private key in docs Co-Authored-By: Claude Opus 4.5 <[email protected]>
Documents connector feature flags including: - Materialization flags (allow_existing_tables_for_new_bindings, retain_existing_data_on_backfill, datetime_keys_as_string) - Capture schema inference flags (use_schema_inference, emit_sourced_schemas) - Migration steps for existing captures to adopt new schema inference behavior Co-Authored-By: Claude Opus 4.5 <[email protected]>
Explains the metadata fields that captures add to documents including standard fields (uuid, op, source) and connector-specific source fields for PostgreSQL, MySQL, SQL Server, Oracle, MongoDB, and Snowflake. Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Add documentation for custom column types feature - Expand caveats for use_existing_table and retain_existing_data_on_backfill - Clarify datetime_keys_as_string default behavior Co-Authored-By: Claude Opus 4.5 <[email protected]>
|
- Note that feature flags are for non-standard/complex situations - Mention some connectors expose them in UI under Advanced settings Co-Authored-By: Claude Opus 4.5 <[email protected]>
… Usage Move metadata-fields, feature-flags, and custom-column-types docs to guides/advanced-usage/ for better discoverability under Using Estuary. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Contributor
Author
|
Hi @aeluce, I'm grouping some doc edits in this PR that I've accumulated over the last week. I'm making a concerted effort to make docs if there's a question that has come up multiple times. I've created a new Using Estuary -> Advanced and Troubleshooting section for these new articles, but I'm not fussed about their exact location. |
- Add capture example for feature flags (not just materialization) - Add explicit JSON paths for LLM/programmatic modification - Show how to append to existing feature_flags string - Clarify how to locate bindings by source collection name Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add comprehensive troubleshooting section covering: - Write vs read schema explanation - Auto-inference behavior (widens only) - NULL values in new columns - Data flow interruption diagnosis - Transient validation errors during schema changes - onIncompatibleSchemaChange configuration - NoSQL type inference issues Based on analysis of 316 schema-related support issues. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Contributor
Author
|
Added schema evolution troubleshooting guide to
Based on analysis of 316 schema-related support issues from July-Dec 2025. |
Documents how schema inference complexity limits work (1,000 fields for most sources, 10,000 for SQL) and how to re-add pruned fields by adding them to the readSchema's third allOf entry. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add 3 new troubleshooting sections to schema-evolution.md: - Schema not yet available for new collections - Binding not found - Unsupported DDL operations (MySQL/MariaDB CDC) These sections provide documentation targets for upcoming error message improvement tickets that will add links to relevant troubleshooting docs. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
_metafields documentation explaining capture metadata (uuid, op, source fields by connector)use_existing_table,retain_existing_data_on_backfill, etc.Test plan
🤖 Generated with Claude Code