[DISCO-3809] Establish a common Elasticsearch adapter for providers and jobs

[misaniwere]

References

JIRA: DISCO-3809

Description

This PR introduces a common Elasticsearch adapter. As more components rely on Elasticsearch, we want to standardize how clients are initialized and how common operations (index lifecycle, bulk, search, etc.) are handled, while still leaving room for future flexibility.

Direction

We are keeping the Elasticsearch adapter configurable, allowing it to be initialized with an arbitrary DSN and API key. If Merino providers and jobs are using the same Elasticsearch cluster, the intended direction is that they should also use a shared credential set (DSN + API key), rather than defining provider-specific credentials. We may need to distinguish read vs write access (e.g., separate API keys). We can confirm if we do, but the goal is to keep the number of credentials small and well-defined.

As a follow-up, we plan to:

• Consolidate Elasticsearch credentials into one or two shared config variables (e.g. read and write).
• Update providers and jobs to consume these shared credentials by default.
• Update Airflow jobs to use the unified credentials.

This approach allows us to:

• Standardize behavior and reduce duplication.
• Avoid locking ourselves into a single global client prematurely.
• Preserve flexibility for future multi-cluster or least-privilege needs.

Followup

• update sports provider to use the new adapter
• consolidate credentials for all providers (make them point to one config)

PR Review Checklist

Put an x in the boxes that apply

• This PR conforms to the Contribution Guidelines
• The PR title starts with the JIRA issue reference, format example [DISCO-####], and has the same title (if applicable)
• [load test: (abort|skip|warn)] keywords are applied to the last commit message (if applicable)
• Documentation has been updated (if applicable)
• Functional and performance test coverage has been expanded and maintained (if applicable)

┆Issue is synchronized with this Jira Task