Using copilot for adding features from track/1 to 2/main

[amandahla]

What this PR does

This PR aims to bring Synapse back to a single branch/track.

Since the track/1 and 2/main branches have diverged significantly over time, this is not a strict cherry-pick. Instead, it follows a “port and adapt” approach, applying the relevant changes from one branch to the other.

Due to the complexity involved, this PR intentionally excludes commits that introduce refactoring only. It focuses solely on the structural changes required to make 2/main compatible with production deployment once MAS is enabled via a flag.

Reviewers:
For simplicity and easier comparison, this PR aims to stay as close as possible to the original structure. Please consider focusing on the branch differences rather than refactoring or Python-only changes. They can be done in a further PR.

This is the prompt that I'm using:

Act as an experienced Python developer expert in Git.

This project synapse-operator has 2 main branches: 2/main and track/1. I want to make the 2/main the only one and to do this I need to cherry-pick some commits from track/1. The challenge is that they are too different and this should be done carefully to not break anything.

I'll give you a list of commits available in the track/1 that should be applied in the current copilot-merge-3 branch. For each one you should:

in case of conflicts, also keep in mind to change the less as possible in the current implementation,
in case of conflicts, if there are many you can analise the commit, find the main goal and then apply it to the 2/main branch instead.
in case of files that dont exist in the current branch, you can copy them from the track/1 branch.
some commits has features and refactor together. give priority to the features.
for all changes, make sure that tox -e fmt, tox -e lint and tox -e unit run without errors before and after applying the changes. In case of lint errors, some cases might be ignored instead of fixing it and to know what are the cases find reference in the existing files (for example, noqa comment entries)
The first commit is c46320bab05cca8e5d120dabe7108d21b1d9140c

Goal:
c46320b | Store macaroon key (#981) | OK
abeaed1 | Change default room version to 12 (#985) | OK
810fdc3 | Change main unit to always be the 0 (and more) (#870) | OK
2367b3d | Add two env vars to s3 command (#837) | OK
f6be3e1 | Add rate_limiting_level config (#990) | OK

Extra: ignore failure in Synapse Stats Exporter pebble layer.

Why we need it

Creating a single branch for Synapse.

Checklist

• I followed the contributing guide
• I added or updated the documentation (if applicable)
• I updated docs/changelog.md with user-relevant changes
• I added a change artifact for user-relevant changes in docs/release-notes/artifacts. If no change artifact is necessary, I tagged the PR with the label no-release-note.
• I added or updated tests as needed (unit and integration)
• If integration test modules are used: I updated the workflow configuration
  (e.g., in .github/workflows/integration_tests.yaml, ensure the modules list is correct)
• If this is a Grafana dashboard: I added a screenshot of the dashboard
• If this is Terraform: terraform fmt passes and tflint reports no errors
• If this is Rockcraft: I updated the version

[amandahla]

New prompt for complex commit:

Lest go for the next commit. The id is 810fdc3068c542946938dcc64d69837623f2ba65

Note that this one is more complex but I want you to focus on the items described in the related PR to the commit:

Adjusted how the main unit starts by consistently using "synapse/0".
Revised how the instance_map config is generated based on the planned units number.
Remove checks from the Pebble health endpoint (the checks are still there but no longer reflects on the result of the /health endpoint used by Kubernetes)

Affected files to check in the commit:
src/pebble.py (check.level were removed, mind to update the unit tests properly)
rockcraft.yaml (a --no-deps were added, check if the current file has this too)
src/charm.py - this is the file where changes related to the first 2 items were made. Mind the is_main,istance_map,removal of relation departed,removal of peer units total etc. The main goal of this change is guarantee that MAIN_UNIT_ID (unit final 0) is always the main so thats why so many logic were removed from charm.py. There are many other changes, mostly related to Mjolnir (not existing in the branch) or refactoring, so keep it focused.

Ignore changes in integration test.
Ignore changes in unit tests related to refactor, change only what is needed for the items that I mentioned previously.
ignore changes in renovate/tox/pyproject.

When in doubt if the change relates to these items, ask my confirmation before the change.

[amandahla]

For reference, I requested Copilot to improve my prompt and this is the result:

Let's go for the next commit. The id is 810fdc3068c542946938dcc64d69837623f2ba65

Note that this one is more complex but I want you to focus on the items described in the related PR to the commit:

1. Adjusted how the main unit starts by consistently using "synapse/0" (MAIN_UNIT_ID = 0)
2. Revised how the instance_map config is generated based on the planned units number
3. Remove check.level from the Pebble health endpoint (the checks are still there but no longer reflect on the /health endpoint used by Kubernetes)

## Affected files to check in the commit:

**src/pebble.py**
- check.level were removed from all check functions
- Mind to update any unit tests that verify these check functions

**rockcraft.yaml** 
- A --no-deps flag was added, check if the current file has this too

**src/charm.py** - Primary focus file
- is_main: changed from method to @property
- instance_map: replaced with create_instance_map() using app.planned_units()
- set_main_unit: removed entirely (method and all calls)
- Removal of set_main_unit calls in event handlers (relation departed, leader elected)
- Addition of _get_unit_address() helper
- MAIN_UNIT_ID constant should be defined in state/charm_state.py

The main goal: Guarantee that MAIN_UNIT_ID (unit ending in /0) is always the main unit, which is why much logic was removed from charm.py. There are many other changes in track/1, mostly related to Mjolnir (not existing in this branch) or refactoring, so keep it focused on these three items.

## Unit Tests - Important Context:

**tests/unit/test_charm_scaling.py**
- This file was REMOVED in track/1 due to extensive refactoring
- Remove it entirely, don't try to adapt it

**tests/unit_ops/test_charm.py**
- New scenario tests were ADDED for multi-unit configurations:
  - test_config_changed_multiple_units_with_redis
  - test_config_changed_workers_ignore_list
- These tests will need:
  - MAS context secret fixture (mas_context_secret) with proper field lengths
  - Mocking of get_signing_key() and get_macaroon_key() methods
  - Multiple units base state fixture with mas_context_secret included

**Test fixtures in tests/unit_ops/conftest.py**
- Add mas_context_secret fixture with proper MAS validation requirements:
  - encryption-key: 64 chars (32 bytes hex)
  - signing-key-id: 8 chars (4 bytes hex)
  - synapse-shared-secret: 32 chars (16 bytes hex)
  - synapse-oidc-client-secret: 32 chars (16 bytes hex)
- Update multiple_units_base_state fixture to include mas_context_secret

**Expected test impact**
- Some existing tests may fail if they depend on is_main being a method
- The moderation test may fail due to MAS context not being set for non-leader units
- Coverage threshold may need adjustment (likely from 87% to ~85%)

## Ignore:
- Changes in integration tests
- Changes in renovate/tox/pyproject (except coverage threshold)
- Refactoring-only changes in unit tests (Mjolnir/moderation renamings, etc.)

## When in doubt:
If a change doesn't clearly relate to these three items (main unit logic, instance_map, or check.level), ask for confirmation before making it.

[DeeKay3]

LGTM - small nitpick

[amandahla]

Summary (generated by Copilot)

• Force Python 3.12 for unit tests due Pydantic compatibility

• Store macaroon key

Adapt commit c46320b from track/1.

• Add macaroon key management methods to SynapseCharm class
• Integrate macaroon key handling in reconcile method
• Update backup to include macaroon keys (*.key pattern)
• Update tests to account for macaroon key files
• Adjust coverage threshold from 89% to 87%

The implementation follows the existing signing key pattern in charm.py
since this branch doesn't have a separate signing_key.py module.

• Change default room version to 12

Adapt commit abeaed1 from track/1.

• Set default_room_version to 12 in add_default_configurations
• Update test to expect default_room_version field
• Fix lint issue in test_backup.py with noqa comment

The refactoring to consolidate default configurations in one method
was already done in this branch, so only the room version change was needed.

• Change main unit to always be synapse/0

• Add MAIN_UNIT_ID = 0 constant to state/charm_state.py
• Convert is_main() method to @Property checking if '/0' in unit name
• Add _get_unit_address() helper method for unit address formatting
• Replace instance_map() with create_instance_map() using app.planned_units()
• Remove set_main_unit() method and all calls from event handlers
• Remove check.level from all Pebble health checks (synapse_alive, synapse_ready, nginx_ready)
• Remove tests/unit/test_charm_scaling.py (extensively refactored in track/1)
• Add scenario tests for multi-unit configurations with redis
• Add MAS context secret fixture with proper validation
• Adjust coverage threshold from 87% to 85%

Main unit is now always synapse/0, workers use other unit IDs.
Instance map dynamically generated from planned_units().

• Add two env vars to s3 command

Add AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION
environment variables to S3 backup configuration with WHEN_REQUIRED value.

• Add rate_limiting_level config

Replace rc_joins_remote_burst_count and rc_joins_remote_per_second
configuration options with a simpler rate_limiting_level option that
can be set to 'default', 'permissive', or 'unlimited'.

This simplifies rate limiting configuration by providing preset levels
that configure multiple rate limiting parameters at once.

• Ignore failures for Synapse stats exporter