Skip to content

Conversation

@adamlazik1
Copy link
Contributor

What changes are you introducing?

Documenting the SSH certificate feature https://projects.theforeman.org/issues/38478)

Why are you introducing these changes? (Explanation, links to references, issues, etc.)

Anything else to add? (Considerations, potential downsides, alternative solutions you have explored, etc.)

Contributor checklists

  • I am okay with my commits getting squashed when you merge this PR.
  • I am familiar with the contributing guidelines.

Please cherry-pick my commits into:

  • Foreman 3.17/Katello 4.19
  • Foreman 3.16/Katello 4.18 (Satellite 6.18; orcharhino 7.6)
  • Foreman 3.15/Katello 4.17
  • Foreman 3.14/Katello 4.16 (Satellite 6.17; orcharhino 7.4; orcharhino 7.5)
  • Foreman 3.13/Katello 4.15 (EL9 only)
  • Foreman 3.12/Katello 4.14 (Satellite 6.16; orcharhino 7.2 on EL9 only; orcharhino 7.3)
  • We do not accept PRs for Foreman older than 3.12.

@github-actions github-actions bot added Needs tech review Requires a review from the technical perspective Needs style review Requires a review from docs style/grammar perspective Needs testing Requires functional testing labels Jan 16, 2026
@adamlazik1 adamlazik1 force-pushed the ssh-cert-support branch 2 times, most recently from 631b8b4 to 105f147 Compare January 16, 2026 13:24
@adamlazik1 adamlazik1 changed the title Add SSH cerrtificates documentation Add SSH certificates documentation Jan 16, 2026
@adamlazik1 adamlazik1 force-pushed the ssh-cert-support branch 3 times, most recently from f883cd4 to c957b28 Compare January 16, 2026 14:33
@adamlazik1
Copy link
Contributor Author

Style review appreciated :)

Copy link
Contributor

@maximiliankolb maximiliankolb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add [role="_abstract"] to each module before the first paragraph.

:client-provisioning-template-type: {parent-client-provisioning-template-type}
endif::[]

include::modules/con_using-ssh-certificates.adoc[leveloffset=+1]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you feel about moving this to a new assembly assembly_using-ssh-certificates.adoc?

I also suggestion to add "[leveloffset=+1]" to the assembly, "[]" to the first concept, and "[leveloffset=+1]" for all other modules.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I second Maximilian's suggestion: Adding some sort of structure to separate the new use case from the other sections would help with navigation. "Configuring and setting up remote jobs" already has dozens of sections (😱) so extra help navigating would surely be appreciated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, applied.

You can disable SSH host certificate verification on {SmartProxy} and revert to the Trust On First Use (TOFU) model.

.Procedure
* On {SmartProxy}, enter the following command:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* On {SmartProxy}, enter the following command:
* On {SmartProxy}, disable SSH host key verification:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied, slightly reworded.

:client-provisioning-template-type: {parent-client-provisioning-template-type}
endif::[]

include::modules/con_using-ssh-certificates.adoc[leveloffset=+1]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I second Maximilian's suggestion: Adding some sort of structure to separate the new use case from the other sections would help with navigation. "Configuring and setting up remote jobs" already has dozens of sections (😱) so extra help navigating would surely be appreciated.

* You have a Certificate Authority for signing SSH certificates.

.Procedure
Perform these steps on each host that uses the {SmartProxy} for remote execution and was registered before you enabled SSH certificate authentication on the {SmartProxy}.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The .Procedure must contain only an uninterrupted sequence of steps to comply with the DITA migration tool. You can move this to the module introduction.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied.

Comment on lines 16 to 26
. Enter the following commands:
+
[options="nowrap", subs="+quotes,verbatim,attributes"]
----
# mkdir -p /etc/ssh/sshd_config.d
# echo 'TrustedUserCAKeys _My_CA_Public_Key_File_Path_' > /etc/ssh/sshd_config.d/60-foreman-user-ca.conf
ifndef::foreman-deb[]
# restorecon /etc/ssh/sshd_config.d/60-foreman-user-ca.conf
endif::[]
# systemctl restart sshd
----
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each command should have its own step, with a description of what that command does. Can you please update this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the commands are related, you could also introduce them as substeps. So something like:

. Update the `sshd` configuration:
.. Step 1 mkdir
.. Step 2 echo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied.

Copy link
Member

@aneta-petrova aneta-petrova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A second round of feedback based on my reading up on the feature and its purpose. This is mostly centered around making sure the use case is explained properly in the introductions/abstracts.

I also had this thought: https://issues.redhat.com/browse/SAT-28038 (public) provides a hint on what to do when the user key or CA cert need to be rotated. Do you think we should add a procedure for that?


Host authentication::
Hosts present signed certificates when {SmartProxy} connects to them.
This provides verification that {SmartProxy} is connecting to legitimate hosts, preventing man-in-the-middle attacks.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This provides verification that {SmartProxy} is connecting to legitimate hosts, preventing man-in-the-middle attacks.
This provides verification that {SmartProxy} is connecting to legitimate hosts, helping prevent man-in-the-middle attacks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be "helping to prevent"?

Copy link
Contributor

@Lennonka Lennonka Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"to" is optional and both are grammatically correct.

Copy link
Contributor Author

@adamlazik1 adamlazik1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @aneta-petrova and @maximiliankolb for reviews! Here are some of my thoughts:


Host authentication::
Hosts present signed certificates when {SmartProxy} connects to them.
This provides verification that {SmartProxy} is connecting to legitimate hosts, preventing man-in-the-middle attacks.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be "helping to prevent"?

Comment on lines 23 to 26
The file name must be `_My_Host_Private_SSH_Key_-cert.pub`.
The location must be the same as the location of the private key.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file gets created automatically with the correct name, this should be something like a warning to not change it. Any ideas on how to reword it?

= Authenticating {SmartProxy} with SSH certificates

You can configure {SmartProxy} to authenticate with an SSH certificate signed by a Certificate Authority (CA).
When enabled, {SmartProxy} presents a signed certificate to hosts instead of a plain SSH public key.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like it's repeating itself, how about merge the two sentences into one like this?

"You can configure {SmartProxy} to present an SSH certificate signed by a Certificate Authority (CA) to hosts instead of a plain SSH public key."

. If the CA is on a different server than your {SmartProxyServer}, import the public key of the {SmartProxy} to the CA server.
. Sign the {SmartProxy} SSH public key with your CA.
+
The certificate name must be `__My_{smart-proxy-context-titlecase}_Private_Key__-cert.pub`.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably rephrase this somehow, the file gets created with a correct name but it's important to not change it.

You can disable SSH host certificate verification on {SmartProxy} and revert to the Trust On First Use (TOFU) model.

.Procedure
* On {SmartProxy}, enter the following command:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied, slightly reworded.

:client-provisioning-template-type: {parent-client-provisioning-template-type}
endif::[]

include::modules/con_using-ssh-certificates.adoc[leveloffset=+1]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, applied.

Comment on lines 16 to 26
. Enter the following commands:
+
[options="nowrap", subs="+quotes,verbatim,attributes"]
----
# mkdir -p /etc/ssh/sshd_config.d
# echo 'TrustedUserCAKeys _My_CA_Public_Key_File_Path_' > /etc/ssh/sshd_config.d/60-foreman-user-ca.conf
ifndef::foreman-deb[]
# restorecon /etc/ssh/sshd_config.d/60-foreman-user-ca.conf
endif::[]
# systemctl restart sshd
----
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied.

* You have a Certificate Authority for signing SSH certificates.

.Procedure
Perform these steps on each host that uses the {SmartProxy} for remote execution and was registered before you enabled SSH certificate authentication on the {SmartProxy}.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied.

@adamlazik1
Copy link
Contributor Author

adamlazik1 commented Jan 19, 2026

I also had this thought: https://issues.redhat.com/browse/SAT-28038 (public) provides a hint on what to do when the user key or CA cert need to be rotated. Do you think we should add a procedure for that?

Yes, that's a good point. I'll get working on that.

Copy link
Contributor

@lhellebr lhellebr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds foreman-installer instructions to foremanctl version of documentation.

. On the {SmartProxyServer}, locate the SSH key pair that {SmartProxy} uses for remote execution.
You can find the private key location in `/etc/foreman-proxy/settings.d/remote_execution_ssh.yml` under the `:ssh_identity_key_file:` setting.
By default, the location is `/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy`.
. If the CA is on a different server than your {SmartProxyServer}, import the public key of the {SmartProxy} to the CA server.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By "import", you probably just mean "copy the file" but this makes an impression that you want the user to import the key to some keystore. This is also true for other occurences of "import" in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied.

+
The certificate name must be `__My_{smart-proxy-context-titlecase}_Private_Key__-cert.pub`.
By default, the created SSH certificate will have the name `id_rsa_foreman_proxy-cert.pub`.
. If the CA is on a different server than your {SmartProxyServer}, ensure the {SmartProxyServer} has the SSH certificate and the CA public key file:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the CA is on a different server...
This really means: if you copied the sat's public key to a different machine (your CA) and signed it, you need to copy back

  1. the resulting cert
  2. and also the CA's public key

It's confusing and 2) also happened in the preceding import the public key of the {SmartProxy} to the CA server part, although perhaps not to the correct directory.
This should be somehow rephrased together with the aforementioned part.

Also, we shouldn't be too specific because signing a key should be user's responsibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Signing has to be done with private CA key, but smart proxy only requires the public CA key. I think we should keep the mention of what files need to be copied for this to work on an external CA server. But I did change "import" for "copy" to be more clear as per your other comment.

@adamlazik1
Copy link
Contributor Author

I also had this thought: https://issues.redhat.com/browse/SAT-28038 (public) provides a hint on what to do when the user key or CA cert need to be rotated. Do you think we should add a procedure for that?

Yes, that's a good point. I'll get working on that.

Added rotating certs procedure.

. If the CA is on a different server than your {SmartProxyServer}, ensure the {SmartProxyServer} has the SSH certificate and the CA public key file:
.. Import the created SSH certificate back to the {SmartProxyServer} into the same directory as the original SSH key pair.
.. Import the CA public key file to the {SmartProxyServer}.
. Ensure that the `foreman-proxy` user has read access to the CA public key file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How? Or what user, group, mod, selinux context is expected?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly if the user doesn't change anything specific or use a non-standard directory then this will work out of the box. The cert has 644 permissions by default.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So perhaps just a note of some sorts that by default, foreman-proxy has read access, while notifying the users not to change the permissions? It's true that writing "Ensure that ..." invites users to take some sort of action, and if that's not expected here, rephrasing would be a good idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this into admonition, please see if it is better.

* You have a configured CA for signing SSH certificates.

.Procedure
. On the {SmartProxyServer}, generate a new SSH key pair inside the `/var/lib/foreman-proxy/ssh/` directory:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In one of the previous section, the user is instructed to find the correct path in /etc/foreman-proxy/settings.d/remote_execution_ssh.yml but here it is hardcoded.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/var/lib/foreman-proxy/ssh is the default dir for ssh config. While installer provides the option to change it, I am not aware that we have a documented procedure for this. Regardless, some hands-on users could have potentially changed their proxy ssh key, so looking in the settings is a surefire way to identify the correct key. About this procedure, I don't want to instruct users to use a different directory because I have a hunch it can break a lot of REX stuff. There are not only keys in that directory but also known hosts files and host CA known hosts file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get that, but why does one procedure check that config and the other doesn't?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I misunderstood. Fixed.

----
# chown foreman-proxy:foreman-proxy /var/lib/foreman-proxy/ssh/_My_New_SSH_Key_-cert.pub
----
. Configure {SmartProxy} to use the new SSH key:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be run even if the key name is the same? If yes, maybe point this out here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. that is a good point. I will make a mention of that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

. On each host that uses this {SmartProxy} for remote execution, prepare the SSH certificate for the host:
.. Export one of the public keys of the host to the CA server.
The public keys are located in `/etc/ssh/`.
.. Sign the public key with your CA to create an SSH host certificate.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although signing is the user's responsibility, consider pointing out that -h option and sometimes principal need to be specified.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, I will think about it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ended up adding the mention about the -h option, but I think we can probably stop there. Signing is user's responsibility.

[options="nowrap", subs="+quotes,verbatim,attributes"]
----
# {foreman-installer} \
--foreman-proxy-plugin-remote-execution-script-ssh-host-ca-public-keys-file-path _My_Host_CA_Public_Keys_File_Path_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean: foreman-proxy-plugin-remote-execution-script-ssh-host-ca-public-keys-file?
In other places, too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching it. Fixed.

@aneta-petrova
Copy link
Member

This PR adds foreman-installer instructions to foremanctl version of documentation.

This is still okay right now. The list of foremanctl guides we expose upstream doesn't yet include REX documentation:

"title": "Foreman with Katello (containerized)",
"header": "Foreman {FOREMAN_VER} and Katello {KATELLO_VER} (containerized)",
"filename": "index-foremanctl-katello.html",
"sections": {
"Release notes and upgrading": [
["Upgrading_Project", "Upgrading Foreman to {FOREMAN_VER}"]
],
"Quickstart": [
["Quickstart", "Quickstart guide"]
],
"Deploying Foreman": [
["Installing_Server", "Installing Foreman Server"]
],
"Administering Foreman server": [
["Configuring_User_Authentication", "Configuring user authentication"]
],
"Administering hosts": [
],
"Reference": [
Although I recognize that the fact that the GHA preview includes foremanctl preview links is confusing... but users won't be able to see those documents on the docs website.

@adamlazik1
Copy link
Contributor Author

Applied the second round of feedback.

Copy link
Member

@aneta-petrova aneta-petrova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more thoughts after the latest updates. It's looking good! I'm going to pause my style review for now, to give proper space to tech/testing review now. I'd like to take one final look after tech/testing review is done.

= Rotating {SmartProxy} SSH certificate

[role="_abstract"]
You can rotate the SSH certificate used by {SmartProxy} for remote execution by generating a new SSH key pair and having it signed by your configured CA.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you think of a use case that could help users identify as something applicable to their situation? Such as:

Suggested change
You can rotate the SSH certificate used by {SmartProxy} for remote execution by generating a new SSH key pair and having it signed by your configured CA.
If the SSH certificate used by {SmartProxy} for remote execution is no longer valid, rotate the certificate by generating a new SSH key pair and having it signed by your configured CA.

It's a simple thing to add, but helps users find their place in the procedures.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, you can rotate the cert at any time for various reasons, it does not have to be upon cert expiration.

@Lennonka Lennonka added the Waiting on contributor Requires an action from the author label Jan 21, 2026
Copy link
Contributor

@adamruzicka adamruzicka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also explicitly say that this also affects ansible

You can configure SSH certificate authentication in the following ways.
You can configure one or both of these authentication methods depending on your security requirements:

{SmartProxy} authentication::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ssh lingo, this would probably be "Using user certificates"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied.

When your {SmartProxy} connects to hosts, it presents a signed certificate.
Hosts trust the CA that signed the certificate, eliminating the need to distribute individual SSH public keys to each host.

Host authentication::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ssh lingo, this would probably be "Using host certificates"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied.

After you enable SSH certificate authentication on your {SmartProxy}, you can configure hosts to accept SSH certificates from the {SmartProxy}.
This ensures the hosts will no longer accept plain SSH keys during a remote execution connection.

You can use remote execution to apply these changes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use rex to do that at any time or does it have to be done at a specific time in the process otherwise you cut yourself off?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be done at any time, smart proxy retains connection to the hosts registered before enabling the feature via plain ssh keys.


[WARNING]
====
After enabling this feature, SSH certificate authentication works on all newly registered or provisioned hosts.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This (hosts getting configured on provisioning and registration) should be mentioned explicitly at a more prominent place than in a warning.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I move it to the abstract instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would advise against moving this to the abstract. An abstract is just a summary of the contents of a module, it shouldn't contain any unique information.

What about moving it to the end of the module? It describes the end result of the procedure, right? So then it could work as the summary of the new state after the user follows through with the procedure.

You could even split the warning into the description of the new state and the link. The description of the new state could work as a summary after the last step of the procedure (it needs to be connected to the last step with + to be compliant with DITA migration). And the link to the other procedure could work as part of a .Next steps section.

@pr-processor pr-processor bot added Needs re-review and removed Waiting on contributor Requires an action from the author labels Jan 21, 2026
Co-authored-by: Adam Růžička <[email protected]>
Co-authored-by: Aneta Šteflová Petrová <[email protected]>
@adamlazik1
Copy link
Contributor Author

adamlazik1 commented Jan 21, 2026

This should also explicitly say that this also affects ansible

Added in the concept module.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs style review Requires a review from docs style/grammar perspective Needs tech review Requires a review from the technical perspective Needs testing Requires functional testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants