feat: Sandbox UDF

[KKould]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Databend’s Python UDFs are currently implemented via in-process calls using PyO3, which introduces several key limitations:

• GIL constraints: true multi-core parallelism cannot be achieved.
• Environment conflicts: Python dependencies required by different UDFs on the same node are prone to conflicts.
• Stability risks: segfaults in the Python layer can directly crash the query process.
• Resource contention: Python’s memory usage is not well controlled and can interfere with the SQL executor’s buffer usage.

This PR introduces support for remotely and dynamically executing Python UDFs in Databend, organized into three layers:

• Control Plane (Cloud Control Plane): responsible for resource scheduling, permission validation, and sandbox lifecycle management.
• Execution Plane (Databend Query): acts as the client and issues computation requests via the Arrow Flight protocol.
• Compute Plane (Sandbox Workers): lightweight Python environments isolated with gVisor, running the databend-udf service.

Workflow

------------------------+   ApplyResource   +------------------------+
|   Databend Query       | ----------------> |   Cloud Controller      |
|  (Execution Plane)     | <---------------- |  Resource Manager       |
|  - UDF Planner         |    Endpoint+Token |  Image Cache/Warm Pool  |
+-----------+------------+                   +-----------+------------+
            |   Arrow Flight (DoExchange)                |
            |                                            | Provision
            v                                            v
+------------------------+                   +------------------------+
|   Sandbox Worker Pod   | <---------------  |   K8s + runsc (gVisor)  |
|  (Compute Plane)       |                   +------------------------+
|  databend-udf service  |
+------------------------+

Config

Query to configure the startup of Cloud Python UDF

[query]
enable_udf_cloud_script = true
cloud_control_grpc_server_address = "http://0.0.0.0:50051"

Set the expiration time of the presigned URL in the session.

SET udf_cloud_import_presign_expire_secs = 1800;

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is 

[github-actions]

🤖 CI Job Analysis

Workflow: 21282089344

📊 Summary

• Total Jobs: 85
• Failed Jobs: 5
• Retryable: 0
• Code Issues: 5

❌ NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

• ❌ linux / test_unit: Not retryable (Code/Test)
• ❌ linux / test_stateless_cluster: Not retryable (Code/Test)
• ❌ linux / test_stateful_standalone: Not retryable (Code/Test)
• ❌ linux / test_stateless_standalone: Not retryable (Code/Test)
• ❌ linux / test_stateful_cluster: Not retryable (Code/Test)

---

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

[forsaken628]

UdfAsset

[forsaken628]

Why embed JSON in protobuf?

[KKould]

Following this method ‘build_udf_cloud_spec’, pass the spec to the mock server so that it controls the specific construction of the Dockerfile.

[forsaken628]

Why JSON? Protobuf itself has a lot of compatibility designs, there is no need to use embedded JSON

[KKould]

Docker specs are usually constructed in JSON to represent nested parameters, which makes structural adjustments convenient. Using Protobuf to pass nested structures is relatively more cumbersome. Does @everpcpc think using Protobuf would be a better choice?

[everpcpc]

I think if the structure is relatively fixed and won’t change much, I would prefer to use Protobuf. Although JSON is more flexible for handling nesting and structural adjustments, Protobuf’s strong type definitions are clearer to work with for stable structures.

[forsaken628]

In fact, for adjusting the structure of how to parse the json, the specification is completely undefined, everything depends on the implementation, there may be subtle differences in each library, json there are some details of the issue of creating countless bugs, for example, [] can be represented by null is a typical example.
pb, on the other hand, is much more explicit in how it handles this, as it is written in the specification.