WIP: Test Suite #11
Open
WIP: Test Suite #11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
we will establish a testing system using multimodal LLM capabilities. These TUI charts can be rendered in a test program and rendered with the installed vhs cli tool (https://github.com/charmbracelet/vhs). You can use the directory tests/multimodal for scripts and documentation and test corpus. You can use a tests/multimodal/Taskfile.yml to help maintain tasks, create tasks as you deem necessary but keep it simple and understandable to a human. Think about the set of sample programs to demonstrate the correctness of this library. Create a JSON file to maintain the test list for tooling. Also put that list as markdown in tests/multimodal/README.md For each test, make a set of directories in tests/multimodal/refs with a short kebab-case summary name, put a prompt markdown file, a test go program, and a VHS file. Make a task to verify the existence of this structure for each test. You can use "vhs validate" to lint the file. In execution mode, you will run the go program, driven by vhs, store the output in tests/multimodal/results/test-name-out.gif. you will then examine the picture for correctness and report any issues. in fix mode, you would also fix the library per the issue. Signed-off-by: Evan Wies <[email protected]>
Human did the .keepme and Taskfile includes ;) prompt: look at the tests/multimodal directory. we are going to use the same test corpus as tests.json, but we will evaluate the View() string to a reference string. first restructure the directory so that the tests.json live in tests/tests.json and the test programs live in tests/cmd/test-name/main.go the taskfile should be updated accordingly then tests/multimodal will have tests/multimodal/refs will eventually have bona fide and fully reviewed reference pictures. create a tests/terminal directory . for each test create a tests/terminal/refs/test-name.txt reference string. update the test programs to also include a reference string test. that output will be used by our test harness. Signed-off-by: Evan Wies <[email protected]>
Signed-off-by: Evan Wies <[email protected]>
initial prompt: look at github issue #7 read the issue and understand it. create a correct test reference using the example in the issue. then understand the issue and fix it Signed-off-by: Evan Wies <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a test suite, both for terminal-based testing and multi-modal LLM testing.
The reference corpus isn't validated yet.
I have already used the terminal suite along with Gemini world knowledge to fix #7.