Check new GitHub pull requests for a description and a test plan
Build a Kovca workflow that starts when a GitHub pull request is opened, checks the title and description against a policy you write, and on a gap comments and adds a needs-description label.
On most teams, a new pull request shows up with an empty description, or one line, and no note on how the change was tested. There is often no linked issue either. A reviewer has to ask in a comment before they can start, and the pull request sits. A pull request template or a checklist helps only when someone fills it in.
This guide builds a workflow that starts when a pull request is opened, checks the title and the description against sentences you write, and, when the decision is non_compliant or insufficient_information, posts a short comment and adds a label such as needs-description. A compliant pull request is left alone. A requires_review decision is left alone too. That branch does not post the comment and does not add the label. You'll start from a template, test it, and activate it on the repository.
The template already uses the GitHub Pull request opened or changed trigger, Comment on issue, and Add labels to issue. It does not store a GitHub account. You choose the account when you activate. Until then the draft saves, and Studio tests simulate the GitHub steps.
Triage and label new GitHub issues is a different workflow. That one starts from an issue, classifies it, and asks for steps to reproduce when the category is technical. This one starts from a pull request and checks a description against a policy you write.
If the repository only gets a few pull requests a week, a pull request template in the repo, and a person who sends back the empty ones, is less machinery than this. A GitHub Action can also reject a body that is missing a heading. That is covered at the end. The workflow is for the repository where the same request for a summary, a test note, and a linked issue gets typed by hand.
What you'll build
The input card is GitHub: Pull request opened or changed. The title is cut off by the card. Opened? has a true handle and a false handle. The trigger card shows one warning, because no GitHub account is connected.
The false branch of Opened? goes straight to Result. A delivery whose action is not opened does not call Policy Evaluation, and it does not comment or label. The ready branch and the review branch of Compliant? also go straight to Result. A compliant pull request gets no comment and no label. A requires_review decision gets no comment and no label either.
Watch the test
Watch the Studio test (MP4). It is a screen recording of the production Studio at kovca.io, in light mode, about 8 seconds. The frame is the page: the workflow header, the Test panel, and a strip of the canvas. Browser chrome is not in the frame. The click through the scroll is played a little slower than it was captured, so Running and the result stay readable. Nothing is sped up. No GitHub account is connected, and Send real requests stays off. On this recording Comment and Add label are skipped. Nothing is posted on GitHub, and no label is added. The clip holds on the output.
The draft is not published. Comment and add_label are skipped. decision is requires_review and suggested_action is manual_review. The UI rule is satisfied. route is review. comment, comment_id, commented, and labeled are null.
The recording follows these steps:
- The Test tab is open on GitHub PR description check. The draft is not published. The header says 1 Decision AI evaluation max and 3 connections. The input is the trigger sample: action
opened, number 1291, title "Include shipping in cart total for international orders", description "Adds the shipping fee to the cart total for international orders. Fixes #1287." Send real requests is off. The switch's own text says action steps return their sample result and nothing is sent. - Run test. The panel says Running. The previous result is cleared until this one finishes.
- The status is 200. The panel shows 5 credits used for that one Decision AI step.
- The panel is then scrolled to the output.
decisionisrequires_review.suggested_actionismanual_review. The UI rule issatisfied. The summary rule and the missing-summary rule arenot_satisfied.missing_informationis[].routeisreview.comment,comment_id,commented, andlabeledare null. Comment and Add label are skipped.
That recording is one Studio test of the trigger sample. The UI rule came back satisfied, because the description does not say the pull request changes the UI. The decision on this recording was still requires_review, and the route was review, so the not-ready comment and the label did not run. Another run of the same sample returned non_compliant and reject, with route default. On that run the UI rule was satisfied as well, and Comment and Add label returned their samples (comment_id 2471029384, commented true, labeled true). A third test kept the same title and number and replaced the description with: "Summary: adds the shipping fee into the cart total for international orders so the charged amount includes shipping. Test plan: placed an international order in the test suite and confirmed the total equals merchandise plus the shipping fee. Fixes #1287." That run returned compliant and proceed, route ready, with all three rules satisfied, and comment, comment_id, commented, and labeled null. What Policy Evaluation returns on another description depends on the sentences and the text. The JSON later in this guide is the template's unit test for the shipping-fee input, where the test does not read the policy text and the decision comes back insufficient_information. It is not the decision on this recording.
Build it in the Studio
1Start from the template
Click Use this template above, or open GitHub PR description check in Workflows.
The draft opens on the canvas. The components library is on the left. Triggers, AI, Action in an app, Data transformation, Flow, Core, and Human review are the groups. This workflow uses GitHub, so GitHub is under Triggers and under Action in an app once the draft uses it. Apps you have not connected, and that this workflow does not use, stay behind Connect another app. The pictures below are the production Studio, in light mode, with no GitHub account connected. The template's step titles are the ones in the table above. The last card on the canvas is Result.
Triggers. General and GitHub. GitHub's line says it starts workflows from issues, pull requests, and commits, and can open, comment on, or label issues. The count is 46. The line under the list points at Connect another app.
Pull request opened or changed is selected. Its description says it is triggered when a pull request is opened, closed, or synchronized. Issue opened or changed and New commit are in the same list.
Comment on issue is under Action in an app, GitHub. The card says it creates a comment on an existing GitHub issue or pull request.
Add labels to issue says it adds labels, provided in the request body, to a repository issue. The card does not mention a pull request.
2The workflow starts from a pull request
Select the input step. It is GitHub: Pull request opened or changed, not an API request. The account line asks you to connect GitHub. Leave it empty while you test.
Connect GitHub is still to do. Repository shows acme / storefront.
Owner and Repo are required. Until an account is connected, those two settings show as one Repository field, with the stand-in acme / storefront. The trigger's sample event is a pull request on that repository (https://github.com/acme/storefront/pull/1291). Replace both with your owner and repository name. Connect GitHub and, when the account's repositories can be read, that field lists them. Picking one fills both. The same owner and repo have to be set again on Comment and Add label, as fixed values on those steps, because those steps do not read the trigger settings. The event does not include owner or repo.
The description on the step says it is triggered when a pull request is opened, closed, or synchronized. action is a string. The sample sets it to opened. The sample does not include another action value, so this guide does not name the other strings the field can hold. Opened? continues only for the sample's value.
Event fields later steps read:
A manual run or an API call can omit fields. Opened? needs action. Comment and Add label need number.
A repeat of the same delivery does not start a second run. This trigger does not name an event id of its own, so a later delivery for the same pull request is a new run. Opened? is what keeps that later run from commenting and labeling when its action is not opened.
3Only a newly opened pull request continues
Select Opened?. A condition checks one or more rules and continues on its true or false branch. This one has a single rule: input.action equals opened. Continue when all rules match.
input.action equals opened. Continue on true when all rules match.
The trigger's sample event uses opened. The template follows that. The false branch does not call Policy Evaluation, so that delivery uses no evaluation.
4Write the text the policy reads
Pull request builds one block from the number, the title, and the description. That block is Policy Evaluation's description. The title is also mapped as a fact, so a sentence can name it as a fact and not only inside the block.
The true handle reaches Pull request. Pull request connects to Policy Evaluation, which is marked Decision AI.
The template is the number, a colon, the title, a blank line, and the description.
For the trigger sample the block is:
1291: Include shipping in cart total for international orders
then a blank line, then Adds the shipping fee to the cart total for international orders. Fixes #1287.
A missing title or description is an empty spot in that block. The fact title is left off when title is missing. A title that is not text, a number, a boolean, or null makes the policy step reject the input. A fact value can be at most 1000 characters, and the event's title allows 8,000, so a very long title stops the run at the policy step. The description the policy reads can be at most 2000 characters. The number, the title, and the description together can cross that. Shorten the template if you see that.
5Put your pull request policy in
Select Policy Evaluation. The description comes from Pull request. title comes from the event. The policy is a fixed value, so the rules live in the workflow. The inspector's request line says it is not set, because the request is filled by the fields under it (request.description and request.facts.title) rather than by one mapping of the whole request. The policy field is one line, so the picture shows the start of the JSON. The full value is the block below.
policy is a fixed value and the line starts with the id. request is not set. request.description is nodes.pull_request.output.text. request.facts.title is input.title.
The node splits that text into rules on sentence boundaries. Each rule is one sentence. id and version are labels. They come back on the result when you set them. Rewrite the sentences in your own words. Keep each rule one sentence, and keep the policy to 40 sentences or fewer. The policy text can be at most 12000 characters.
The template's three sentences say:
- A pull request may proceed when the description includes a summary of what changed and why, a note on how it was tested, and either a linked issue such as Fixes #123 or a reason there is no linked issue.
- If the pull request says it changes the UI, the description includes a screenshot or a note on the visual change; otherwise this rule does not apply.
- A pull request that is missing the summary, the test note, or the issue reference cannot proceed.
The second sentence is one rule. The clause after the semicolon stays in that sentence, so a pull request that does not say it changes the UI is not asked for a screenshot. The title is inside the pull request text, and it is also the fact named title, so a UI change named in the title is the pull request saying so.
The node does not look up the diff, the review comments, or the linked issue. If those words are not in the title or the description, the sentences have nothing else to go on. There can be at most 50 facts.
decision is compliant, non_compliant, requires_review, or insufficient_information. suggested_action is proceed, reject, request_information, request_approval, or manual_review. requirements is a list of the rules that were judged, each with a status of satisfied, not_satisfied, requires_review, or insufficient_information. missing_information is the rules that were short of facts. What comes back depends on the sentences and the pull request text. Run pull requests you have actually seen before you treat a route as settled.
6Leave a compliant pull request alone
Select Compliant?. The switch reads nodes.policy.output.decision. The node's own line says it continues on the branch of the first case that matches, or on default. Under Cases, the editor says the first case whose values include the decision wins, and that each case is a branch to connect.
Route on nodes.policy.output.decision. The case ready matches compliant. The case review matches requires_review. Anything else continues on default.
Compliant? has a ready handle, a review handle, and a default handle. The edge labeled default reaches Gaps comment. The ready and review lines continue past Gaps comment, to Result.
The ready branch and the review branch go to Result with no comment and no label. The switch continues on the first case whose values include the decision, and each case is its own branch. requires_review has a case named review, so route is review and the gaps comment does not run. Default posts the comment that says the description is not ready, and it adds needs-description. That branch is non_compliant, insufficient_information, and any other value. A case can list more than one value. This template uses a second case so a requires_review decision stays visible in route, and the not-ready comment stays on default.
7Comment, and add the label, when it does not pass
Gaps comment, Comment, and Add label run only on default.
Gaps comment is this template:
The template names the decision and the suggested action, then inserts the requirements list and the missing list.
The requirements and the missing list are the node's own lists, inserted as JSON. An empty list shows up as []. The comment does not invent a missing test note when the node did not report one.
Comment is GitHub's Comment on issue action. The action's own description says it creates a comment on an existing GitHub issue or pull request. The Issue Number field's own description says it identifies the issue or pull request for the comment. The action's sample in the catalog is an issue comment URL (/issues/1287#issuecomment-2471029384), not a pull request page. This template passes the pull request's number. A Studio test returns that sample id. It does not post a comment.
If it fails, the run stops. Owner and Repo are the fixed values acme and storefront. Issue Number is input.number. The field says it identifies the issue or pull request. Body is nodes.gaps_comment.output.text.
The body is GitHub Flavored Markdown, at most 8,000 characters. This template's text is well under that unless the requirements list is very long. Replace acme and storefront here too. If it fails is Stop the run. Add label does not run if this comment fails. The step's own line says it is sent once: Kovca does not retry it. A second run with a new idempotency key can post another comment. Choose the GitHub account on this step before you activate. Studio tests simulate it and do not post. The simulated result is ok: true and id: 2471029384. This template reads ok and id only.
Add label is GitHub's Add labels to issue action. It runs after the comment. The action's own description says it adds labels to a repository issue. The Issue Number field's own description says it identifies the issue. Neither line mentions a pull request. This template still passes input.number, the same number Comment on issue accepts as a pull request. That a pull request number is accepted by Add labels is an assumption. This guide has not made a live call. The label name is a stand-in. The action does not say it creates a label that the repository does not already have. Create needs-description in the repository, or change the value to a label that is already there, before you activate.
If it fails, the run stops. Repo is storefront and Owner is acme. Labels is the JSON array needs-description. The field says it is an array of label names to add to the issue. Issue Number is input.number, and that line says it identifies the issue.
Keep the labels value a JSON array. Typing needs-description without the brackets stores the word, and Add labels expects a list. Labels accepts at most 100 names. If it fails is Stop the run. If the comment has already been posted and the label then fails, the comment is already on the pull request. The step's own line says it is safe to retry: Kovca retries it if GitHub is briefly unavailable. Studio tests simulate it and do not change labels. The simulated result is ok: true and result: {}. This template reads ok only.
Comment's subtitle is GitHub: Comment on issue. Add label's subtitle starts GitHub: Add labels and is cut off by the card. Both cards show one warning, because no GitHub account is connected.
8Test it
Open the Test tab. Shipping fee and Update checkout are the template's examples. Sample pull request opened or changed is the trigger's own sample, which is the same pull request as Shipping fee. Leave Send real requests off. When Comment or Add label runs, it returns its sample and is marked sample. Nothing is written to GitHub. That sample is not what a published API call returns. The Test panel still charges Policy Evaluation when that step succeeds. It does not charge the run fee, and it does not charge a simulated Comment or Add label. See What it costs.
Send real requests is off. The input is the trigger sample. The run returned 200 and used 5 credits. The steps list is just starting.
Comment and add_label are skipped. decision is requires_review and suggested_action is manual_review. The UI rule is satisfied. route is review. comment, comment_id, commented, and labeled are null.
What Policy Evaluation returns depends on the sentences and the pull request text. The run in the pictures returned 200 and used 5 credits. decision was requires_review and suggested_action was manual_review, so route was review. Comment and Add label were skipped, and the comment fields came back null. Another run of the same sample returned non_compliant and reject, so route was default, and Comment and Add label succeeded and were marked sample. The description with a summary, a test plan, and Fixes #1287, and no mention of the UI, returned compliant and proceed, so route was ready. A compliant result takes the ready branch to Result, and Gaps comment, Comment, and Add label are skipped. A requires_review result takes the review branch to Result, and those three steps are skipped there too. An input whose action is not opened skips the policy step too.
Before you activate, run pull requests you have actually seen, including ones that should pass and ones that should not. When the routes look right, turn on Send real requests once, with the GitHub account chosen, on a pull request where an extra comment is harmless. Add label will apply the label for real on a result that takes the default branch.
Publish, call it, and activate it
The same workflow can be called over the API. The body uses the trigger's field names. Create an API key under API keys and keep it on your backend.
That curl calls a published workflow. The call runs Comment for real, and Add label as well, on the default branch: non_compliant, insufficient_information, and any other decision that is not compliant or requires_review. The steps need the GitHub account you chose on them. If that account is missing, the action fails and, with If it fails set to stop, the run stops. A published call does not return the sample you see with Send real requests off.
The JSON below is the output this template's unit test expects for the shipping-fee input when Comment and Add label are simulated and the policy step returns insufficient_information with suggested_action manual_review. The test does not read the policy text. That is one result the step can return. It is not a response from a published API call, and it is not the decision on the Studio recording. In the example, commented is true and comment_id is 2471029384 because the simulated step's sample says so. That does not mean a comment was posted on GitHub. labeled is true because the simulated Add label sample says ok: true. The comment is Gaps comment filled in from that result. Update checkout is the other example, a description that is only the word wip. On this same test it also takes the default branch.
Look up the run later. This returns status, timing, and steps. It does not return the output. Keep the execute response if you need the decision or the comment. Runs are kept for 30 days.
1291 is a key you choose. Keys are kept for 24 hours. A retry with the same key and the same request returns the original run instead of posting another comment. A new key can comment again, and can add the label again, because Comment on issue is not idempotent. Add labels is safe to retry. Kovca does not store the pull request or the output. On a delivery whose action is not opened, opened is false and the decision, the comment, and the label come back null.
To run it from the repository, open Activate. The checklist is what's left: connect GitHub, and, once an account is connected, choose that account on the trigger, on Comment, and on Add label. It says up to 15 credits per event. The activate row is Publish and activate: run on every pull request opened or changed. The button on that row says Publish & activate. That publishes the draft and starts it. A live automation runs that pinned version. Later edits don't reach it until you publish them to the automation.
The automation is still a draft. Up to 15 credits per event. Connect GitHub is still to do. Tested is checked. Publish and activate runs on every pull request opened or changed. The button is not clicked.
Connect GitHub from the checklist, or from the trigger, when the routes in the Test tab look right. Until then, leave the account empty and keep Send real requests off. GitHub sign-in opens a connection page. Kovca keeps an ID for the account.
A new pull request starts a run when GitHub sends the event. Follow runs on the workflow's Executions page (Studio → Usage → Executions). While an automation is paused, events that arrive are skipped. Kovca does not keep them to run later. A failed run is not retried on its own, because the comment or the label may already have happened.
If a template and a heading check are enough
You don't need this if a person can look at new pull requests and the list is short. A file at .github/pull_request_template.md can ask for a summary, a "How tested" note, and a linked issue before the pull request is opened. That file does not check that anyone filled it in.
A GitHub Action can go one step further and fail the check when the body does not match a pattern: a Summary heading, a How tested heading, and either something like Fixes # or a line that says there is no linked issue. That is enough when the team only needs those headings to be present. It does not read whether the test note describes a test, and it does not treat "How tested: TBD" as a gap unless you write the pattern that way. It also does not post the comment from a decision. It fails a check.
What the workflow adds is the same sentences on every opened pull request, a comment that includes the decision and the node's own lists, and one label when the switch takes the default branch. You still choose the GitHub account, replace acme and storefront, and read a sample of results before you activate it.
Make it yours
- Rewrite the sentences. The three sentences are a fixed value on Policy Evaluation. Keep each rule one sentence. A path or a word the description does not contain is something the node cannot see.
requires_reviewdoes not get the missing-description comment. The decision can come backrequires_review, with suggested actionmanual_review, when a rule is uncertain. This template matches that value with a case namedreviewand connects the branch to Result. It does not post the gaps comment and it does not addneeds-description. Add a step on that branch if a person should see those pull requests.- Use your label name.
["needs-description"]is a stand-in. Keep it a JSON array, and use a label the repository already has. - Add a label on the ones that pass. The ready branch goes to Result and does nothing else. Add another Add labels step on that branch, with a value such as
["ready-for-review"], if a passing pull request should be marked. That step has the same account and the same owner and repo as the other one. - Leave the other actions alone. The trigger description says it fires when a pull request is opened, closed, or synchronized. This template continues only when
actionisopened, which is the value in the sample. Change the rule only if you have seen the other action strings on a delivery you trust. - Point Owner and Repo at the repository you actually watch.
acmeandstorefrontare the sample's repository. Set them on the trigger, on Comment, and on Add label. - Roll it out on a copy of the repository. Use a test repository for a week and compare the comments with what the team would have written.
- Change it without surprising a live repository. Edit the draft and publish the changes to the automation. The running version stays pinned until you do.
What it costs
Credits, and only for steps that succeed. Billing is the list.
A Studio test skips the run fee. It does not skip Policy Evaluation. That step is 5 credits when it succeeds, including a test with Send real requests off. A simulated action is not charged, and an action that fails is not charged. With Send real requests off, Comment and Add label are simulated, so they add nothing.
- A GitHub event: 6 credits for the run.
- A published API call: 1 credit for the run.
- Policy Evaluation: 5 credits when it succeeds, and only when
actionisopened. - Comment on issue: 2 credits when it runs for real and succeeds, and only on the default branch.
- Add labels: 2 credits when it runs for real and succeeds, and only on the default branch.
The condition, the switch, and the text steps use no credits. A branch that doesn't run isn't charged. The Activate panel adds every step together, so its number is the cost of the branch that does the most. With this template that number is 15 credits per event: the 6 credit event fee, Policy Evaluation, Comment, and Add label. A live event on the ready branch or the review branch is 11 credits (the event fee and Policy Evaluation, and no GitHub action). A live event whose action is not opened is 6 credits. The response's credits field is the run's total. See the workflows API, HTTP actions, and automations.