doc: create ai-guidelines and include to CONTRIBUTING

[RafaelGSS]

As discussed in today's TSC meeting.

cc: @nodejs/tsc @BridgeAR

[nodejs-github-bot]

Review requested:

• @nodejs/tsc

[joyeecheung]

There may be some ideas we can borrow from https://llvm.org/docs/AIToolPolicy.html - for example "good first issue" should not be picked up by AI is a good one.

[RafaelGSS]

There may be some ideas we can borrow from https://llvm.org/docs/AIToolPolicy.html - for example "good first issue" should not be picked up by AI is a good one.

I took inspiration from https://github.com/zulip/zulip/blob/main/CONTRIBUTING.md#ai-use-policy-and-guidelines

[mcollina]

Do we have a template? I thought those are for issues, not PRs.

[legendecas]

Not strictly a template: https://github.com/nodejs/node/blob/main/.github/PULL_REQUEST_TEMPLATE.md?plain=1

[MikeMcC399]

It's not possible to fulfil the instructions "Complete pull request templates fully" based on the contents of https://github.com/nodejs/node/blob/main/.github/PULL_REQUEST_TEMPLATE.md?plain=1 so it looks like this sentence needs to be removed.

[gurgunday]

lgtm

[indutny]

I'd be against this contribution policy update. While many different opinions exist on the licensing terms of the code produced by LLMs, my opinion is that the generated code isn't explicitly licensed and attributed to the original authors so it cannot be considered open source regardless of the used prompt.

[mcollina]

Suggested change

	* **Verify accuracy** of any LLM-generated content before including it in a
	PR description or comment.
	* **Complete pull request templates fully** rather than replacing them with
	LLM-generated summaries.
	* **Verify accuracy** of any LLM-generated content before including it in a
	PR description or comment.

[joyeecheung]

In #61478 (comment) , regarding the usage of Claude Code, @mcollina suggested:

As it seems that you have a different position on this issue, I will bring this to the attention of the @nodejs/tsc for an official vote. On a side note, I would also officially summon the topic at the next OpenJS board meeting.

I added it to the TSC agenda tomorrow for awareness/context collection before moving to a proper vote. @indutny sorry about the short notice since this is just one day ahead of the meeting, but if you'd like to join the meeting to present your points please let us know.

[bmuenzenmeyer]

this first sentence seems intentionally to be the same as the full policy document, which now has a suggested change. marking this here to resolve whether or not the language should continue syncing, if it should change

[Ethan-Arrowood]

What is / can we link to our official policy on "stale" or stalled PRs? Like regardless if its involving AI do we have a policy to close out PRs after a certain amount of time? If not, should we?

[avivkeller]

Right now our automation closes PRs if they are both older than a year and have had no activity in the past sixth months, but IMO we should make that just six months (no year clause)

[mcollina]

@RafaelGSS Can you please add a link to https://openjsf.cdn.prismic.io/openjsf/aca4d5GXnQHGZDiZ_OpenJS_AI_Coding_Assistants_Policy.pdf (official AI policy)?

This document already aligns for the most part, which should mention the Assisted-by: git commit tag (like the kernel).

[timfish]

While I don't personally want to see a blanket ban on AI assisted PRs, isn't the real risk here that parts of Node.js end up no longer under its current MIT license?

Both the US and the EU have stated that purely AI derived work is not copyrightable. The January US ruling says "prompts alone do not provide sufficient human control". The EU paper says that there are varying interpretations of human input requirements in different member states.

This means AI contributions could be considered public domain.

Chad Whitacre wrote a great article covering the copyright/licensing issues from the perspective of source available licenses but this applies equally to open source licensed work if you want to retain that license!
https://blog.sentry.io/fair-source-software-in-the-ai-age/

[JakobJingleheimer]

@RafaelGSS should this be put on hold until the session at the collab summit (openjs-foundation/summit#484)?

[mcollina]

Not necessarily, we'll cover this in the next TSC session and see.

Likely it would drag out naturally to that, but unless otherwise specified this is not necessary, given the Foundation overall AI policy.

[jasnell]

I think given the conversation around all of this so far, I would structure the guideline differently.

The core principal should simply be: Node.js expects contributions to come from people.

People are free to use whatever tools they want when generating contributions. However the contribution is created, contributors are expected to understand and take full responsibility for every change they propose.

If any contribution was generated with the assistance of any automation or tool (AI-based or otherwise) then that should be acknowledged honestly as part of the contribution so that those reviewing the change have appropriate context going into the review.

If it becomes apparent that individual contributors are relying too much on these tools, or aren't understanding or taking responsibility for the changes they are proposing, or it becomes clear that the contributor is being dishonest about the use of automated assistance when creating PRs, then that's going to influence whether or not the project considers accepting further contributions from that person.

PRs should never be opened by automated tooling not specifically approved in advance by the project.

While I do not believe a blanket ban on AI contributions is at all necessary or beneficial to the project, I would absolutely accept a policy that allows a contributor to be blocked from further contributions if their only contributions are AI, they show no actual understanding of the project or processes, or are caught being habitually dishonest about their use of automation.

I would even accept language saying that while me may discourage Automation/AI-assisted contributions, we will not reject them solely on that basis; every PR must still be evaluated on the technical merits of the change following our existing established review processes.

[RafaelGSS]

Applied all suggestions I could find. Please, re-review.

[mcollina]

This should be expanded a bit, see https://docs.kernel.org/process/coding-assistants.html#attribution., I would just drop the tools, so:

Assisted-by: AGENT_NAME:MODEL_VERSION

[joyeecheung]

Personally I find some of the tool mentions in PRs these days already somewhat annoying, it's like "sent from my iPhone" advertisements everywhere and more marketing in the commit logs. Especially when issues are found and the explanation is just "this (proprietary) AI tool did it/made me believe it", as if the tool is an excuse because it's supposed to be the best. I think if people let the tools do most of the work that it requires reviewer's awareness, it's much better to include your prompt and roughly explain your process than just leaving what (likely proprietary) model and tool you use. Also the list can be very long if they use multiple tools and it's just more advertisement/marketing noise.

[jasnell]

Including the prompt is impractical in most cases. For example, in the PRs where I use AI assistance the "prompt" is typically a multi-session chat with many prompts. I don't consider the prompts really to be all that useful.

[joyeecheung]

I don't think it needs to be the exact same prompt, but if we are talking about usefulness, "here's the process of how I used the tools to come to this diff" is much more useful and relevant to determine the soundness of the diff than "I used this specific model and this specific tool" but without any description of how it arrives at that diff. If the process is sound, you could probably arrive at the same diff with just any tool/model or no AI at all with varying level of efficiency, but mentioning one specific tool/model just feels like marketing if we don't care about "how long it takes to arrive at this diff", only "the quality of this diff". Even pre-AI, I don't think people would say "I use IntelliJ IDEA to refactor these names in the code base" in the commit logs, they just describe the refactoring rules and motivations in the logs. If the process is not sound, there's little difference in what model/tool/human you use to arrive at that diff.

[jasnell]

Personally, I think mentioning which model was used is important to establishing context for the review. Frankly, some models are better than others at producing good code. I don't need to know what prompt you used as long as the PR description follows good practices in explaining the changes or the change is clear enough about what has changed. But if you used an AI-agent, knowing which agent was used it helpful in establishing context for which patterns to look for.

Unless there's a specific rule proposed and accepted, I plan to continue to use the format Assisted-By: Opencode:{model+version} (opencode since that's the agent tool I happen to use).

[joyeecheung]

I think that's only useful if the commit does not have human involvement, which is what the guidelines are discouraging. If the human needs to be 100% responsible of the process then that hardly matters, whatever pattern there is should be the author's pattern, not the tools. If they let the tool dictate their own pattern then they are not following the guidelines. If they think it's absolutely useful then that could go into PR descriptions, but commit logs shouldn't be a marketing ground especially for commercial brands of proprietary tools/models.

[RafaelGSS]

So, what's the status here? Should we change something? If so, which suggestion do we agree on?

[joyeecheung]

I think we can recommend anonymise the tool: a CLI coding agent backed by a reasoning model, etc. Would be enough. Use your discretion when mentioning a name that's a commercial brand or a trademark.

(TBH I feel flaunting trademarks in commit logs is rather inappropriate, but if it's relevant enough it's okay)

[bnb]

After today's Summit discussion, I'd like to recommend that we adjust messaging to be "discouraged but allowed". I think that would generally represent the sentiment that was expressed and hit the median of the collaboratorship's opinion.

If people are chill with that, happy to submit prose suggestions.

[tniessen]

Suggested change

	Pull requests that consist of AI-generated code the contributor has not
	Pull requests that contain AI-generated code the contributor has not

[tniessen]

Suggested change

	every change they propose. Pull requests consisting of AI-generated code the
	contributor has not personally understood, tested, and verified will likely be closed
	without review.
	every change they propose. Pull requests containing AI-generated code the
	contributor has not personally understood, tested, and verified will likely be closed
	without review.

[tniessen]

Some hallucination machines may write tests based on the implementation of a feature instead of considering the expected behavior of the feature.

Suggested change

	* **Test thoroughly.** AI-generated code must pass the full test suite and
	any manually written tests relevant to the change. Existing tests should not
	be removed or modified without human verification. Do not rely on the LLM
	to assess correctness.
	* **Test thoroughly.** AI-generated code must pass the full test suite and
	any manually written tests relevant to the change. Existing tests should not
	be removed or modified without human verification. Do not rely on the LLM
	to assess correctness. It is crucial to manually verify the correctness of
	tests against the expected behavior of the feature being tested,
	independently of the feature's implementation.

[ChALkeR]

Can we extract this and also apply to human-written code too? (but keep it here also)

I believe we have existing decade-old tests that should be subject to this

Mostly "increase coverage" PRs are highly subject to documenting bugs instead of fixing them (not just in Node.js, but overall)

[JakobJingleheimer]

The moderation policy would also need to be updated to align with this new policy doc (the moderation policy expressly forbids AI contributions from non-collaborators).

[bmuenzenmeyer]

The moderation policy would also need to be updated to align with this new policy doc (the moderation policy expressly forbids AI contributions from non-collaborators).

Opened nodejs/admin#1059

[ChALkeR]

I would prefer for mostly-ai-driven (but reviewed by human) contributions to be explicitly labeled as mostly-ai-driven.

Assisted-by obscures that to an extent.

E.g. if the code is mostly auto-generated, it should be the main author and the human only a co-author and a Signed-off-by to signal acceptance/origin in terms of DCO.

[ChALkeR]

Hm.

The OpenJSF policy (at https://openjsf.cdn.prismic.io/openjsf/aca4d5GXnQHGZDiZ_OpenJS_AI_Coding_Assistants_Policy.pdf) specifies Assisted-by for AI assistants indeed.

Still, non ideal, but likely not something that can be solved at a project level?

[panva]

E.g. if the code is mostly auto-generated, it should be the main author and the human only a co-author and a Signed-off-by to signal acceptance/origin in terms of DCO.

I disagree with having tooling be the contribution's main author. This isn't aligned with taking responsibility for the contributions (signed-off-by !== author) and doesn't sit well with changelog entries.

While you may use assistants for the content it is ultimately you, the human, we expect a contribution from.

[mcollina]

an option could be to also add a label